RETROSPECTIVE: Sparse ReRAM Engine: Joint Exploration of Activation and Weight Sparsity in Compressed Neural Networks

When we began this work in 2018, deep neural networks (DNNs) had gained significant popularity, and researchers were actively exploring innovative architectural designs to achieve energy efficiency. To address the intensive computing and memory demands of DNNs, several studies proposed computing-in-memory (CIM) designs to accelerate DNN inference. Particularly, in ISCA 2016, two papers [O9][O40] first introduced this idea to the computer architecture community. These pioneering studies suggested leveraging the inherent matrix-vector multiplication computing capability of emerging memristor devices, such as resistive random access memory (ReRAM), to overcome the memory wall challenge and enable highly parallel computations. The revisit of the CIM concept [1], along with its successful combination with AI workloads, has opened up a prominent research domain. Despite memristor-based CIM showing promising potential for AI acceleration, there remain challenges to overcome in practice. First, the non-ideality of memristor devices makes it difficult for an analog-to-digital converter (ADC) to accurately read out the matrix-vector multiplication results on bitlines. Our analysis, conducted through DL-RSIM [O31] (a simulation framework we developed), revealed that the inference accuracy drops considerably when too many wordlines are activated concurrently due to the accumulated per-cell current deviation (e.g., 16 wordlines, even with technology advances enabling cell variation reduction). Second, deploying a highfrequency ADC with high bit-resolution to enable the entire memristor crossbar array to operate in one cycle leads to significant power and area overhead. Considering the non-ideality of memristor devices and the overheads associated with ADC, practical matrix-vector multiplication must proceed at a smaller granularity in memristorbased CIM. This was demonstrated by the state-of-the-art ReRAM macro designed for DNN acceleration at that time, which operated only a 9x8 sub-region in a crossbar array per cycle [O6]. However, when we developed our idea, existing designs published in the computer architecture community overlooked these hardware constraints and assumed that the entire crossbar array could be operated in a single cycle. We were the first to recognize that it is more practical to operate only a smaller section of a crossbar array in a single cycle, and we defined this small section that can be operated per cycle as the Operation Unit (OU). Although the OU-based architecture is more practical, it is not necessarily better than the over-idealized one that operates an entire crossbar array in a single cycle when considering the impact on performance. In comparison to the over-idealized architecture, an OU-based design benefits from better accuracy and lower resolution requirements on ADCs, as fewer wordlines are activated per cycle. However, while an OU-based design could potentially achieve a shorter cycle time by adopting lower-resolution ADCs with faster sensing speed, it also requires more cycles to complete the same amount of computations due to limited wordline and bitlinelevel parallelism. Consequently, the significantly increased number of cycles overshadows the advantage of a shorter cycle time, making an OU-based design likely to deliver lower performance than an over-idealized counterpart if no optimization mechanism is applied to reduce computations. The comparison between the OU-based design and the overidealized counterpart involves a tradeoff between accuracy and performance. This motivates us to explore whether we can resolve this dilemma through architectural design, and the collaboration between academics and industry enables us to derive a practical yet efficient solution. During our investigation, we discovered that the OU-based architecture naturally allows the exploitation of fine-grained sparsity to skip more redundant computations, as each OU operates independently. By effectively harnessing the fine-grained sparsity offered by the OU-based design, it becomes possible to achieve a practical design with satisfactory inference accuracy while attaining comparable performance and energy efficiency with the over-idealized counterpart. To accomplish this objective, we propose the Sparse ReRAM Engine, which stands as one of the pioneering works that avoids overlooking the impact of hardware constraints and aims to develop a practical yet energy-efficient memristor-based CIM design.