# Approximate LSTM Computing for Energy-Efficient Speech Recognition
## 4 Accelerator Design for Approximate LSTM Processing
More precisely, a PE is first initialized by reading 16 bits from the on-chip SRAM and divides the data into two W registers, each of which contains eight bits. When we perform the normal LSTM cell mode, two W registers together denote a weight value, whereas the approximate mode handles two 8-bit weight values separately. Whenever a PE receives the data input (DI) in order to perform the normal LSTM cell operation, which is an element of the current input vector, note that we can turn on two internal \(8\times 8\) multiplications followed by a summation unit to support a \(16\times 16\) multiplication. Figure 5: Proposed LSTM accelerator architecture. Figure 6: Overall architecture of LSTM engine providing multiple operation modes. Otherwise, for the case of approximate LSTM mode, we disable the red-colored modules in Figure 7, and then only the leftmost/rightmost \(8\times 8\) multipliers are involved in the cell operation, supporting two \(8\times 8\) multiplications used for the delta-based LSTM operation with reduced computing precision. Note that the delta generator in Figure 5 stores the approximate input values into on-chip memories, reducing the unnecessary memory accesses associated with the high-resolution values. As a result, the proposed LSTM engine structure offers an attractive energy efficiency by eliminating all the operations for the skipped LSTM cells and also by reducing the number of processing cycles for the approximate computing cases. As we adopt the delta-based processing from [20] to develop the processing scenario of approximate mode, some elements of input vector are changed to zero values before starting the LSTM cell operation, increasing the sparsity of matrix-to-matrix multiplications for gate operations shown in (1)-(3). If we directly use the same sequential memory accessing for pseudo-skipping LSTM operations as what we apply to the normal mode, as illustrated in Figure 8, the increased sparsity results the additional redundant on-chip memory accesses for taking meaningless weight parameters. To solve this problem, in this work, we newly introduce a small-sized on-chip SRAM to keep the accessing sequence of **W** matrices. This address buffer is initialized at the run time by marking the nonzero element positions whenever the delta generator module in Figure 5 produces the survived input elements with reduced resolution. As we only apply the delta processing for the partially similar inputs, the number of survived elements are naturally less than that of the original delta-RNN processing [20], allowing to reduce the size of this additional SRAM. For the prototype design, we only utilize the address buffer of 64 entries for the prototype design, minimizing the additional hardware complexity. By accessing only meaningful columns of **W**, as a result, the proposed accelerator maximizes the energy efficiency even for supporting the approximate LSTM operations. After performing the MAC-oriented operations at the LSTM engine, the proposed accelerator enables the non-linear operators such as sigmoid and hyperbolic tangent functions. We realize these non-linear operators by adopting a number of look-up tables, which are basically obtained by piece-wise linear approximations. It is also possible to re-utilize internal MAC operations at the LSTM engine for performing five FC layers; the first three for the pre-processing and the rest two for the post-step as depicted in Figure 2. As a result, the proposed accelerator design shown in Figure 5 successfully supports all the processing sequences defined by the state-of-the-art DeepSpeech network while even allowing multiple LSTM operating modes to save energy consumption by eliminating redundant processing parts as many as possible. Figure 7: Detailed PE architecture.