# Memristor-only LSTM Acceleration with Non-linear Activation Functions

Rafael Moura, João Lima, Luigi Carro

► To cite this version:

Rafael Moura, João Lima, Luigi Carro. Memristor-only LSTM Acceleration with Non-linear Activation Functions. 7th International Embedded Systems Symposium (IESS), Nov 2022, Lippstadt, Germany. pp.89-100, 10.1007/978-3-031-34214-1_8. hal-04645291

HAL Id: hal-04645291

https://inria.hal.science/hal-04645291v1

Submitted on 11 Jul 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
Memristor-only LSTM acceleration with
non-linear activation functions

Rafael Fão de Moura¹, João Paulo C. de Lima¹, and Luigi Carro¹

Informatics Institute - Federal University of Rio Grande do Sul - Porto Alegre, Brazil
{rfmoura,jpclima,carro}@inf.ufrgs.br

**Abstract.** Long Short-Term Memories (LSTMs) applied to Speech Recognition are an essential application of modern embedded devices. Computing Matrix-Vector Multiplications (MVMs) with Resistive Random Access Memory (ReRAM) crossbars has paved the way for solving the memory bottleneck issues related to LSTM processing. However, mixed-signal and fully-analog accelerators still lack in developing energy-efficient and versatile devices for the calculus of activation functions between MVM operations. This paper proposes a design methodology and circuitry that achieves both energy efficiency and versatility by introducing a programmable memristor array for computing nonlinearities. We exploit the inherent capability of ReRAM crossbars in computing MVM to perform piecewise linear interpolation (PWL) of non-linear activation functions, achieving a programmable device with a smaller cost. Experiments show that our approach outperforms state-of-the-art LSTM accelerators being 4.85x more efficient using representative speech recognition datasets.

**Keywords:** Long Short-Term Memory (LSTM) · Resistive Random Access Memory (ReRAM) · Near-sensor computing.

# 1 Introduction

With the widespread usage of mobile devices and virtual assistants, the bulk of Speech Recognition applications has increased [14]. Long Short-Term Memories (LSTMs) figure as an implementation mechanism for such Machine Learning (ML) problems and have regained attention due to their capability to solve the time-dependency issues that Recurrent Neural Networks (RNNs) present. As for other Neural Network (NN) models, LSTMs are memory-intensive, having their performance bounded by the memory bandwidth available on-chip [6].

One of the solutions to suppress the memory bottleneck found on LSTM lies in bringing these applications closer to data sources. To perform such a processing in-memory computation (i.e., in-situ Matrix-Vector Multiplications (MVMs)), Resistive RAM (ReRAM) crossbars have been employed as they demonstrate higher efficiency compared to mixed-signal and digital designs [6, 9, 16, 17]. Although efficiently implementing the aforementioned computation, ReRAM crossbars can only be used for linear computations. Moreover, the LSTMs perform on average 20 times more calls to non-linear computations (i.e., activation
functions) for a single neuron compared to other NN models, such as AlexNet and Lenet, culminating in high power dissipation and energy consumption regardless of the design approach.

To bring efficiency on the execution of activation functions, several works have been proposing custom circuits, achieving high energy benefits. However, such custom logic lacks versatility, preventing reprogramming different activation functions in the same design. Also, several works have shown that depending on the input, LSTMs can exploit varied non-linear functions to improve the accuracy of their predictions [18]. In this scope, custom logic applied to LSTMs drastically reduces the possibilities of combinations of different nonlinearities.

On the other hand, Field-Programmable Arrays promise ubiquitous programmable engines to directly solve the versatility requirements of current LSTMs. Compared to custom Integrated Circuits (ICs), these configurable devices provide more flexibility and enable practical engineering for ML solutions, achieving a design cost at least 10x smaller [7]. Even though Field-Programmable Arrays bring versatility, they fall far from achieving the performance and energy efficiency levels of ICs since extra costs related to communication, routing, and IO comes out. Therefore, the trade-off between energy efficiency and versatility of LSTM designs is still an open issue to be maximized.

This work introduces a method to enable the execution of non-linear computations using ReRAM crossbars, bringing higher energy efficiency compared to Field-Programmable circuits. At the same time, this work provides a programmable-enabled activation function design that brings versatility compared to custom LSTM circuits. Given that, our main contributions are:

- We are the first to reduce the energy gap between a Field-Programmable non-linear environment and custom circuits while keeping the versatility and high accuracy provided by the Field-Programmable approach.

- We introduce the PNLU to reprogram and calculate activation functions at a smaller cost. To achieve that, we combine the inherent efficiency of ReRAM crossbars to compute MVM with Piecewise Linear interpolation (PWL).

- We compare our new LSTM design over a custom circuit (fixed activation function) and an FPAA circuit (dynamic activation functions). By using a representative speech recognition dataset, we show that our approach brings up to 2.24x computing efficiency gains over the FPAA implementation counterpart, while presenting 15% more accuracy in comparison to a custom circuit design.

The rest of this paper is organized as follows: Section 2 presents background on ReRAM crossbars for LSTM processing. Section 3 discusses state-of-the-art approaches to accelerate LSTMs. Section 4 introduces the proposed architecture and its details. Section 5 discusses the methodology used to evaluate this work and the results. Finally, Section 6 compiles future works and conclusions.
## 2 Background

### 2.1 ReRAM for in-memory NN computing

<figure><img src="image_3.png" /><figcaption>Fig. 1: a) Physical model of typical conductive filament-based ReRAM cell, b) Parallel dot-products implemented in a ReRAM crossbar array, c) a dot-product implemented as a sum of currents, d) LSTM cell structure</figcaption></figure>

ReRAM is the most attractive Non-Volatile Memory (NVM) for NNs computing, since its cells provide many states of data representation at a low-voltage, low-latency, and dense storage [4]. Fig. 1a) illustrates the physical model of a ReRAM cell. Each ReRAM cell consists of a Resistive Switching (RS) material sandwiched between two electrodes, and stores data either through the formation (set) or destruction (reset) of the conductive filament in the oxide layer by applying a programming voltage between the two electrodes [5].

The most employed representation of synapses with ReRAMs is the synaptic array. This structure is composed of a $N \times N$ ReRAM cell arrangement, which can easily implement the computation model of NNs, as illustrated in Fig. 1b). This arrangement is more efficient in chip area and power consumption and allows the design of novel accelerators that can perform MVM in the analog domain in $O(1)$ time complexity [2]. In Fig. 1c), each ReRAM cell represents a synaptic weight trough its conductance $g_i$, siting between a horizontal wordline (WL) and vertical bitline (BL). Here, $v_i$ is the input voltage signal with different amplitudes applied to each WL of the crossbar, and the output signal of each neuron is the sums-of-product of input signals with their corresponding synaptic weight. Thus, the sum-of-products is implemented as the sum of currents $I$ through each BL of the crossbar.

### 2.2 LSTM architecture

LSTMs are modified RNN architectures that exhibit a dynamic-temporal behavior, which grants them the ability to learn order dependence in sequence-prediction problems, such as machine translation, speech recognition, and others. These networks take both the current sample ($x_t$) and the previously calculated network state ($h_{t-1}$) as input. This feedback loop creates an internal state, which
allows the network to remember and make a decision based on previous information. Fig. 1d) presents a typical LSTM containing three gates (input, forget, output), block input, a memory cell, an output activation function, and peephole connections.

## 3 Related work

Prior works compute non-linear functions in the digital domain, with the costs of the peripheral circuit reaching up to 58% and 43% of the total tile power, respectively [17, 11]. On the other hand, the analog implementation of activation functions for LSTMs appears as the most prominent solution to minimize the related costs of calculating those nonlinearities [1]. ERA_LSTM [6] adopts analog approximate activation functions implemented by NNs to reduce AD conversions. However, the approach adopted on ERA_LSTM only bypasses the calculus of activation functions to another NN, which also relies on the calculus of those functions. Adam et al. [1] present a fully analog LSTM system with custom modules, such as memristor crossbars, activation functions, analog multipliers, and memory units. However, current approaches to achieve the full-analog implementation of LSTMs with ReRAMs lack either versatility or energy efficiency. Designing specialized circuits for each activation function undoubtedly achieves the best energy-efficiency but compromises the device compatibility to support custom LSTM models.

Alternatively, FPAA devices offer a promise of ubiquitous reconfigurable devices for ultra-low-power machine learning that can directly solve the enormous energy requirements of current fielded machine learning applications, as well as enable what is typically considered cloud-based machine learning in edge devices [7]. Compared to custom ICs, these configurable devices provide more flexibility and enable effective engineering cost designs for machine learning solutions. Typically the design cost for a new design is at least 10× the mask cost, requiring 10× the expected financial return to even attempt such a venture. The resulting price for designing an IC is often far too high for most engineering applications to hope to reach these financial returns. Therefore, an **optimal solution** should address the analog implementation of nonlinearities with energy efficiency and versatility.

## 4 Proposed architecture

Fig. 2a describes the overall architecture for the energy-efficient, programmable execution of LSTMs. The accelerator design is composed of tiles that are interconnected through an NoC. Each tile comprises DACs, memory buffers, a control unit, and ADCs (i.e., elements to handle the interface with other tiles and external memories). Also, an MVM Unit, a Hadamard Product Unit (HPU), and a Programmable Non-Linear Unit (PNLU) are placed inside a tile to perform the coarse grain of computations alongside the processing of an LSTM cell.
The execution flow of an LSTM described in Fig. 1 drives the data flow among the blocks inside each tile as follows: DACs convert the current input ($x_t$) and the previous hidden state ($h_{t-1}$) brought from the Local Buffers, and send them to MVM Unit. The MVM Unit computes the dot-product operations related to the four gates of the LSTM cell in parallel. The MVM results are sent to the PNLU to perform the computation of activation functions related to the four gates. The previous memory cell ($c_{t-1}$) is brought from the Local buffers, and converted to the analog domain. HPU computes the memory cell ($c_t$) output value. PNLU performs the calculus of the tanh over the $c_t$ value. HPU computes the hidden state value ($h_t$). Finally, both the $c_t$ and $h_t$ are converted to the digital domain and stored in the Local Buffers for the next iterations.

<figure><img src="image_4.png" /><figcaption>Fig. 2: a) Overview of the proposed architecture for the energy-efficient versatile acceleration of LSTMs, b) MVM Unit in detail: organization of the ReRAM crossbar arrays.</figcaption></figure>

## 4.1 MVM Unit

The implementation of the MVM Unit is presented in Fig. 2b. The MVM Unit stores the weights of the LSTM gates and efficiently computes in parallel the four dot-product operations described in Fig. 2a. Before each MVM computation, DACs take place to convert the current input data ($x_t$) and the previously-stored hidden state ($h_{t-1}$) from the memory buffer into the analog domain represented by analog voltages. Two crossbar arrays are employed to represent negative weights, containing only the weights module: one for the positive and the other for the negative weights. Then, analog inverters place at the input of the negative-weight crossbars to invert the polarity of the input voltage. Then, the MVM results are sent to the PNLU still in the analog domain to perform the computation of activation functions, depending on the requirement of the LSTM gates. To avoid drop in the signal caused by a voltage division in the connection between the MVM Unit and the PNLU, a voltage follower amp-op circuit acts by grounding the BLs and isolating neighbor layers.
The current technology of ReRAMs can deliver only cells with a limited range of conductance (typically, cells with four (2-bit) or eight (3-bit) levels). Hence, several implementations employ multiple ReRAM crossbars to compose the weight bits and combine the results by shift-and-add operations. However, such an approach may introduce two drawbacks to the proposed design: the shift-and-add operations will require unwanted AD-DA steps in the execution flow if implemented in the digital domain. Second, the shift-and-add could be implemented through the linear scaling of the voltage levels, introducing noise sources to the analog signal. Therefore, we adopt the add method proposed by [8] to represent the entire weight bits. In this method, several memory cells with smaller levels are added together to represent a weight with a higher number of levels. For example, to compose a 5-bit (32 levels) weight cell with 2-bit (4 levels) ReRAM cells, eight 2-bit ReRAM cells are necessary. Generalizing, for a ReRAM crossbar in which cells have $L_c$ levels, an LSTM in which weights are represented with $L_w$ levels, and $L_c < L_w$, then $L_w \div L_c$ ReRAM cells are necessary to represent the target weight.

## 4.2 Programmable Non-Linear Unit (PNLU)

To enable the calculus of non-linearities inside the ReRAM crossbars, we rely on representing the target activation function as multiple segments of linear functions. Such a method is called PWL, and it defines the interpolation of the original function ($f(x)$) by $N$ line segments as defined below, and illustrated by Fig. 3b.

$$f(x) \approx \begin{cases} g_1(x) = a_1x + b1 & x_0 \le x < x_1 \\ g_2(x) = a_2x + b2 & x_1 \le x < x_2 \\ g_n(x) = a_nx + bn & x_{n-1} \le x < x_n \end{cases}$$

Fig. 3a presents the circuit schematic for the PWL computation. The circuit is split into two parts: the computation of linear functions by ReRAM crossbars, and a mux to bypass the correct value of the interpolated function according to the current input value ($x$). In the ReRAM crossbar, the inputs are represented by the column-vector $[x, 1]$ and perform a MVM to generate $N$ linear functions in the format $g_i(x) = a_i x + b_i$. In this arrangement, all the $g_i(x)$ functions are computed in parallel through the sum of currents on each BL, followed by an op-amp in the voltage follower configuration. The distribution of weights in the crossbar (i.e., the $a_i$ and $b_i$ values) follows the same structure as the one presented in the MVM Unit, using the add method to compose memory cells with a higher number of bits.

After the parallel computation of the $N$ linear functions, a mux bypasses only the $g_i(x)$ output that matches the current input $x$ value. The analog circuit that generates the control signal for the mux is illustrated on the right-side of Fig. 3a. This analog circuit is composed of a series of parallel analog comparators. Each op-amp compares the input signal ($x$) to a unique reference voltage level ($V_{ref}$). $V_{ref}$ is a stable reference representing the maximum $x$ value in the domain of the interpolated function. As the analog input voltage exceeds the reference voltage
<figure><img src="image_5.png" /><figcaption>Fig. 3: a) The Programmable Non-linear RLU in detail: circuit schematic for the PWL computation. All programmable elements are shown in the orange color, b) Original, PWL, and simulated PWL version of sigmoid and tanh functions.</figcaption></figure>

at each comparator, the comparator outputs will sequentially saturate to a high state. Then, an encoder generates a signal control for the mux based on the highest-order active input, ignoring all other active inputs.

In our design, a commonplace priority encoder is not necessary to generate the control signal for the mux, as such a device introduces complexity and extra area and energy costs to the circuit. Hence, due to the nature of the parallel comparators, where each op-amp saturates high in sequence from the lowest voltage level to the highest voltage level in the voltage divisor circuit composed of the programmable resistors $R_{1:n}$, a highest-order-input selection may be realized with the employment of a set of XOR gates, followed by the usage of a simpler non-priority encoder. Last but not least, the encoder circuit itself can be made from a matrix of diodes for more simplicity.

In term of costs, to interpolate any given function with $N$ segments of lines, $(3N) - 1$ programmable resistors, $(2N) - 1$ op-amps, $N - 1$ XOR gates, and a $N$: 1 mux are needed. Such a hardware cost may look expensive in comparison with custom circuits, but this design has a much smaller cost related to any FPPA implementation. Also, since our design employs four PWL engines in the PNLU to calculate the five calls to activation functions in the LSTM flow, only $(12N) - 4$ values corresponding to the programmable resistors should be configured in contrast to thousands of elements to be configured inside an FPAA. Further discussion and comparison regarding the costs and benefits that come from the employment of the PNLU are provided in Section 5.

## 4.3 Hadamard Product Unit (HPU)

To calculate the Memory cell ($c_t$) and Hidden state ($h_t$) values for each LSTM iteration, the Hadamard or element-wise product is computed in the HPU. Since the computation flow of $c_t$ and $h_t$ contains a true dependence that hinders both from being computed in parallel, the Hadamard Unit comprises only two parallel
circuits represented as two green boxes in Fig. 2a, each one capable of performing the element-wise product of two vectors. We adopt the Four-Quadrant Analog Multiplier Circuit from [10] in our design. To complete the execution flow of an LSTM cell, the Hadamard Product Unit receives the analog voltage signals that come from the PNLU and computes the $c_t$. Then the PNLU is used to perform the tanh over $c_t$. After that, the Hadamard Product Unit is activated to calculate the output for the $h_t$ gate, completing the current iteration of the LSTM cell.

## 4.4 Local buffers

The local buffers are composed of two blocks: a memory buffer and a cache memory. Together, the memory buffers and cache comprise the memory hierarchy of a tile, which is a commonplace practice in prior ReRAM-based architectures. We employ eDRAM as the memory buffer in the proposed design and SRAM as the cache memory. Local buffers keep the input data fed to the MVM Unit, and the activation of output gates from the HPU. The cache size is set to have all the necessary input/output data of computing components mapped for one tile. By mapping the LSTM gates that share the same input onto one tile, buffer sharing can reduce the hardware overhead of local buffers.

## 4.5 Controller

The Control Unit drives the execution and data flow on each tile. This unit coordinates the data transmission between the memory buffer and cache and generates control signals for the DACs, the MVM Unit, PNLU, HPU, and ADCs. Each tile can operate in two modes: activation mode and non-activation mode. When an LSTM cell is mapped onto a single tile, it is said that this tile is operating in the activation mode. In the activation mode, the MVM products go through the PNLU and HPU. Then, the activation outputs are converted by ADCs and stored in the local buffers until the next iteration. Whenever an LSTM cell is mapped onto multiple tiles (denoted by T), T-1 tiles operate in the non-activation mode performing only MVM operations. Then, the last tile operates in the activation mode, collecting the partial sums from the corresponding T-1 tiles and proceeding with the execution flow.

# 5 Experimental setup and results

## 5.1 Methodology

To evaluate the proposed work, we built a simulation framework based on DNN+NeuroSim [15]. DNN+NeuroSim simulates the behavior of in-situ analog computing, such as the variability of the ReRAM, and error of ADC and DAC converters within a PyTorch wrapper, to obtain the inference accuracy. Then, the framework employs an analytical model of the ReRAM crossbar computation, including converters, memristor non-idealities, and weight mapping to compute
the design's energy, area, and timing statistics. Initially, DNN+NeuroSim did not support LSTM cells or weights composed of several ReRAM cells, so we added these features and the simulation of the PNLU for the Pytorch wrapper. All activation functions were interpolated by a PWL function composed of five line segments.

**Simulation parameters** We use the power, area, and timing model of ReRAM crossbar arrays and the Op-Amp from [11] to design the MVM Unit and the PNLU. For the HPU, the components are extracted from related work[10]. We scale some parameters of the circuits (i.e., transistor sizes) accordingly to work with the 32 nm model. The ReRAM devices have a resistance range of 50$k\Omega$ - 800$k\Omega$, and the read voltage is 0.15 V. We set the sub-array size to be 128$\times$128 for the MVM Unit and 32x5 for the PNLU. Each ReRAM cell has four bits (16 available conductance levels). Hence, 16 cells are used to represent an 8-bit weight value with the add method. The ADC and DAC models are from a fabricated ADC chip [9], and an analytical model of DACs [16]. CACTI 6.5 [13] is used to model the overhead of buffers. We limit the frequency of tiles to 10 MHz to hold the critical path of our design, which is the MVM Unit. The size of the local buffers and the number of DACs/ADCs are determined accordingly to match the throughput of the MVM Unit and the PNLU. The NN weights are quantized to 8-bit under the WAGE8 format before being mapped to the device conductance values. We use the Gaussian noise to represent the device variation for the MVM Unit and the PNLU. It has been measured that the variation is normally 0.2% for the ReRAM cells, 0.3% for the op-amps, and half LSB for the AD-DA converters[11, 9, 16].

**Benchmarks** We use the bidirectional LSTM architecture [3] for the TIMIT dataset, and the LSTM model presented in [18] for the IMDb dataset as evaluation benchmarks. Regarding the employment of activation functions, the standard LSTM models for the TIMIT and IMDb datasets are built with the original tanh and sigmoid nonlinearities, refereed as the TMIT-std model and the IMDb-std model, respectively. Also, we evaluate the IMDb with the activation function comb-H-sine from [18], named IMDb-comb-H-sine model.

## 5.2 Power and Area Analysis

Fig. 4 presents the power and area breakdown of a tile, and the power improvement by comparing our approach with two state-of-the-art ReRAM-based accelerators: [11], and ERA-LSTM [6]. Based on the overhead of each component, we provide a few insights into architectural details and trade-offs: DACs and ADCs consume a large part of the overall power and area in ReRAM-based architectures. As for ERA-LSTM, the number of DAC-ADCs is reduced due to the analog processing of the whole LSTM flow, which contributes the most to the resource-saving. The MVM unit in the proposed design also shows improvement over the digital and analog designs. The main reason is that the Op-Amps,
<figure><img src="image_6.png" /><figcaption>Fig. 4: Power and area breakdown, and power dissipation comparison with related work.</figcaption></figure>

the major power consumers in the MVM Unit, are shared in the proposed design. This implies the advantage of the add method, since the BL currents can be added together instead of being shifted and added after DA conversion. Using the **PNLU and the HPU** dissipates more power than its analog counterpart. However, the programmable analog computation of activation functions is necessary to achieve the best model accuracy for each dataset, as demonstrated in the next section.

## 5.3 Inference accuracy

Table 1 presents the inference accuracy comparison. Results from [11] and ERA-LSTM are listed in the *Digital imp.* and *Analog imp.* columns, respectively. The most important result regards the accuracy levels achieved by the IMDb-comb-H-sine scenario: an increase of up to 15% compared to the IMDb-std. Here, the simple change in the activation function leads to a significant improvement in accuracy. Moreover, among all accelerator designs in this comparison, only our approach can achieve these accuracy levels, since this can reprogram itself to allow experimentation of different activation functions for LSTMs. In the *Retrained LSTM* column, we present the accuracy of our technique after retraining the LSTM models with the interpolated functions, where our design can reach the best accuracy levels. This result reveals a second level of versatility that only our work can bring to the LSTM acceleration: both LSTM and activation function model can be tuned for a given dataset with our device, achieving the best inference accuracy possible without extra hardware costs if we change any parameter of our LSTM model.

## 5.4 Computing efficiency

Table 2 compares the computing efficiency of our design with state-of-the-art LSTM accelerators, including an FPAA-based implementation of activation functions [12] plus a ReRAM crossbar to perform MVM operations, a Digital
<table><caption>Table 1: Inference accuracy comparison</caption><thead><tr><th rowspan="2">Scenario</th><th colspan="5">Inference accuracy(%)</th></tr><tr><th>Baseline</th><th>Digital imp.</th><th>Analog imp.</th><th>This work</th><th>Retrained LSTM</th></tr></thead><tbody><tr><td>TIMIT-std</td><td>84.21</td><td>83.60</td><td>84.24</td><td>82.15</td><td>84.31</td></tr><tr><td>IMDb-std</td><td>76.12</td><td>76.01</td><td>75.97</td><td>75.92</td><td>76.10</td></tr><tr><td>IMDb-comb-H-sine</td><td>91.32</td><td>N/A</td><td>N/A</td><td>90.78</td><td>91.29</td></tr></tbody></table>

ReRAM-based LSTM accelerator [11], and ERA-LSTM [6]. Compared with the digital design, our approach achieves 4.85x its efficiency. ERA-LSTM is 1.27x more efficient than our design compared to the analog counterpart. This stems from the fact that ERA-LSTM employs analog custom circuits for the calculus of activation functions, which are known to be more efficient than a programmable circuit. However, our design is the only one that can achieve higher levels of accuracy due to its versatility in the choice of a programmable non-linear unit. Compared to the FPAA-based design, our approach achieves 2.24x more efficiency. Such a result can be achieved due to its simple logic inside the PNLU, which has considerable smaller costs of re-programmability than a conventional FPAA circuit. Hence, our device can deliver the versatility to model an LSTM with a smaller cost, reducing the energy gap between an FPAA dynamic activation function environment and custom circuits, while keeping the versatility and high accuracy provided by the FPAA approach.

<table><caption>Table 2: Computing efficiency comparison</caption><thead><tr><th>Study</th><th>ReRAM-based</th><th>Computing Efficiency (GOP/s/W)</th></tr></thead><tbody><tr><td>Digital design</td><td>Yes</td><td>116</td></tr><tr><td>ERA-LSTM</td><td>Yes</td><td>714</td></tr><tr><td>FPAA</td><td>Yes</td><td>250</td></tr><tr><td>This work</td><td>Yes</td><td>562</td></tr></tbody></table>

## 6 Conclusion and future work

This paper presented PNLU, a programmable ReRAM unit to accelerate non-linearities for LSTMs in the analog domain. The proposed unit combines the inherent energy-efficient capacity of ReRAMs to perform MVM with PWL to reduce the energy gap between an FPAA dynamic activation function environment and custom circuits, while keeping the versatility and high accuracy provided by the FPAA approach. Our results show that our approach brings up to 2.24x computing efficiency gains over an FPAA implementation counterpart (dynamic activation functions), while presenting 15% more accuracy than a custom circuit design (fixed activation function). In future work, we intend to expand our approach to different NN models, and the dynamic selection of activation functions during NN inference.
References

1. Adam, K., Smagulova, K., James, A.: Generalised analog lstms recurrent modules for neural computing. Frontiers in computational neuroscience p. 85 (2021)

2. Ankit, A., et al.: Panther: A programmable architecture for neural network training harnessing energy-efficient reram. IEEE Transactions on Computers (2020)

3. Evangelopoulos, G.N.: Efficient hardware mapping of long short-term memory neural networks for automatic speech recognition. Ph.D. thesis, KU Leuven Leuven, Belgium (2016)

4. Grossi, A., et al.: Experimental investigation of 4-kb rram arrays programming conditions suitable for tcam. IEEE VLSI **26**(12), 2599-2607 (2018)

5. Halawani, Y., et al.: Reram-based in-memory computing for search engine and neural network applications. IEEE JETCAS (2019)

6. Han, J., Liu, H., Wang, M., Li, Z., Zhang, Y.: Era-lstm: An efficient reram-based architecture for long short-term memory. IEEE TPDS **31**(6), 1328-1342 (2019)

7. Hasler, J.: The potential of soc fpaas for emerging ultra-low-power machine learning. Journal of Low Power Electronics and Applications **12**(2), 33 (2022)

8. Ji, Y., et al.: Fpsa: A full system stack solution for reconfigurable reram-based nn accelerator architecture. In: ACM ASPLOS. pp. 733-747 (2019)

9. Kull, L., et al.: A 3.1 mw 8b 1.2 gs/s single-channel asynchronous sar adc with alternate comparators for enhanced speed in 32 nm digital soi cmos. IEEE JSSC **48**(12), 3049-3058 (2013)

10. Li, S.C.: A symmetric complementary structure for rf cmos analog squarer and four-quadrant analog multiplier. Analog Integrated Circuits and Signal Processing **23**(2), 103-115 (2000)

11. Long, Y., Na, T., Mukhopadhyay, S.: Reram-based processing-in-memory architecture for recurrent neural network acceleration. IEEE VLSI (2018)

12. Moreno, D.G., Del Barrio, A.A., Botella, G., Hasler, J.: A cluster of fpaas to recognize images using neural networks. IEEE TCAS II (2021)

13. Muralimanohar, N., Balasubramonian, R., Jouppi, N.: Optimizing nuca organizations and wiring alternatives for large caches with cacti 6.0. In: 40th IEEE/ACM MICRO. pp. 3-14. IEEE (2007)

14. Park, S.H., Kim, B., Kang, C.M., Chung, C.C., Choi, J.W.: Sequence-to-sequence prediction of vehicle trajectory via lstm encoder-decoder architecture. In: 2018 IEEE IV. pp. 1672-1678. IEEE (2018)

15. Peng, X., Huang, S., Jiang, H., Lu, A., Yu, S.: Dnn+ neurosim v2. 0: An end-to-end benchmarking framework for compute-in-memory accelerators for on-chip training. IEEE TCAD **40**(11), 2306-2319 (2020)

16. Saberi, M., Lotfi, R., Mafinezhad, K., Serdijn, W.A.: Analysis of power consumption and linearity in capacitive digital-to-analog converters used in successive approximation adcs. IEEE TCAS-I **58**(8), 1736-1748 (2011)

17. Shafiee, A., et al.: Isaac: A convolutional neural network accelerator with in-situ analog arithmetic in crossbars. ACM SIGARCH **44**(3), 14-26 (2016)

18. Vijayaprabakaran, K., Sathiyamurthy, K.: Towards activation function search for long short-term model network: a differential evolution based approach. Journal of King Saud University-Computer and Information Sciences (2020)