Efficient recurrent architectures through activity sparsity and sparse back-propagation through time
Authors: Anand Subramoney
Khaleelulla Khan Nazeer
Mark Schöne
Christian Mayr
David Kappel
Venue: ICLR 2023 notable top 25% (2023)
Keywords: RNN
GRU
recurrent network
language modeling
dvs
gesture recognition
activity sparsity
efficiency

Abstract: Recurrent neural networks (RNNs) are well suited for solving sequence tasks in resource-constrained systems due to their expressivity and  low computational requirements. However, there is still a need to bridge the gap between what RNNs are capable of in terms of efficiency and performance and real-world application requirements. The memory and computational requirements arising from propagating the activations of all the neurons at every time step to every connected neuron, together with the sequential dependence of activations, contribute to the inefficiency of training and using RNNs. We propose a solution inspired by biological neuron dynamics that makes the communication between RNN units sparse and discrete. This makes the backward pass with backpropagation through time (BPTT) computationally sparse and efficient as well. We base our model on the gated recurrent unit (GRU), extending it with units that emit discrete events for communication triggered by a threshold so that no information is communicated to other units in the absence of events.  We show theoretically that the communication between units, and hence the computation required for both the forward and backward passes, scales with the number of events in the network. Our model achieves efficiency without compromising task performance, demonstrating competitive performance compared to state-of-the-art recurrent network models in real-world tasks, including language modeling. The dynamic activity sparsity mechanism also makes our model well suited for novel energy-efficient neuromorphic hardware. Code is available at https://github.com/KhaleelKhan/EvNN/.

Published as a conference paper at ICLR 2023
EFFICIENT RECURRENT ARCHITECTURES THROUGH ACTIVITY SPARSITY AND SPARSE PACK-PROPAGATION
THROUGH TIME
Anand Subramoney1 Khaleclulla Khan Nazeer", Maze Schöne", Christian Mayr Davidd Kappel*
Institute for Neural Computation, Ruhr University Bochum, Germany
2 Royal Holloway, University of London
3 Faculty of Electrical and Computer Engineering, Technische Universität Dresden, Dresden, Germany
(CeTTI,Techhisse Universität Dresden, Dresden, Germany lanand. subramoney: david. kappel @ini. rub. de
Centre for Tactile Internet with Human-in-the-Loow
(khaleelull khan nazeer, mark. schoene, christian mayr -tu-dresden de
ABSTRACT
Recurrent neural networks (RNNs) are well suited for solving sequence tasks in ressurce-constrained systems due their expressivity and low computational requirements. However, there is still need to bridge the gap between what RÑNs are capable of terms of efficiency and performance and real-world application requirements. The memory and computational requirements arising from propagating the activations of all the neurons at every time step to every connected neuron, together with the sequential dependence of activations, contribute to the inefficiency of training and using RNNs. We propose a solution inspired by biological neuron dynamics that makes the communication between RNN units sparse and discrete. This makes the backward pass with backpropagation through time (BPTT) computationally sparse and efficient well. We base our model on the gated recurrent unit (GRU), extending it with units that emit discrete events for communication triggered by threshold so that no information is communicated other units in the absence of events. We show theoretically that the communication between units, and hence the computation required for both the forward and back- ward passes, scales with the number of events in the network. Our model achieves efficiency without compromising task performance, demonstrating competitive performance compared to state-of-the-art recurrent network models in real-world tasks, including language modeling. The dynamic activity sparsity mechanism also makes model well suited for novel energy effeeietf neuromorphic hardware. Code available at https:/ /titthb........ com/ Khalee Khan EENN//
INTRODUCTION
Large scale models such as GPT-3 (Brown et al., 2020) and DALL-E (Ramesh et al., 2021) have demonstrated that scaling up deep learning models to billions of parameters improve not just their per- formance but lead to entirely new forms of generalisation. But for resource constrained environments, transformers are impractical due to their computational and memory requirements during training as well as inference. Recurrent neural networks (RNNs) may provide a viable alternative in such low-resource environments, but require further algorithmic and computational optimizations. While is unknown scaling up recurrent neural networks can lead to similar forms of generalization, the limitations on scaling them up preclude studying this possibility. The dependence of each time step's s computation on the previous time step's output prevents easy parallelisation of the model computation. Moreover, propagating the activations of all the units in each time step is computationally inefficient and leads to high memory requirements when training with backpropagation through time (BPTT).
While allowing extraordinary task performance, the biological brain's recurrent architecture is extremely energy efficient (Mead, 2020). One of the brain's 's strategies to reach these high levels of efficiency is activity sparsity. In the brain, (asynchronous) event-based and aetivity-sparse communication results from the properties of the specific physical and biological substrate on which
Work done while at Ruhr University Bochum

---

Published as a conference paper at ICLR 2023
the brain is built. Biologically realistic spiking neural networks and neuromorphic hardware aim to use these principles to build energy-effieien software and hardware models (Roy et al., 2019; Schuman et al., 2017). However, despite progress in recent years, their task performance has been relatively limited for real-world tasks compared to recurrent architectures based on LSTT and GRU.
In this work, we propose an activity sparsity mechanism inspired by biological neuron models, to reduce the computation required by RNNs at each time step. Our method adds a mechanism to the recurrent units to emit discrete events for communication triggered by a threshold so that no information is communicated to other units in the absence of events. With event-based communication, units in the model can decide when to send updates to other units, which then trigger the update of receiving units. When events are sent sparingly, this leads to other units most of the time, leading to substantial computational savings during training and inference. We formulate the gradient updates of the network to be sparse using a novel method, extending the benefit of the computational savings to training time. We theoretically show, in the continuous time limit, that the time complexity of calculating weight updates is proportional the number of events in the network. We demonstrate these properties using Gated Recurrent Unit (GRU) (Cho et al., 2014) as a case study, and call our model Event- based Gated Recurrent Unit (EGRU). We note, however, that our dynamic activity-sparsiee mechanism can be applied to any RNN architecture.
where most units do not send updates to
In summary, the main contributions of this paper are the following:
1. We introduce variant of the GRU with event-generatie mechanism, called the EGRU.
2. We theoretically show that, in the continuous time limit, both the forward pass computation and the computation of parameter updates in the EGRU scales with the number of events (active units).
3. We demonstrate that the EGRU exhibits tesk--efformance competitive with state-of-the recurrent network architectures on real-world machine learning benchmarks.
4. We empirically show that EGRU cxhibits high levels of activity-sparsity during both inference (forward pass) and learning (backward pass).
We note here that methods for training with parameter sparsity improving handling of long-term dependencies are both orthogonal to, and can be combined with our approach (which we plan to do in future work). Our focus, in this paper, is exclusively on using activity-sparsity to increase the efficiency of RNNS, specifically the GRU. We expect our method to be more efficient but not better at handling long-range dependencies compared to the GRU.
The sparsity of the backward-pase overcomes one of the major roadblocks in using large recurrent models, which is having enough computational resources to train them. We demonstrate the task performance and activity sparsity of the model implemented in PyTorch, but this formulation will also allow the model to run efficiently on CPU-based nodes when implemented using appropriate software paradigms. Moreover, implementation on novel neuromorphic hardware like Davies et al. (2018); Höppner et al. (2017), that is geared towards avent-based computation, can make the model orders of magnitude more energy efficient (Ostrau et al., 2022).
RELATED WORK
Activity sparsity in RNNs has been proposed previously (Neil et al., 2017: 2016; Hunter et al., 2022), but only focusing on achieving it during inference. Conditional computation is form of activity sparsity used in Fedus et al. (2022) to scale a feedforward transformer architecture to trillion parameters. An asynchronous event-based architecture was recently proposed specifically targeted towards graph neural networks (Schaefer et al., 2022). QRNNs (Bradbury et al., 2017), SRUs (Lei et al., 2018) and IndRNNs (Li et al., 2018) target increasing the parallelism in recurrent network without directly using activity-sparsity. Unlike Fedus et al. (2022) that used separate network to decide which sub-networks should be active (Shazeer et al., 2017), our architecture uses a unit-local decision making process for the dynamic activity-sparsiny The cost of computation is lower in our model compared to Neil et al. (2017), and can be implemented to have parallel computation of intermediate updates between events, while also being activity sparse in its output.
Models based on sparse communication (Yan et al., 2022) for scalability have been proposed recently for feedforward networks, using locality sensitivity hashing to dynamically choose downstream units
2

---

Published as a conference paper at ICLR 2023
for communicating activations. This is dynamic form of parameter-sparsitn (Hoefler et al., 2021). which is orthogonal to and complementary with our method for activity-sparriee and can be combined for additional gains. The use of rectified linear units (ReLU) in recurrent networks as in the IRNN (Le et al., 2015) is closely related, and can lead to sparse activations. But current literature lacks an analysis of such models from the context of efficiency. Moreover, the use of ReLU activation requires careful weight initialization, and it seems to lag behind in task performance in comparisons with other RNN models (Li et al., 2018).
Biologically realistic spiking networks (Maass, 1997) are often implemented using event-based updates and have been scaled to huge sizes (Jordan et al., 2018), albeit without any task-related performance evaluation. Models for deep learning with recurrent spiking networks (Bellec et al., 2018; Salaj et al., 2021) mostly focus on modeling biologically realistic memory and learning mechanisms. Moreover, units a spiking neural network implement dynamics based on biology and communicate solely through unitary events, while units in an EGRU send real-valued signals to other units, and have different dynamics. A sparse learning rule was recentl y proposed (Bellec al., 2020) that is a local approximation of backpropagation through time, but not
The theoretical analysis of the event-based learning rule for the continuous time EGRU is inspired by, and a generalization of the analysis in Wunderlich and Pchle (2021) for spiking neurons. As in that paper, we use the adjoint method for ordinary differential equations (ODEs) Pontryagin al. (1962); Chen et al. (2018) combined with sensitivity analysis for hybrid discrete/continuous systems (Galán et al., 1999; Chen et al., 2020). Using surrogate gradients for backpropagating through the di-fferreentill threshold function was originally proposed for feedforward spiking networks in Esser et al. (2016) and developed further in Bellec et al. (); Zenke and Ganguli (201 8). The sparsity of learning with BPTT when using appropriate surrogate gradients in a discrete-time feed-forward spiking neural network was recently described in Perez-Nieves and Goodman (2021).
A continuous time version of sigmoidal RNNs was proposed in Beer (1995) and for GRUs in De Brouwer al. (2019). The latter used a Bayesian update for network states when input events were received, but the network itself was not event-based. As in Neil et al. (2016); Lechner and Hasani (2020), the focus there was on modeling irregularly spaced input data, and not on event-based network simulation activity-sparse inference and training. Chang et al. (2019) also recently proposed continuous time recurrent network for more stable learning, without event-based mechanics. GRUs were formulated in continuous time for analyzing its autonomous dynamics in Jordan et al. (2021).
EVENT-BASED GRU
3.1 TIME-SPARSE GRU FORMULATION
We base our model on the GRU (Cho et al., 2014), illustrated for convenience in Fig. 1A. It consists of internal gating variables for updates (u) and reset (r), that determine the behavior of the internal state y. The state variable z determines the interaction between external input x and the internal state. The dynamics of layer of GRU units, time step I, is given by the set of vector-valued update equations:
(0) o Wa (0 100yy. 1t 1- +b.)
( = Wr ( 1y (t-1)
10] +b..)
(4)
10 11- +bs). (1) = ( 1 oo (4) + +(1+++111111 (1- )o (1-1)
where W a/r/2/ bufr/z denote network weights and biases, @ denotes the element-wise (Hadamard) product, and a (-) is the vectorized sigmoid function. The notation [x(*), (t-13 denotes vector concatenation. The function y(-) is an element-wise nonlinearity. typically the hyperbolic tangent.
We introduce an event generating mechanisms by augmenting the GRU with rectifier (a thresholding function). See Fig. IB for illustration. With this addition the internal state variable y1(h) is nonzero only when the internal dynamics reach a threshold d and is cleared immediately afterwards, thus making y; (t) event-based. Formally, we add an auxiliary internal state ci to the model, and replace
(*)==())
= (v)* .y2 ,...) with the bvent-based form
(t)
v o.0 H (1))) -0.) with d" 2022(0 (0)())(11
(0))(0110 (t-1) +(1 (1-11)0)) u. )o -1 (y1--)
(1-1)
(2)
3

---

Published as a conference paper at ICLR 2023
A
C
GRU
EGRU
H(c)
dH
event generating muchanism
dc
EE
+
D
forward
backward
Active neuron
y
Inactive
Forward transmission of event between units
Backward transmission of gradlent between units
unit
una
Forward and backward transmission of state within unit
time -
time
Figure 1: Illustration of EGRU. A: A single unit of the original GRU model adapted from Cho et al. (2014), B: EGRU unit with event generating mechanism. C: Heaviside function and surrogate gradient. D: Forward state dynamics for two EGRU units (i and j). E: Activity-sparse backward dynamics for two EGRU units (i and j). Note that we only have to backpropagate through units that were active or whose state was close the threshold each time step.
where (-) is the Heaviside step function and 0; > 0 is a trainable threshold parameter. (-) is the threshold gating mechanism here, generating single non-zero output when c crosses the threshold it) 00. The - -,²"" term in Eq. (2) makes 8, That is, at all time steps i with ( < Vi, we have v,
emission of multiple consecutive events by the same unit unlikely, hence favoring overall sparse activity. With this formulation, each unit only needs to be updated when input is received either
externally or from another unit in the network. This is because, if both z(") =y," = for the i-th
unit, then (t) (e) (t)
(e) ri 24 are essentially constants, and hence the update for y, can be retroactively
calculated efficiently on the next incoming event.
3.2 SPARSE APPROXIMATE BPTT
The threshold activation function Hee) to decide whether to emit an event in Eq. (2) is not differentiable at the threshold úr We define surrogate gradient at that point for calculating the backpropagated gradients. The surrogate gradient defined as a piece-wise linear function that is non-zero for values of state c; between 0; E and 0.- zero whenever the internal state of the unit below 0, E, the backpropagated gradients are also 0 for all such units, making the backward-pass waass-pass sparse (see Fig. ID, E for an illustration). Note that the case where the internal unit state is above 0, +e tends to occur less often, since the unit will emit an event and the internal state will be cleared the next simulation step in that case.
shown in the inset in Fig. 1C. Since the surrogate gradient is
3.3 COMPUTATION AND MEMORY REDUCTION DUE TO SPARSITY
For the forward pass of the EGRU, an activity sparsity of (i.e. an average of a events per simulation step) leads to the reduction of operations, since they are by far the most expensive compute operation in these models. MACs are also relevant for our focus on resource constrained systems which are more likely to be based on CPUs rather than GPUs. If optimally implemented an activity sparsity of 80% will require 80% fewer MAC operations compared to a standard GRU, Computation related to external input is only performed at input times, and hence is as sparse as the input, both time and space. During the backward pass, a similar factor of computational reduction is observed, based on the backward- pass sparsity 3 which is, in general, less than a. This is because, when the internal state value is not within the of the threshold V, the backward pass is skipped, as described in section 3.2. Since wackward pass is also sparse,
accumulate operations (MAC), by factor a. We focus on MAC
4

---

Published as a conference paper at ICLR 2023
need to store only / fraction of the activations for later use, hence also reducing the memory usage.
In all our experiments, we report activittysparriity values calculated through simulations.
4 THEORETICAL ANALYSIS OF THE EGRU
To further analyze the dynamics of EGRU we develop a continuous time version of the model. This allows us to study the model parameters as dynamical system.
4.1 LIMIT TO CONTINUOUS TIME
Eq. (1) of the discrete time model considers the GRU dynamics only at integer time points, to = 0, 11 = 1, 12 = 2..... However, in general it is possible to express the GRU dynamics for an arbitrary time step At, with in n-1 + At. The discrete time GRU dynamics can be intuitively interpreted as an Euler discretization of an ordinary differential equation (ODE) (Jordan et al., 2021) (see Supplement), which we extend further to formulate the EGRU. This is equivalent taking the continuous time limit At + to get dynamics for the internal state c(t). In the resulting dynamical system equations inputs cause changes the states only at the event times, whereas the dynamics between events can be expressed through ODEs. To arrive at the continuous time formulation we introduce the neuronal activations an (t), a, and a. (f), with
utt) aa (t)),
r(t) (a, (t)), z(t) = g(a (t)).
(3)
with dynamics
āx = ax bx XE x
and Tm Tm))(t - u()))) c(t)) = Ftt.a,.a,,a...
(4)
where and Tm are time constants, ctt), n(() and z(() are the continuous time analogues to e(1) (!)) and (t)) and åx denotes the time derivative of ax. The boundary conditions are defined for t = as axx()) = c0) = O. The function F in Eq. (4) determines the behavior of the EGRU between event times, i.e. when x x() 0 and y y() (0)=00) O. Nonzero external inputs and internal events cause jumps in c(l) and ax (1). For theoretical tractability, we add decay term -ay to the ODE in Eq. (3), which is implemented with small or zero time constant in the discrete time model.
To describe these dynamics we introduce the set of internal events e, e, E 6, = (s,, (s,, where s, are the continuous (real-valued) event times, and n, denotes which unit got activated. The formulation of the event generating mechanisms Eq. (2) introduced above can be expressed as event & that is triggered whenever C., (t) reaches 9. More precisely:
(82, Gr (8L)=
(5)
where the superscript... - (.+) denotes the quantity just before (after) the event. The clearing mechanism
in continuous time is expressed as resetting (s(s) to zero right after event times This is because in
continuous time the exact time at which the internal variable ci(s) (s) reaches the threshold (c:(s) )) = 0.)
can be determined with very high precision. Therefore, the value of cs(s) and the instantaneous amplitude of (s) simultancously approach 12 at time point so that the ys term in Eq. (2)
effectively resets (s) to zero, right after an event was triggered.
At the time of this event, the activations of all the units m connected to unit n, experiences a jump in its state value. The jump for ax,m is given by:
at (s,) = -x (s.) Wx, Fx,n, Co, (su),
(6)
where X (u, r, 2], I'x =x= when X E { and r'x r when x= (r). This is equivalent to y = CT being the output of each network unit. A similar jump is experienced on arrival of an external input, using the appropriate input weights instead (see Supplement for specifics). The event-based asynchronous nature of this update scheme can be formalized in the following proposition.
Let the dynamics of u(t) and e(t) be given by (3) and (4). The unperturbed dynamic (in the absence of input) of internal states c(t) are decoupled (the variables of c(t) do not interact with each other).
Proposition 1.
Proof. The proof follows directly from the definition (4). While variables of single units interact (e.g. u, (t) and (t)), the dynamics of variables of all units are decoupled from each other in the absence of inputs.
5

---

Published as a conference paper at ICLR 2023
4.2 EVENT-BASED GRADIENT- USING ADJOINT METHOD
To show that the EGRU gradient updates are event-based, we define the loss over duration T as So fc(c(t),t)dt where (t is the instantancous loss at time t. I' is a task-specific time duration within which the training samples are given to the network as events, and the outputs are read out. In general (c((tt),t) may depend arbitrarily on c(t), however in practice choose the instantaneous loss to depend on the EGRU states only at specific output times to adhere to our fully event-based algorithm.
The loss is augmented with the terms containing the Lagrange multipliers Ae; Aoz to add constraints defining the dynamics of the system from Eqs. (3), (4). The total loss L thus reads
C= c= (dcc(t),)) F aa c))+
Σ tT dt.
(7)
XE(urra)
The Lagrange multipliers are referred to as the adjoint variables in this context, and may be chosen freely since both Tm c()) ar, and T,åx +ax are everywhere zero by construction.
can can choose dynamics and jumps at events for the adjoint variables in such a way that they can be used to calculate the gradient dC
dwy, Calculating the partial derivatives taking into account the discontinuous jumps at event times depends on the local application the implicit function theorem, which requires event times to be a differentiable function of the parameters. See the Supplement for full derivation.
The time dynamies of the adjoint variables is given by the following equations with a boundary condition of (T) Au, (T)
OFF
OF Oax
A Am
Aax
Ac-Taaa- =0,
(
ac
for X {u,r,z), and A denoting the transpose of the matrix M. The event updates for the adjoints are described in the Supplement. In practice, the integration of À is done backwards in time. In analogy to Proposition 1 we find the following property of the forward dynamics.
Proposition 2. Let At( and Xay (t) be given by the adjoint dynamics Eq. (8). Let initial conditions of Xe be given by (s Then the system of differential equations for 8, <tt& 8, is decoupled (the variables of X do not interact with each other).
Proof. The proof follows from (8). Since the dynamics of ax and e are decoupled (unit-wise) between events as shown in Proposition 1, the dynamics of A are also decoupled. The term (OF/0a, X reduces to a vector without cross-unit unit interactions shown in the Supplement.
For the recurrent weights Wx.ij from the different parameter matrices Wx for E we can write the weight updates using only quantities calculated at events eg as Awx,ij the following property of the weight updates.
â L(W). We find 01x,,35
Theorem 1. Let (t and >a, (t) be given by the adjoint dynamics Eq. (8) such that Propositions I and 2 hold. Let (xx(s)) = independent of euch other and fully determined by pix (8)) and XT, (82), the set of EGRU states evaluated at event times 8,, i.e. Awx,i - L (s.) X (s2).
(s..). Then the weight updates Awx,ss Twx,ss L(W) are
Proof sketch. Here, @ is the outer product, cm refers to the value of c(t) just before event Cx. r = 0 for X E } and equal to the value of r r(t) just before event eR for X = {r]. Xt, refers to the value of the adjoint variable Ao just after the event ex, and le| the total number events. Thus, the values of r()),c() need to be stored only at event times, and Any (1) needs to be calculated only at these times, making the gradient updates event- event-based. See the Supplement for more details to the proof and for update rules for the input weights and biases. Finally we find the following corollary for the algorithmic complexity of the learning algorithm.
Gäven 2 (sx) and (s,(s) the time complexity of parameter update computation AWx,sj grows linearly with K (the number of events).
Corollary 1.
Proof. The proof follows from Theorem 1. The computation required for the outer product #x (s& & A (st) depends on the network size and is independent the total number of events K. The sum thus grows linearly with K.
n

---

Published as a conference paper at ICLR 2023
5
RESULTS
5.1
GESTURE PREDICTION
A
B
input time
Input
olf events
activity
1 layer
unit
30
2 layer
x
unit
200
600
800
time (ms)
1000
20
step
00
Illustration of DVS gesture classification data for example class (right hand wave). On (red) and off (blue) events are shown over time and merged into a summary image for illustration (not presented the network). B: Sparse activity of input and EGRU units (random subset of 30 units shown for each layer).
Figure 2: A:
We evaluated our model on gesture prediction, which is a popular real-world benchmark for RNNS and widely used in neuromorphic research. The DVS128 Gesture Dataset (Amir et al., 2017), provides sparse event- event-based imputs which enables to demonstrate our model's performance and computational efficiency. The dataset contains 11 gestures from 29 subjects recorded with a DVS DVS128 event camera (Lichtsteiner et al., 2008). Each event encodes a relative change of illumination and is given as spatio-temporal coordinates of X/Y position the 128 128-pixel sensor and time stamp. We combined the raw event times into 'frames' by binning them over time windows of 25 ms, and then downscaled them to 32 x 32 pixels using maxpool layer.
model
hidden dim
para- meters
effective MAC
test accuracy
activity sparsity
backward sparsity
LSTM (He et al., 2020)
512
7.4 M
3.9 M
86.8%
AlexNet+LSTM--A
256
M
601.3 M
97.7%
(Innocenti et al., 2021)
GRU EGRU
1024
15.7 M
10.6 M
88.1 1%
0%
512
5.5
0.98 M
88.0%
83.8%
46.8%
GRU+DA
1024
15.7 M
10.6 M
95.1%
0%
EGRU+DA
1024
15.7
3.2
97.1%
78.8%
58.2%
CNN+GRU+DA
136
2.1 M
79.1 M
97.4%
0%
CNN+EGRU+DA 795
4.8
80.6 M
97.8%
76.4%
72.3%
Table 1: Model comparison for the DVS Gesture recognition task.
Effective number of MAC
operations as described in section 3.3. DA stands for Data Augmentation.
Comparison of model performance on gesture prediction presented in Table The backward sparsity as described in Section 3.3 was calculated at epoch 100. EGRU consistently outperformed GRU networks of the same size on this task by a small margin. Adding data augmentation (DA) by applying random crop, translation, and rotation, previously done in Innocenti et al. (2021), ), increased the perfor- mance of pure RNN EGRU architecture to over 97%, coming close to state-of-the-an architectures with- out costly AlexNet pre-processing. We improved this result with a CNN feature extraction head adapted from ALexNet, DropConnect (Wan et al., 2013) applied to the hidden- to to-hidden weights and Zone- out (Krueger et al., 2017), outperforming Innocenti et al. (2021) with far fewer parameters and order of magnitude reduction in MAC operations. Further experimental details, ablation studies and statistics over different runs can be found in the supplement sections 1. E. E.1.1 and tables S1, S4 respectively.
7

---

Published as a conference paper at ICLR 2023
5.2 SEQUENTIAL MNIST
We evaluated the EGRU on the sequential MNIST and permuted sequential MNIST tasks (Le al., 2015), which are widely used benchmarks for RNNs. In the sequential MNIST task, the MNIST handwritten digits were given as input one pixel at time, and at the end of the input sequence, the network output was used to classify the digit. For the permuted sequential MNIST task, the pixels permuted using fixed permutation before being given input in the 1 -llayer EGRU with 590 units (matching the number of parameters with a 512 unit LSTM). We did not any regularisation to increase sparsity in this task, that could test how much sparsity, both forward and backward, arises naturally in the EGRU
way. We trained
In Table 2, we report the results of discrete-tinn time EGRU along with other state-of sate-of-ther the art architectures. EGRU achieved a task performance comparable to previous architectures while using much fewer operations (more than an 5-fold reduction in effective MAC operations compared to GRU). Further experimental details, and statistics over different runs can be found in the supplement sections E.2 and table S5 respectively.
dataset
model
hidden dim
para- meters
effective MAC
test accuracy
activity sparsity
backward sparsity
coRNN
256
134 K
262 K
99.4%
et al., 2020)
512
I 1
M
98.9%
SMNIST
GRU
590
I M
M
98.8%
EGRU
590
I M
226 K
98.3%
72.1%
27.4%
CoRNN
256
134
262 K
97.3%
PSMNIST (Gu et al., 2020)
512
I IM
M 95.1%
EGRU
590
IM
195 K
95.1%
82.0%
8.4%
Table Model comparison on sequential MNIST (sMNIST) and permuted sequential MNIST (psMNIST) task. Top-1 test scores, given as percentage accuracy, where higher is better. coRNN is the model described in Rusch and Mishra (2021).
5.3 LANGUAGE MODELING
We evaluated our model on language modeling tasks based on the PrnnTreebank (Mareus al., 1993) dataset and the WikiTeext-2 dataset (Merity et al., 2017). We focused exclusively on the RNN model this work, and did not consider techniques such as neural cache models (Grave et al., 2017), Catture-of-Sftmaaae
(Yang et al., 2018 8) or dynamic evaluation (Krause et al., 2018), all of which can be used on top of our model. strong baseline for gate- RNN architectures was established by Merity et al. (2018). Similarly, our models consisted of three stacked EGRU cells without skip connections. DropConnect (Wan et al., 2013) was applied to the hidden-to to-ec--tt-dddeen weights. The weights of the final softmax layer were tied to the embedding layer (Inan et al., 2017; Press and Wolf, 2017). All experimental details, and statistics over different runs can be found in the Supplement in sections 1 and Table $6 respectively.
The results presented in Table 3 show that EGRU achieved performance competitive with AWD- LSTM (Merity et al., 2018). At the same time, EGRU inherently exhibited activity sparsity that reduced the required computational operations (calculated analytically). Another desirable property of EGRU for resource constrained systems is a natural approach to model compression through pruning of least active units. We discuss experimental findings for this pruning technique using language modeling as a case study in the Supplement (see Sec. E.4).
DISCUSSION
This work introduces the EGRU, a new form of a recurrent neural network that uses a biologically inspired activity sparsity mechanism. The EGRU extends the GRU with event-generating mechanism using a threshold and uses surrogate gradients to train the model with BPTT, which leads to computationally sparse forward and backward passes. We theoretically proved that in the continuous time limit, the model's inference and training scales proportionally to the number of events generated and the number of neurons, but not with the number parameters in conventional RNN models.
8

---

Published as a conference paper at ICLR 2023
dataset
model
hidden
para-
effective MAC
test
activity sparsity
backward sparsity
dim
meters
Merity et al. (2018)
1150
24 M 52 M
24.0 M
57.3
Liet al. (2018)
51.6 M
55.2
PTB
GRU EGRU EGRU
1350
28 M
27.6 M
66.3
1350
33 M
9.9 M
57.2
79.7%
46.3%
2000
55 M
12.7 M
57.0
84.8%
42.9%
Melis al. (2018)
24 M
65.9
Merity et al. (2018)
1150
33 M
32.0 M 65.8
WikiText-2
GRU
1350
39 M
38.8 M
71.8
EGRU
1350
51 M
17.5 M
70.6
76.8 %
44.2 % 42.2 %
EGRU
2000
74 M
20.1 M
68.9
82.7 %
Table 3: Model comparison on Penn Treebank and WikiText-2. Test scores are given as perplexities, where lower is better. Effective MAC operations are given for single time step and consider the layer-wise sparsity in the forward pass. Activity sparsity given for the trained model to resemble inference sparsity. Backward sparsity is averaged over the whole training. Model parameters were optimized on Penn Treebank and transfered to WikiText-2.
The EGRU achieved competitive task performance on tasks such as gesture recognition, sequential image classification and language modeling while achieving activity sparsity up to 85% (15% of the units active on average). Scaling up networks for language modeling has shown some of the most promising results in the last few years (Brown et al., 2020), hence, our choice of this task, albeit on smaller scale. Considering the need for extensive hyperparameter search (Melis et al., 2018) for language modeling. our model achieved promising results while maintaining a high degree of setivity-sparsity. For example, our EGRU with 1350 hidden units reached perplexities better than reported by LSTM and GRU baselines, while maintaining the high level of activity-sparrity To the best of our knowledge, this the first demonstration of such activity sparsity mechanisms that yields strong benchmark performance compared to baselines.
While we base our model on the GRU due to its simplicity, this formulation can easily be extended to any arbitrary network dynamics, including the LSTM, allowing specialized architectures for different domains. The adjoint method for hybrid systems that we use for analysis here can also be used as a pow- erful general-purpor purepose tool to formulate cotinuous time models that activity-sparse and trained using event-based gradient descent updates for any recurrent neural network architectures. Another novel outcome of this paper is that this theory handle inputs in continuous time as events, which is very intuitive, hence providing an alternative to the more complex controlled differential equations (Kidger al., 2020). The EGRU can also be used for irregularly spaced sequential data quite naturally.
The compute efficiency of this model can directly translate into gains in energy efficiency when implemented using event-based software primitives. The model will work well on heterogeneous compute resources, including pure CPU nodes, neuromorphic devices such as Intel's Loihi (Davies et al., 2018) and SpiNNaker 2 (Höppner et al., 2017). that can achieve orders of magnitude higher energy efficiency, well as deep learning hardware that support dynamic sparsity, such as the Graphcore system (Jia et al., 2019). On neuromorphic devices with on-chip memory in the form of crossbar array, the activity sparsity directly translates into energy efficiency. For larger models that need off-chip memory, activity-pparsity needs to be combined with parameter-sparsity to reduce energy-intensive memory access operations.
In summary, with the motivation of building scalable, energy energy-efficient deep recurrent models, we pre- sented an activity sparsity mechanism that reduces the required compute for both inference and learning. We demonstrated that the EGRU our CRU-based model using this activity sparsity mechanism - is a competitive alternative to GRU and LSTM models, especially for resource constrained systems and neuromorphic devices. In future work, we plan to combine this activity sparsity mechanism with meth- ods for parameter sparsity and learning long-range dependencies and implement them on neuromorphic hardware realise the efficiency gains and scalability of recurrent neural network architectures.

---

Published as a conference paper at ICLR 2023
7 REPRODUCIBILITY
We ensure that the results presented in this paper are easily reproducible using just the information provided in the main text well as the appendix. Details of the models used in our experiments are presented in the main paper and further elaborated in the appendix. We provide additional experimental details, ablation studies, and statisties over multiple runs in the appendix Section E. We use publicly available libraries (Appendix section G) and Datasets (Appendix section F) in our experiments. We will further provide the source code to the reviewers and ACS in an anonymous repository once the discussion forums opened. The included code will also contains "readme" texts to facilitate easy reproducibility. The theoretical analysis provided in Section is derived in the appendix along with the event- based learning rule.
ACKNOWLEDGEMENTS
The authors gratefully acknowledge the GWK support for funding this project by providing computing time through the Center for Information Services and HPC (ZIH) at TU Dresden. We acknowledge the use of Fenix Infrastructure resources, which are partially funded from the European Union's Horizon 2020 research and innovation programme through the ICEI project under the grant agreement No. 800858. AS funded by the Ministry of Culture and Science of the State of North Rhine Westphalia, Germany. KKN is funded by the German Federal Ministry of Education and Research (BMBF) within the KI-ASIC project (1 (I6ES0996). MS is fully funded by a grant of the Bosch Research Foundation. CM receives fundeding from the German Research Foundation (DFG, Deutsche Forschungssemmeinschaffii as part of Germany's 's Excellence Strategy EXC00 - Project ID 390696704 Cluster Excellence "Centre for Tactile Internet with Human-ing -araan-----th--oygg (CeTTl of Technische Universität Dresden. DK is funded by the German Federal Ministry of Education and Research (BMBF) within the project EVENTS (16ME0733). The authors would like thank Darjan Salaj. Melika Payvand, Markus Murschitz, Franz Scherr, Robin Schiewer for helpful comments on this manuscript. AS would also like to thank Darjan Salaj and Franz Scherr for insightful early discussions, and Laurenz Wiskott for institutional support.
of
REFERENCES
A. Amir, B. Taba, D. Berg, T. Melano. J. McKinstry, C. Di Nolfo, T. Nayak, Andreopoulos, G. Gar- reau, M. Mendoza, et al. A low power, fully event- gesture recognition system. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7243-7252, 2017.
R. D. Beer. On the Dynamics of Small Continuous- MaminnuuusTTime Recurrent Neural Networks. Adaptive Behavior, ((4):469-50) Mar. 1995. ISSN 1059-7123, doi:
G. Bellec, D. Salaj, A. Subramoney, R. Legenstein, and W. Maass. Long short-term memory and Learning-to-learm in networks of spiking neurons. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 787-797. Curran Associates, Inc., 2018
G. Bellec, F. Scherr, A. Subramoney, E. Hajek, D. Salaj, R. Legenstein, and W. Maass. A solution to the learning dilemma for recurrent networks of spiking neurons. Nature Communications, 11 (1):3625, July 2020. ISSN 2041-1723. doi:
J. Bradbury. S. Merity, C. Xiong, and R. Socher. Quasi-recurrent neural networks. In International Conference on Learning Representations, 2017. URL https: //openreview.oo net. =o
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language Models are Few-Shot Learners. arXiv:2005.1 14165 [cs], July 2020.
B. Chang, M. Chen, F. Haber, and F. H. Chi. AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks. arXiv: (99.0099899 [cs, stat), Feb. 2019.
R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural Ordinary Differential Equations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Gamett,
10

---

Published as a conference paper at ICLR 2023
editors, Advances in Neural Information Processing Systems 31, pages 6572-6583. Curran Associates, Inc., 2018.
R. Q. Chen, B. Amos, and M. Nickel. Learning Neural Event Functions for Ordinary Differential Equations. In International Conference Learning Representations, Sept. 2020. URL https: //tpee/eeewwwweeeeeee/eeee net Coorummdd=k__ppppe
K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7724-1734 Doha, Qatar, Oct. 2014. Association for Computational Linguistics. doi: D0.1115/11/D14-111 URL wwwph:ttpst.. a antthnthoooooo www.olygyyyyoooooooooooogggggege /tgpD14-1//
M. Davies, N. Srinivasa, T.-H. Lin, G. Chinya, Y. Cao, S. H. Choday, G. Dimou, P. Joshi, N. Imam, S. Jain, et al. Loihi: neuromorphic manycore processor with on-chip learning. Ieee Micro, 38 (1):82 99, 2018.
E. De Brouwer, J. Simm, A. Arany, and Y. Moreau. GRU-ODE- Continuous Modeling of Aporadically-Observe Time Series. Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: http:::://proceedings. wwwpiinnnnn.nuirss.eee nwwrips. cc/ paper/201 20/9//1110000 hash/ haash/455cbe 455c426577777 aaa (tt.cce222622.eeee22eeeeeee www.aaeeeeaa0000bb5000e003 ratt mastraacthtl..
S. K. Esser, P. A. Merolla, J. V. Arthur, A. S. Cassidy, R. Appuswamy, A. Andreopoulos, D. J. Berg, L. McKinstry, T. Melano, D. R. Barch, C. di Nolfo, Datta, A. Amir, B. Taba, M. D. Flickner, and D. S. Modha. Convolutional networks for fast, energy-efficient neu- romorphic computing. Proceedings of the National Academy of Sciences, 14441 doi: 10.1073/pnas.160855113. URL
11446, Nov. 2016. ISSN 0027-8424, 1091-6490.
http:/ www. www...na........
W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(12)) 2022. URL http: / jmlr www..rrrr..o...............ooo.r papers httpr///v//////// html.
S. Galán, W. F. Feehery, and P. I. Barton. Parametric sensitivity functions for hybrid discrete/continuous systems. Applied Numerical Mathematics, 31(1):17-47, Sept. 1999. ISSN 0168-9274. doi: 10.
E. Grave, A. Joulin, and N. Usunier. Improving neural language models with a continuous cache. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net 2017. URL https:/ hitts::/cpenrevatt wpwpreeeeeeeeeeeeeeeeeeee net/ Forum?1d==18
A. Gu, C. Gulcehre, T. Paine, ing Mechanism of Recurrent Neural Networks. In Proceedings of the 37th Interna- tional Conference on Machine Learning, pages 3800-3809. PMLR, Nov. 2020. URL https: http::s:/pproeeednnng..
Hoffman, and R. Pascanu. Improving the Gat-
mlr www.m.rr.rrrs...mmmmmmmmmmm. ptrress/vi9/// gtt.//gu...httte.m
W. He, Y. L., Deng, L. H. Wang, Y. Tian, W. Ding. W. Wang, and Y. Xie. Comparing snns and mas on neuromorphic vision datasets: Similarities and differences. Newral Networks, 132:108-120, 2020.
T. Hoefler, D. Alistarh, T. Ben-Nun, N. Dryden, and A. Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241): 1-2224,.... 2021.
S. Höppner, Y. Yan, B. Vogginger, A. Dixius, J. Partzsch, F. Neumärker, S. Hartmann, S. Schiefer, S. Scholze, G. Ellguth, al. Dynamic voltage and frequency scaling for neuromorphic many-core systems. In 2017 IEEE International Symposium Circuits and Systems (ISCAS), pages 1-4. IEEE, 2017.
K. Hunter, L. Spracklen, and S. Ahmad. Two sparsities are better than one: unlock- ing the performance benefits of sparse-sparse networks. Neuromorphic Computing and Engineering, 2(3):034004, jul 2022. doi: 10.1088226444488/ncc..... URL ftttp::/dxx.oi..cret 2333

---

Published as a conference paper at ICLR 2023
H. Inan, K. Khosravi, and R. Socher. Tying word vectors and word classifiers: A loss framework for language modeling. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017. Conference Track Proceedings. OpenReview.net, 2017. URL https: /tpp///peeeeeeeeeeeeeeeeeeeeeepe ew net forum? b-rl aPbsFle.
S. U. Innocenti, F. Becattini, F. Pernici, and A. Del Bimbo. Temporal binary representation for event-based action recognition. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 10426-10433 10432. IEHE, 2021.
Z. Jia, B. Tillman, M. Maggioni, and D. P. Scarpazz.. Dissecting the graphcore ipu architecture via microhenchmarking. arXiv preprint arXiv: 191203311, 2019.
I. D. Jordan, P. A. Sokót, and I. M. Park. Gated recurrent units viewed through the lens of continuous time dynamical systems. Frontiers in computational neuroscience, page 67, 2021.
J. Jordan, T. Ippen, M. Helias, I. Kitayama, M. Sato, J. Igarashi, M. Diesmann, and S. Kunkel. Extremely Scalable Spiking Neuronal Network Simulation Code: From Laptops Exascale Computers. Frontiers in Neuroinformatics, 12:2, 2018. ISSN 1662-5196.
Kidger, J. Morrill, J. Foster, and Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33:6696-6707, 2020.
B. Krause, E. Kahembwe, Murray, and Renals. Dynamic evaluation of neural sequence models. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2766-2775. PMLR, 10-15 Jul 2018. URL https: http:///occeedddeeeeo mlr www.prsss............m.m..mmm presss/ v aaa.aeeaaaaaaa............
D. Krueger, T. Maharaj, J. Kramar, M. Pezeshki, N. Ballas, N. R. Ke. A. Goyal, Y. Bengio, A. Courville, and C. Pal. Zoneout: Regularizing RNNs by randomly preserving hid- den activations. In International Conference on Learning Representations, 2017. URL
https://cgpt httts:://openeeiew net feett/ouundd---ggppeoao
Q. V. Le, Jaitly, and Hinton. A Simple Way to Initialize Recurrent Networks of Rectified Linear Units, Apr. 2015.
00941.
M. Lechner and R. Hasani. Learning Long-Termi Dependencies SBti:2000.044116 [cs, stat), Dec. 2020. URL http::tttt.x....oooooo hbtp://bbb//0000 2006. 04418.
i in Iregulllt Errguularly-ammpled Time Series.
Lei, Y. Zhang, Wang, H. Dai, and Y. Artzi. Simple recurrent units for highly parallelizable recurrence. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4470-4481, Brussels, Belgium, Oct.-Nov. 2018. Association for Computational Linguistics. doi: 10.18653/v 1/D13/////111/// 1477.
ptatttss://ccaantholooooo
org/D18 1477.
Li, W. Li, C. Cook, C. Zhu, and Y. Gao. Independently recurrent neural network (indrnn); Building longer and deeper rnn. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
P. Lichtsteiner,, C. Posch, and T. Delbruck. A 128x 128 120 dB 15 latency asynchronous temporal contrast vision sensor. journal of solid-state state circuits, 43(2):566-55) 6766,,5766666 2008
W. Maass. Networks of spiking neurons: the third generation of neural network models. Neural networks, 1009):1659-1671 1997.
M. P. Marcus, M. A. Marcinkiewicz. and B. Santorini. Building large annotated corpus of english: The penn treebank. Compur. Linguist., 19(2):313-33.juu 1993. ISSN 0891-2017.
C. Mead. How created neuromorphic engineering, Nature Electronics. 3(7):434 435, 2020.
G. Melis, C. Dyer, and P. Blunsom. On the state of the art of evaluation in neural language models. In 6th International Conference Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https: https://cpenret http://ppeeeevveeeeeee net forum? BBy
Learning Representations, ICLR 2018, Vancouver. BC,
12

---

Published as a conference paper at ICLR 2023
S. Merity, C. Xiong, J. Bradbury. and R. Socher. Pointer sentinel els. International Conference on Learning Representations. 2017. URL https: //tp////////// eeeenreveewnnt//eeee wttp:m/eee.ommmmdd.yyyyaa.a
mixture mod-
S. Merity, N. S. Keskar, and R. Socher. Regularizing and optimizing LSTM language models. In International Conference on Learning Representations, 2018. URL
https:// /ttp:///ppeeeeewww net foor/oormmm/// Doruumid-SyyGGPOzZ
D. Neil, M. Pfeiffer, and S.-C. Liu. Phased Istm: Accelerating recurrent network training for long event-based sequences. In D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https: htttp::/pproceddiiii http://pptceeeng.seeeeeee neurips ott.a///eeea...ooroooooo///e//// File/Sbce843dd 6b
D. Neil, J. H. Lee, T. Delbruck, and S.-C. Liu. Delta Networks for Optimized Recurrent Network Com- putation. In International Conference on Machine Learning, pages 2584-2599 PMLR, July 2017.
C. Ostrau, C. Klarhorst, M. Thies, and U. Rückert, Benchmarking neuromorphic hardware and its energy expenditure. Frontiers in neuroscience, 16, 2022.
N. Perez-Nieves and D. M. Goodman. Sparse spiking gradient descent. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URI / /openrevi ew net. / forum? wtt.rmmmm.daaaa22ettetee d=a1F2sEtMMXX
L. Pontryagin, V. Boltyanskiy, V. Gamkrelidze, and MISHCHENKO, Mathematical theory of optimal processes. CRC press, 1962.
O. Press and I.. Wolf. Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter the Association for Computational Linguistics: Volume Short Papers, pages 157-163, Valencia, Spain, Apr. 2017. Association for Computational Linguistics. httptt:::/ http://///alaaathhooocooo
A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot tex--to-image generation. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceed- ings of Machine Learning Research, pages 8821-8831. PMLR, 18 24 Jul 2021. URL http:://pooeeeeing.s.iemmem www..pres//v... http:////////r/a/ee/ee/ ramesh2me1h11aa a.mml.
K. Roy, A. Jaiswal, and P. Panda. Towards spike- spike-based machine intelligence with neuromorphic computing. Nature, 575(7784):007-6177 6172,,, 2019.
T. K. Rusch and S. Mishra. Coupled oscillatory recurrent neural network (co(rnn) )): An accurate and (gradient) stable architecture for learning long time dependencies. In International Conference on Learning Represenaaionss 2021. URL https: /oppeeeeviww.eeeeeeeeeee htte:/oorumiii--33s3..cc
D. Salaj, A. Subramoney, C. Kraisnikovic, G. Bellec, R. Legenstein, and W. Maass. Spike frequency adaptation supports network computations on temporally dispersed informa- tion. eLife, 10:e65459, July 2021. ISSN 2050-084X. doi: 10.7554/eLife.65559 URL
https://doi. /10. 75541......555 7554/eLife, 65459.
Schaefer, Gehrig, and D. Scaramuzza. Aegnn: Asynchronous event-based graph neural networks. arXiv preprint a2Xiv: /7149, 2022.
C. D. Schuman, Plank. A survey of neuromorphic computing and neural networks in hardware. arXiv preprint arXiv:1705.0696: 069000 2017.
Potok, M. Patton, Birdwell, M. Dean, S. Rose, and J.
N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Le, G. Hinton, J. Dean. Outrageously large neu- ral networks: The sparsely-gated mixture-of-experr layer. In International Conference on Learning Representations, 2017. URL ht wewereeeeeeeeeeeeeeeeeeee net. forum? d==1 ckMDql
13

---

Published as a conference paper at ICLR 2023
L. Wan, M. Zeiler, S. Zhang, Y. Le Cun, and R. Fergus. Regularization of neural networks using dropconnect. International Conference Machine Learning, volume 28 of Proceedings of Machine Learn- ing Research, pages 1058-1066, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR. URL
Dasgupta and D. McAllester, editors, Proceedings of the 30th
https: http:////roeeeeeeee ngs ml www.sr.pre///ee http////////aa/33 html.
C. Wunderlich and Pehle. Event-bused backpropagation can compute exact gradients for spiking neural networks. Scientific Reports, 11(1):12829, June 2021. ISSN 2045-2322. doi:
URL. https: htttps://www.aauuuue nwwur. com/ www./rtiiileeeeeeeoooeee.eee..ee
84 s41598- 020---0011111----1111111111-111-
M. Yan, N. Meisburger, T. Medini, and A. Shrivastava. Distributed slide: Enabling training large neural networks on low bandwidth and simple cpu-clusters via model parallelism and sparsity. arXiv preprint arXiv:2201, /2677 2022.
Z. Yang, Z. Dai, R. Salakhutdinov, and W. W. Cohen. Breaking the softmax bottleneck: A high-rank RNN language model. In International Conference on Learning Representations, 2018. URL ht https: http//pppeeeeeeeeeeeeeeeeeeeeeep .nt/ HL
Zenke and S. Ganguli. SuperSpike: Supervised Learning in Multilayer Spiking Neural Networks. Neural Computation, 3066):111-1541, Apr. 2018. ISSN 0899-7667. doi: 10. 12.1162//cc_ a 010086.
URL https:://///.. /tttpp://doi.ooeee htt.//oi..ogrgg// 1262/1162/0 newo//ecccccccccco a 01086.
14


Reviews:
[{"review_id": "VVYTSqVwQw", "reviewer": "Reviewer_MdUd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors show that by including the biologically grounded constraint that neural activity is sparse, learning in a recurrent neutral network can be improved, both in the areas of computation (memory footprint,...) and accuracy. Using a continuous time extension of the recurrent neural network model, they theoretically prove the properties of the proposed model, and in particular the modulation of the computational cost with sparsity of the activity. The numerical results provide encouraging results for applying this methodology to generic networks and in particular their use in neuromorphic hardware.\n", "summary_of_the_review": "In summary, the authors provide an important and novel contribution to recurrent neural networks that have important applications to neuromorphic hardware. The work could be further improved by showing mechanically why their solution is improved over SOTA.\n", "strength_and_weaknesses": "The strength of the paper lies in the focus on the sparsity inducing mechanism and its theoretical and numerical application. In particular, the success of several benchmarks such as DVS gesture, sequential MNIST and natural language processing clearly demonstrate the advantage of this method. \nOne weakness of the paper lies in the lack of analysis of the encoding resulting from the introduction of the sparsity induction mechanism. For example, Figure 2 shows the activity produced by a segment of the DVS gesture dataset, the output of which in the different layers is qualitatively very different from the raster plots observed in biology. Further analysis of the average frequency of completion could help to understand how this could be improved, for example by incorporating homeostatic mechanisms that optimize the information carried by each neuron (for simplicity, a neuron that does not respond or always responds carries no information about the sensory input). Also, what is the influence of thresholding on accuracy? This would be an added value to the article.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OL6YBjm8b-h", "reviewer": "Reviewer_SZU5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a biologically inspired event-generating mechanism that they integrate within GRU units for enhancing activity sparsity and reducing train/test compute budget. They show that the time complexity for computing weight updates in the continuous time limit is proportional to the number of events generated in EGRU. The authors show competitive performance of their sparse EGRU network on multiple machine learning benchmarks related to RNNs despite the network producing high activity sparsity and low effective MACs (highly desirable).", "summary_of_the_review": "This work proposes an effective solution to training RNNs with less computational requirements using activity sparsity found in biological neurons. The authors have theoretically proven that the time complexity to calculate EGRU's weight updates grows linearly with number of events, and evaluated their EGRU model rigorously on a variety of RNN-relevant benchmarks and compared performance against a set of well chosen baselines. It is clear that the proposed EGRU model is capable of achieving high performance with high activity sparsity and potential gains are even larger when EGRU is implemented on suitable neuromorphic hardware. This work is highly promising and is of great relevance to ICLR audience and hence, I recommend acceptance of this paper.", "strength_and_weaknesses": "Strengths:\n1) While recurrent neural networks come with several advantages such as low parameters (owing to weight tying), ability to effectively integrate long-range dependencies and model sophisticated functions with nonlinear dynamics, their relevance for applications is restricted by their high computational budget especially during training caused by BPTT. Proposed work addresses this central issue of RNNs by taking inspiration from sparsity in biological neurons. \n2) Evaluations performed on DVS gesture recognition, sequential MNIST and language modeling highlight that EGRU can match/outperform related RNN models in task performance while using a relatively much lesser computational requirements.\n3) The potential to use neuromorphic hardware to maximize the gains produced by sparse activations of EGRU is very exciting.\n\nWeaknesses:\n1) Some of the compared baselines aren't strong enough. E.g. in Table.1, it looks like DA produces a big gain in performance on EGRU. Why did the authors not evaluate GRU or LSTM with DA? What is the CNN+EGRU adding to the comparison? It looks like the effective MACs are much higher on CNN+EGRU networks in return for a marginal gain in test accuracy. On a related note in Table 3, I don't see any difference in number of parameters between EGRU and GRU with the same hidden dim in Table 1. Why is there a difference between matched GRU and EGRU model number of parameters in table 3?\n2) What are some of the limitations of the proposed EGRU model? It would be great if the authors could please further highlight room for improvement of EGRU apart from combining with neuromorphic devices in the discussion.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6CJss-sfUg", "reviewer": "Reviewer_KSan", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present an event-based version of GRUs that allows for sparse forward and backward passes that achieves similar (sometimes even better) task performance than vanilla GRU but at a lower compute cost. The authors also present theoretical results indicating that the computation complexity doesn't scale with the network size, but instead with the number of events. Although it is unclear how does the number of events scale with network size (I personally feel that it would depend on the task and training loss function, and is therefore hard to make a generic statement), the implications of this result are significant in terms of efficiently scaling up network size for solving complicated tasks. In this work, the authors support their claims with empirical evidence on gesture recognition, sequential MNIST and language modeling tasks. ", "summary_of_the_review": "Overall, I would recomment accepting the paper. I feel the paper can be slightly improved by updating the motivation by talking about connections to ReLU non-linearity for RNNs, which are also supposed to sparsify the activations. I have currently rated the paper to be marginally above the acceptance threshold, but if the authors can address some of the weaknesses mentioned above, I am happy to increase it to 8.", "strength_and_weaknesses": "Strengths:\n1. The paper presents a strong motivations for their solution and is overall well-written.\n2. The theoretical results present strong evidence of why their solution enables efficient scaling up of models.\n3. The EGRU architecture contributes to reducing the memory footprint of BPTT, which is indeed a significant step towards making BPTT implementable in long sequences (instead of relying of truncated BPTT). It would be interesting to see how EGRU performs with some of the biologically-proposed temporal credit assignment rules, that rely on erasing out credit signals carried over long temporal horizons.\n\nWeaknesses:\n1. I feel the EGRU is quite similar to the Identity-RNNs proposed by Le, Jaitly & Hinton in 2015, wherein they used a ReLU non-linearity with RNNs (already cited by the author in the context of sequential MNIST experiments). Although EGRU has a mechanism to avoid multiple firing of the same unit, which is different from the Identity-RNN formulation, I would be curious to know any corresponding links/ comparison to the IRNN architecture. \n2. Since one of the central claims that the authors make relates to scalability, it would be nice to see how the performance and activity/backward sparsitiy changes when the number of hidden units in EGRU are changed. A trend to empiricially demonstrate how these quantities change with scaling up the network would be useful.\n3. A minor point (feel free to ignore if it is more along the directions in future work): it would be interesting to see how the sparsity changes across training. The authors currently report the average sparsity of the network but I would be curious to know if the operational sparsity is lower than that of an untrained network because the network eventually converged to such a solution. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%