EGRU: Event-based GRU for activity-sparse inference and learning
Authors: Anand Subramoney
Khaleelulla Khan Nazeer
Mark Schöne
Christian Mayr
David Kappel
Venue: NeurIPS 2022 Submitted (2022)
Keywords: activity sparse
event based
scalable
recurrent networks
GRU

Abstract: The scalability of recurrent neural networks (RNNs) is hindered by the sequential dependence of each time step's computation on the previous time step's output. Therefore, one way to speed up and scale RNNs is to reduce the computation required at each time step independent of model size and task. In this paper, we propose a model that reformulates Gated Recurrent Units (GRU) as an event-based activity-sparse model that we call the Event-based GRU (EGRU), where units compute updates only on receipt of input events (event-based) from other units. When combined with having only a small fraction of the units active at a time (activity-sparse), this model has the potential to be vastly more compute efficient than current RNNs. Notably, activity-sparsity in our model also translates into sparse parameter updates during gradient descent, extending this compute efficiency to the training phase. We show that the EGRU demonstrates competitive performance compared to state-of-the-art recurrent network models in real-world tasks, including language modeling while maintaining high activity sparsity naturally during inference and training. This sets the stage for the next generation of recurrent networks that are scalable and more suitable for novel neuromorphic hardware.

EGRU: Event-based GRU for activity-sparye inference and learning
Anonymous Author(s) Affiliation
Address
email
Abstract
The scalability of recurrent neural networks (RNNs) is hindered by the sequential dependence of each time step's computation on the previous time step's output. Therefore, way speed up and scale RNNs is to reduce the computation required at each time step independent of model size and task. In this paper, we propose model that reformulates Gated Recurrent Units (GRU) as event-based activity-sparse model that we call the Event-based GRU (EGRU), where units compute updates only on receipt of input events (event-based) from other units. When combined with having only a small fraction of the units active at a time sectivity-sparrsee this model has the potential be vastly more compute efficient than current RNNs. Notably, activity- parameter updates during gradient descent, extending this compute efficiency the training phase. We show that the EGRU demonstrates competitive performance compared to state-of-the-ar recurrent network models real-world tasks, including language modeling while maintaining high activity sparsity naturally during inference and training. This sets the stage for the next generation of recurrent networks that are scalable and more suitable for novel neuromorphic hardware.
2
10
in our model also translates into sparse
11
13
14
15
Introduction
17
18
Large scale models such as GPT-3 [8], switch transformers [17] and DALL-E [52) have demonstrated that scaling up deep learning models billions of parameters cannot i of these models but lead to entirely new forms of generalisation. For example. GPT-3 can do basic translation and addition even though was trained only on next word prediction. While is unknown if scaling up recurrent neural networks can lead to similar forms of generalisation, the limitations on scaling them up preclude studying this possibility. The dependence of each time step's computation on the previous time step's s output is the source of significant computational bottleneck, preventing RNNs from scaling well. Therefore, in recent years, RNNs, despite their many desirable theoretical properties [15] such the ability to process much longer context and their computational power [57, 60], have been supplanted by feedforward network architectures.
19
improve the performance
20
21
23
24
25
27
28
By reducing the computation required at each time step, independent of model size and task, we can speed up and better scale RNNs. We propose do this by designing general-purpose -puposee event-based recurrent network architecture that is naturally activity-sparse. Dubbed the Event-based Gated Recurrent Unit (EGRU), our model an extension of the Gated Recurrent Unit (GRU) [12]. With bvent-based communication. units in the model can decide when to send updates to other units, which
29
30
31
32
33 then trigger the update of receiving units. Therefore, network updates are only performed at specific, most units do not send updates to other 35 units most of the time, leading substantial computational savings during training and inference.
dynamically determined event times. With activity
34
We formulate the gradient updates of the network to be sparse using a novel method, extending the benefit of the computational savings to training time.
37
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). not distribute.

---

38
The biological brain, which relies heavily on recurrent architectures and is at the same time extremely energy efficient [43]. is a major source of inspiration for the EGRU. One of the brain's 's strategies to reach these high levels of efficiency is activity-sparsiee
39
40
In the brain, (asynchronous) event-based communication just the result the properties of the specific physical and biological substrate on which the brain is built. Biologically realistic spiking neural networks and neuromorphic hardware also aim to use these principles to build energy-efficient software and hardware models [53, 58J. However, despite progress in recent years, their task performance has been relatively limited for real-world tasks compared to state-of-the-e recurrent architectures based on LSTM and GRU. We view the EGRU as generalisation of spiking neural networks, moving away from modeling biological dynamics toward a more general-purpe purpose recurrent model for deep learning.
41
42
43
44
45
47
In this paper, first introduce version of EGRU based on a principled mathematical approach that for- mulates the dynamics of the internal states the network in continuous time. The units of the network communicate solely through message events triggered when the internal state of a unit reaches thresh- old value. This allows us to derive exact gradient descent update equations for the network analogous (BPTT) that mirrors the activity-sparsity of the forward pass.
49
51
52
58 We then introduce a discrete simplification of this continuous-ting model that is also event-based and activity-sparse while being easier implement on today's prevailing machine learning libraries and thus directly comparable to existing implementations of GRU and LSTM. The backwards pass here uses an approximate version of BPTT, and these updates are also sparse.
54
55
56
57
The sparsity of the backward-past overcomes one of the major roadblocks in using large recurrent models, which is having enough computational resources to train them. We demonstrate the task performance and activity sparsity of the model implemented in PyTorch, but this formulation will also allow the model run efficiently on off-the-shelf hardware, including CPU-based nodes when implemented using appropriate software paradigms. Moreover, an implementation on novel neuromorphic hardware like [13, 27], that is geared towards event-based computation, can make the model orders of magnitude more energy efficient [48].
58
59
60
61
62
63
64
In summary, the main contributions of this paper are the following:
65
1. We introduce the EGRU, an event-based continuous-time variant of the GRU model.
66
2. We derive an event-based form of the error-back- proopaaat-oooaattiionnn algorithm for EGRU.
3. We introduce discrete-time version of EGRU that can be directly compared to current LSTM/GRU implementations.
68
4. We demonstrate that the EGRU exhibits task perffrrrfome competitive with state-of the-art recurrent network architectures (based on LSTM, GRU) on real-world machine learning benchmarks.
70
71
72
5. We show that EGRU exhibits high levels of activity-sparsity during both inference and learning.
73
74
2
Related work
75
Activity sparsity in RNNs has been proposed previously in various forms [28, 46, 47], but only focusing on achieving it during inference. Conditional computation is a form of activity sparsity used in [17) to scale to 1 trillion parameters. This architecture is based on the feedforward transformer architecture, with a separate network making the decision of which sub-networks should be active [59]. An asynchronous event-based architecture was recently proposed specifically targeted towards graph neural networks [56]. QRNNs [7], SRUs[38] and IndRNNs [39] target increasing the parallelism in a recurrent network without directly using activity-sparsin Unlike [17], our architecture uses a unit-local -pparstyyy specifically for recurrent architecture. The cost of computation is lower in an EGRU compared to [47], and can be implemented to have parallel computation of intermediate updates between events, while also being activity sparse in its output.
77
78
79
80
81
82
decision making process for the dynamic
83
84
85
Models based on sparse communication [64] for scalability have been proposed recently for feedforward networks, using locality sensitivity hashing to dynamically choose downstream units for communicating activations. This is a dynamic form of parameter-sparsity [25]. But, parameter/model-sparree is, in general, orthogonal to and complementary with our method for activity-sparsity, and can easily be combined for additional gains.
86
87
88

---

90
Biologically realistic spiking networks [41] are often implemented using event-based updates and have been scaled huge sizes [33], albeit without any task-relater performance evaluation. Models for deep learning with recurrent spiking networks [3, 55] mostly focus on modeling biologically realistic memory and learning mechanisms. Moreover, units in spiking neural network implement dynamics based on biology and communicate solely through unitary events, while units in an EGRU send real-valued signals to other units, and have more general dynamics. A sparse learning rule was recently proposed [4] that is local approximation of backpropagation through time, but not event-based.
91
93
94
96
The event-based learning rule for the continuous time EGRU is inspired by, and generalization of, the event-prop learning rule for spiking neurons [63]. As in that paper, we use the adjoint method for ordinary differential equations (ODEs) to train the continuous time EGRU [10, 50] combined with sensitivity analysis for hybrid discrete/continuous systems [1 1, 19). Using pseudo-derivativn use for our discrete-time EGRU, was originally proposed for feedforward spiking networks in neuromorphic hardware in [16] and developed further in [3, 65]. The sparsity of learning with BPTT when using appropriate pseudo-derivatives in a discrete-time feed-forward spiking neural network was recently described in [49].
98
100
for back-
propagating through the non-differ -in-differential threshold function,
102
103
104
105
A continuous time version of sigmoidal RNNS was first proposed in [2] and for GRUs in [14]. The latter used Bayesian update for network states when input events were received, but the network itself was not event-based. As in [37, 46], the focus there was on modeling irregularly spaced input data, and not on event-based network simulation or activity-sparse inference and training. [9] also recently proposed continuous time recurrent network for more stable learning, without event-based mechanics. GRUs were formulated in continuous time in [32], but purely for analyzing its autonomous dynamics.
108
107
108
109
110
Event-based GRU
111
112
3.1
Time-sparse GRU formulation
A
GRU
C
Input: (1.0)
readout
time
X;
pattern X;
recall cue X2
c,(1)
Z
y
state (c)
C2(t)
y
convolved ouput trace
B
D
readout: (1.01
EGRU
input: (0,1)
X:
pallern X2
recall cue Xs
c,(t)
state (c)
c,(0)
X
clear
convolved puput
event generating
mechanism
time
readout: (0.11
Figure 1: Illustration of EGRU. A: A single unit of the original GRU model adapted from [12]. B: EGRU unit with event generating mechanism. C.D: Dynamics EGRU internal state variables for the delay-copy task with input (1,0) (C) and (0,1) (D). Colors are matched for neurons in both plots.
113 We base our model on the GRU [12], illustrated for convenience in Fig. IA. It consists of internal
114 gating variables for updates (u) and a reset (r), that determine the behavior of the internal state
115 The state variable z determines the interaction between external input x and the internal state. The
116
dynamics of layer of GRU units, at time step t. is given by the set of vector-valued update equations:
(0) = o -o(w., W. x) ( 11-- + +ba).
1 = -o(=oW, W, (x
(t-1)) +b.).
(I)
W (0) (1) 11- (bb)) +bs). ym m oo (1) (1 )o)
3

---

117 where W u///2/ Dujr/= denote network weights and biases, & denotes the element-wise (Hadamard) product, and (-)) 1) is the vectorized sigmoid function. The notation [x(*) -y(+")] denotes vector concate- nation. The function g()) is an element dlement-wise nonlinearity (typically the hyperbolic tangent function).
118
120
We introduce an event generating mechanisms by augmenting the GRU with a rectifier and a clearing mechanism (see Fig. 1B for an illustration). This introduces
wrent-based variant of the internal state variable y; (t) that is nonzero when the internal dynamics reach a threshold tt and is cleared immediately afterwards. Formally, this can be included in the model by adding an auxiliary internal (8) (t)
122
123
state of and replacing y (0) = (v1 :3/2 ,...) with the event-based form
124
(1) (0) = ci H ( -0. yi
(0) (0) 0.) with (0) = u 21 (1 (0) )c
125
where (.) the Heaviside step function and 0. > is a trainable threshold parameter. This form is well suited for time sparsity, since (-) acts here as gating mechanism, by generating a single non-zero output when cf crosses the threshold v), That is, at all time steps t with ci < Vi, we have //8h 00. The -y; - term in Eq. (2) makes emission of multiple consecutive events by the same unit unlikely, hence favoring overall sparse activity. With this formulation, each unit only needs to be updated when an input is received either externally or from another unit in the network. This is
126
127
128
129
(t) are essentially constants, and because, both x0(0) =y (t-1) 0 for the i-th unit, then ( r 2,
131
(t) hence the update for 1/3 can be retroactively calculated efficiently on the next incoming event.
132
133
3.2 Limit to continuous time
134
The discrete time model Eq. (1) considers the GRU dynamies only at integer time points, to -==0, 0, f1 = 1,,2=2,.... However, in general it is possible to express the GRU dynamics for an arbitrary time step At. with f, = 1- + At. The discrete time GRU dynamics can be intuitively interpreted to get dynamics for the internal state c(t) starting from the discrete time EGRU model outlined above.
135
136
137
as an Euler discretization of an ordinary differential equation (ODE) [32] (see Supplement), which
138
we extend further to formulate the EGRU, This equivalent to taking the continuous time limit At - +
139
In the resulting dynamical system equations inputs cause changes to the states only at the event times,
140
whereas the dynamics between events can be expressed through ODEs. To arrive at the continuous time formulation we introduce the neuronal activations a. (t), a, (t) and a (t), with
142
u(t) = (t) r(t) = ((a, (t)), z(t) = g(a (t)).
(3)
with dynamics
T, = -a - -x, x (u,r,,)
143 and
Tm é(t = -(())==u())))-cll/))))) (c(())))ccc() F(t.a,,,,
(4)
where Tu and Tm are time constants, c(t), nnt) and ztt) are the continuous time analogues to c(0) (1 and (*)) and åx denotes the time derivative of ax. The boundary conditions are defined for t= as ax (0) = c0) = O. The function F in Eq. (4) determines the behavior of the EGRU between event times, i.e. when x(t) = 0 and (L) = O. Nonzero external inputs and internal events cause jumps in
144
145
146
147
148
c()) and (t).
Furthermore, the formulation of the event generating mechanisms Eq.(2) introduced above can be expressed in straightforward manner in continuous time. Note that in continuous time the exact time at which the internal variable (s)s) reaches the threshold (c, (s) = 0)) can be determined with very high precision. Therefore, the value of (s) and the instantameous amplitude of y (s) simultaneously approach 3; at time point 8, so that the "/ term in Eq. (2) effectively resets e; (s) to zero, right after an event was triggered. To describe these dynamies we introduce the set of internal events e, e E e, = (e), where 8, are the continuous (real-valued) event times, and n. denotes which unit got 155
149
150
151
152
154
156
activated. An event e, is triggered whenever Cn. (f) reaches O. More precisely:
(R), (s+)= On,;
(5)
where the superscript. - (.+) denotes the quantity just before (after) the event. Immediately after an event has been generated the internal state is cleared: ct, (s.) = 0. At the time of this event, the
157

---

159
activations of all the units m m n connected to unit n, experiences a jump in its state value. The jump for xx is given by:
180
xx. = (m, mm, ax Cr (8a),
(6)
where E {u, r, 2], rx rx= when E {u,2) and rx when x= (r)- This is equivalent to y/- Co. being the output of each network unit. A similar jump is experienced on arrival of an external input, using the appropriate input weights instead (see Supplement for specifies).
162
163
164
The continuous time event event-based state update is illustrated in Fig. IC and D for the delay-copy task described in Section 4.1. Two EGRU units are used here and states (4) and event times are shown. At the beginning of the trial an input pattern (x1 = 1, 22 = O, and T1 = 0, 22 = in Fig. IC and D, respectively) has to be memorized in the network and retrieved again after the recall cue (233 = 1) was given. The parameters are trained with the event- updates described in Section 3.3. The required memory is stored in the internal events and state dynamics. State updates can be performed in an event based fashion, i.e. by jumping from one event time & to the next 8,+1- be-bee state values follow the state dynamics (4) and their values not needed to perform the updates (but are shown here for the sake of illustration). By construction, the state updates for external and internal events only happen on receipt of event. Since Eqs. (3), (4) are linear ODEs, the intermediate updates due to autonomous state dynamics can also be performed cumulatively and efficiently just at event times, hence avoiding any computation in the absence of incoming events.
165
166
168
169
170
172
173
174
175
178
3.3
Event-based gradient using adjoint method
177
To derive the event- based gradient updates for the EGRU we define the loss over duration T as So (e),())))),, where (c(())))) is the instantaneous loss time I. T is a task-specific time duration within which the training samples are given to the network as events, and the outputs are read out. In
178
179
general (c((),) ))) may depend arbitrarily on c(t), however in practice we choose the instantaneous loss
180
181
to depend on the EGRU states only at specific output times to adhere to our fully event-based algorithm.
182
The loss is augmented with the terms containing the Lagrange multipliers λe; Aa, to add constraints defining the dynamics of the system from Eqs. (3), (4). The total loss L thus reads
183
C- c-ff"
(e(c(t),t)+ +Am (Tm (tt,,,,
Σ XEfu,r,>)
Aax (T +xx dt.
(7)
184
The Lagrange multipliers are referred to as the adjoint variables in this context, and may be chosen freely since both Tm é(t) ((,t))F(t,a,,... at, and 7,åx + ax are everywhere zero by construction.
185
186
We can choose dynamics and jumps at events for the adjoint variables in such a way that they can de_ Calculating the partial derivatives taking into account the
be used to calculate the gradient duy. discontinuous jumps at event times depends on the local application of the implicit function theorem, which requires event times to be differentiable function of the parameters. See [10, 19, 63] for description of applying the adjoint method for hybrid discrete/continuous time systems with further theoretical background, and the Supplement for derivation specific to the EGRU.
188
189
190
182
The time dynamics of the adjoint variables is given by the following equations with boundary condition of (T) = =xa, (T)=
193
(FF)
T
OF
T
Am-Tmd.
Ao,
+
Ac- To-TsAa.== =0,
(8)
dc
0ax
for e {u.r,z), and M denoting the transpose of the matrix M. The event updates for the adjoints are described in the Supplement. In practice, the integration of À is done backwards in time.
195
198
For the recurrent weights Wx.ig from the different parameter matrices Wx for X E u,,,2 we can write the weight updates using only quantities calculated at events Cg as:
197
Aux.ij
L(W) Exxije
awxsij
198
The corresponding value of Ex,ik = (Ex,a) is given by the following formula, written in vector form for succinctness:
199
Sx, = - (rx oc (s.)) 8 (s))
(10)
5

---

200
where @ is the outer product, c refers to the value of c(t) just before event Cx, rx =0 for X {u.z} and equal to the value of r(t)) just before event EN for X = {r), A refers to the value of the adjoint variable Xay (t) just after the event ej.- Thus, the values of (t)) needs to be stored only at event times, and Aag (t) needs be calculated only at these times, making the gradient updates event-based.
201
202
203
204
See the Supplement for the update rules for the input weights and biases.
205
3.4
Sparse approximate BPTT in discrete time
A H(c)
forward
event generation
B
dH dc
backward
22
+
ym
yI
yP
yi
y
y
AGS
unit
cD
C
C
y
yI
yp
y
y11
yP
y12
y
unit
C/D
time
time
Figure 2: Illustrate the discrete time state dynamics for two EGRU units (i and j). A: Forward dynamics. Information only propagates from units that generate an event. B: Activity-sparse backward dynamics. Insets show threshold function H (c) and pseudo derivative thereof.
206 In discrete time, the network uses threshold activation function (c) to decide whether emit an event as described in Eq. (2). Since H is not differentiable at the threshold 0; we define a pseudo-derivative that point for calculating the backpropagated gradients. The pseudo-derivative is defined as a piece- wise linear function that is non-zero for values of state q; between 0, + and 1. p as shown in the inset in Fig. 2B. Since the pscudo-derivative is zero whenever the internal state of the unit is below 1, -E, the backpropagated gradients are also 0 for all such units, making the backward- -ill-pass sparse (see Fig.2 for an illustration). Note that the case where the internal unit state is above 0, + tends to occur less often, since the unit will emit event and the internal state will be cleared (Eq. (2)) at the next simulation step.
208
210
212
214
3.5
Computation and memory reduction due to sparsity
215
For the forward pass of the discrete time EGRU, an activity sparsity of a (i.e. an average of a events per simulation step) leads to the reduction of multiply-accummn
216
operations (MAC), by factor a. We focus on MAC operations, since they are by far the most expensive compute operation in these models. optimally implemented an activity sparsity of 80% will require 80% fewer MAC operations compared to GRU that is not activity-sparse. Computation related to external input is only performed
217
218
219
220
at input times, and hence is as sparse as the input, both time and space. During the backward pass, a similar factor of computational reduction is observed, based on the backward paaa sparsity 3 which is, in general, less than a. This is because, when the internal state value is not within dec of the threshold 0, the backward pass is skipped, described in section 3.4. Since our backward pass is also sparse,
222
224
expect to need to store only 3 fraction of the activations for later use, hence also reducing the memory usage. In all our experiments, report aetivity-sparsiny values calculated through simulations.
Results
226
4.1
Delay-copy task
228
To illustrate the behavior of the continuous time EGRU model (Fig. 1C.D) we used simple delay-copy task (also called the copy memory task [24]). binary vector was presented to the network the input
229
230
time. This was followed by a delay period, after which the network was given cue input indicating that it should recall the input seen before. A small network with only two EGRU units was used here, trained with the event-based learning rules described in Section 3.3. Right after the cue input, the network had to report the memorized input pattern. EGRU outputs yi emitted at network event times were convolved with an exponential kernel to retrieve output traces, which were then used to retrieve
231
233

---

235
the stored binary patterns based on their relative magnitudes. The kernel time constant was chosen be significantly lower than the delay time such that the network had to retain the memory in the event dynamics. The binary cross-entropy loss was used to train this model until it reached perfect (100%) bitwise accuracy on this task. Fig. IC,D shows the dynamies of the continuous-time model after training, well as the output trace and events. The network has learned to generate events such that output traces reliably encode the stored input patterns. Supplemental Table SI shows the robustness of the training for different sizes of inputs, networks, delay periods, all for multiple runs.
236
238
240
4.2
Gesture prediction
A
B
input events across time
input
on events
off events
unit
0.
30
1 layer
unit
9.0
2nd layer
unit
200
400
1000
time (ms)
time step
Figure 3: A: Illustration of DVS gesture classification data for an example class (right hand wave). On (red) and off (blue) events shown time and merged into summary image for illustration (not presented to the network). B: Sparse activity of input and EGRU units (random subset of 30 units shown for each layer).
243
We next evaluate our model on gesture prediction, which is popular real-world benchmark for RNNs. Here and in the remainder of the experiments we used the discrete time version of EGRU, since is easier to implement and use while retaining most of the advantages of the continuous time model. We use the DVS128 Gesture Dataset [1], where the inputs are defined as events. This dataset is widely used
244
246
247
in neuromorphic research and enables us to demonstrate our model's performance and computational
efficiency on event-based data. The dataset contains gestures from 29 subjects recorded with
248
DVS128 event camera [40]. Each event encodes a relative change of illumination and is given as spatio-temporal coordinates of X/Y position on the 128 128-pixel sensor and time stamp. Raw event times were combined into 'frames" by binning them over time windows of 25 ms. Frames were then downscaled 32x x 32 pixels using a maxpool layer.
250
reference
architecture
para- meters
effective MAC
aceu- racy
activity sparsity
backward sparsity
(# units)
He et al. [23] Innocenti et al. [30]
LSTM (512)
7.35M
7.34M
86.81%
AlcxNeHHLSTHHA
9,99M
638.25M
97.73%
ours
GRU (1024)
15.75M
15.73M
88.07%
0%
ours
EGRU (512)
5.51M
4.19M
88.02%
83.79%
53.55%
ours
EGRU (1024)
15.75M
10.54M
90.22%
82.53%
56.63%
ours
EGRU+DA (1024)
15.75M
10.77M
97.13%
78.77%
58.20%
Table 1: Model comparison for the DVS Gesture recognition task. Effective number of MAC operations as described in section 3.5.
252
253
Unlike previous approaches that focused on feedforward/RNN hybrid [1, 20, 30), we focused on pure RNN based architectures following the work of [23]. A binary cross-entropy loss was applied with an additional regularization loss on the output gate to produce 5% activity and the state variable e to be slightly below the threshold. The models were trained using the Adam optimizer for 1000 epochs to
255
7

---

257
verify their stability but typically reached plateau performance after 200 epochs (see Supplement for further details). Due to this, backward sparsity described in Section 3.5 was calculated epoch 100.
258
259
Comparison of model performance on gesture prediction is presented in Table 1. The model had inherent activity-sparsity of 70% which the regularization increased to 90% without significant performance decrease. EGRU consistently outperformed GRU networks of the same size on this task by a small margin. Adding data augmentation (DA) by applying random crop, translation, and rotation, as previously done in [30], increased the performance to over 97% of this pure RNN architecture, coming close to state-of-the-art architectures that even include costly AlexNet pre-processing. Further experimental details, ablation studies and statistics over different runs can be found in the supplement
260
261
262
263
264
285
266
sections D. D.1.1 and tables S3, S4 respectively.
267
4.3
Sequential MNIST
268
Next, we tested the EGRU on the sequential MNIST task [36], which is a widely used benchmark for recurrent networks. In this task, the MNIST handwritten digits were given as input one pixel at a time, and at the end of the input sequence, the network output was used to classify the digit. We trained a -layer EGRU with 590 units (matching the number of parameters with a 512 unit LSTM). We did not use any regularisation to increase sparsity in this task, so that we could test how much sparsity, both forward and sackward, arises naturally in the EGRU. In Table 2. we report the results of discrete- time
269
271
273
EGRU along with other state-of wef-the-art art architectures. EGRU achieved a task performance comparable to previous architectures while using much fewer operations (more than an f-lld reduction in effective MAC operations compared to GRU). Further experimental details, and statistics over different runs can be found in the supplement sections D.2 and table S5 respectively.
275
277
reference
architecture (# units)
parameters
effective MAC
test accuracy
activity sparsity
Rusch and Mishra [54]
coRNN (256)
134K 1M
262K
99.4%
Gu et al. [22]
LSTT (512)
1M
98.8%
ours ours
GRU (590)
IM
IM
98.8%
EGRU (590)
1M
226K
98.3%
72.1%
Table 2: Model comparison on sequential MNIST task. Top-1 test scores, given as percentage accuracy, where higher is better.
278
4.4
Language Modeling
279
Natural language processing a popular domain for benchmarking recurrent neural networks. We evaluated our model on language modeling task based on the Penn Treebank [42] dataset to validate the functionality of our model. While techniques such as neural cache models [21] or dynamic
280
281
evaluation [35] have been shown to improve language models, we focused on the RNN model itself in
283
this work, taking [45] as our baseline. Following [45], our models consists of a dense 400-dimensional embedding layer, and three stacked RNN cells with DropConnect applied to the hidden-to-hiddee weights [61]. The weights of the final softmax layer were tied to the embedding layer [29, 51]. All our models are optimized with Adam for 1000 epochs, and parameters were tuned for each model individually. Details on training and model parameters can be found in the Supplement. Results are shown in Table 3. In our experiments, GRUs did not reach the performance of LSTM variants this task, which, to the best of our knowledge, is consistent with recent RNN language modeling literature [44, 45]. At the same time, EGRU slightly outperformed GRU, while maintaining high levels of activity sparsity. Further experimental details, and statistics over different runs can be found in the supplement sections D.3 and table $6 respectively.
285
287
289
281
282
5
Discussion
293
294
We have introduced EGRU, a new form of a recurrent neural network that is competitive with current deep recurrent models yet can efficiently perform both inference and learning. To achieve this we first formulated the GRU in continuous time and converted it to an event- event-based form that achieved activity-sparsity naturally. Furthermore, the gradient-descent updates on this time-continuous EGRU mirrored the activity sparsity of the inference. We then demonstrated a discrete-time simplification of this model that also exhibited event-based activity-sparse inference and learning while being easier to implement with popular ML frameworks such as PyTorc or Tensorflow.
295
296
297
299
8

---

reference
architecture
para- meters
effective MAC *
validation
test
activity
(# units)
sparsity
Gal et al. [18]
Variational LSTM
24M
77.3
75.0
Melis et al. [44]
layer LSTM
24M
61.8
59.6
Merity et al. [45]
AWD-LSTM
24M
24M
60.0
57.3
ours
GRU (1350)
24M
24M
71.2
68.8
ours
EGRU (1350)
24M
4.7M
67.4
64.5
88.0%
ours ours
EGRU (2000)
45M
6.6M
66.5
63.7
90.4%
EGRU (2700)
77M
8.1M
66.4
63.5
93.2%
Table 3: Model comparison on Penn'rreebank. Validation and test scores are given as perplexities, where lower is better. Sparsity refers to activity-sparsity of the EGRU output, and effective MAC operations consider the layer-wise sparsity in the forward pass." *
The EGRU achieved competitive task performance various real-world tasks such gesture recognition and language modeling while achieving a sparsity of up to 80% for the gesture recognition task and 90% on the language modeling task. Scaling up networks for language modeling has shown some of the most promising results in the last few years [8, 17] Hence our choice of task, albeit on
302
303
304
305
a smaller scale, was validate the funetionalityy of the model. Considering the need for extensive hyperparameter search [44] for language modeling, our model achieved promising results while maintaining a high degree of activity-sparsity. For example, our EGRU with 1350 hidden units reached perplexities comparable with LSTM and GRU, while maintaining an activity
308
307
308
of 86 % (14% of the units active on average). The amount of computation used by an EGRU also scales sub-linearly with an increase in the size of the network and number of parameters, making it scalable alternative to LS STM//GUU based architectures (see Supplement).
309
310
311
312
While we use the GRU as the basis for our model due to its simplicity, this formulation can easily be ex- tended to any arbitrary network dynamics, including the LSTM, allowing specialized architectures for different domains. The adjoint method for hybrid systems that we use here is a powerful general- purppse forms of various recurrent neural network architectures. Another novel outcome of this paper is that this theory can handle inputs continuous time as events, which is very intuitive, hence providing an alternative to the more complex controlled differential equations [34]. The EGRU can also be used for irregularly spaced sequential data quite naturally.
313
314
tool for training event-based activity-sparss
316
317
318
319
The compute efficiency of this model can directly translate into gains in energy efficiency when implemented using event-based software primitives. These same properties would also allow the model to work well on heterogenous compute resources, including pure CPU nodes, and neuromorphic devices such as Intel's Loihi [13] and SpiNNaker 2 [27], that can achieve orders of magnitude higher energy efficiency. The EGRU model will also perform well in more mainstream deep learning hardware that is enabled for dynamic sparsity, such as the Graphcore system [31]. On neuromorphic
321
323
324
325
devices with on-chip memory in the form of a crossbar array, the activity sparsity directly translates into energy efficiency. For larger models that need off-chip memory, activity- sparssity needs to be combined with parameter-sparsity to reduce energy-intensive memory access operations.
326
327
328 In summary, starting with the motivation of building scalable, energy-efficient deep recurrent models, we demonstrated the EGRU, which reduces the required compute for both inference and learning by enhancing sparsity in the network. This approach lays the foundation for exploring novel capabilities that can emerge from scaling up RNNs similar to what has been seen for feed-forward architectures in recent years.
329
330
331
332
333
Potential negative societal impact: The proposed model is new variant of the previously published GRU and would therefore essentially inherit all potential negative societal impacts from that model, including the potential risks that come with automated surveillance systems, vulnerability fraud and adversarial attacks, etc. (see [6] and [62] for critical reviews). However, the model also provides the
334
335
336
337
potential societal benefit of making these models more energy-efficient and thus reducing the energy and carbon footprint of machine learning. Scaling this model to larger sizes, especially for language The
339
modeling, can lead to the same problems as current large language models [5]. The effect of activity sparsity on prediction bias needs to be studied further in the same way as for parameter sparsity [26].

---

References
341
342
[1] A. Amir, B. Taha, D. Berg, T. Melano, J. McKinstry, C. Di Nolfo, T. Nayak, A. Andreopoulos, G. Garreau, M. Mendoza, et al. A low power, fully event-baser gesture recognition system.
343
3M
In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
345
7243-7252, 2017.
[2] R. Beer. On the Dynamics of Small Continuous- Timee Recurrent Neural Networks. Adaptive Behavior, 3(4):469550 Mar. 1995. ISSN 1059-7123. doi: 10.1177/05971
347
[3] G. Bellec, D. Salaj. A. Subramoney, R. Legenstein, and W. Maass. Long short-term memory and Learning-to-learm in networks of spiking neurons. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 787-797. Curran Associates, Inc., 2018.
349
350
351
352
[4]
G. Bellec, F. Scherr, A. Subramoney, E. Hajek, D. Salaj, R. Legenstein, and W. Maass. A solution to the learning dilemma for recurrent networks of spiking neurons, Nature Communications,
353
354
1 11(1):3625, July 2020. ISSN 2041-1723, doi: 10. 0038//41467-020-1111
[5] E. M. Bender, T. Gebru, A. McMillan-Major and S. Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 202/ ACM Conference on Fairness, Accountability, and Transparency, pages 610-623, 2021.
356
358
[6] A. Birhane, P. Kalluri, D. Card, W. Agnew, R. Dotan, and M. Bao. The values encoded in machine learning research. arXiv preprint arXiv:2 /5590, 2021.
360
[7] Bradbury, S. Merity, C. Xiong, and R. Socher. Quasi-Recurrent Neural Networks. ar 1rX1v:1611.01576 [cs]. Nov. 2016.
381
[8]
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess,
363
385
J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, Language Models are Few-Shot Learners arXiv:2005. 14/65 [cs], July 2020.
366
367
[9] B. Chang, M. Chen, E. Haber, and F. H. Chi. AntisymmetricRNN: A Dynamical System View on Recurrnt Neural Networks. arXiv: 1902.0969 [cs, stat], Feb. 2019.
369
[10] R. T. Chen, Y. Rubanova, J. Bettencourt, and K. Duvenaud. Neural Ordinary Differential Equations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31. pages 6572-6583 Curran Associates, Inc., 2018.
370
371
372
[11] R. T. Q. Chen, B. Amos, and M. Nickel. Learning Neural Event Functions for Ordinary Differential Equations. In International Conference on Learning Representations, Sept. 2020. URL. htt.ttps::/ http::/ss////ppereeeeeee/ wwwppeerrewiie.eeettrurmrmmo L
374
375
376
[12] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representationn using rnn encoder-decoder for statistical machine translation. arXiv preprint /406, 1078, 2014.
377
378
379 [13] M. Davies, N. Srinivasa, T.-H. Lin, G. Chinya, Y. Cao, S. H. Choday, G. Dimou, P. Joshi, N. Imam, S. Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning, leee Micro. 38(1):82-99 2018
380
381
[14] E. De Brouwer, J. Simm, A. Arany, and Y. Moreau. GRU-ODE-Bayes Continuous Modeling of cetuddicall--bbeeevee Time Series. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL htttt::::/ppptttt p/pooeeeee///oooeeene/e.. nww.ips.ccc... cc/ paper/2015
383
384
385
html.
[15] M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and L. Kaiser. Universal Transformers. In International Conference on Learning Representations, Sept. 2018 URL https:// wwt.
387
10

---

389
[16]
S. K. Esser, P. A. Merolla, J. V. Arthur, A. S. Cassidy, R. Appuswamy, A. Andreopoulos,
390
D. J. Berg, J. L. McKinstry, T. Melano, D. R. Barch, C. di Nolfo, P. Datta, A. Amir,
B. Taba, M. D. Flickner, and D. S. Modha. Convolutional networks for fast, energy-efficient neuromorphic computing. Proceedings of the National Academy of Sciences, 113(41): 11441-11446, Nov. 2016. ISSN 0027-8424, 1091-6490. doi: 10. 10.1073/pnas. 160485001 13. URL http://www. pww.ppaaa..... org/ ctt////nttt/////11/////
382
383
394
385
177] W. Fedus, B. Zoph, and N. Shazeer. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, arXiv:2101.0396/ [cs], Jan. 2021.
396
397
[18] Y. Gal and Z. Ghahramani.
theoretically grounded application of dropout in recur- rent neural networks. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, Curran As- sociates, Inc., 2016. URL https: proceddingg pwwpa/eps..cppppp///////e/eee/ee .aper.
398
389
400
401
402
[19] S. Galán, W. F. Feehery, and P. Barton. Parametric sensitivity functions for hybrid discrete/continuous systems. Applied Numerical Mathematics, 31(1:177-7, Sept. 1999. ISSN 0168-9274. doi: 10. 10.1016/S0168 68- (27008997799992225....
404
405
[20] R. Ghosh, A. Gupta, A. Nakagawa. A. Soares. and N. Thakor. Spatiotemporal filtering for
408
event-based action recognition, 2019. URL https: /aapss://arxvv..o /arxiv.xvvv org/ 0tbs/1903.000 07067.
407
[21] E. Grave, A. Joulin, and N. Usunier. Improving neural language models with a continuous cache. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https://openrevier
409
[22] A. Gu, C. Gulcehre, T. Paine, M. Hoffman, and R. Pascanu. Improving the Gating Mechanism of Recurrent Neural Networks. In Proceedings of the 37th Interna- tional Conference on Machine Learning, pages 3800-3809. PMLR, Nov. 2020. URL
412
413
https: attps://proceedinggs mlr press/v119 hatps//vv///g/aaa../ gu20a, html.
414
[23]
W. He, Y. Wu, L. Deng, G. Li, H. Wang, Y. Tian, W. Ding, W. Wang, and Y. Xie. Comparing snns and rnns on neuromorphic vision datasets: Similarities and differences. Neural Networks, 132:108-123 120, 2020.
416
418
[24] S. Hochreiter and J. Schmidhuber. Long Short-Term Memory. Neural Computation, 9())
419
1735-1780, Nov. 1997. ISSN 0899-7667. doi: 10. 1111/ncco.199.. 1997.9...
420
[25] T. Hoefler, D. Alistarh, T. Ben-Nun, N. Dryden, and A. Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):1-124 2021.
421
422
[26] S. Hooker, A. Courville, G. Clark, Y. Dauphin, and A. Frome. What do compressed deep neural networks forget? arXiv preprint arXiv:1911.05220 2019.
424
425
[27] S. Höppner. Y. Yan, B. Vogginger, A. Dixius, J. Partzsch, F. Neumärker, S. Hartmann, S. Schiefer, Scholze, G. Ellguth, al. Dynamic voltage and frequency scaling for neuromorphic many-core systems. In 2017 IEEE International Symposium on Circuits and Systems (ISCAS), pages 1-4. IEEE, 2017.
426
427
428
429
[28]
K. L. Hunter, L. Spracklen, and S. Ahmad. Two sparsities are better than one: Unlocking the performance benefits of sparse-sparse networks, 2021.
[29]
H. Inan, K. Khosravi, and R. Socher. Tying word vectors and word classifiers: A loss framework for language modeling. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. wttp:/ooomdddddraaaoo.o...
432
433
434
URL
[30] S. U. Innocenti, F. Becattini, F. Pernici, and A. Del Bimbo. Temporal binary representation for ovent-based action recognition. In 2020 25th International Conference on Pottern Recognition (ICPR), pages 10426-10432. IEEE, 2021.
436

---

438
[31] Z. Jia, B. Tillman, M. Maggioni, and D. P. Scarpazza. Dissecting the graphcore ipu architecture via microbenchmarking, arXiv preprint arXiv: 1912.0399120331333. 2019.
439
440
[32] I. D. Jordan, P. A. Sokót, and I. M. Park. Gated recurrent units viewed through the lens of
continuous time dynamical systems. Frontiers in computational neuroscience, page 67, 2021.
442
[33] Jordan, T. Ippen. M. Helias, I. Kitayama, M. Sato, Igarashi, M. Diesmann, and S. Kunkel. Extremely Scalable Spiking Neuronal Network Simulation Code: From Laptops to Exascale Computers. Frontiers in Neuroinformatics, 12:2, 2018, ISSN 1662-5196.
443
444
445
[34] P. Kidger, J. Morrill, J. Foster, and T. Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 01166666-7001 2020.
447
[35] B. Krause, Kahembwe, I. Murray, and S. Renals. Dynamic evaluation of neural sequence models. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 Proceedings of Machine Learning Research, pages 2766-2775 PMLR, 10-15 Jul 2018. URL https: prt.eeeeeeeeess....oo mlr, press /vesss//v//// 2e00//rauue888 html.
448
449
450
[36] Q Le, N. Jaitly, and G. E. Hinton. A simple way to initialize recurrent networks of rectified linear units. arXiv preprint arXiv: 150400044 2015.
452
453
[37] M. Lechner and R. Hasani. Learning Long- Dependencies in Iregullrry-SSappled Time Series. arXiv:2006. 04418 [cs, stat], Dec. 2020. wwt.htttp///xxiii http://awxvvv..r/bbbee/ss/e/// arg/abs/2000 04418.
454
455
[38] T. Lei, Y. Zhang, S. I. Wang, H. Dai, and Y. Artzi. Simple Recurrent Units for Highly Parallelizable Recurrence. arXiv:1709 00..599..275..0. [cs], Sept. 2017.
457
[39] Li W. Li. C. Cook, C. Zhu, and Y. Gao. Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN. 17X3iv:803.044 [cs], May 2018.
458
[40] P. Lichtsteiner, C. Posch, and T. Delbruck. A 128x 128 120 dB 15 latency asynchronous temporal contrast vision sensor. IEEE journal of solid-state circuits, 43(2))566 576, 2008
480
461
[41] W. Maass. Networks of spiking neurons: the third generation of neural network models. Neural networks, 1099):659-1171, 1997.
462
463
[42] M. P. Marcus, M. A. Marcinkiewicz, and B. Santorini. Building a large annotated corpus of english: The penn treebank. Comput. Linguist., 19(2):313330,, jun 1993. ISSN 0891-2017.
485
[43] C. Mead. How we created neuromorphic engineering, Nature Electronics. 3(7):434 435, 2020.
466
[44] G. Melis, C. Dyer, and P. Blunsom. On the state of the art of evaluation in neural language models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL Uttht:ps:/ / wwwpoopwwweew.eeeoooe net/ httph//wwwne/foouuneeeeee forum?id-By.HHuug...
468
470
[45] S. Merity, N. S. Keskar, and R. Socher. Regularizing and Optimizing LSTM Language Models. 17Xiv:1708.02111 [cs]. Aug. 2017.
[46] D. Neil, M. Pfeiffer, and S.-C. Liu. Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences. 1rXiv:1610.0951: [cs]. Oct. 2016.
473
474
[47] D. Neil, J. H. Lee, T. Delbruck, and S.-C. Liu, Delta Networks for Optimized Recurrent Network Computation. In International Conference on Machine Learning, pages 2584-2593. PMLR, July 2017.
475
478
[48] C. Ostrau, C. Klarhorst, M. Thies, and U. Rückert. Benchmarking neuromorphic hardware and its energy expenditure. Frontiers in Neuroscience, page 732.
478
479
[49] N. Perez-Nieves and D. F. M. Goodman. Sparse Spiking Gradient Descent. arXiv:2105.088// [cs, q-bio], May 2021.
480
481
[50] I.. Pontryagin, Boltyanskiy, R. V. Gamkrelidze, and MISHCHENKO, Mathematical theory of optimal processes. CRC press, 1962.
12

---

483
[51] O. Press and L. Wolf. Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 157-163, Valencia, Spain, Apr. 2017. Association wwt.:/y/.rr/////////////
484
486
for Computational LiniitatitonannLLiiiiiitiiit Linguisties. UKt.https::/aclantbbo..
487
[52] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-Shot Text-to-Image Image Generation. 1rxxi:2102.12092 [cs], Feb. 2021.
[53] K. Roy, A. Jaiswal, and P. Panda. Towards spike- -ased machine intelligence with neuromorphic computing. Nature,57 575(7784);607-617,22 2019.
480
481
[54] T. K. Rusch and S. Mishra. Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependencies. arXiv:2010.009) 00950 [cs, stat], Mar. 2021. URL http: / /arx/vxvvvvvv///i///////aaa/// org/ wwtgg/abs/211000 00951.
482
493
484
[55] D. Salaj, A. Subramoney, C. Kraisnikovic, G. Bellec, R. Legenstein, and W. Maass. Spike frequency adaptation supports network computations on temporally dispersed informa- tion. eLife, 10:e65459, July 2021. ISSN 2050-084X.
495
doi: 10.7554/eLife.65455 URL
497
attps://dii.orgg w.tg 7554/elife..5445
498
[56] S. Schaefer, D. Gehrig, and D. Scaramuzza. Aegmn: Asynchronous event-based graph neural networks. arXiv preprint arXiv:2203.17149) 2022.
499
500
[57] A. M. Schäfer and H. G. Zimmermann. Recurrent Neural Networks Are Universal Approximators In S. D. Kollias, A. Stafylopatis, W. Duch, and E. Oja, editors, Artificial Neural Networks - ICANN 2006, Lecture Notes in Computer Science, pages 632-640, Berlin, Heidelberg, 2006. Springer. ISBN 778-3-540038627-
501
502
503
doi: 10.1007/1 840817_66
(58) C. D. Schuman, T. E. Potok, R. M. Patton, J. D. Birdwell, M. E. Dean, G. S. Rose, and J. S. Plank. A survey of neuromorphic computing and neural networks in hardware. arXiv preprint arXiv: 1705.06963 2017.
505
506
[59] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously Large Neural Networks: The Sparsely- Gated Mixture-of-Experr Layer. arXiv:1700 701.06538 [cs, stat], Jan. 2017.
508
510
[60]
H. T. Siegelmann and E. D. Sontag. On the Computational Power of Neural Nets. Journal of Computer and System Sciences, 50(1):13--550, Feb. 1995. ISSN 0022-0000. doi: 10. (9cbijcss.195.1000 htthttp://wwr. http://www..ceenndrreecc.cc 50022000085510013.
511
512
513
514
[61]
L. Wan, M. Zeiler, S. Zhang, Y. Le Cun, and R. Fergus. Regularization of neural networks using dropconnect. In S. Dasgupta and D. McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 1058-1066, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR. URL https:/ htteee///oooceeedee mlr ppess/v28/wan1/ html.
515
517
[62] J. Whittlestone, R. Nyrup, A. Alexandrova, K. Dihal, and S. Cave. Ethical and societal implications of algorithms, data, and artificial intelligence: a roadmap for research. London: Nuffield Foundation, 2019.
520
522
[63] T. C. Wunderlich and C. Pehle. Event-based bawkpropagation can compute exact gradients for spik- ing neural networks. Scientific Reports, 11(1):12829) June 2021. ISSN 2045-2322. doi: 10. 1038/ s41598 141598-021-99 786-2.. URL https: ///w///www wwwuur...e............. cwm httm///artteeee//111000000--1--0
523
524
525
[64] M. Yan, N. Meisburger, T. Medini, and A. Shrivastava. Distributed slide: Enabling training large neural networks on low bandwidth and simple cpu-clusters via model parallelism and sparsity. arxiv preprint 2rXiv:2201./2667, 2022.
526
527
528
[65] F. Zenke and S. Ganguli. SuperSpike: Supervised Learning in Multilayer Spiking Neural Networks. Neural Computation, 30(6): Apr. 2018 ISSN 0899-7667. doi: 1162//ecoo___0088
530
10. 166/neco a 01086. URL //doii.://///i/// org/10. 1ttg:/...11.....
13

---

Checklist
532
1. For all authors...
538
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
534
contributions and scope? [Yes] See end of Section for a summary of main claims.
635
(b) Did you describe the limitations of your work? [Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 5.
537
538
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] The authors have read and approved the ethics guideline
539
540
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 3 and the Supplement
542
543
(b) Did you include complete proofs of all theoretical results? [Yes] See Section and the Supplement
545
3. If you ran experiments...
548
(a) Did you include the code, data, and instructions needed to reproduce the main experimen- tal results (either in the supplemental material as URL)? [Yes] In the Supplement
547
548
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Some details in the main text and further details in Supplement
550
(c) Did you report error bars (e.g., with respect the random seed after rumning experiments multiple times)? [Yes] In the Supplement
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] In the Supplement
553
4. If you are using existing assets (e.g., code, data, models) curating/releasing assets...
555
(a) your work uses existing assets, did you cite the creators? [Yes]
556
(b) Did you mention the license of the assets? [Yes]
557
(c) Did you include any new assets either in the supplemental material or as a URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you'ne using/curating? [N/A]
559
560
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
561
582
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
564
585
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
566
567
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "YdPqvrVoZkz", "reviewer": "Reviewer_eSbV", "summary": "In this paper, the authors have introduced an event-based continuous-time variant of the gated recurrent model and derived an eve-based form of backprop to train the system. Extensive experiments on a wide variety of benchmarks show that the proposed model EGRU can achieve comparable performance with SOTA recurrent models and exhibits high levels of activity-sparsity during the training and inference phase. ", "questions": "* The comparison should be made with at least two models from [1-5], first, one proposed sparse RTRL to train neural networks, that can potentially solve the scalability issues shown by BPTT.  Other [2] focuses on other recurrent architecture and learning approaches to solve this issue. \n\n* For models only based on BPTT, prior work has shown that sparse versions of recurrent models have the potential to outperform dense versions. Hence it is important to show how efficient the current approach is compared to other sparse RNNs.\n\n*Hyper-parameter optimization- No detail about hyper-parameter optimization, authors should provide reasoning how these parameters were chosen. \n\n* Statistical significance: - Experiments should be conducted for K trials and average performance and the standard error should be reported.\n* Can authors comment on the robustness and stability of the given model? Based on different seeds, and settings, the optimizer is model stable compared to dense models trained using BPTT.\n\n* Authors have mentioned that EGRU is computationally efficient, can authors provide numbers in FLOPS for both training and inference compared with other sparse and dense RNNs\n\n[1] https://arxiv.org/abs/2006.07232\n[2] https://arxiv.org/abs/1602.03032\n[3] https://arxiv.org/pdf/1901.09208.pdf\n[4] https://link.springer.com/article/10.1007/s00521-021-05727-y\n[5] https://proceedings.mlr.press/v139/liu21p.html\n", "limitations": "Highlighted above", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n* Approach is novel\n* Well-written paper\n* Experiments are good\n\n# Weakness\n* Ablation study missing\n* Experimental section is missing key details\n* Missing some key citations", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GB1jLG9W1WU", "reviewer": "Reviewer_RdzD", "summary": "This paper proposes EGRU, an event-based GRU for activity-sparse inference and learning. The proposed model is more efficient as only a select set of units are active at any given time. ", "questions": "* In terms of activity sparsity during inference, how does it compare against pruning a GRU, such as the one mentioned in \"EXPLORING SPARSITY IN RECURRENT NEURAL NETWORKS\"?", "limitations": "The authors have discussed inheriting negative societal impacts and risks existed in GRU. ", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality**:\n\n* To my knowledge, the idea proposed in this article is novel and original. \n\n**Quality**:\n\n* The paper technically sounds correct and claims well supported by theoretical analysis and experimental results.\n* Related works are covered and discussed.\n* Experiments are conducted extensively in different type of tasks and the results are discussed thoroughly and compared in terms of accuracy,#MAC, and sparsity. \n\n\n**Clarity**:\n\n* This paper is well written and organised.\n\n\n**Significance**:\n* the proposed EGRU, as the authors have suggested, can be more energy-efficient and scalable as it reduces the required compute for both inference and learning. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4MS-onvnKnQ", "reviewer": "Reviewer_egpJ", "summary": "This paper proposes an event-based continuous-time variant of the GRU model, named EGRU, motivated by building scalable, energy-efficient deep recurrent models. The EGRU unit only outputs events when the internal states reach a trainable threshold, which makes it activity-sparsity naturally. Furthermore, the authors derive corresponding BP algorithms for updating EGRU and extend it to a discrete-time version that is easier to implement by popular machine learning architecture. As the experiments show, the EGRU model exhibits comparable performance to state-of-the-art recurrent network architectures for several tasks.", "questions": "This paper presents a novel GRU architecture, which balances the performance and computational complexity. \n- However, in the downstream application with EGRU, some experiments’ setups are unclear, as listed in Weakness. I expect the authors to clarify these points. \n- Besides, the authors apply EGRU in some special cases. Since the LSTM/GRU are applied widely in CV/NLP applications, can they be replaced by EGRU to reduce the computation complexity in general? \n- As GRU is simplified from LSTM, can event generating mechanism transfer to the LSTM model?\n", "limitations": "The authors have addressed the limitations and potential negative societal impact of their work in discussion section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is well-written, consistent, and produces good results. By introducing the event generating mechanism, the EGRU model not only significantly reduces the multiply-accumulate operations but also preserves the performance that is comparable to the more computation-expensive GRU-based model. \n\nStrengths:\n+ The event generating mechanism is a novel module inserted into the GRU model, which is proved to effectively reduce the computation of the original model.\n+ The authors conduct three downstream applications to validate the effectiveness of the EGRU model, which are convincing and with comparable performances to previous methods.\n\nWeakness:\n- For the delay-copy task, the authors only show a simple example. How about increasing the number of EGRU units? Besides, did the events generating mechanism improve the memory ability, compared to the GRU model?\n- In gesture prediction, the authors down sample frames into 32x32, which may affect the performance. I wonder how the input size affects the prediction accuracy with EGRU model. In general, the larger-resolution images contain more detailed information, which could be helpful for classification. \n- For language modeling experiments, the performance decreases with the increment of EGRU units. Can the authors give some explanation or the trend of performance related to the number of EGRU units?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject