Training biologically plausible recurrent neural networks on cognitive tasks with long-term dependencies
Authors: Wayne WM Soo
Vishwa Goudar
Xiao-Jing Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: neuroscience
recurrent neural network
neural circuits
cortical circuits
cognitive tasks
working memory
TL;DR: We train biologically-plausible recurrent neural networks on tasks requiring long-term dependencies without artificial elements such as LSTMs.

Abstract: Training recurrent neural networks (RNNs) has become a go-to approach for generating and evaluating mechanistic neural hypotheses for cognition. The ease and efficiency of training RNNs with backpropagation through time and the availability of robustly supported deep learning libraries has made RNN modeling more approachable and accessible to neuroscience. Yet, a major technical hindrance remains. Cognitive processes such as working memory and decision making involve neural population dynamics over a long period of time within a behavioral trial and across trials. It is difficult to train RNNs to accomplish tasks where neural representations and dynamics have long temporal dependencies without gating mechanisms such as LSTMs or GRUs which currently lack experimental support and prohibit direct comparison between RNNs and biological neural circuits. We tackled this problem based on the idea of specialized skip-connections through time to support the emergence of task-relevant dynamics, and subsequently reinstitute biological plausibility by reverting to the original architecture. We show that this approach enables RNNs to successfully learn cognitive tasks that prove impractical if not impossible to learn using conventional methods. Over numerous tasks considered here, we achieve less training steps and shorter wall-clock times, particularly in tasks that require learning long-term dependencies via temporal integration over long timescales or maintaining a memory of past events in hidden-states. Our methods expand the range of experimental tasks that biologically plausible RNN models can learn, thereby supporting the development of theory for the emergent neural mechanisms of computations involving long-term dependencies.

Training biologically plausible recurrent neural networks on cognitive tasks with long-term dependencies
Wayne W.M. Soo Department of Engineering University of Cambridge wmws20cam. ac uk
Vishwa Goudar Center for Neural Science New York University vishwa. gowdaronud.rr.u.ooooooomm
Xiao-Jing Wang Center for Neural Science New York University xj xjwangonyu.ed
Abstract
Training recurrent neural networks (RNNs) has become go-to approach for gen- erating and evaluating mechanistic neural hypotheses for cognition. The case and efficiency of training RNNs with backpropagation through time and the availabil- ity of robustly supported deep learning líbraries has made RNN modeling more approachable and accessible neuroscience. Yet, a major technical hindrance re- mains. Cognitive processes such as working memory and decision making involve neural population dynamics over a long period of time within a behavioral trial and across trials. It is difficult to train RNNs to accomplish tasks where neural represen- tations and dynamics have long temporal dependencies without gating mechanisms such as LSTMs GRUs which currently lack experimental support and prohibit direct comparison between RNNs and biological neural circuits. We tackled this problem based on the idea of specialized skip-connections through time to support the emergence of task-relevant dynamics, and subsequently reinstitute biological plausibility by reverting the original architecture. We show that this approach enables RNNs to successfully learn cognitive tasks that prove impractical if not impossible to learn using conventional methods. Over numerous tasks considered here, we achieve less training steps and shorter wall-clock times, particularly in tasks that require learning long-term dependencies via temporal integration over long timescales or maintaining a memory of past events in hidden-states. Our meth- ods expand the range of experimental tasks that biologically plausible RNN models can learn, thereby supporting the development of theory for the emergent neural mechanisms of computations involving long-term dependencies.
Introduction
The adoption of deep learning methods neuroscience research has fostered entirely new approaches to modeling neural systems and developing scientific hypotheses [[[] Moreover, rapid advances in network initialization and optimization techniques has reduced the threshold knowledge for designing and training neural networks and improved the accessibility of model development in neuroscience. Consequently, it is now far more feasible to formulate theories and even take a theory-first approach neuroscience. Technological advances in experimental neuroscience are also making it easier to study increasingly complex neural computations, escalating the demand for models that support much longer time-scale dependencies than conventionally modeled with biologically plausible RNNs (see Appendix A). However, fundamental physiological and anatomical properties of the brain constrain the neural network architectures that are permissible for study in neuroscience. First, the brain's connectivity is highly recurrent at multiple spatial scales, making recurrent neural networks (RNNs) [5]
the modeling tool of choice in neuroscience
The training and analysis of RNNs
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

has advanced theory on the neural basis of perception 77,9,, motor behavior [10H13], cognition
Second, physiological properties of neurons places a greater demand on the emergent computational properties of neural populations. Specifically, in the absence of inputs, a neuron's 's response rapidly decays away due to its passive membrane properties. Consequently, "leaky" RNN models rely on the reverberation of activity throughout population of units to integrate inputs over time and to maintain this information in the absence of ongoing inputs. This has even contributed to the population doctrine gaining traction in neuroscience that neural populations rather than single neurons form fundamental computation units 30 31.
[7] memory and navigation [28 29
1.1 Gradient instability in recurrent neural networks
The stability challenges of training biologically-plasiibe (leaky) RNNs with backpropagation through time are widely understood the larger the number of time steps the network must compute over, the more likely is that the gradients become catastrophically large or small which arrests further learning. Long-short-term memory (LSTM) [32] and Gated Recurrent Units (GRU) [33] based RNNs were developed precisely mitigate this adversity posed by exploding or vanishing gradients, doing so by storing memory in self-sustaining cells rather than leaky population activity. Unfortunately, there is dearth of evidence in neuroscience in support of the gating mechanisms that help LSTMs and GRUs overcome gradient stability issues. There exist multiple biological mechanisms for gating in the brain [44, but none of them is known to turn on and off the neuronal leak, input and output flows in such unconstrained ways as presumed by LSTM or GRU, Moreover, memory function through emergent population dynamics (rather than via a population of self-sustaining units) are a central hypothesis in neuroscience. An extended review of other aspects of biological implausibility of gating mechanisms can be found in Appendix B, which highlights issues with gating properties, case of training, network dynamics as well as mechanisms underlying memory storage. Other methods that help RNNs learn long-term tergterrm&on dependencies have been proposed in the past that do not involve gating mechanisms. A straightforward way would be simply initialize the recurrent weight matrix as an identity matrix 35 When applied to ReLU RNNs, these networks were competitive with LSTMs on speech and language benchmarks. Similarly, RNNs whose recurrent weight matrices are constrained to have absolute eigenvalues of have been proposed [36] 37). at the cost of lower convergence rates [38]. At their cores, both methods alleviate the problem by reducing the base multiplier in which gradients explode or vanish. Auxiliary losses targeting intermediate time steps 339 have also been proposed.
1.2 Skip connections through time
is difficult to justify any architectural modifications to biologically plausible RNNs, especially when it involves artificial elements. The alternative, which we propose, is to introduce surrogate elements which improve convergence rates during training and subsequently remove them after training to reinstitute biological plausibility test time. The ease of adding (and eventually removing) skip connections makes it an ideal candidate for this objective. We present and compare a set of engincering solutions using specialized skip connections for training biologically plausible RNNs on array of commonly studied cognitive tasks [16]. The idea of using skip connections to improve gradient- based model training gained mainstream popularity when its implementation in deep convolutional neural networks (CNNs) achieved state-of of the-art art performance in image classification [40). Since then, skip connections have become staple feature of newer architectures in machine learning, such as transformers [41] and MLP-mixers 42). Skip connections of this nature are also referred to as residual connections. In this context, skip connections solve the degradation problem, where increasing the depth of the model reduces the maximum attainable training performance by direct backpropagation 44). A conceptually similar idea of skip connections through time involves adding a direct connection between two non-consecutive states in sequential models, thereby facilitating shortened path during backpropagation and alleviating gradient stability problems. One of the earliest formulations of this idea was proposed more than two decades ago in the training of NARX networks (43) The architecture of NARX networks naturally contains the mathematical equivalent of skip connections through time, which were referred to as "jump-ahead connections". Network architectures and investigations based on skip connections through time continue to surface even in recent years 44 45 For example, dilated skip connections reduce the computational complexity of the RNN, in contrast to regular skip connections which requires additional computations [55 LSTMs and GRUs are not mutually exclusive with skip connections [46] 47).

---

Input population units)
Oulput population (20 units)
Modality A
Output ring (32 units)
Modality B (32 units)
Fixation (1 unit)
Fixation (1 unit)
Auxiliary population (10 units)
B
Perforance
go
ddmd
Performance
1.0
Training steps
(10,000)
dgo
0,8
Maanii rigo
anii
go
modm
dncc
dmd
drime
dm
drms
dms
ddmdd
msiddm danti dgo
Training steps (10,000)
Tasks with long-term dependencies
o Tasks
without long-term dependencies
Figure 1: Limitations training biologically plausible RNNs for tasks in neuroscience. A. Overview of RNN model. B. RNN performance on three cognitive tasks against the number of training steps. RNNs require more training steps in order to successfully perform tasks that require the learning of long-tert dependencies. C. Number of training steps required to train biologically plausible RNNs to attain 0.99 performance on 16 standard tasks [16] in neuroscience (Appendix D).
2 Background
2.1 Biologically plausible recurrent neural networks
The continuous-time dynamics of biologically plausible leaky ("vanilla") RNNs are described by:
T - + (W + + b hes + +7)
(I)
di
where T is diagonal matrix whose elements are the time constants of each individual neuron (typically constant across all neurons), and is the firing rate vector (the hidden-state of the RNN), Wre: is the recurrent synaptic weight matrix, b is the bias, hest is the external input vector and represents additive noise present the system. I the activation function (or input-output function), which we choose to be ReLU for our simulations. Alternative functions include the hyperbolic tangent (tanh) and softplus. The output of the network is a straightforward readout layer given by
ye = g (Wooott Te)
for activation function g. which we choose be the sigmoid function. Other common choices include softmax, softplus or linear, depending on the decoding objective. These models reflect the diversity in function across the brain areas they model, such perceptual functions in early sensory areas such as the visual cortex [9], maintenance of items in working memory, decision-making, context-dependent
mule-dependent responses higher-order cortical areas such as the prefrontal cortex [14 16, [9,, and motor control of eye saccades or arm movements in motor cortical areas 13. Consequently. the task structure, requisite computations, RNN inputs and outputs vary widely according to the function. We review the values of some of these parameters from previously- published models
[1 29
48 60 in Appendix and Table S1 in the supplementary.
2.2 Difficulty in learning tasks with long-term dependencies
The suite of cognitive tasks used in our benchmark tests have been adapted from [16], which includes straightforward tasks involving responding after stimulus presentation (go, anti, decision making, decision making with distractors, multi-sensory decision making), reaction time-based tasks (react go/anti), as well as more complex tasks that involve storing information in memory (delayed match/non-maut match to sample/category, delayed go/anti, delayed decision making). Exact
3

---

task descriptions and setup are explained in Appendix D. It has been previously shown that RNNs can learn these tasks without inter-neural interactions, where recurrent connections are restricted to only self-coupling of each neuron [6]]. To prevent this class of semi-degenerate and biologically unrealistic solutions, we designated distinct populations of neurons to interface with the input and output units respectively, with the rest of the unassigned neurons designated as auxiliary neurons to aid computations (Figure [T]. This ensures that, at a minimum, there must be connections between the input and output populations, leading to the emergence of meaningful solutions found by the trained networks. For all tasks, the primary inputs and outputs have a ring structure of 32 units, with each unit i having preferred direction 2/22 An input stimulus is characterized by modality, magnitude y and a direction şi such that:
1 2
8/6 3/l
h2 7 0.8 exp
The input consists of two separate rings, representing two different stimulus modalities. The output is a single ring characterized by a direction v,, representing a saccadic response,
18/b v/sl )
y/ 0.8 exp
+ 0.05
(4)
When cued "fixate" by a binary fixation input unit, the RNN must respond by maintaining the value of its fixation output umit, constrained to lie between O and 1, above 0.5. The desired input- output mapping at other times in a trial are task dependent. The loss function to be minimized the mean-squared error between the network output and the target output. The network is optimized with respect to the input weights Winps output weights W out, recurrent weights W rec and network bias b. For each task, we train 100 models in parallel with time steps ranging from 1500 to 2500 depending on the task, time constant of 100 ms and dt- = 5 ms, which together represents a higher temporal resolution than typical models (see Table S2 for detailed model parameters). We train each set of 100 models until the average performance of the networks in the 10th - 90th percentile reaches 0.99 or until 40,000 training steps have been reached (Figure B-CC). The number of training steps required for the tasks span more than an order of magnitude (Table []. thus depicting their varying difficulties. Importantly, we find that tasks that involve some form of long-term dependency typically require more training steps and are harder to train in general. In fact, RNNs trained on two tasks that require long- term dependencies were not able attain the required threshold performance within 40,000 training steps, but instead maintained a low but constant performance at the later stages of training (Figure BB).
Methods
The most common way of training biologically plausible RNNs involves simulating its dynamics using Euler's method for the forward pass (Figure A
re+at f r, + dr,
A
(5)
di
Here, At is the discretization time step that plays a central role in performing backpropagation through time on the simulated dynamics. Consequently, iterating this over time steps results in:
dr At
base T.t rr Σ dt
(6)
base which we will denote as the dynamics of the base model (-001- We explore several ways in which this process can be modified, only during training, in order to help with the learning of long-term dependencies.
3.1 Coarsened discretization (CD)
CD begins by training the network with large step size to support stable gradients while learning long-term dependencies (Figure 2B). Here, (5) is modified to:
dr. EHeA + r, di
CD
A 0
for some discretization factor 0 1. Thereafter, 0 is gradually reduced over the course of training until 0 = 1 at the end of training, resulting in a reversion to the base model described by (5.

---

A
10
drp
dr
dre-
x
x0
0
Fi
Te10
Base model
Coarsened discrelization (CD)
Training steps
C
constant
dr,
dr++]
dr
X
3
T++1
K2+6
0.1
Skip connection through time (SCTT)
x(1
(3)
Training steps
dr:
drei3
dr:10
constant
x,8
Te+1
Xe+0
8
dr:
0.1
x6
x(1
Training steps
Dynamics-aligned skip connection (DASC)
Figure 2: Proposed algorithms for training RNNs to perform tasks requiring long-ferm depen- dencies. Base model: simulating the RNN using Euler's method and training by backpropagation through time. B. Coarsened discretization (CD): training in the same way the base model, ex- cept with a larger discretized time step (by factor of () and then gradually reducing 0 as training progresses until 0 = 1. C. Skip connections through time (SCTT): implementing skip connection every 8 time steps which takes a weighted average of the hidden state on the two ends the skip con- nection with weights 3 and - B, and then gradually increasing until 8 1. D. Dynamics-alignas skip connections (DASC): SCTTs but with a CD step before taking the weighted average.
3.2 Skip connections through time (SCTT)
SCTT similarly confines the well-established idea of skip connections through time to training only (Figure 2C). For some mixing ratio 8, skip connection between time steps t and 0 alters the hidden state as:
SCTT 1140
(1 31 1 Brttoe
The skip length remains unchanged throughout training, while the mixing ratio 0 < 3 < 1 starts at small value. This provides shorteuts for gradient backpropagation, thus reducing the risk of gradient instability, Similar to CD, the mixing is gradually reduced (i.e. , is gradually increased to 1) until we left with trained network without skip connections (5).
3.3 Dynamics-aligned skip connections (DASC)
We propose additional method, DASC, which combines the essence of CD with SCTT, in order to respect dynamical properties of the system (Figure(p)) DASC achieves this by simulating large time steps through its skip connections. The skip connections between / and L + 0 alters the hidden state as:
DASC EHOA + (1 B) rrtoA: + 9 base Brttos
CD
(9)
DASC aims to combine the benefits of and SCTT while limiting the risk of gradient instability. Unlike CD, DASC simultaneously trains the model with large and appropriately small time steps: unlike SCTT, mitigates misalignment in the network's dynamics when it mixes hidden state estimates through the paths with and without skip connections.
5

---

A
CD 7
Dynamics
alignment
Neural state
dr, dr x0
DASC Y
space
Time
B
drei2
dr
0
++1
SCTT
dr:
True dynamics
Dynamics
attenuation
B
C
0.2
Mean- squared (nomalized)
Performance discrepancy
H
Discretization Skip interval [time constant)
Diacretization Skip interval (time constant)
Figure 3: Effects of proposed algorithms on network dynamics. A. Diagram of proposed dynamics in activity subspace over time. Coarsened discretization (CD, red) typically results in large discretization error. Skip connections through time (SCTT, green) result an attenuation in dynamics. aaaamics--alined skip connections (DASC, blue) make its hidden-states more aligned with the true dynamics compared to CD. B. Normalized mean--quared error between true and modified dynamics for B = 0.5, 0 {5, 10, 20, 30, 40). Shaded area (0 > 20) implies time discretization greater than the time constant. C. Difference in task performance computed between modified and true dynamics.
3.4 Network dynamics
We define the neural dynamics of an RNN the absence of skip connections or time-step over- discretization its true dynamics; RNNs simulated with any of the methods under study (CD, SCTT or DASC) will thus result in modified dynamics. We provide an intuition of the resultant modified dynamics underlying each method with reference to the true dynamics (Figure 3 A). Under reasonable circumstances, CD typically provides a relatively poor estimation the true network activity result of extrapolating over longer time interval. DASC bridges this discrepancy by taking a weighted average between the true neural activity and neural activity produced by CD (Figure blue), and is therefore more aligned to the true dynamics than CD by design. We refer to this property as being rdynamics-aligned" (hence its name). SCTT, on the other hand, takes a weighted average between the network state two different time points. From the perspective of the later time point, the dynamics can be said to be attenuated (Figure green).
In order to quantify the effects of modified dynamics on trained networks, train 100 networks the aforementioned 16 standard tasks [4] and evaluated the networks with performance in the 10th - 9hth percentile over 400 training steps using all three proposed methods. For SCTT and DASC, mixing ratio 3 constant at 0.5, while varying the skip length E (5. 10, 20, 30,40) 40), where the time constant 20 times the base discretization step. For CD. vary the discretization factor, also referred to as 0, using the same values. For all three methods, 0 represents the number of skipped time steps which effectively shortens the minimum number of steps over which backpropagation through time is performed to alleviate gradient stability problems. Therefore, they are plotted on the same axis (Figure BB-C). We find that it difficult train networks with CD using values greater than the time constant. We provide an empirical rationale for this in Appendix E. In contrast, SCTT and DASC remained stable even with skip lengths equal twice the time constant, which highlights key advantage in using skip connections. To compare the accuracy of the methods, we compute the mean- squared distance in neural activity between each of their true and modified dynamics for each network and average this distance over all networks and across all tasks (Figure BB). SCTT's discrepancy is consistently large due to the fact that offers a zeroth-order approximation of DASC, as shown the first terms of the RHS of and (9.. Conversely, DASC uses a first-order approximation to improve numerical accuracy along the skip connections. These results are mirrored by the discrepancy in output performance, averaged here in the same manner the nquare distance (Figure3C)
keep the

---

Table I: Training biologically plausible RNNs on cognitive tasks requiring long-term dependen- cies. For each task, 100 models were simultaneously trained by backpropagation through time until the average performance of the networks in the 10th 90th percentile reaches 0.99 or when 40,000 training steps have been reached. Control represents the base model in Figure A The other columns are methods introduced in Figure
Tasks that require long-term dependencies
Task
Control
CD10
SCTT
SCTT40
DASC10 DASC40
Training steps (x10*)
dmc
0.196
0.251
1.089
0.907
0.244
0.343
dnmc
0.203
0.282
1.089
0.916
0,252
0,558
dnms
1.167
1.311
1.269
1.299
0,993
1.065
dms
1.324
1.079
1.293
1.282
1,469
1.041
ddmd
1.900
1.465
1.270
1.018
1.757
1.059
ddm
2.413
2.030
1.283
1.088
2.129
1.055
msddm
3.434
2.096
1.269
1.071
2.196
1.121
danti
failed
failed
1.268
1.088
failed
1.005
dgo
failed
failed
1.106
1.268
failed
1.003
Wall-clock training time (x10*s)
dmc
0.085
0.016
0.191
0.156
0.119
0.167
dnmc
0.088
0.018
0.171
0.172
0.125
0.273
dnms
0.501
0.237
0.498
0.554
0.491
0.517
dms
0.565
0.105
0.515
0.499
0.723
0.508
ddmd
0.820
0.326
0.484
0.595
0,863
0.518
ddm
1.030
0.632
0.548
0.664
1.023
0.523
msddm
1.464
0.683
0.634
0.595
1.079
0.543
danti
failed
failed
0.402
1.345
failed
0.409
dgo
failed
failed
0.325
0.261
failed
0.412
Floating point operations (x10
dme
0.114
0.080
0.229
0.187
0.144
0.201
dnme
0.119
0.090
0.207
0.207
0.148
0.327
dnms
0.682
0.416
0.599
0.685
0.585
0.623
dms
0.773
0.343
0.625
0.609
0.865
0.609
ddmd
1.110
0.465
0.582
0.706
1.034
0.620
ddm
1.409
0.645
0.667
0.804
1.254
0.617
msddm
2.005
0.666
0.768
0.727
1,293
0.656
danti
failed
failed
0.575
2.062
failed
0.588
dgo
failed
failed
0.465
0.386
failed
0.587
Results
We train networks on the same set 16 cognitive tasks [16 as in the previous section. For each method, train 100 separate networks across different hyperparameter values (see Appendix F) until the average performance of the networks in the 10th - 90 percentile reaches 0.99, or until a maximum of 40,000 training steps. We then verify that the performance of each network that failed to attain the required performance threshold has converged and remains constant, albeit at lower level of performance (for example, Figure 33 blue). The hyperparameters are:
Step count. CD reverts the base model when 1, while SCTT and DASC revert to the base model when 3 = 1. Training starts at some initial value 6u > 1 (CD) 3g < 1 (SCTT or DASC). 3 up to a value of (Figure right). The number of
We progressively update the value steps over which we carry out this progressive update through the course of training is defined the step count. We consider three values of step counts: {1,5, 10). This annealing process performed over the first 20,000 training steps in equal intervals.
7

---

Skip length. As described in the previous section, all methods have some way of shortening the number of backpropagation steps during training. For SCTT and DASC, this is facilitated by the skip connections. The skip length is therefore defined as the temporal length of the skip connection, which takes three values 0 E {10, 20, 40), representing half, one and twice the time constant respectively. For CD, the skip length is defined as initial discretization factor (i.e. prior to progressive updates), Oo. which can only take the value of 10 but not 20 and 40 due to network instability problems, as explained in Section 3.4andd Appendix E.
Initial ratio. For SCTT and DASC, we also vary the initial weighting ratio 30 E {0.2, 0.5, 0.8). There is no equivalent term for CD.
As an example, suppose we train using DASC with step size 5 and initial ratio 0.5. Then, 8 would start at 0.5 and then progressively take the values of 0.5, 0.6, 0.7, 0.8, 0.9 and finally 1.0 in fixed intervals over the course of training. In summary. we trained 4,800 networks using CD and 43,200 networks each using SCTT and DASC, totalling over 90,000 trained networks (Table S3). For each training session, we recorded the number of training steps (either due to attaining 0.99 performance or hitting the training step limit), the wall-clock time spent in training, and the number of floating point operations involved during training. We highlight the advantage of using skip connections over timescales longer than the time constant by comparing six training configurations: the base model (control), CD with discretization factor 10 (CD1o). SCTT with skip lengths 10 and 40 (SCTT10 10. aCTT and DASC with skip lengths 10 and 40 (DASC DASC.o) on tasks that require long-term dependencies (9 out of 16 tasks) in Table [ Detailed results can be found in Appendix F.
In the following analyses, we compare the training performance of methods we introduce here, namely SCTT and DASC, previously known methods, namely CD and direct training (although no explicit literature was found for CD with the best of our efforts). Skip connections alleviate gradient stability problems during training which improves training efficiency by facilitating shorter path for backpropagating through time. The most suitable metric to evaluate this objective is the number of training steps needed to learn the tasks (Table T top). RNNs trained with SCTT or DASC required fewer training steps than control or CD networks on 7 of the 9 tasks that require long-term dependencies. Moreover, control and CD networks were unable learn 2 of these 9 tasks. This presents a clear advantage of our proposed methods. However, in general, when evaluating training algorithms, other metrics such as wall-clock training time and total floating point operations are more practical and may be more emphasized (Table bottom). In most tasks, while CD requires more training steps. the wall-clock time and floating point operations required for each step is significantly less (determined by the discretization factor). However, CD networks are not always successful at learning long-term dependency tasks. Also, SCTT and DASC achieved shorter training times and required fewer floating point operations than contol networks on 7 of the 9 tasks. Taken together, we believe that these considerations tilt the advantage in favor of SCTT and DASC. Note that the results in Table [ are computed using optimal hyperparameter configurations. Hyperparameter-speciie analyses can be found in Figure S3 and Appendix G.
4.1 Strategic placement of skip connections
Another feature of skip connections is that they do not need to be regular or periodic, as implemented in the networks explored thus far. Skip connections can be placed anywhere over the task duration, where they may enhance training performance or inadvertently disrupt the true network dynamics. However, poorly placed skip connections can produce dynamical instabilities during training. Con- versely, we show that by introducing additional yet strategically-placee skip connections, we can exploit its properties to improve training efficiency (Figure 4D). To best illustrate this, we train bio- logically plausible RNNs on rule-based visuomotor association task 662 ized by the need to maintain long-term dependencies across trials so that it may maintain the current rule in memory across trials and detect implicit changes in the task rule to update this memory. There- fore, in contrast to the tasks studied thus far where networks are trained on single task trials, for this task networks must be trained sequences of consecutive trials. Consequently, these tasks pose higher level of difficulty for training biologically plausible RNNs due to the sheer number of time steps involved.
63). This task is character-
The network architecture is different compared to the previous models (Figure 4A) in that there is no separation of input and output populations, the input structure is changed, and the output of the
8

---

A
B
Biologically plausible (50 units)
Rule
Rule 2
Stimulus (2 units)
Stimulus
Response (2 units)
Feedback (2 units)
Response
C
Trial
Trial
Trial 5
Stimulus
Stimulus
Stimulus
Stimulus
Stimulus
Response
Response
Response
Response
Response
Feedback
Feedback
Feedback
Feedback
Feedback
True rule:
True rule:
True
RNN rule:
RNN rule: 2
RNN rulo: 2
RNN rule: 2
RNN rule:
D
Inter-trial interval n-1
Trial
Inter-trial interval
Trial n-1
1
1
F5
+1
x
Trial n+1
CD SCTT
DASC
Skip connection through time
x - (
Figure 4: Strategic placement of skip connections in multi-trial tasks. A. Overview of RNN model. B. Two hidden task rules. C. Example of a trial where uncued rule switching occurs, D. The task is trained implementing one of our proposed methods (CD, SCTT, DASC, marked with colored dots) with additional strategically-placel skip connections. tasks that encompass multiple trials, adding skip connections between the start of every trial provides minimal disruption to network dynamics.
network now passes through softmax in order to read out decision:
softmax W )
(10)
The network receives a 2-dimensional one-hot vector representing one of 2 stimuli as inputs. The target response is dictated by one of hidden rules. Under Rule the network must associate stimulus with response and stimulus with response 2, and under rule these associations are reversed (Figure44) 4B). The network may infer the current rule based on the feedback receives in the form another 2-dimensional one-hot vector that indicates positive or negative feedback following the network's 's response. Each rule persists across multiple trials, and then switches the other rule an uncued manner. Similar to the monkeys in (2), our trained models detect and adapt their behavior within a single trial after the rule switch (Figure 4C). We trained RNNs
consecutive trials, which necessitates learning dependencies that are 5x longer than the previous tasks. In light of this challenge, we define a new metric for performance: the number of networks out 100 that successfully learn to perform the task more than 0.99 accuracy. By introducing strategic skip connections between the start of consecutive trials, the proportion of trained networks increased across all methods (Table thereby demonstrating the effectiveness and flexibility of skip connections.
Table 2: Proportion of networks successfully trained to perform a rule reversal task. 100 models trained independently with their respective best hyperparameter configuration for each of the proposed methods and the proportion of successfully trained models are presented as baseline results. This is then repeated with additional strategically-place skip connections (Figure
Proportion of trained networks
Control
CD10 10
SCTT 40
DASC 19
Baseline
0.67 (0000))
0.19 (+0.02)
0.71 (+0.02)
0.71 (#0.02)
Strategic SC
0.76 (# 0.02)
0.25 (= 0.02)
0.81 (+ 0.02)
0.78 (#: 0.02)

---

5 Discussion
can be challenging to train biologically plausible RNNs on cognitive tasks requiring long-term dependencies without the aid of artificial gating elements. In "vanilla" RNN, an intuitive way to alleviate this difficulty is to coarsen the time discretization (CD) in order to reduce the number of time steps of the overall simulation, thereby shortening the path for backpropagation. However, we show that over-discretization can produce poor approximations the true continuous-time dynamics (red curves in Figure 33,C), thus hindering model reliability. While our implementation of CD overcomes this by progressively refining the discretization, our results suggest that it does not offer an improvement in training efficiency (Table []) even if some time savings is achieved. We also find that its benefits are constrained by the single unit time constant, beyond which the network quickly destabilizes during training (Figure BB. Finally even with CD, networks may fail to learn challenging long-term dependency tasks (Tables 1 In contrast, SCTT and DASC offer significant improvements in training efficiency and reliability. However, the two approaches accomplish this in different ways. While it provides a shorter path to backpropagate gradients, SCTT inadvertently attenuates the network's 's dynamics (Figure (A)). This significantly misaligns the network's 's dynamics in the presence versus absence of the skip connections (Figure 3 B,CC), which compels more learning as the network must change the skip connections are annealed away. Instead, DASC emphasizes an alignment of the network's 's dynamics with (modified) and without (true) skip connections (Figure while still supporting shorter paths for backpropagation.
Experimental advances in neuroscience are facilitating the study of tasks involving increasingly complex and lengthy temporal dependencies. Concurrently, technological advances are enabling simultaneous recordings from increasingly large neural populations. This underscores the need for complementary theory through fighliidlitty biologically plausible models that perform challenging computations over lengthy timescales. By introducing methods that enable the development of such models without sacrificing biological plausibility, our work will serve to broaden the applicability of deep learning methods in systems neuroscience. This includes the development of model for evidence accumulation (64] spatial navigation (65], scene processing [66), rule-guided behavior (67 68
complex rule learning [69]. few-shot learning and adaptive behavior [70]. An important reason for using large time steps when training biologically plausible RNNs that decreases the computational resources or wall-clock time required for training. While true for simpler tasks, we have shown that there are harder tasks that cannot be learned with conventional methods regardless of time step size.
is noteworthy that we focus on RNNs built to be directly comparable with neural circuits in the brain, rather than biological plausibility of training algorithm itself; a neural circuit in the brain is sculpted by not only learning but also development and evolution. In general, training biologically plausible RNNS is resource-intensive undertaking, and while methods improve training efficiency and model fidelity, they do not alleviate the resource load per-training step. Additionally, at the present time approaching human-level computational competency requires architectures such transformers unknown in neuroscience. More work will be necessary to develop biologically plausible yet computationally powerful mechanisms for learning long-term dependencies.
Conclusion
We have proposed and evaluated a set of methods to learn long-term dependencies in biologically plausible leaky RNNs. These methods improve model fidelity while offering substantial learning performance improvements over conventional training approaches, and even facilitate learning of tasks that are difficult to learn with conventional methods.
Author Contributions
W.S. developed the algorithm. W.S. and V.G. performed the experiments. All authors designed the study, took part in discussions, interpreted the results, and wrote the paper.
Acknowledgments and Disclosure of Funding
This work was partly supported by the NIH grant RO1 MH062349 and ONR grant --0014-23-1-
10

---

References
[1] Yamins, D. L.. DiCarlo, J. Using goal-driven deep learning models understand sensory cortex. Nature neuroscience 19, 356-365 (2016).
[2] Barak, O. Recurrent neural networks as versatile tools of neuroscience research. opinion in neurobiology 46, 1-6 (2017).
Current
[3] Richards, B. A. et al. A deep learning framework for neuroscience. Nature neuroscience 22, 1761-1770 (2019).
[4] Yang. G. R. & Wang, X.-J. Artificial neural networks for neuroscientists: primer. Neuron 107, 1048-101 1070 (2020).
[5] Elman, J. L. Finding structure in time. Cognitive science 14, 179-211 (1990).
[6] Valente, A., Pillow, J. W. & Ostojic, S. Extracting computational mechanisms from neural data using low-rank rnns, Advances Neural Information Processing Systems 35, 24072-24086 (2022).
[7] Lindsay, G. W., Mrsic-Flogel, T. D. & Sahani, M. Bio-inspired neural networks implement different recurrent visual processing strategies than task-trained ones do. bioRxiv 2022-03 (2022).
[8] Nayebi, A. et al. Recurrent connections in the primate ventral visual stream mediate a trade-off between task performance and network size during object recognition. Neural Computation 34, 1652-1675 (2022).
[9] Soo, W. & Lengyel, M. Training stochastic stabilized supralinear networks by dynamics-neutral growth. In Advances in Neural Information Processing Systems, vol. 35, 29278-29291 (2022).
[10] Willett, F. R. et al. Hand knob area of premotor cortex represents the whole body in compositional way. Cell 181, 396-409 (2020).
[11] Sussillo, D., Churchland, M. M., Kaufman, M. T. Shenoy, K. V. A neural network that finds naturalistic solution for the production of muscle activity. Nature Neuroscience 18, 1025-1033 (2015).
[12] Michaels, J. A., Schaffelhofer, S., Agudelo-Toro, A. & Scherberger, H. A goal-driven modular neural network predicts parietofrontal neural dynamics during grasping. Proceedings of the National Academy of Sciences 117, 32124-32135 (2020).
[13] Saxena, S., Russo, A. A., Cunningham, J. & Churchland, M. M. Motor cortex activity across movement speeds is predicted by network-level strategies for generating muscle activity. eLife 11 (2022).
[14] Mante, V., Sussillo, D., Shenoy, K. V. & Newsome, W. T. Context-dependent computation by recurrent dynamics in prefrontal cortex. Nature 503, 78-84 (2013).
[15] Kleinman, M., Chandrasekaran, C. & Kao, A mechanistic multi-area recurrent network model of decision-making. In Advances in Neural Information Processing Systems, vol. 34, 23152- 23165 (2021).
[16] Yang. G. R., Joglekar, M. R., Song, H. F., Newsome, W. T. & Wang. X.-J. Task representations in neural networks trained to perform many cognitive tasks. Nature Neuroscience 22, 297-306 (2019).
[17] Song, H. F., Yang, G. R. & Wang. X.-J. Reward-based training of recurrent neural networks for cognitive and value- tasks. eLife 6 (2017).
[18] Driscoll, L., Shenoy, K. Sussillo, D. Flexible multitask computation in recurrent networks utilizes shared dynamical motifs. bioRxiv 10.1101, 19..088155.50388 (2022).
[19] Masse, N. Y., Yang, G. R., Song. H. F., Wang. X.-. & Freedman, D. J. Circuit mechanisms for the maintenance and manipulation of information in working memory. Nature Neuroscience 22, 1159-1167 (2019).

---

[20] Cueva, C. J. er al. Low-dimensional dynamics for working memory and time encoding. Proceedings of the National Academy of Sciences 117, 23021-23032 (2020).
[21] Goudar, V., Peysakhovich, B., Freedman, D. J., Buffalo, E. A. & Wang, X.-J. Schema formation in neural population subspace underlies dearning-to-lean in flexible sensorimotor problem- solving. Nature Neuroscience 26, 879-890 (2023).
[22] Orhan, A. E. & Ma. W. J. A diverse range of factors affect the nature of neural representations underlying short-term memory. Nature Neuroscience 22, 275-283 (2019).
[23] Wang, J., Narain, D., Hosseini, E. A. & Jazayeri, M. Flexible timing by temporal scaling of cortical responses. Nature Neuroscience 21, 102-110 (2017).
[24] Zhou, Y. et al. Distributed functions of prefrontal and parietal cortices during sequential categorical decisions. eLife 10 (2021).
[25] Chaisangmongkon, W., Swaminathan, S. K., Freedman, D. J. & Wang, X.-J. Computing by robust transience: How the fronto-parietal network performs sequential, category-based decisions. Neuron 93, 1504-1517.c4 (2017).
[26] Finkelstein, A. et al. Attractor dynamics gate cortical information flow during decision-making. Nature Neuroscience 24, 843-850 (2021).
[27] Fascianelli, V. et al. Neural representational geometries correlate with behavioral differences in monkeys and recurrent neural networks. bioRxiv 10.1101 2022.10.05.55 1024 (2023).
[28] Whittington, J. C. al. The tolman-eichenbaum machine: unifying space and relational memory through generalization in the hippocampal formation. Cell 183, 1249-1266 (2020).
[29] Cueva, C. & Wei, X.-.. Emergence of grid-like representations by training recurrent neural networks to perform spatial localization. arXiv 10.48550, 1803.07770 (2018).
[30] Ebitz, R. B. & Hayden, B. Y. The population doctrine in cognitive neuroscience. Neuron 109, 3055 3068 (2021).
[31] Vyas, S., Golub, M. D., Sussillo, D. & Shenoy, K. V. Computation through neural population dynamics. Annual review of neuroscience 43, 249-275 (2020).
[32] Hochreiter, S. Schmidhuber, J. Long short-term memory. Neural computation 9, 1735-1780 (1997).
[33] Cho, K. et al. Learning phrase representations using mn encoder-decoder for statistical machine translation. arXiv preprint 140. (2014).
[34] Wang, X.-J. & Yang, G. R. A disinhibitory circuit motif and flexible information routing in the brain. Current Opinion Neurobiology 49, 75-83 (2018).
[35] Le, Q. V., Jaitly, N. & Hinton, G. F.. A simple way to initialize recurrent networks of rectified linear units. arXiv 10.48550, 1504.00941 (2015).
[36] Arjovsky, M., Shah, A. & Bengio, Y. Unitary evolution recurrent neural networks. Proceedings of The 33rd International Conference on Machine Learning 48, 1 1120-1128 (2016).
[37] Henaff, M., Szlam, A. & LeCun, Y. Recurrent orthogonal networks and long-memors tasks. Proceedings of The 33rd International Conference Machine Learning 48, 2034-0240000 2042 (2016).
[38] Vorontsov, E., Trabelsi, C., Kadoury, S. Pal, C. On orthogonality and learning recurrent networks with long term dependencies. Proceedings of the 34th International Conference Machine Learning 70, 3570-3578 (2017).
[39] Trinh, T., Dai, A., Luong, T. & Le, Q. Learning longer-term dependencies in RNNs with auxiliary losses. Proceedings of the 35th International Conference on Machine Learning 80, 4965-4974 (2018).
12

---

[40] He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. arXiv 10.48550, 1512.03385 (2015).
[41] Vaswani, A. et al. Attention is all you need. arXiv 10.48550, 1706.03762 (2017).
[42] Tolstikhin, et al. Mlp-mixer. An all-MLP architecture for vision. arXiv 10.48550, 2105.01601 (2021).
[43] Lin, T., Horne, B. G., Tino, P. & Giles, C. L. Learning long-term dependencies is not difficult with narx recurrent neural networks. Advances in Neural Information Processing Systems (1998).
[44] Zhang, et al. Architectural complexity measures recurrent neural networks. In Advances in Neural Information Processing Systems, vol. 29 (2016).
[45] Chang, S. et al. Dilated recurrent neural networks. In Advances in Neural Information Processing Systems, vol. 30 (Curran Associates, Inc., 2017).
[46] Yu, A. W., Lee, H. Le, Q. Learning to skim text. arXiv 10.48550, 1704.06877 (2017).
[47] Gui, T. et al. Long short-term memory with dynamic skip connections. Proceedings of the AAAI Conference on Artificial Intelligence 33, 6481-6488 (2019).
[48] Kim, R. & Sejnowski, T. J. Strong inhibitory signaling underlies stable temporal dynamics and working memory spiking neural networks. Nature Neuroscience 24, 129-139 (2020).
[49] Echeveste, R., Aitchison, I.., Hennequin, G. & Lengyel. M. Cortical-like dynamics in recurrent circuits optimized for sampling-basee probabilistic inference. Nature Neuroscience 23, 1138- 1149 (2020).
[50] Dubreuil, A., Valente, A., Beiran, M., Mastrogiuseppe, F. & Ostojic, S. The role of population structure in computations through neural dynamics. Nature Neuroscience 25, 783-794 (2022).
[51] Schuessler, F., Mastrogiuseppe, F., Dubreuil, A., Ostojic, S. & Barak, O. The interplay between randomness and structure during learning in rnns. Advances Neural Information Processing Systems, vol. 33, 13352- 13362 (2020).
[52] Stroud, J. P., Watanabe, K., Suzuki, T., Stokes, G. & Lengyel, M. Optimal information loading into working memory in prefrontal cortex explains dynamic coding, bioRxiv 10.1101, 2021.11.16.468360 (2021).
[53] Murray, J. M. Local online learning in recurrent networks with random feedback. el.ife 8 (2019).
[54] Zhang, X., Long, X., Zhang, S.-J. & Chen, Z. S. Excitatory-inhibioon recurrent dynamics produce robust visual grids and stable attractors. Cell Reports 41, 111777 (2022).
[55] Rajakumar, A., Rinzel, J. Chen, Z.S. Stimulus-driven and spontaneous dynamics in excitatory- inhibitory recurrent neural networks for sequence representation. Neural Computation 33, 2603- 2645 (2021).
[56] Liebe, S. et al. Phase of firing does not reflect temporal order in sequence memory of humans and recurrent neural networks. bioRxiv 10.1101, 2022.09.25.0. 25.925500777 (2022).
[57] Kim, R., Li, & Sejnowski, T. Simple franework for constructing functional spiking recurrent neural networks. Proceedings of the National Academy of Sciences 116, 22811 12811-18 1-22820 (2019).
[58] Keller, A. J. al. A disinhibitory circuit for contextual modulation in primary visual cortex. Neuron 108, 1181-1193.c8 (2020).
[59] Song, H. F., Yang, G. R. & Wang, X.-J Training excinatory-nnhiiiorry recurrent neural networks for cognitive tasks: simple and flexible framework. PLOS Computational Biology 12, c1004792 (2016).
13

---

[60] Kalidindi, H. T. er al. Rotational dynamics in motor cortex are consistent with a feedback controller. eLife 10 (2021).
[61] Khona, M., Chandra, S., Ma, J. & Fiete, Winning the lottery with neurobiology: faster learning many cognitive tasks with fixed sparse mns. arXiv 10.48550, 2207.03523 (2022).
[62] Asaad, W. F., Rainer, G. & Miller, E. K. Neural activity in the primate prefrontal cortex during associative learning. Newron 21, 1399-1407 (1998).
[63] Fusi, S., Asaad, W. F., Miller, E. K. & Wang, X.-J A neural circuit model of flexible sensorimo- tor mapping: learning and forgetting on multiple timescales, Neuron 54, 319-333 (2007).
[64] Morcos, & Harvey, C. D. iistory-deppndent variability population dynamics during evidence accumulation in cortex. Nature neuroscience 19, 1672-1681 (2016).
[65] Terry Jr, A. V. Spatial navigation (water maze) tasks. Methods of behavior analysis in
neuroscience 2, 153-166 (2009).
[66] Epstein, R. A. & Baker, C. Scene perception the human brain. Annual review of vision science 5, 373-397 (2019).
[67] Bernardi, S. et al. The geometry of abstraction in the hippocampus and prefrontal cortex. Cell 183, 954-967 (2020).
[68] Kaefer, K., Nardin, M., Blahna, K. & Csiesvari, J. Replay of behavioral sequences in the medial prefrontal cortex during rule switching. Neuron 106, 154-165 (2020).
[69] Grant, D. A. Berg, E. behavioral analysis degree of reinforcement and ease shifting to new responses in a weigl-type card-sorting problem. Journal of experimental psychology 38. 404 (1948).
[70] Wang, J. X. et al. Prefrontal cortex as a meta-reinffrrcemen learning system. Nature neuro- science 21, 860 868 (2018).
14


Reviews:
[{"review_id": "UhkQ4ZLB9R", "reviewer": "Reviewer_CT76", "summary": "This paper proposes a way to speed up biological RNN training by up to 3x using well placed skip connections during training time.", "questions": "Does the training procedure affect the type of synaptic connectivity solution learned? This should be a interesting direction to explore.", "limitations": "none", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper will be of interest to researchers training biological neural networks, especially for hard cases where a 3x time speed up can be achieved. Presentation is clear and the work should be relatively easy to reproduce.", "weaknesses": "I find a 3x speed up only for really difficult cases to be of modest improvement. Given currently powerful computers, training a biological style small recurrent neural network is relatively fast which reduces the practical significance of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> This paper proposes a way to speed up biological RNN training by up to 3x using well placed skip connections during training time. This paper will be of interest to researchers training biological neural networks, especially for hard cases where a 3x time speed up can be achieved. Presentation is clear and the work should be relatively easy to reproduce.\n\nWe thank the author for the review and helpful comments.\n\n> I find a 3x speed up only for really difficult cases to be of modest improvement. Given currently powerful computers, training a biological style small recurrent neural network is relatively fast which reduces the practical significance of this work.\n\nWe understand the concern of the reviewer. Due to the order in which we presented our results, it is possible that too much emphasis may have been unintentionally diverted to the 3x speed up result. While it is one of our significant claims, we have actually presented an even stronger result in our paper that we would like to divert the reviewer’s attention towards. From our abstract, our main claims are:\n\n(1) up to 3x training speedup for biological RNNs on a range of standard cognitive tasks  \n(2) enable convergence on cognitive tasks that prove challenging to train with conventional methods\n\nTo better present our results, we would like to reorder these two claims to better emphasize the (originally) second claim:\n\n(1) our techniques enable RNNs to successfully learn tasks that are otherwise impossible with conventional methods (rephrase of point 2)  \n(2) on simpler tasks where direct training is still possible, we achieve up to a 3.1x reduction in training iterations and 2.7x reduction in training time (rephrase of point 1 with an additional result involving wall-clock training time)\n\nThese results are backed by Table 1 in the main text (training iterations) and Table R1 in the uploaded rebuttal PDF (wall-clock training time). In addition, we have reviewed, in Appendix A, a full range of more complex tasks requiring long-term dependencies in neuroscience that our method can potentially be applied to, which again are impossible to train directly without the use of artificial gating elements. \n\n> Does the training procedure affect the type of synaptic connectivity solution learned? This should be a interesting direction to explore.\n\nWe thank the reviewer for the helpful suggestion. We will discuss this as future work in a future revision if given the opportunity.\n\nOverall, we hope that our response will improve the reviewer’s impression regarding the significance and scope of our work.\n\n"}, {"review_id": "luvZYSH0AQ", "reviewer": "Reviewer_T2Cs", "summary": "This paper describes and tests several methods for training a class of “biologically-plausible” recurrent neural networks (RNN), which are discrete approximations of continuous-time leaky RNNs. This type of gate-less RNN model has difficulty learning dependences over long time intervals, so tweaks are applied during training time to help stabilize and propagate gradients through time. Three such tweaks are explored: coarsened discretization (CD), where the RNN time step is simply increased during training; skip connections through time (SCTT); and the new method dynamics-aligned skip connections (DASC), which cleverly combines CD and SCTT. Some theoretical analyses of these methods are provided, and then each method is evaluated on a variety of realistic cognitive tasks. The results show that the new method, DASC, tends to perform best (measured by how long the model takes to reach a performance criterion), although different tasks require different hyperparameters.\n", "questions": "None noted. (See suggestions under \"weaknesses\" above.)", "limitations": "The discussion of limitations is clear and complete.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "* This paper is quite comprehensive, including clear mathematical descriptions (and nice figures) of the methods, theoretical analyses of their properties, and thorough experimental evaluation.\n* The overall approach—modify the network at training time, then remove the modifications after training—is a clever end-run around issues of biological plausibility. The resulting models are always in the “biologically plausible” class, even though the training procedure that arrived there is not. (And we suspect that training with backpropagation is implausible anyway, so no big change there.) While I’m not sure I’ve seen this approach taken with this type of RNN in the past, the authors might note that similar ideas have been floated for spiking neural networks by using surrogate loss at train time.\n* The results are solid and seem useful going forward!", "weaknesses": "The science in this paper is strong, so all the weaknesses noted here relate to clarity and presentation.\n* I found many points in this paper fairly difficult to understand or under-explained until I read the Appendix. There is quite a bit of information in the Appendix that, by any right, should be in the main paper. (References to earlier skip-connection work, the actual model formulation used, how training and evaluation was performed—it was unclear from the main paper how or when the extra discretization and skip connections were removed when testing models!—and descriptions of the tasks.) I understand that the authors are operating under strict page limits, but should this paper be accepted I strongly urge the authors to consider moving some things that are likely critical information for a reader into the main text.\n* While the figures illustrating the model (Figures 2 & 5) and showing experimental results (Figure 4) were fairly clear, those summarizing earlier work (Figure 1) and illustrating the modified model dynamics (Figure 3) were much more opaque. I’d recommend the authors attempt to simplify or clarify these figures.\n* One issue I was left somewhat uncertain of was novelty. As mentioned above, skip connections are by no means new, but are presented as a novel technique in the main text of this paper. I am not deeply embedded enough in the literature to know how to place DASC. I would have felt better about this if the authors had spent a bit more time  & ink on describing earlier approaches to these issues and connections to their proposed methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the author for the positive review and helpful comments. We also thank the reviewer for the suggested relevance to surrogate gradient approaches used in training spiking neural networks – we will mention this in a revised version if given the opportunity.\n\n> I found many points in this paper fairly difficult to understand or under-explained until I read the Appendix ... I strongly urge the authors to consider moving some things that are likely critical information for a reader into the main text.\n\nWe apologize for including important information in the supplementary. As specific formatting issues are difficult to address in a rebuttal without providing context to the (modified) overall narrative, we refer the reviewer to our general response to all reviewers which details a full outline overhaul.\n\n> While the figures illustrating the model (Figures 2 & 5) and showing experimental results (Figure 4) were fairly clear, those summarizing earlier work (Figure 1) and illustrating the modified model dynamics (Figure 3) were much more opaque. \n\nWe apologize for the confusion regarding Figures 1 and 3. For Figure 1, the data in panels B-D are summarized in Table S1 in the supplementary with their exact values listed. Panel E involves simulations that we have run, whose exact datapoints are summarized in the first data column of Table 1 in the main text. We have also included a step-by-step diagram of how the plots in Figure 3B-C are generated in the rebuttal PDF (Figures R1 and R2), along with further explanation in the general response.\n\n> One issue I was left somewhat uncertain of was novelty.\n\nWe would like to clarify the novelty of our work in this reply, which we will include in a revised version if given the chance. For each of the methods in the manuscript, we further classify them into two versions: (1) the regular feedforward version and (2) the version which confines the method to training only. \n\n**Coarsened discretization (CD)**\n  \nRegular: We do not regard this idea as novel to the paper.  \nThe concept of using a larger time discretization for simulations is well-established since the early days of running simulations on a computer using Euler’s method.    \nTraining only: We do not regard this idea as novel to the paper.  \nIn general, experiments were run with large discrete steps following verification or assumption that using smaller steps does not change the resultant dynamics. \n\n**Skip connection through time (SCTT)**\n  \nRegular: We do not regard this idea as novel to the paper.   \nWe have provided a brief overview of the background of skip connections through time (currently as Appendix B, but will be included in the main text in a revised version).   \n\nTraining only: We regard this idea as novel to the paper.  \nIn existing models, skip connections are either inherent in some mathematically-equivalent form or added as a permanent part of the model architecture. Our unique goal of biological-plausibility has resulted in the rare situation where removing features is justified.\n\n**Dynamics-aligned skip connection (DASC)**\n  \nRegular: We regard this idea as novel to this paper.  \nTraining only: We regard this idea as novel to this paper.  \n\nIn addition, we also regard the following analyses as original work:  \n(1) showing that removing skip connections by gradually reducing their linear weights allows us to successfully bake in long-term dependencies into a vanilla RNN  \n(2) analysis of network activity dynamics and stability – the time skip used in CD cannot exceed the time constant of the neuron due to stability issues, while SCTT and DASC can extend skip connections way beyond this interval  \n(3) hyperparameter analyses of all three proposed methods and the conclusions drawn, such as using short skip connections for simpler tasks and longer skip connections for tasks requiring long-term dependencies  \n(4) strategic placements of skip connections can result in minimal disruption to network activity and improve training efficiency  \n\n> ... describing earlier approaches to these issues and connections to their proposed methods.  \n\nWe present a review of earlier approaches of solving the long-term dependency problem without gating mechanisms, which we plan to add to the introduction, as well as provide a more in-depth review in the supplementary:\n\n“Other methods that help RNNs learn long-term dependencies have been proposed in the past that do not involve gating mechanisms. A straightforward way that was found would be to simply initialize the recurrent weight matrix as an identity matrix (Le et al. 2015). When applied to ReLU RNNs, the networks were competitive with LSTMs on speech and language benchmarks. On the same line of thought, RNNs whose recurrent weight matrices are constrained to have absolute eigenvalues of 1 have been proposed (Arjovsky et al. 2016, Henaff et al. 2016), at the cost of a lower convergence rate (Vorontsov et al. 2017). At their cores, both methods alleviate the problem by reducing the base multiplier in which gradients explode or vanish. Alternatively, auxiliary losses targeting intermediate time steps (Trinh et al. 2018) have been proposed, which reduces the exponents rather than the base of the gradient instability problem. Skip connections fall under this category of solutions.”\n\nLe et al. (2015) doi:10.48550/arXiv.1602.06662  \nArjovsky et al. (2016) doi:10.48550/arXiv.1511.06464  \nHenaff et al. (2016) doi:10.48550/arXiv.1602.06662  \nVorontsov et al. (2017) doi:10.48550/arXiv.1702.00071  \nTrinh et al. (2018) doi:10.48550/arXiv.1803.00144  \n\nOverall, we hope that we are able to clarify any confusion and address the concerns of the reviewer, especially regarding novelty and priority of information in the main text and supplementary."}, {"review_id": "VQJmQWg4TS", "reviewer": "Reviewer_fBiB", "summary": "In this study, the authors propose a couple of architectural innovations to train biological RNNs in a stable manner on a suite of 16 cognitive tasks, some of which require the integration of long-range temporal dependencies. The authors motivate their work from the lack of cortical evidence for gating mechanisms that are crucial to high-performing RNNs such as GRU and LSTM. Instead the authors focus on incorporating skip connections through time in various implementations to learn long-term dependencies. Over various experiments and numerical simulations of recurrent trajectories, the authors show that their proposed DASC method to apply skip connections through time performs effectively, especially when the task requires learning long-term dependencies.  ", "questions": "- My suggestion to the authors is to strengthen their stance against gated recurrent networks; if these gated networks are good models of cognitive tasks and neural data, why should they not be favored over biological RNNs? As such, all these networks are trained with BPTT which also does not have strong biological evidence.  ", "limitations": "The authors have stated limitations of the work adequately.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ RNNs are widely used in encoder-decoder models trained on cognitive tasks or neural data. The proposed work aims to address one of the issues that plagues RNNs - learning long range dependencies. Skip-connections through time aren't actively explored in RNNs for computational neuroscience and I appreciate the authors' effort in studying this mechanism for improving RNN training stability.\n+ The figures in the paper are commendably presented and really help the readers to understand the underlying mechanisms and differences between Coarsened Discretization, Skip Connections Through Time and DASC. ", "weaknesses": "- My main issue with the paper is that I believe it is not well motivated; they mention that using biological RNNs without gates is important but this claim is not backed with any advantage of biological RNNs over ones with gates. As such, I don't think one would consider gating as biologically implausible as this computation has surfaced in various parts of the cortex. \n- A related issue is that, the authors have not shown comparisons of their proposed skip-connections models against the established gated recurrent networks like GRU and LSTM. I find this as an important comparison that is missing, as it allows for relatively measuring the contribution of temporal skip-connections against gating mechanisms.\n- I believe the writing in the paper could be improved further, there is a lot of information that isn't central to the paper's theme such as Section 2.2 (Review of model parameters) which obstructs the paper's flow and makes it harder to glean the main message. \n- Even though the authors mention a variety of usecases for biological RNNs (such as modeling neural responses), they only evaluate their models on a suite of simple cognitive tasks. It would be more holistic if the authors also included observations on whether the alternate usecases of RNNs in neuroscience also benefit from adding dynamic skip connections through time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> ... they mention that using biological RNNs without gates is important but this claim is not backed with any advantage of biological RNNs over ones with gates...\n\nWe thank the reviewer for raising this important issue. LSTM and GRU architectures were introduced to overcome challenges pertaining to training vanilla RNNs with gradient descent. LSTMs and GRUs endow individual network units with adaptive multiplicative gating applied to their inputs, memory computation and outputs (only for LSTMs). This has two important computational benefits. First, the multiplicative gating, particularly in the context of memory computations, strongly curtails gradient stability challenges (Hochreiter and Schmidhuber, 1997). Second, they inherently support adaptive memory timescales and computation at the single unit level. These enhancements make them substantially more computationally powerful than vanilla RNNs, thus inviting their adoption in cognitive models (Storrs and Kriegeskorte, 2019; Feinman and Lake, 2020) and in phenomenological models of neural computation (Pandarinath et al., 2019). However, key biological constraints preclude them from being considered biologically plausible:\n\n**Gating.** Gating of the inputs and outputs of neurons is thought to be mediated by distinct inhibitory interneuron types via their nonlinear interactions at the dendrites and soma, respectively, of excitatory pyramidal neurons (Wang and Yang, 2018). While it is certainly believed that gating may enhance adaptive properties of neural computation, available evidence and existing models do not support the hypothesis that this gating is multiplicative.\n\n**Training.** A major motivation for LSTM and GRU architectures is that they are easier to train than vanilla RNNs due to their improved gradient stability. In contrast, RNNs endowed with interneurons and biologically plausible gating functions are more difficult to train than vanilla RNNs (Song et al., 2016). In future work, we certainly aim to extend DASC to these models in order to improve their practical application in computational neuroscience studies.  \n\n**Dynamics.** A constraint for biologically-plausible RNN models is that they should produce continuous-time dynamics. LSTM and GRU architectures are discrete-time systems by construction. Continuous-time variants of these architectures have been proposed in the neuroscience literature to address precisely this issue (Song et al., 2017).  Moreover, LSTM units typically respond with abrupt activity transitions between consecutive time steps (Storrs and Kriegeskorte, 2019). In contrast, neurons exhibit smooth temporal responses.\n\n**Memory.** LSTM and GRU memory mechanisms seem too powerful relative to known brain mechanisms of short-term memory function. The forget (update) gate of LSTMs (GRUs) endow individual units with arbitrarily small and large memory timescales. In contrast, the neuron membrane time constants are typically quite small and of limited range (Tripathy and Gerkin, 2016). While a variety of Glutamatergic signaling mechanisms are believed to enhance neural integration timescales (Reiner and Levitz, 2018), they are still limited. Moreover, receptor expression levels vary drastically by brain region (Froudist-Walsh, et al., 2023), obviating a general-purpose mechanism to support arbitrary timescales at the single neuron level. Continuous-time neural *population* dynamics offer a complementary mechanisms for arbitrary (and adaptive) timescales in the brain (Vogels et al., 2005). There is little evidence for LSTM/GRU like single-unit timescales in the brain.     \n\nFor these reasons, our study, which focuses on enhancing the ability of biologically-plausible trained RNNs to compute over long timescales, foregoes the analysis of DASC in LSTM and GRU networks. We also intend to include this review in the supplementary, with the main points summarized in the introduction of the main text.\n\nHochreiter, S., & Schmidhuber, J. (1997).  doi:10.1162/neco.1997.9.8.1735  \nFeinman, R., & Lake, B. M. (2020). doi:10.48550/arXiv.2003.08978  \nStorrs, K. R., & Kriegeskorte, N. (2019). doi:10.48550/arXiv.1903.01458  \nPandarinath, C. et al. (2018). doi:10.1038/s41592-018-0109-9  \nWang, X. J., & Yang, G. R. (2018). doi:10.1016/j.conb.2018.01.002  \nSong, H. F. et al. (2016). doi:10.1371/journal.pcbi.1004792  \nSong, H. F. et al. (2017). doi:10.7554/eLife.21492  \nReiner, A., & Levitz, J. (2018). doi:10.1016/j.neuron.2018.05.018  \nFroudist-Walsh, S. et al. (2023). doi:10.1038/s41593-023-01351-2  \nVogels, T. P. et al. (2005). doi:10.1146/annurev.neuro.28.061604.135637  \n\n> I believe the writing in the paper could be improved further...\n\nWe refer the reviewer to our general response which details a full outline overhaul.\n\n> Even though the authors mention a variety of usecases for biological RNNs (such as modeling neural responses), they only evaluate their models on a suite of simple cognitive tasks...\n\nWe believe that our suite of cognitive tasks are a representative sample of what is being trained in the field. The reviewer is right that these are relatively simple tasks; this does not necessarily represent a problem in our work but exposes another problem – that such RNNs are currently unable to learn more complex tasks. Our method expands the range of tasks. We have also provided a review in Appendix A on complex tasks in neuroscience that our method can be applied to.\n\n> ... all these networks are trained with BPTT which also does not have strong biological evidence.\n\nAll references in Table S1 have used BPTT to train their networks in order to draw comparisons with the brain. Our work is intended to support this research direction. While the ideal goal would be to train with biological learning rules, our work focuses first on finding a working solution after which other solutions can be explored by implementing more biological constraints.\n\nWe thank the reviewer and hope we have cleared most doubts.  "}, {"review_id": "0NhvetEC2B", "reviewer": "Reviewer_nbN6", "summary": "The authors suggest a method for accelerating the training of continuous time vanilla recurrent neural networks. Specifically, they add temporal skip connections during training, and gradually remove them throughout the training process. For tasks with long term memory requirements, this can provide up to 3 times less training steps. The authors also numerically examine different variants of such skip connections. In particular, they use an interpolation between coarse-grained and fine-grained dynamics.\n\n", "questions": "The use of the term biological throughout the paper is somewhat misleading. It is true that notations vary between fields and subfields, but still – biological is usually reserved for actual biological networks. Terms such as vanilla or continuous time are perhaps more appropriate. The term biologically plausible could also be used, but that is usually used for adding properties above simple rate networks – separation to E and I neurons, spiking, dendrites, short term plasticity, excitability and so on.\nLine 106: alpha used without definition\nFigure 1: perhaps use log scale for X axis of panels B,D.\nColors in figures are hard to discern (the specific shades of blue and green used).\nIf I understand correctly, skip connections usually come with trained weights. In this work, they have the untrained weight beta, to allow their eventual removal. This should probably be stressed.\nComparison of true and modified dynamics (line 170, Fig 3B) – I couldn’t fully understand the setup. Were these networks with the same random seed? Why should neural activity be identical? There are many symmetries, such as permutations, that are expected to arise while training. Was some alignment done between the neural trajectories?\nFigure 4. The Y axes are very different in every panel. If possible, this should be avoided. If not (different scales of results), this should be stressed to the reader.\nFigure 4B – how do these results look when splitting the tasks?\nFigure 4B, last panel. Is the bimodal distribution a result of different tasks?\nFigure 4B the X labels are not intuitive.\nThe first paragraph of the discussion is titled “existing methods” but has no references.\n", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors highlight an existing problem with RNNs and offer a novel way of addressing it.", "weaknesses": "The main claim – faster training – is not verified. Networks with skip connections might be more computationally expensive. Table S3B gives a hint that there is some saving, but this saving is quite modest. It’s not clear what is the relationship between the algorithms and compute time, and how this scales with parameters such as network size.\nIt was hard to understand the exact form of the equations, due to the use of “before” and “after” labels. I assume (also from figure 3) that the meaning is that the high-resolution dynamics runs in parallel, and the two versions are combined. If this is correct, then equations 7 and 8 seem incorrect. Equation 7 is the case for CD, not for the discretized equation 1. The last term of equation 8 (“before”) also depends on the recurrent connectivity. A rough approximation would probably be $W^\\theta$, because W is applied $\\theta$ times to generate this term. If the eigenvalues of W are larger than 1, this will amplify them.\nThe relation to prior literature is very partial. For instance, skip connections are only described in the supplement – without references in the main text. Other approaches for training RNNs on long dependencies (a few references below as an example) are also not discussed.\n\n\nTrinh, T., Dai, A., Luong, T., & Le, Q. (2018, July). Learning longer-term dependencies in rnns with auxiliary losses. In International Conference on Machine Learning (pp. 4965-4974). PMLR.Le, Q. V., Jaitly, N., & Hinton, G. E. (2015). A simple way to initialize recurrent networks of rectified linear units. arXiv preprint arXiv:1504.00941.\nArjovsky, M., Shah, A., & Bengio, Y. (2016, June). Unitary evolution recurrent neural networks. In International conference on machine learning (pp. 1120-1128). PMLR.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the extensive review. We hope we can address them within the character limit.\n\n> The main claim is not verified.\n\nOur claim was referring to the reduction in number of training iterations. We understand that this would not be sufficient without data on wall-clock training time and total floating point operations, and hence we have included that in Table R1 in the rebuttal PDF. We strengthen this claim to now be up to a 3.1x reduction in training iterations and 2.7x reduction in training time.\n\nThis is not the main claim of this work. From Table 1 (main text) and Table R1 (rebuttal PDF), there are tasks in which an RNN cannot be trained on with conventional methods, but this was made possible using some of our proposed methods. Our strongest result would therefore be that our techniques enable RNNs to successfully learn tasks that are otherwise impossible with conventional methods. \n\nExperiments on how our methods scale with network size will entail future work, but since skip connections involve floating operations that linearly scale with network size, we believe that it will not be as significant as the recurrent dynamics which has quadratic scaling with network size.\n\n> It was hard to understand the equations...\n\nDASC is not strictly two parallel time resolutions since they combine and resplit after the interpolation (Figure 2D).\n\n$$ \\mathbf{r}\\_{t+\\theta\\Delta t}^\\text{SCTT} \\leftarrow (1-\\beta) \\mathbf{r}\\_{t} + \\beta \\mathbf{r}\\_{t+\\theta\\Delta t} $$\n\n$$ \\mathbf{r}\\_{t+\\theta\\Delta t}^\\text{DASC} \\leftarrow (1-\\beta)  \\left( \\mathbf{r}\\_{t} + \\frac{d\\mathbf{r}\\_t}{dt} \\Delta{t} \\times \\theta \\right) + \\beta \\mathbf{r}\\_{t+\\theta\\Delta t} $$\n\nWe observe empirically that when we increase the discretization step size for CD, the network becomes unstable. The threshold is approximately the time constant. To model this, we consider a linear approximation of the network dynamics:\n$$\n    \\mathbf{T}\\frac{d\\mathbf{r}\\_t}{dt} \\approx - \\mathbf{W}\\_\\theta^* \\mathbf{r}\\_t\n$$\n\nThe dominant eigenvalue of $\\mathbf{W}\\_\\theta^*$ is dependent on $\\theta$, where $\\theta\\Delta t$ is the time discretization used to simulate the network. $\\alpha\\Delta t$ is the time constant. From this framework, this eigenvalue will exceed 1 when $\\theta$ reaches some threshold $\\theta\\_\\text{thres}$. We also define $\\mathbf{W}\\_1^*$ for the base case.\n\nWe ask: are the skip connections in SCTT and DASC subject to the same threshold? Our simulations suggest that skip connections can extend beyond that. To gain an intuition, we apply the same model to DASC, where:\n\n$$\n\\mathbf{r}\\_{t+\\theta\\Delta t} \\approx e^{-\\frac{\\theta}{\\alpha}\\mathbf{W}\\_1^{*}}\\mathbf{r}\\_t \n\\approx \\mathbf{r}\\_t - \\frac{\\theta}{\\alpha} \\mathbf{W}\\_1^\\* \\mathbf{r}\\_t\n$$\n\nso that the update equation becomes:\n\n$$\n\\mathbf{r}\\_{t+\\theta\\Delta t}^\\text{DASC} \\leftarrow (1-\\beta)  \\left( \\mathbf{r}\\_{t}- \\frac{\\theta}{\\alpha} \\mathbf{W}\\_\\theta^\\* \\mathbf{r}\\_t \\right) + \\beta \\left( \\mathbf{r}\\_t - \\frac{\\theta}{\\alpha}\\mathbf{W}\\_1^\\*\\mathbf{r}\\_t \\right) = \\mathbf{r}\\_t - \\frac{\\theta}{\\alpha}\\left[ (1-\\beta) \\mathbf{W}\\_\\theta^\\* + \\beta\\mathbf{W}\\_1^\\* \\right] \\mathbf{r}\\_t\n$$\n\nThis is equivalent to a linear system with $\\mathbf{W}^* = (1-\\beta) \\mathbf{W}\\_\\theta^\\* + \\beta\\mathbf{W}\\_1^\\*$ which is not always an unstable system. We will include this in the supplementary.\n\n> The relation to prior literature is very partial...\n\nWe have briefly summarized related works on this topic and will add the following review to the introduction, as well as include an extended review in the supplementary:\n\n“Other methods that help RNNs learn long-term dependencies have been proposed in the past that do not involve gating mechanisms. A straightforward way that was found would be to simply initialize the recurrent weight matrix as an identity matrix (Le et al. 2015). When applied to ReLU RNNs, the networks were competitive with LSTMs on speech and language benchmarks. On the same line of thought, RNNs whose recurrent weight matrices are constrained to have absolute eigenvalues of 1 have been proposed (Arjovsky et al. 2016, Henaff et al. 2016), at the cost of a lower convergence rate (Vorontsov et al. 2017). At their cores, both methods alleviate the problem by reducing the base multiplier in which gradients explode or vanish. Alternatively, auxiliary losses targeting intermediate time steps (Trinh et al. 2018) have been proposed, which reduces the exponents rather than the base of the gradient instability problem. Skip connections fall under this category of solutions.”\n\nLe et al. (2015) doi:10.48550/arXiv.1602.06662  \nArjovsky et al. (2016) doi:10.48550/arXiv.1511.06464  \nHenaff et al. (2016) doi:10.48550/arXiv.1602.06662  \nVorontsov et al. (2017) doi:10.48550/arXiv.1702.00071  \nTrinh et al. (2018) doi:10.48550/arXiv.1803.00144  \n\n> The use of the term biological throughout the paper is somewhat misleading. \n\nWe agree with the reviewer that the nomenclature should be intuitive rather than be specific to any field or subfield. We will refer to our networks as biologically plausible (instead of biological) in a revised version to reflect the incorporation of realistic neuron time constants and lack of artificial gating mechanisms.\n\n> Comparison of true and modified dynamics...\n\nWe have included a flowchart in Figure R2 in the rebuttal PDF, plus an explanation in the general response. Networks have different seeds and no alignment was performed.\n\n> Figure 4...\n\nSee Figure R3 in the rebuttal PDF and the accompanying explanation in the general response. The reviewer’s intuition is correct regarding the bimodal distribution, and we have split the tasks according to long-term dependency requirements in Figure R3.\n\nWe have also fixed all other minor issues raised by the reviewer, such as color contrast, undefined symbols and issues with axis labels. We hope that we can convince the reviewer about the significance of our work.\n"}, {"review_id": "cM8ZOTnfBS", "reviewer": "Reviewer_cfQC", "summary": "The paper proposes a method for extracting long-term dependencies in biologically plausible leaky RNNs. The motivation of this work is well-presented and has contribution to the neuroscience field.", "questions": "Please consider the things listed in the “Weaknesses” section.\n\nAlso please consider providing information regarding any potential future improvements.", "limitations": "To my knowledge and understanding, there is no potential negative societal impact of this work. Other limitations please see “Weaknesses”.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n\n* The paper demonstrates significant improvements in terms of performance when compared to other baseline methods.\n\n* The paper is well structured and has a smooth and concrete flow.\n\n* I find the incorporation of biological domain knowledge in the algorithm quite interesting and insightful.", "weaknesses": "Weaknesses\n\n* In terms of presentation, it would be helpful to list the properties of datasets used in the experimental part.\n\n* Comparison with LSTMs and GRUs-based methods should be a part of the experiments in order to prove the superiority of the suggested approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive comments.\n\n> In terms of presentation, it would be helpful to list the properties of datasets used in the experimental part.\n\nAll cognitive tasks used to train our networks are currently explained in Appendix C in the supplementary. We will elaborate further on the neuroscientific basis of each task, as well as provide a summary of the experimental data that have been reported in literature in a revised version if given the opportunity. \n\n> Comparison with LSTMs and GRUs-based methods should be a part of the experiments in order to prove the superiority of the suggested approach.\n\nWe agree with the reviewer. As a baseline (and tangential) argument, we first would like to establish the biological implausibility of LSTMs and GRUs. This has been summarized in our reply to reviewer fBiB. The next step would naturally be to compare the capabilities of our proposed methods against gating mechanisms, which will entail future work. \n\nWe thank the reviewer again for their time."}]

Author rebuttal:
We thank all reviewers for their time and effort in reviewing our submission. We will address common issues here. 

Several reviewers have suggested moving information between the main text and supplementary. After also taking into account additional work requested by reviewers, we have arrived at the following outline:

**Current outline**  
Main text
1. RNNs as models of the brain
2. Gradient stability issues
3. RNN architecture
4. Review of model parameters
5. Difficulty in learning long-term dependencies
6. Introduction to CD, SCTT, DASC
7. Network stability
8. Network dynamics
9. Training steps required on cognitive tasks
10. Hyperparameter exploration
11. Strategic skip connections
12. Discussion  

Supplementary
1. Long-term dependency tasks in neuroscience
2. Review of skip connections
3. Exact model setup and task descriptions
4. Full training results
5. Rule reversal task

**Updated outline**  
Main text
1. RNNs as models of the brain
2. Gradient stability issues
3. Brief review of gating mechanisms and other solutions (new)
4. Brief review of skip connections (previously in supplementary)
5. RNN architecture and model setup (previously in supplementary)
6. Difficulty in learning long-term dependencies
7. Introduction to CD, SCTT, DASC and their dynamics (previously two separate sections)
8. Network stability (proof in supplementary)
9. Training steps and wall-clock time results (new)
10. Stating results from hyperparameter exploration
11. Rule reversal task with strategic skip connections (previously in supplementary)
12. Discussion  

Supplementary
1. Long-term dependency tasks in neuroscience
2. Extended review on biological plausibility of gating mechanisms (new)
3. Review of other existing methods (new)
4. Review of model parameters (previously in main text)
5. Task descriptions
6. Mathematical proof on stability results (new)
7. Full training results with hyperparameter exploration (previously in main text)

**Novelty**  
Reviewer T2Cs had expressed concerns about the novelty of this work. In short, we claim that SCTT confined to training only (but not regular SCTT) is an original contribution from this work. All aspects of DASC are also completely novel. All analyses and experiments that follow are also original to this work.

**Wall-clock training time**  
Reviewer nbN6 has raised issues about the validity of using the number of training iterations required as a metric for training speed. To further reinforce our results, we have provided the wall-clock training time and total floating point operations performed during training. These results can be found in Table R1 in the rebuttal PDF (note that only tasks requiring long-term dependencies are shown due to space constraints). Combined with the results from Table 1 of the main text, we make the following conclusions:

On two tasks (delayed decision making and multi-sensory delayed decision making), DASC is able to achieve a 2.0x and 2.7x speed up in terms of wall-clock training time respectively compared to direct training.
On the same tasks, DASC requires 2.3x and 3.1x less training iterations to successfully train the networks.
On two other tasks (delayed go and delayed anti-response), direct training by backpropagation was unsuccessful, but DASC (and SCTT) succeeded.

These results suggest that our proposed methods are able to train RNNs to perform tasks that are otherwise impossible by direct training, and also achieve up to 2.7x speed up in wall-clock time and 3.1x reduction in training iterations on simpler tasks. On the same note, we would also like to address the performance of our methods on the rest of the tasks. These tasks are easily trained by direct backpropagation through time (BPTT) and widely appear in neuroscience literature. Our focus of this work is on tasks where direct BPTT is challenging – these tasks have been extensively reviewed in Appendix A and represent future work.

**Clarification to Figure 3B-C**  
Several reviewers have expressed confusion regarding Figure 3B-C. We first clarify the hyperparameters of each of the three methods, shown in Figure R1 of the rebuttal PDF. 

Coarsened discretization (CD): initial skip length and step count
Skip connection through time (SCTT): skip length, initial skip ratio and step count
Dynamics-aligned skip connection (DASC): skip length, initial skip ratio and step count

The purpose of Figure 3 is to understand how RNNs trained with our methods behave when simulated normally without any modifications to the dynamics. The exact steps required to reproduce Figure 3B-C are shown in Figure R2 in the rebuttal PDF. We first train RNNs for a small number of iterations using either of the three methods. Throughout training, the skip ratio and skip length remain unchanged. The step count is therefore 0 by definition. We then compare the neural activity when simulating the network with and without the method and compute their mean squared error. We do the same for task performance and compute the absolute difference in performance. These metrics represent the degree in which the modified dynamics differ from the true dynamics. The time constant is 20x the base time discretization. We have found empirically that this is approximately the skip limit for CD to remain stable. The data points for CD therefore do not extend into the dark region of the plots in Figure 3B-C.

**Clarification to Figure 4B**  
Reviewer nbN6 has also expressed confusion about the axes of Figure 4B. We have clarified them by using clear color codes in Figure R3 of the rebuttal PDF with reference to Figure R1. The x-axes represent the initial skip ratios of SCTT and DASC, as well as the step counts of all three methods. As requested, we have also split Figure 4B into two rows, where the bottom (top) row represents results from training on tasks that do (not) require long-term dependencies.

We hope our responses to all reviewers help provide a clearer picture on the significance of our work.


Meta-review: Most reviewers are in favor of acceptance due to the clear exposition of the effect of discretizing continuous-time RNNs and the development of their DASC method for combining benefits of SCTT and CD as well as the demonstration of the usefulness of gradually reducing these components over training to allow a test-time continuous RNN with appropriate dynamics.

Two reviewers mentioned that a comparison to LSTMs and GRUs would be necessary/nice to have.   Reviewers are encouraged to add this in the final version.
Decision: Accept (poster)