Emergent mechanisms for long timescales depend on training curriculum and affect performance in memory tasks
Authors: Sina Khajehabdollahi
Roxana Zeraati
Emmanouil Giannakakis
Tim Jakob Schäfer
Georg Martius
Anna Levina
Venue: ICLR 2024 poster (2024)
Keywords: timescales
recurrent neural networks
memory tasks
curriculum learning
Primary area: applications to neuroscience & cognitive science
TL;DR: Emergent mechanisms for long timescales in memory tasks depend on training curriculum.

Abstract: Recurrent neural networks (RNNs) in the brain and \emph{in silico} excel at solving tasks with intricate temporal dependencies.
Long timescales required for solving such tasks can arise from properties of individual neurons (single-neuron timescale, $\tau$, e.g., membrane time constant in biological neurons) or recurrent interactions among them (network-mediated timescale, $\tau_\textrm{\small{net}}$). 
However, the contribution of each mechanism for optimally solving memory-dependent tasks remains poorly understood. Here, we train RNNs to solve $N$-parity and $N$-delayed match-to-sample tasks with increasing memory requirements controlled by $N$, by simultaneously optimizing recurrent weights and $\tau$s. We find that RNNs develop longer timescales with increasing $N$, but depending on the learning objective, they use different mechanisms. Two distinct curricula define learning objectives: sequential learning of a single-$N$ (single-head) or simultaneous learning of multiple $N$s (multi-head). Single-head networks increase their $\tau$ with $N$ and can solve large-$N$ tasks, but suffer from catastrophic forgetting. However, multi-head networks, which are explicitly required to hold multiple concurrent memories, keep $\tau$ constant and develop longer timescales through recurrent connectivity. We show that the multi-head curriculum increases training speed and stability to perturbations, and allows generalization to tasks beyond the training set.
This curriculum also significantly improves training GRUs and LSTMs for large-$N$ tasks. 
Our results suggest that adapting timescales to task requirements via recurrent interactions allows learning more complex objectives and improves the RNN's performance.

Published as a conference paper at ICLR 2024
EMERGENT MECHANISMS FOR LONG TIMESCALES DE- PEND ON TRAINING CURRICULUM AND AFFECT PER- FORMANCE IN MEMORY TASKS
Sina Khajehabdollahi 1.2.*, Roxana Zeraati 1.3,* Emmanouil Giannakakis 1.3
Tim J. Schäfer 1,3 Georg Martius 1.2 Anna Levina 1,3
1 University of Tübingen, Germany
2 Max Planck Institute for Intelligent Systems, Tübingen, Germany
3 Max Planck Institute for Biological Cybernetics, Tübingen, Germany
These authors contributed equally to this work.
( i missttame...... astramme Buni -tuubinngnn.cc
ABSTRACT
Recurrent neural networks (RNNs) in the brain and in silico excel at solving tasks with intricate temporal dependencies. Long timescales required for solving such tasks can arise from properties of individual neurons (single-neuron timescale, T, e.g., membrane time constant in biological neurons) or recurrent interactions among them (network-mediane each mechanism for optimally solving memory-dependent tasks remains poorly understood. Here, train RNNs to solve N-parity and N-delayed match-to- sample tasks with increasing memory requirements controlled by N, by simulta- neously optimizing recurrent weights and rS. We find that RNNs develop longer timescales with increasing N, but depending on the learning objective, they use different mechanisms. Two distinct curricula define learning objectives: sequen- tial learning of single-N (single-head) or simultaneous learning of multiple Ns (multi-head). Single-head networks increase their with and can solve large- N tasks, but suffer from catastrophic forgetting. However, multi-head networks, which are explicitly required to hold multiple concurrent memories, keep T con- stant and develop longer timescales through recurrent connectivity. We show that the multi-head curriculum increases training speed and stability to perturbations, and allows generalization tasks beyond the training set. This curriculum also significantly improves training GRUs and LSTMs for large-N tasks. Our results suggest that adapting timescales to task requirements via recurrent interactions allows learning more complex objectives and improves the RNN's performance.
timescale, Tnet). However, the contribution of
INTRODUCTION
The interaction of living organisms with their environment requires the concurrent processing of signals over wide range of timescales, from short timescales coding sensory stimuli (Bathellier et al. 20088 Panzeri et al., 2010 Safavi et al. 2023) to longer timescales of cognitive processes like working memory (oondes et 2008). The diverse timescales of these tasks are reflected in the dynamics of the neural populations performing the corresponding computations in the brain (Mur-) ray et al. 2014] Cavanagh al.) 2020; Gao al. 2020 Zeraati al.) 2022).
the same time, artificial neural networks performing memory-demandingg tasks (speech (Graves et al.) 2013), hand- writing (Graves] (203),, sketch (Ha & Fekk 2018). language (Bowman et al. 2015). time series prediction (Chung al. 2014, Torres al. (021). music composition (Boulanger- Lorolanngr-Lewwadowwslll ctt al., 2012)) need to process the temporal dependency of sequential data over variable timescales. Recurrent neural networks (RNNs) (Elman 1990, Hochreiter & Schmidhuber. 1997 Lipton al.)
2015 Yu et al. 2019) have been introduced as a tool that can learn such temporal dependencies using back-propagation through time.
In biological networks, diverse neural timescales emerge via a variety of interacting mechanisms. Timescales of individual neurons in the absence of recurrent interactions determined by cellu-

---

Published as a conference paper at ICLR 2024
a
r
Activity AC
Output
time
Single-neuron
Head 3
Network-mediated Tee
Input
log (AC)
10011001100 10100
Head 4
Time lag
Figure 1: Schematics network structure and timescales. a. An outline of the network. A binary sequence is given imput leaky RNN, with each neuron's 's intrinsic timescale being trainable parameter т. The illustration shows the N -ppaiity task with readout heads for and 4. b. An illustration of the manifestation of different timescales (single-neuron and network-mediated) on the autocorrelation (AC) of a network neuron (see also Fig. S2).
lar and synaptic processes (e.g., membrane time constant) that vary across brain areas and neuron types (Gjorgjieva al. 2016 Duarte al. 2017). However, recurrent interactions also shape neural dynamics introducing network-mediated timescales. The strength (Ostojic) 2014: Chaudhuri et al,,
2015 van Meegen & van Albada 202I) and topology (Litwin-Kumin Kumar Doiron 2012] Chaudhuri
et al. 2014| Zeraati et al. 22023] Shi et al. (002)) of recurrent connections give rise to network-
mediated timescales that can be much longer than single-neuron timescales.
Heterogeneous and tunable single-neuron timescales have been proposed mechanism to adapt the timescale of RNN dynamics to task requirements and improve their performance Perez-Niever
et al.
2021 Tallec & Ollivier 2018 Quax et al., 2020; Yin et al.) 22020: Fang et al. 2021] Smith
al. 2023b Jain et al.) 2020). these studies, time constants of individual neurons are trained together with network connectivity. For tasks with long temporal dependencies, the distribution of trained timescales becomes heterogeneous according to the task's memory requirements (Perez- Nieves et al. (021). Explicit training of single-neuron timescales improves network performance in benchmark RNN tasks in rate (Talee & Ollivier! 2018, Quax et al. 2020) and spiking (Yin et al.) 2020 Fang et al. 2021 Perez-Nieves et al. 2021 networks and leads to greater robustness (Perez- Nieves et al., 2021) and adaptability to novel stimuli (Smith et al.) 2023b). While these studies propose the adaptability of single-neuron timescales mechanism for solving time-dependent tasks, the exact contribution of single-neuron and network-mediane timescales in solving tasks is unknown.
Here, we study how single-neuron and network-mediated timescales shape the dynamics and perfor- mance of RNNs trained on long-memory tasks. We show that the contribution of each mechanism in solving such tasks largely depends on the learning objective defined by the curriculum. Challenging common beliefs in the field, we identify settings where trainable single-neuron neuron timescales offer no advantage in solving temporal tasks. Instead, adapting RNNs timescales using network-mediated mechanisms improves training speed, stability and generalizability.
MODEL
We approximate the effect of the membrane timescale of biological neurons by equipping each RNN-neuron with trainable leak parameter т, defining the single-neuron timescale (Fig. [T). The activity of each neuron evolves over discrete time steps governed by:
rt())= {
Al ra(t - At)
AL
+
w ry(t - At) + W. S(t) + R +b!
Ti
where Fa is the leaky ReLU function with negative slope a, given by:
[0] = {
2 0.
(2)
a a.
2

---

Published as a conference paper at ICLR 2024
a
Input
Output
b
N-parity: 9 8(t5p1 t.r mod
Single head
Multi head
N-parity
100
Active heads,
time
or N-DMS
D 0
N-DMS:s; 9pN+1
Curriculum step
Curriculum step
Figure 2: Schematic description of the tasks and curricula a. An outline of the network and tasks. In both tasks, the network receives binary input sequence, one bit each time step. b. In the single- head curriculum, only one read-out head is trained at each curriculum step, while the multi-head curriculum, new read-out head is added each step without removing the older heads.
0.1. We obtain similar results using different type or location of nonlinearity (Appendix [A). l " 6", and W are the recurrent and input weights and biases, respectively, S is the binary input given the network at each time step, and T; 2 1 the trainable timescale of the neuron. Unless otherwise stated, the time step is At = 1 (other At discussed in Appendix B Each RNN has 500 neurons. When r 1, a neuron becomes memory-less (in isolation) as its current state does not directly depend on its past activity, i... memory can only be stored at the network level via interactions. contrast, for T > 1, the neuron's activity depends on its past activity, and the dependency increases with In the limit of 7 co, the neuron's 's activity is constant, and the input has no effect.
For all networks, we
The dynamics of each neuron can be characterized by two distinct timescales: (i) single neuron timescale т, (ii) network mediated timescale Thet T gives the intrinsic timescale of a neuron in the absence of any network interaction, while Thet is shaped by the combination of and the learned connectivity and represents the effective timescale of the neuron's 's activity within the network. Thet is generally function of and recurrent weights: Tnet f(T, W R (Ostojic 2014 Chaudhuri et al. 2014 Shi et al. 2023) and Tnet 2 (Shi et al. 2023). For networks with linear dynamics, Thet can be directly estimated from the eigenvalues of the connectivity matrix normalized by (Chaudhuri al., 2014). For nearest- weerees-neiihhbr connectivity or mean-field dynamics, it is also possible to derive Thet analytically for nonlinear networks. However, general analytical solution does not exist, Instead, Thet can be effectively estimated from the decay rate of the autocorrelation (AC) function. The AC defined as the correlation coefficient between the time series and its copy, shifted by time f', called the time lag. For the neuron's activity, it can be computed as
AC,(!") (T - t7
Σ (r((1))))) (r(( 1') -
where i and o? the sample mean and variance of (4). To estimate Tnet, we drive the network by uncorrelated binary inputs sampled from a Bernoulli distribution.
The AC of neuron's activity, defined by Eq. 1 can be approximated by two distinct timescales which appear as two slopes in logarithmic-lincar coordinates (Fig. [bb) (Shi et al. (023). The steep initial slope indicates 7, and the shallower slope indicates Tnet- In the same way, we characterize the timescale of collective network dynamics by computing population activity (summed activity of all neurons within network) timescale Tpop. which reflects the timescale of network dynamics as a whole. To avoid AC bias our estimates (Zeraati et al., (202). we use long simulations (7 105 time steps). We simulate each network for 10 trials (i.e. 10 distinct realizations of inputs) and compute the average AC of each neuron across trials. To estimate Tnet. we fit the average AC with single- (Tnet 7) and with double-exponential (Tnet 7) decay function using the nonlinear least-squares method. Then, we use the Akaike Information Criterion (AIC) (Akaike 1974 to select the best-fitting model. For most neurons (above 95%), Bayesian information criterion (BIC) selects the same model (Fig. ST) and previous work (Pasula 2023) indicates that AIC provides similar results as the sum of three information criteria (AIC, BIC and Hannan- Quinn- information criteria).
3

---

Published as a conference paper at ICLR 2024
a
curiculum
100
100
Multi-has head N-DMS
10
Maximal trained N
Accuracy
80
20
15
75
N-Parity
Multi-
50
Single-head N-DMS
25
Single-head
N-Parity
0
20
40
50
100
50 100
Epochs
Epoche
Maximal trained
Figure 3: Training performance depends on the curriculum. a. Accuracy training the networks (N (N-parity task) without a curriculum increases slowly, especially when N > 10. For each N, models are independently trained for 50 epochs or until reaching > 98% accuracy. b. Multi-head (dashed) trained networks time (colors in c). c. The maximum trained for each task/curriculum at the end of training (1000 epochs or solving N 101, whichever comes first). Gray lines mean value across 4 networks.
solving larger Ns than single-head (solid) within the same training
When the double-cxponential model is selected, the slowest of two timescales indicates Thet. For most fits, we obtain a large coefficient of determination, confirming good quality of fit (Fig. S2)
3 SETUP
3.1 TASKS
In both tasks (Fig.2p) a binary sequence is given as the input, one bit at each time step. We train the networks sequences with lengths uniformly chosen from the interval L {N + 2, 2,N). N N-delayed match-to-sample (N-DMS): The network outputs 1 or 0 to indicate whether the digit presented at current time matches the digit presented time / N 1. To update the output every time step, the network needs to store the values and order of the last N digits in memory. N N-parity: The network outputs the binary sum (XOR) of the last N digits. N-parity has a similar working memory component as N-DMS, but requires additional computations (binary sum).
3.2 TRAINING
We train single-neuron timescales 7 (II...., 7500). RR W b², and a linear readout layer via back-propagnne proppaaaatnnionn through time using stochastic gradient descent optimizer with Nesterov mo- mentum and a cross-entropy loss. Each RNN trained on a single Nvidia GeForce 2080ti for 1000 epochs, 3 days, until the N 101 task is solved, whichever comes first. RNNs trained without any regularization. Including L2 regularization achieves comparable performance.
Starting with N 2, we train the network reach accuracy of 98%. We then the trained network parameters to initialize the next network that we train for N + 1 (Fig. (p). Multi-head: As with the single- networks, we begin with network solving task for N 2, but once a threshold accuracy of 98% is reached, new readout head is added for solving the same task for N 1, preserving the original readout heads. At each curriculum step, all readout heads are trained simultaneously (the loss is the sum of all readout heads' losses) so that the network does not forget how solve the task for smaller Ns (Fig. 2b).
Single-head:
RESULTS
4.1 PERFORMANCE UNDER DIFFERENT CURRICULA
Necessity of curriculum: Our objective is to learn the largest possible N in each task. First, test whether a good performance can achieved for high N in either task without any curricula. We find that for both tasks (see Appendix B for N-DMS results), networks struggle reach high accuracy for N > 10 (Fig. B. Fig. S13). However, using either curriculum significantly boosts the network's capacity to learn tasks with larger N (Fig. 3)).
4

---

Published as a conference paper at ICLR 2024
a
Single-head
N-parity N-DMS
Multi-head --ppaitty
Single-head
Multi-head
N-DDD
Max trained N
100
2
35
(san(T)
STD(2)
1.0
30
80
0.5
25
Trained T=
2 T=
Trained
0
100
100
t
T=
t=
N
Figure 4: Importance of single-neuron neuron timescales for different curricula. a. The maximum N solved in the N-bit parity task after 1000 epochs (reaching an accuracy of 98%). X-label indicates training constraints: - 1, 2 or 3 fixed with only weights being trained, "Trained" allows training of r. In the single-head curriculum, models rely on training v, whereas the multi-head curriculum, 7 fixed is good training Horizontal bars mean. b. The mean and standard deviation (STD) of the trained TS increase with N in single-head networks. In contrast, in multi-head networks, the mean decreases towards 1, and the STD remains largely constant. The mean and STD are computed across neurons within each network (up to the maximum N shared between all trained networks). Shading variability across 4 trained networks.
Comparison of curricula: Task performance differs significantly between curricula. The single- head networks can reliably reach N R 35 for the N-parity and N z 90 for the N-DMS task. The difference in performance between the two tasks is expected because the N-DMS task is much easier than the N-parity task. However, the multi-head networks can reliably reach N 100 for both tasks and require fewer training epochs to reach 98% accuracy for each N (Fig. 3p, c). Networks that are trained using an intermediate curriculum between the two extremes of single- and multi-head (i.e., solving simultaneously H < N tasks for N. - 1I.... - H + 11 with gradually increasing Ns) exhibit an intermediate performance (Appendix(F) F... Overall, the multi-head networks outperform the single-head ones in terms of performance (maximum N reached) and the required training time. Moreover, the multi-head curriculum significantly improves the training of other recurrent architectures such as GRU and LSTM to perform large-N tasks (Appendix G.. suggesting that the multi-objective curriculum can generally improve learning long-memory tasks.
The superior performance of the multi-head networks may be counter-intuitive since they solve the task for every N S m at the m-th step of the curriculum, whereas the single-head networks only solve it for exactly N
However, we find that the single-head networks suffer from catastrophic forgetting: a network trained for larger N cannot perform the task for smaller Ns was trained for, even after retraining the readout weights (Appendix H.. These results suggest that explicit prevention of catastrophic forgetting by learning auxiliary tasks (i.e. tasks with smaller than objective, N < m) facilitates learning large Ns. Interestingly, training directly on a multi-N task without using an explicit curriculum results in the emergence of the multi-head curriculum: networks learn first solve small-N tasks and then large-N tasks (Appendix Tb. supporting the use of the multi-head step-wise strategy.
Necessity of training r: We examine the impact of training single- neuron timescales on training performance. We compare the training performance of networks with a fixed E {1, 2,3) shared across all neurons versus networks with trainable timescales. In the single-head curriculum, the training performance with fixed is significantly worse than when we train r (Fig. Ta). On the other hand, In multi-head networks, training performance is the same for fixed T = 1 and trainable 7 cases, but steeply declines for fixed > 2 (Fig. 4p). See (Fig. STO) for N-DMS results. These results indicate that single- head networks rely on J for solving the task, whereas multi-hean networks only to track the timescale of updating the input and rely other mechanisms to hold the memory.
4.2 MECHANISMS UNDERLYING LONG TIMESCALES
To uncover the mechanisms underlying the difference between the two curricula, we study how т, Thet, Tpop and recurrent weights change with increasing task difficulty N. One can expect that as the timescale of the task (mediated by N) increases, neurons would develop longer timescales to integrate the relevant information. Such long timescales can arise either by directly modulating T for each neuron or through recurrent interactions between neurons reflected in Teet and Tpop-
5

---

Published as a conference paper at ICLR 2024
d
5
=
Single-head
Multi-head
109
40
N 15
-V 36
N-DMS
15
Mean(Tnel
N-parity
10-1
20
10
5
10-2
10
20
25
50
75
50 -N-DMS -N-parity
100
Time lag
N -=5
Density
0.50
Single-head Multi-head
30
30
0.25
STD()
20
20
0.00
10
15
20
N = 30
Dansity
0.25
10
0.00
0
0
10
15
20
25
50
75
50
100
Tnat
N
Figure 5: The emergence of network-mediater timescales depends the curriculum. Example average ACs of all the neurons within a single-head network, N N-parity task. The ACs of individ- ual neurons' activity decay slower with increasing N. b. Distributions of the network-mediated timescales Thet for single and multi-head networks solving N N-parity task for = and N = 30. The distribution becomes broader for higher N. c, d. The mean and STD of the network-mediated timescale Thet increase with N in both tasks. The mean and STD are computed across neurons within each network. Shades variability across 4 trained networks.
Dependence o on N: The two curricula adjust their to N in distinct ways: single-head networks increase their 7 with N, but multi-head networks prefer - 1 (Fig. 4p). For the single-head curriculum, the mean and variance of F increase with N, suggesting that not only 7s become longer as the memory requirement grows, but they also become more heterogeneous. We obtain similar results for networks trained without curriculum (Fig. (S3). On the contrary. in multi-head networks, the average decreases with N, approaching that multi-head networks with fixed T performed as well as networks with trained T.
1. The trend of r 1 is consistent with the fact
Dependence of net and ppp on N: Network-mediated timescales Thet and Tpop generally increase with in both curricula (Fig. 5 Fig. S4). Thet reflects the contribution
recurrent weights in dynamics of individual neurons. In single-head networks, the mean and variance of Tnet follow similar trend as r (p). suggesting that changes in Tnet can arise from changes in т. The mean and variance of Tnet in multi-head networks increase with N up to some intermediate N. but the pace of increase reduces gradually and saturates for very large Ns (Fig. 5 top). Given the small in multi-head networks, long Tnet only arise from recurrent interactions between neurons. Tpop is the timescale of collective network dynamics (sum of all neurons' activations) and arises from interactions between neurons within the whole network. Tpop exhibits a clear increase with N for both tasks and curricula with comparable values (Fig. (54). These results indicate that in both curricula, collective network dynamics become slower with increasing N, but due differences in the two curricula employ distinct mechanisms to achieve this.
Dependence of connectivity on N: Multi-head networks have, on average, almost the same total incoming positive and negative weights (with a slight tendency towards larger total negative weights as N increases), leading to relatively balanced dynamics (Fig. (F)..6a). On the other hand, hendle-head networks have stronger bias towards more inhibition (negative weights) N increases. The strong negative weights in single-head networks are required to create stable dynamics in the presence of long single-neuron timescales (Appendix D).
Dimensionality of dynamics: of principal components that explain 90% of the variance) increases almost linearly with N in the N N-parity task but sub-linearly in the N-DMS task, using both curricula, reflecting distinct com- putational requirements for each task (Fig. 6b. $20 Appendix K. Since computations should be
The dimensionality of population activity (measured as the number

---

Published as a conference paper at ICLR 2024
Single-head
Multi-head
N-parity
N-DMS
b
N-parity
N-DMS
Average incoming weight Wii)
Dimensionality
40
0
40
-5-
20
20
-2
0
10
20
30
0 50
100
100
N
N
N
N
Figure 6: Learned recurrent connectivity and dimensionality of population activity. a. The average incoming weight neuron remains close zero for multi-head networks but becomes strongly negative as N increases in single-head networks (example RNN). Error bars - + STD. b. The dimensionality of the activity increases linearly with N in N-parity and sub-linearly in N-DMS task. Dashed lines linear fit computed for all Ns (N- (N-parity task) and independently for N E [0, 20] (gray line- extension for visual guidance) and N E [20, 100] (N-DMS task). Blue line- - 20.
performed at every time step, the increase in dimensionality may be required map different input patterns (that grow with N) to the same outputs.
Our findings suggest that both curricula give rise to networks with slower and higher dimensional collective dynamics with increasing N, but via distinct mechanisms. In single-head networks, single-neuron properties r play an important role in creating slow dynamics, which are then sta- bilized by stronger inhibition in the network. However, in multi-head networks, the slow dynamics should arise from recurrent network interactions. The significant difference in performance of the two curricula suggests that the second mechanism is more effective in solving the task.
4.3 IMPACT OF DIFFERENT CURRICULA ON NETWORKS ROBUSTNESS
To compare the robustness and retraining capability between the two curricula, we investigate changes in network accuracy resulting from ablations, perturbations, and retraining networks on unseen N. We measure the effects on network performance using a relative accuracy metric with respect to the originally trained network, defined as aCCzel == (acc 0.5)/ 0.5), where aCCbase represents the accuracy of the network before any perturbations or ablations, acc is the measured accuracy after the intervention, and 0.5 chance level used for the normalization. acC,el = 1, the intervention did not change the accuracy; when acCret R 0, the intervention reduced the accuracy to chance level. All accuracies are evaluated on the maximal trained N.
To examine the relative impact of neurons with different trained on network perfor- mance, we ablate individual neurons based on their T and measure the performance without retrain- ing (Appendix D Specifically, we compare the effect of ablating the 20 longest (4% of the network) and 20 shortest timescale neurons from the network (Fig. b).. For small N and both curricula, ab- lating individual neurons has only minimal effect (less than 1%) on accuracy. However, for larger N, we observe a considerable difference in the importance of neurons. Single-head networks rely strongly long-timescale neurons (Fig. 7a), such that ablating them reduces the performance much more than for short-timescale neurons. In contrast, multi-head networks exhibit greater robustness against ablation, and their accuracy more affected when short-timescale neurons (i.e., neurons with r = 1) are ablated (Fig. (b). Note that in single-head networks, the average of the 20 longest single-neuron timescales is 2.7 times longer than in multi-head networks.
Ablation:
Perturbation: We perturb W and J with strength as Ww et al. 2020)
W R +
Ew
II.
F=
& ||,,
|/rl//
Ew (((0I"xx)) & ~((,T").
(4)
||. || represents Frobenius norm and] |- the absolute value. is perturbed positively to avoid < 1. Multi-head networks are more robust to perturbations (Fig. dd.d). The robustness to changes in WR is noteworthy since these networks rely connectivity to mediate long timescales.
Retraining: We evaluate the performance of networks that solve N = curriculum (without training intermediate Ns) for an arbitrary higher N, after 20 epochs. Multi-
16, when retrained without
7

---

Published as a conference paper at ICLR 2024
Single-head
Multi-head
Multi-head
Relative accuracy
1.0
Short
1.0
1.0
1.0
1.0
Single-head
Long
Chance level
0.8
0.8
0.5
0.5
0.5
0.0
0.0
0.0
0.6
0.6
1.0
5.28
1.0
1.95
10
10
10
10
16
26
36
Average of
Average of ablated neurons
Perturbation of Wa
Perturbation E of r N of single-head retraining
ablated neurons
Figure 7: Multi-head networks are more robust to ablation and perturbation, and better retrainable. (a, by Ablating long-timescale neurons largely decreases the performance of single-head networks (a), while multi-head networks (b) are more affected by the ablation of short-timescale neurons (N = 30). (c, d) Multi-head networks are more robust against perturbations of recurrent connec- tivity WR and than single-head networks (note the log-scale x-axis, N 30). (e) Multi-head networks retrain faster: They achieve higher relative accuracy when retrained for 20 epochs without
curriculum for higher N. Networks trained solve N 16 are retrained for 20 epochs solve higher N without a curriculum to compare re-trainability. Bars mean; dots and lines 4 networks for each curriculum; error bars and shades STD.
head networks show superior retraining ability compared single-head networks for at least 10 N beyond N = 16 (Fig. (p). This finding suggests that multi-head networks are better at learning the underlying task and adjust faster to a new, larger N even when skipping intermediate Ns.
RELATION TO NEUROSCIENCE
Continuous- time setting: So far, we described the tasks and network dynamics in discrete time with time-step At However, to make the connection to more realistic settings, such as neuroscience, we need to describe the dynamics in continuous time. For this purpose, we consider that each input digit is presented the network for certain time T. In neuroscience experiments (e.g.. DMS task), T is often set to 250-500 ms (Meyer et al. 2011 Qi et al. 2011: Kim & Sejnowski 2021).
First, we show that for fixed T, the discretization time step At does not affect the networks' performance and dynamics. We train the networks with each curriculum using variety of At (Appendix B). these networks, the performance on the test data is comparable for different At, even for values much smaller than what was included during the training, mimicking continuous- time dynamics (Fig. SS c-d). Moreover, similar to our previous results, we find that single-head networks increase their J with N, while multi- heat networks try to reach - T (Fig. [S3]a,b).
Next, we test how changes in input presentation time affect the dynamics. We set T = kat, where At 1. We find that for all k single-head networks increase with N, whereas multi-head networks' r tends to k thus matching the timescales of the input changes (Appendix C
Timescales and learning in the biological neural networks: Our findings suggest that networks that are required to solve tasks with larger memory requirements should develop longer timescales. This largely agrees with findings in the brain: higher cortical areas that are involved in cognitive processes with larger memory requirements (e.g., working memory, evidence accumulation) have longer timescales than sensory areas (Murray et al., 2014). Moreover, we show that developing longer network timescales via changes in network connectivity is a superior solution (in terms of performance and stability) than using longer single-neuron timescales. This is consistent with find- ings that neural timescales in primate visual cortex adapt to task demands via recurrent network interactions rather than biophysical time constants Zeraati et al. 2023). Moreover, this result aligns with the learning strategy of biological neural networks, which primarily relies on changes in synap- tie strengths rather than modifying the biophysical time constants of individual neurons. While such changes do happen in biology (via protein turnover (Sun & Schuman 2022), calcium currents (Tiganj et al. 2013). and other mechanisms ), synaptic strength modification is overwhelmingly the mechanism by which biological networks learn.
8

---

Published as a conference paper at ICLR 2024
6 RELATED WORK
Previous works independently investigated the role of neuronal and network-mediated timescales in solving memory tasks and proposed inconsistent solutions. Studies focusing on neuronal aspect suggested heterogeneous and adaptable neuronal properties (e.g.. membrane time constant) as an optimal mechanism (Perez-Nieves et al. 202T] Mahto al. 2021 Smith et al., 2023a Quax al., 2020). At the same time, other studies presented that network-mediated mechanisms like balanced dynamics (Lim & Goldman 2013), strong inhibition (Kim & Sejnowski (02T) or homeostatic plas- ticity (Cramer et al. [2200, 2023) can create timescales required for memory tasks. For single neuron modeled with multiple memory units, long timescales were shown to be instrumental in solving memory tasks (Spieler et al. 2023). Here, we explicitly compare these mechanisms and show that while both can be useful for learning long-memon tasks, applying network-mediater mechanisms leads to faster training and more robust solutions.
We find that the difference between mechanisms is revealed mainly in the context of distinct learn- ing objectives defined by curricula. This is an important distinction with previous work, since the role of timescales has been often studied when RNNs solve a single task (e.g.. single-head DMS), without considering learning dynamics or the potential for catastrophic forgetting. We relate the mechanisms of task-dependent timescale with the learning dynamics of RNNs across curricula. The use of curricula in our study is inspired by previous work suggesting curriculum learning as fitness landscape-smoothing mechanism that can enable the gradual learning of highly complex tasks (EI-) man 1993] Bengio al., [2099] Krueger& Dayan 2009) and be used to uncover distinct learning mechanisms (Kepple et al. 2022 Dekker et al. 2022). Here, we extend these findings by demon- strating how different curricula can push networks towards adopting different strategies to develop slow collective dynamics required for solving long-memon tasks.
DISCUSSION
We find that to solve long-memory tasks, RNNs develop high-dimensional activity with slow timescales via two distinct combinations of connectivity and single-neuron timescales. While single- head networks crucially rely on the long single-neuron timescales to perform the task, melti-head networks prefer constant single-neuron timescale and solve the task relying only on the long timescales emerging from recurrent interactions. We show that developing long timescales via re- current interactions instead of single-ncuron properties is optimal for learning memory tasks and leads to more stable and robust solutions, which can be a beneficial strategy for brain computations.
Our findings suggest that training networks on sets of related memory tasks instead of a single task improves performance and robustness. By progressively shaping the loss function with a curriculum to include performance evaluations sub-tasks that are known to correlate with the desired task, can smooth the loss landscape of our network to allow training for difficult tasks that were previously unsolvable. In this way, choosing an appropriate curriculum can act powerful regularization.
Limitations: Our study considers two relatively simple tasks with explicitly controllable mem- ory requirements. In follow-up studies, it would be important to test our observations in more sophisticated tasks and investigate whether our results apply to other architectures and optimizers. Additionally, our approach suitable only for a set of tasks with controllably increasing memory requirements, where the different versions of the same task can be simultaneously performed on the same data imulti-head training). This is a relatively strong constraint, and future research expand- ing our findings could focus on generalizing the multi-head curriculum for training more realistic tasks. Time series reconstruction is a potential task that can be used to uncover generative dynamical systems from data (Durstewitz et al. 2023). We proposed a potential experiment in Appendix D
Our current model is a crude approximation of biological neural networks, and more plausible archi- tectures (spiking models, distinct neuron types) could be studied. Finally, biological neural networks can produce long timescales via various other mechanisms we did not consider here (short-term plasticity (Hu et al.) 2021), adaptation (Salaj et al., 2202 Beiran & Ostojic 2019), synaptic delays, etc). A follow-up study could investigate whether our findings extend to more plausible networks incorporating such additional mechanisms.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by Sofja Kovalevskaja Award from the Alexander von Humboldt Founda- tion, endowed by the Federal Ministry of Education and Research (SK, RZ, EG, AL), the Deutsche Forschungsgemeinschaatt (DFG, German Research Foundation) under Germany's 's Excellence Strat- EXC number 2064/1 Project number 390727645 (RZ, EG), and Else Kröner Medical Scientist Kolleg "n Artificial Intelligence for Clinical Brain Research" (TJS). We acknowledge the support from the BMBF through the Tübingen AI Center (FKZ: 01IS (8039B) International Max Planck Research School for the Mechanisms of Mental Function and Dysfunction CMMPRS-MMFD and International Max Planck Research School for Intelligent Systems (IMPRS-IS). We thank Victor Buendía for valuable discussions.
REFERENCES
H. Akaike. new look at the statistical model identification. IEEE Transactions on Automatic Con- trol, 19(6):11-7733, 1974. doi: 10.119//AAC.194.1110077 URLhttps:: Stttps::/iieexppoxse
Brice Bathellier, Derek L. Buhl, Riccardo Accolla, and Alan Carleton. Dynamic ensemble
odor coding in the mammalian olfactory bulb: Sensory information at different timescales.
Newron, 57(4):586-598, 2008. ISSN 0896-6273. doi: https//dbo..ogg/0..111......
2008.02.011. URL https:///wwwtt /wwp://wwww/... www.sccieccciccccc www.neeiiree.....o rect. com /tcm://ciennce/attcle///..
$0896627308001320
Manuel Beiran and Srdjan Ostojic. Contrasting the effects of adaptation and synaptic fil- tering on the timescales of dynamics recurrent networks. PLOS Computational Bi-
ology, 15(3):e1008893, March 2019. ISSN 1553-7358. doi: 10. 10.137//joumal.pehi 1006893. URL Sttebi.1006890 1006893
plos. https///owoocooooo////aaeeeeeeee
1371/journal pcwipcc.i.......
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference Machine Learning, ICML '09, pp. 41-48, New York, NY, USA, 2009, Association for Computing Machinery. ISBN 9781605585161. doi: 1551447/553374.1553999. URL httpp:://ccioog/// 1145/ 1553374.155338
Nicolas Doullangr-Lewanddoee Yoshua Bengio, and Pascal Vincent. Modeling temporal depen- dencies in high-dimensionn sequences: Application to polyphonic music generation and tran- scription. arXiv preprint arXiv: 1206.6/2066639. 2012.
6392
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Ben- gio. Generating sentences from a continuous space. arXiv preprint arXiv: 1.10::34411. 2015. URL
https: wrg.o/g////o..oo abs http://b//11111 06349
Sean E. Cavanagh, Laurence T. Hunt, and Steven W. Kennerley. A Diversity of Intrinsic Timescales Underlie Neural Computations. Frontiers in Neural Circuits, 14, 2020. ISSN 1662-5110. doi: 33388//fcii.222261666//// 10. (3.9339/FF 1/389/fncirr fncir. 2020 Neural Cesural_Crrciittt Circuits Publisher: Frontiers.
URL https: wwwp///www/////m///////////ww frontiersin htt.:/errrsi..oorrrrrteeeeeeeeee
Frontiers _euuurrrrr_____uaaal_____e
Rishidev Chaudhuri, Alberto Bernacchia, and Xiao-Jing Wang. A diversity of localized timescales in network activity. eLife, 3, January 2014. ISSN 2050-084X. doi: 10.7554/eLife.0122 012.99 URL
https:// ncbi. mmm. nih. http:h/h../////aaaataaa/////////
Rishidev Chaudhuri, Kenneth Knoblauch, Marie-Alice Gariel, Henry Kennedy, and Xiao-Jing Wang. A Large-Scale Circuit Mechanism for Hierarchical Dynamical Processing in the Pri-
mate Cortex. Neuron, 88(2):419 431, October 2015. ISSN 0896-6273. doi: 10.1016/j. com/ htt.:/et..coooceenceeeeeeeeeeeee wwtaacc/eeenneeeeeee/e
neuron. neuron.2015.099000 URL http: /tttp://www http://wwwwwwwww..e...tttt
e117508966273315007777
10

---

Published as a conference paper at ICLR 2024
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014. URL httpp::/aaxxii...... wttp.//pp/pbbb/1...35555555
Benjamin Cramer, David Stöckel, Markus Kreft, Michael Wibral, Johannes Schemmel, Karlheinz Meier, and Viola Priesemann. Control of criticality and computation in spiking neuromorphic networks with plasticity. Nature communications, 11(1):2853, 2020. URL https: //www
nature. com caoom/artiilee wwwaaaatiiesssee s414s7///4445--/// 020 16548-3
Benjamin Cramer, Markus Kreft, Sebastian Billaudelle, Vitali Karasenko, Aron Leibfried, Eric Müller, Philipp Spilger, Johannes Weis, Johannes Schemmel, Miguel A Muñoz, et al. Auto- correlations from emergent bistability in homeostatic spiking neural networks neuromorphic hardware. Physical Review Research, 5(3):033335, 2023. URL httppttts///:orraa aps.. aps.
Ronald B Dekker, Fabian Otto, and Christopher Summerfield. Curriculum learning for hu- man compositional generalization. Proceedings of the National Academy of Sciences,
119(41)xc2205822119 2022. 22C5582119
URL https: Detts::/www.pneet org/ (ttp//oo/Iin 10.3///3./....1
Renato Duarte, Alexander Seeholzer, Karl Zilles, and Abigail Morrison. Synaptic patterning and the timescales of cortical dynamics. Current Opinion in Neurobiology, 43:156-165, April 2017.
ISSN 0959-4388. doi: (0.1016/f.comb.2017.0222222 URL https: /wwww scinsseieeeccdire......
com/ Cttp://ccooo//rrrceee///////3330
Daniel Durstewitz, Georgia Koppe, and Max Ingo Thurm. Reconstructing computational system dynamics from neural data with recurrent neural networks. Nature Reviews Neuroscience, pp-
1-18, 2023. URL http:::/wwww natura..ur......... com http:aatticlesseeemeee /sleee///41533 023 00740-7
Jeffrey L Elman. Finding structure in time, Cogmitive science, 14(2): 1990. URL https:
wi 1ey. www.coo//oooooo wwwoo///oiaaae aetoo//bs/100/ 12011 http:/aabb/1..1111115555501 http:////./////../...//////////.
Jeffrey L Elman. Learning and development in neural networks: The importance of starting small.
Cogmition, 48(1):71-99, 1993, URL httpsss/www.seeeeeeeedeee http://dttrrtttcocccooooooooooot scienc/
aartcleeppi/0010oe 00302/////00007779351
Wei Fang, Zhaofei Yu, Yanqi Chen, Timothee Masquelier, Tiejun Huang, and Yonghong Tian. Incor- porating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks, August 2021. URL /tctt://arxivv
abs/2000 008000057788 05785 Sit:20007.05788 [cs].
Richard Gao, Ruud L van den Brink, Thomas Pfeffer, and Bradley Voytek. Neuronal timescales are functionally dynamic and shaped by cortical microarchitecttree eLife, 9:e61277, November 2020. ISSN 2050-084X. doi: 10.7554/el.ffe.611 61277. URL htttts/tt::///tdt orge 10. 61277
/0t.r////.....1/11... 7554/el.
Publisher: eLife Sciences Publications, Ltd.
Julijana Gjorgjieva, Guillaume Drion, and Eve Marder. Computational implications of biophys- ical diversity and multiple timescales in neurons and synapses for circuit performance. Cur-
rent Opinion in Neurobiology, 37:44-52, April 2016. ISSN 0959-4388. doi: 10.1016/j.comb.
2015.12.008. URL https:/ htt::://wwwwwse. www.sccieced... ieceeeeiieeeec.eeo com wtt.:////cccieeeeaartee////////a
809594388 500 865
Alex Graves. Generating sequences with recurrent neural networks.
arXiv preprint
0rXiv:1308.0833 2013. URLtt:ts: org abs /1////30.... 0850
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recur- rent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6645-6649. leee, 2013. htl.tttps: http:///xxxvv......e http:s///aaaav.orr/////3333..3
David Ha and Douglas Eck. A neural representation of sketch drawings. In International Confer- ence on Learning Representations, 2018. URL
net/ ffrttriiffrrdd
Hy6GHpkCи
11

---

Published as a conference paper at ICLR 2024
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997. ottpp//rpp/aaaaaaa.....aooa/aoaa 211.12200.2.......caaa/aa/rrraar
Brian Hu, Marina E Garrett, Peter A Groblewski, Douglas Ollerenshaw, Jiaqi Shang, Kate Roll, Sahar Manavi, Christof Koch, Shawn R Olsen, and Stefan Mihalas. Adaptation sup- ports short-term memory in visual change detection task. PLoS computational biology, 17
(9):e100224,, 2021. URL/ttts:://jouraaal plos wttp://ooocoomiic///rr.r.oo. id 10.100.11101 jouraa/oooaaaaaaa www..................
Shailee Jain, Vy Vo, Shivangi Mahto, Amanda LeBel, Javier S Turek, and Alexan- der Huth. Interpretable multi-timescaal models for predicting fMRI responses
to continuous natural speech. Adv. Neural Inf. Process. Syst. 13749, 2020.
URL https::/proceedii neurips wttppuuiss..cpppppppeeeeeee pctppappe//22///as..... 9e9a30b74cs ott.0///000....7010000000000011. htw.bbeeeee..cooooooootto...... Awstraat.
John Jonides, Richard I.. Lewis, Derek Evan Nee, Cindy A. Lustig, Marc G. Berman, and Kather- inc Sledge Moore. The Mind and Brain of Short-Term Memory. Annual Review of Psychol-
ogy, 59()):193-224, 2008. doi: 10.1 40.11////ueuee.psychh... http://wwwsspc.....0000000000000
URL https:
7/doi.orge 10..111//nnnreee
annurev psych. 59. 103006 093615
Daniel R. Kepple, Rainer Engelken, and Kanaka Rajan. Curriculum learning tool to uncover learning principles in the brain. In International Conference on Learning Representations, 2022.
URL https: htttps://ppenrevieee wet.eeewweeeeeeeeeeeeeeeeeeeeeee wtt..eeeteooommmmddddooeeeeeeeee
Robert Kim and Terrence J. Sejnowski. Strong inhibitory signaling underlies stable temporal dy- namics and working memory in spiking neural networks. Nature Neuroscience, 24(1):129- 139, January 2021. ISSN 1546-1726.
doi: 10.10088/41193--000000533---5--3 URL https:
//www. nature. Com http:/atticclseeeme a
Kai A. Krueger and Peter Dayan. Flexible shaping: How learning in small steps helps. Cog- nition, 11003)380--994 2009. ISSN 0010-0277.
doi:
2008. 11.044. 80C 00010007708002000
Sukbin Lim and Mark Goldman. Balanced cortical microeircuitry for maintaining information in working memory. Nature neuroscience, 16, 2013. doi: 10. 10.18//n..344.. URL httpt:t////// : /www.
nature. www.e/oo.aaattiee
3492
Zachary C Lipton, John Berkowitz, and Charles Elkan. A critical review of recurrent neural networks for sequence learning. arXiv preprint Sttv::1506.000//99 2015. httlhttss://aaa org//
abs/ abs/1506.. 1106.11660000 00019
Ashok Litwin-Kumar and Brent Doiron. Slow dynamics and high variability in balanced cortical networks with clustered connections. Nature Neuroscience, 15(11):1488-555, November 2012. ISSN 144611726. doi: 10.10388nn..220 URL htthtt/: //www nature com/articles/ nn. 3220
Number: 11 Publisher: Nature Publishing Group.
Shivangi Mahto, Vy Ai Vo, Javier S. Turek, and Alexander Huth. Multi-timescale representation learning [Istm] language models. In International Conference on Learning Representattonss 2021. URL https: http///pppp......p////p/..
Travis Meyer, Xue-Lian Qi, Terrence R Stanford, and Christos Constantinidis. Stimulus selectivity in dorsal and ventral prefrontal cortex after training in working memory tasks. Journal of neuro- science, 31(17):626666276 2011. URL https: /wwww jneurosci. www.urocci.ogg/oootttetooooeeeee .ogg/content//11... 1776266
John D. Murray, Alberto Bernacchia, David J. Freedman, Ranulfo Romo, Jonathan D. Wallis, Xinying Cai, Camillo Andoa-Schioppr Tatiana Pasternak, Hyojung Seo, Daeyeol Lee, and Xiao-Jing Jing Wang. A hierarchy of intrinsic timescales across primate cortex. Nature Neuro- science, 17(12):16111166 1-11666116666.. December 2014. ISSN 1546-1722 1726. doi: 10..038/m...862. URL
https: www. nature. com ietiiicl 3862
12

---

Published as a conference paper at ICLR 2024
Srdjan Ostojic. Two types of asynchronous activity in networks of excitatory and inhibitory spiking neurons. Nature Neuroscience, 17(4):594 600, April 2014. ISSN 1546-1726 doi: 10.1038/mn. 3658. URLpttps::/ nature com/ www.aa/tiieeee//cccooooo.oco.aet
3658
Stefano Panzeri, Nicolas Brunel, Nikos K. Logothetis, and Christoph Kayser. Sensory neural codes using multiplexed temporal scales. Trends Neurosciences, 33(3) March 2010. ISSN 0166-2236. doi: 10.1016/j.tins 200.. 12.001. URL http: www www.eceiencccrrcc..oo com/ wwwence/articcoeo Coceaae/Aatticee/////66666666200
Pranay Pasula. Real world time series benchmark datasets with distribution shifts: Global crude oil price and volatility, arXiv preprint urXiv:2308.10846 2023, URL.https://acxivooo
abs / 2308 10846
Nicolas Perez-Nieves, Vincent C. H. Leung, Pier Luigi Dragotti, and Dan F. M. Goodman. Neural heterogeneity promotes robust learning. Nature Communicationss 12(1):5791, October 2021.
ISSN 2041-1723, doi: 10. 10.1088/444447--22222602-23 URL mittps://ee //ww:////www//// nature. com/ articles/s4114 021-2222-2--22222
Xue-Lian Qi, Travis Meyer, Terrence R Stanford, and Christos Constantinidis. Changes in pre- frontal neuronal activity after learning to perform spatial working memory task. Cerebral cor-
lex. 21(12))2722-2732 2011. URL httts::/accademi oup. com http:///roooorrleeeee.e.
21/12/2122/255413
Silvan C. Quax, Michele D'Asaro, and Marcel A. J. in recurrent neural networks. Scientific Reports, 10(1):11300, July 2020. ISSN 2045- 2322. doi: 10. 01018/444198--02-6888111 141598-020-681652 Number: Publisher: Nature Publishing Group.
Gerven. Adaptive time scales
URL/ttps:: /www. nature. wwt.//ta/taiii
Shervin Safavi, Matthew Chalk, Nikos Logothetis, and Anna Levina. Signatures of criticality in efficient coding networks. bioRxiv, pp. 2023-02, 2023,
content/10. 1t1pee/220..111222...000
528465v1
Darjan Salaj, Anand Subramoney, Ceca Kraisnikovic, Guillaume Bellec, Robert Legenstein, and Wolfgang Maass. Spike frequency adaptation supports network computations on temporally dispersed information. Elife, 10:e65459, 2021. URL httpss://ellessccierceee org/
articles/65459
Yan-Liang Shi, Roxana Zeraati, Anna Levina, and Tatiana A Engel. Spatial and temporal correlations in neural networks with structured connectivity. Physical Review Research, (1):013005, 2023. URL httt:::////::rrrtt L1C7/BbysRerReeearch..... 01300
aps. wtwpaaas..ppppppssesseeeeeeeeeee wtt.:o///prreeseaaaasstsstr////e
013005
Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for se- quence modeling. In The Eleventh International Conference on Learning Representations, 2023a.
URL httph:ht
httt:p:://ppeereveee www.//etoouureee id=Ai BHw
Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for quence modeling. In The Eleventh International Conference on Learning Representations, 2023b.
URL URR/https::/ppeeeeviiee
Aaron Spieler, Nasim Rahaman, Georg Martius, Bernhard Schölkopf, and Anna Levina. The elm neuron: efficient and expressive cortical neuron model can solve long-horizon tasks. arXiv preprint 2rXiv:2306.1693 2023. URL htthtttss///xiii org abw.gg/abbbaee 2205. 110086666692 16922
Chao Sun and Erin M. Schuman. Logistics of neuronal protein turnover: Numbers and mechanisms. Molecular and Cellular Neuroscience, 123:103793, 2022. ISSN 1044-7431. doi: https://doi.
org/10. www.///.......... 2022. 103793. URL httpt:tt////.m /twp///ww//////////////////////. sci http://wwww.eeciiireeee....s com/ eecieeee///
Corentin Tallec and Yann Ollivier. Can recurrent neural networks warp time? arXiv: 1804. /88, 2018. UR..nttppss//rrxtvvon////ass 11188
arXiv preprint
13

---

Published as a conference paper at ICLR 2024
Zoran Tiganj. Michael E. Hasselmo, and Mare W. Howard. A simple biophysically plausible model for long time constants single neurons. Hippocampus, 25(1):27-37, 2015. doi: https://doi.
URL https: https://onnineelrraayy wwtpey...wllye.. ctt
1002/hip. 22347
José F Torres, Dalil Hadjout, Abderrazak Sebaa, Francisco Martinez-Alvarez, and Alicia Troncoso.
Deep learning for time series forecasting: a survey. Big Data, 9(1):3 21, 2021. URL https:
//www. wwwbertpub.bbb.... com www/coi/co/////oooooooocooooo 108./110..91...00 2020 0159
Alexander van Meegen and Sacha J. van Albada. Microscopic theory of intrinsic timescales in spiking neural networks. Physical Review Research, 3(4):043077, October 2021. doi:
0011033PhyssceRReaaad0..000000 PhysRevResearch. 3. 043077
URL
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. Advances in Neural Information Processing Systems, 33:2958- 2969, 2020. URL hntps::/pprocecinngs hew.rooeeedingg..eeiii....... neurips. http://ps....pppppr////////////
Bojian Yin, Federico Corradi, and Sander M. Bohté. Effective and Efficient Computation with Mulsiple-timescale Spiking Recurrent Neural Networks. In International Conference Neu- romorphic Systems 2020, ICONS 2020, pp. 1-8, New York, NY, USA, July 2020. Association doi: 10.1 01114//34071973447222 URL htttps://dl.acm org/doi 0100///1..115//37717777711 3407222
for Computing Machinery. ISBN 978 1 40-555003888-1-1
Yong Yu, Xiaosheng Si, Changhua Hu, and Jianxun Zhang. A review of recurrent neural net- works: Lstm cells and network architectures. Neural computation, 31(7):1235-1270 2019. URL
ht https://eeeexplcee
leee org/ document/ 8737887
Roxana Zeraati, Tatiana A. Engel, and Anna Levina. A flexible Bayesian framework for unbiased estimation of timescales. Nature Computational Science, 203):193-24,, March 2022. ISSN 2662- 8457. doi: 10. (0388/43388-0022-002211 0225888022-001---
URL www.ltttsspm/ www http://wwwwww...eeeeeaaa www. www/artii//a//e////.. crlsscssttis///caccccccc
Roxana Zeraati, Yan-Liang Shi, Nicholas A Steinmetz, Marc A Gieselmann, Alexander Thiele, Tirin Moore, Anna Levina, and Tatiana A Engel. Intrinsic timescales in the visual cortex change with selective attention and reflect spatial connectivity. Nature Communications, 14(1):1858, 2023.
URL Sutph:tt:///et nature. com/ 1tt.teee..e//rrraeeeeaa/1/110-30
14

---

Published as a conference paper at ICLR 2024
APPENDIX
A DIFFERENT TYPES AND LOCATIONS OF NONLINEARITY
In order verify that our results are robust with respect to the type of nonlinearity used in the network, we train RNNs using two of the most commonly used nonlincarities: ReLU and Tanh. We find that in both cases, the training performance is similar to leaky ReLU, and the development of single-neuron and detwork-mediates timescales follow the same trajectory as N increases (Fig. SS7).
In some implementations of leaky-RNN, the neural self-interaction linear and located outside of the nonlinearity (cf. equ
(1)
At Ty 1 (1))+++++
At
Σ W ry(1 - -1) + W/ 5(1) inf
5())+(((++))))) + + )
(5)
Ti
with the explicit time discretization A.. The input is presented for time duration T k&l with input-update time steps k. In the main text, we chose k = 1 and At = 1. We discuss k > 1 in Appendix(aamm and At in Appendix
We verify that training RNNs with this implementation gives similar training dynamics and trajec- tories of T and Thet with increasing N (Fig. (S8), for both curricula. Furthermore, we find that, for large N, ablating neurons with long in single-head networks and neurons with short 7 in multi- head networks reduces the performance significantly, compatible with the findings in the main text (Fig. [S9] cf. Fig. 77.. We also verify that the performance of the model depends on the initialization of T and its trainability in the same way regardless of the location of the nonlinearity (Fig. STT cf. Fig.
B CHANGING TIME DISCRETIZATION
In computational neuroscience, the single neuron dynamics are typically captured by the differen- tial equations that need to discretized for running numerical simulations and training networks. However, the discretization can be important for stability and internal representation of the model and the task. In the main text, we used Eq. 5 with At = = 1. For simplicity of notation, we take in the rest of this section T 1. We train networks with different values of At (a different At for each training batch), so they can perform the same task independent time discretization. We take AL 1/n with E N and train the network while keeping the duration of each stimulus presentation in units of time constant (which means that with larger n, it would be presented for more time steps). The flexible framework for time discretization allows simultancously. Then, we test whether the network can solve the same task but with At not included in their training set.
train with multiple At
We find that in networks trained with multiple At, the single-neuron timescales т follow a similar trajectory as the results in the main text, independent of At (Fig. S55bbb compared to Fig. 4p). Multi- head networks adjust their 7 to converge to nAt 1, while single-heat networks increase their individual neuron timescale. Moreover, the networks can generalize (without retraining) the task to smaller AL than what was included in their training set, in single- and multi-head networks. Interestingly, the performance decreases slowly when At becomes smaller than the training set, but abruptly when it becomes larger (Fig. (...d). The performance is best when training with multiple At, but qualitatively, the result is similar for a single, small enough At (Fig. S5b).
C CHANGING THE DURATION OF THE INPUT PRESENTATION
In our tasks, the input contains two timescales. First is the duration of presentation each input digit T' = A Tmin. with Tmin a minimal considered duration of stimulus presentation measured in milliseconds. Second is the timescale of the task's 's memory N. In the main text, we consider time-rescaling parameter and defines
the situation of k = 1, but in general, à acts time for the task performance. Here, we train the RNNs with different values of E {2, 3, 5, 103 and check the trajectories of changing with N depending k. We find that similar to the case
unit of
15

---

Published as a conference paper at ICLR 2024
with & 1, single-head networks trained with k >> 1 increase their with N, while multi-head networks try to keep close to k (Fig. (6). Moreover, tasks with > are generally more difficult to solve since the imput needs to be tracked over N - time steps. Hence, as k grows, RNNs would reach smaller N within the same number of training epochs. The changes in values of 7 after rescaling with k might be due nonlinear interactions in the network arising from the combination of different N and k.
D PROPOSED ADDITIONAL TASK: TEMPORAL PATTERN GENERATION
For future research, the task variety can be extended to include the temporal pattern generation, which is a continuous-time task that is often used to evaluate RNNs (Durstewitz et al. 2023). The classic variation the task involved an RNN receiving either random noise or no input and having produce target time series as output (usually
of sine waves with different frequencies).
A variation of the task we could consider for testing our model is the following:
Single-head: On the first step of the curriculum, train the network to produce single sine wave with frequency f1, setting the target sequence to be yN=1 = sin(2 f1 -t).
Then, for each curriculum step, complexify the target sequence by setting the new target as:
m
VN= Σ sin ft-t),
for / > /2 >. >..>> fm. In this way, as the newly added frequencies decrease, a need arises for the network to develop longer timescales.
Multi-head: Unlike the single-head network where the RNN needs to produce only one target time series =N=m at the mnth step of the curriculum, the multi-head curriculum, the network produces m output time series Y = ,
E EFFECTS OF TRAINING WITHOUT A CURRICULUM ON THE NDDMS TASK
We investigate the negative effects of not using a curriculum during training for the N-DMS task to extend our results from Fig. 3h. We show in Fig. [S13 that similar the N-Parity task, networks rapidly lose the ability to solve the N-DMS task as N increases when training without a curriculum. Interestingly, the two tasks differ in the way they fail to be solved despite using identical optimizers. In all of our results, the NDDMS task tends to be easier to solve for larger N. However, despite the relative success these networks have with the NDDMS task, their drop-off in training these networks
is much steeper when comparing the curves from Fig. 3 and Fig. S13] In Fig. [S13 tasks N < 15 get solved in only
epochs, however between 15 < N 20 the networks rapidly slow down in their ability to train until completely failing for N > 20 even when given longer training time.
We can infer from these results that different tasks have varying degrees to which they benefit from particular curriculum.
F INTERMEDIATE CURRICULA: MULTI-HEAD WITH A SLIDING WINDOW
The two curricula discussed in the main text (single-head and multi-head) represent two extreme cases. In the single- head curriculum, each step of the curriculum, RNNs are trained to solve new N without requiring to remember the solution to the previous Ns. On the other hand, in the multi-head curriculum, RNNs need to remember the solution all the previous Ns in addition to the new N. Here, we test the behavior of curricula that lie in between the two extreme cases.
The intermediate curricula involve the simultaneous training of multiple heads, similar to the multi- head curriculum, but instead of adding new heads at each curriculum step, train fixed number of heads and only shift the Ns. which they are trained for according to a sliding window. We consider the number of heads to be 10, and start the training for N E .2..... 11]. In the next steps of the curriculum, we use the already trained network to initialize another network which we train for N + (e.g., N [2 + w, ...... 11 + wl). where (1,3,5) indicates the size of the sliding
16

---

Published as a conference paper at ICLR 2024
window. For each w, we train 4 different networks (i.e., different initialization). For the following analyses, we trained the networks on the N --parity task.
We find that networks trained with the multi-head-sliding curriculum generally demonstrate in- between behavior compared with the extreme curricula, but the results also depend on the size of the sliding window. Within 1000 training epochs, the maximal N these networks solve (with > 98% accuracy) is in between the maximal N of single- and multi-head curricula, depending on the sliding window. Networks with larger sliding window can solve a higher maximal N, indicating that large sliding window not only does not slow down the training but also provides a networks, more efficient curriculum to learn higher Ns (Fig. (s Moreover, in multi-head-sliding single-neuron (T) and network-mediater (Tnet) timescales have values in between single-heat and multi-head curricula (Fig. (S12)). However, both т and Thet grow with N similar to single-head networks, with the pace of growth reducing for larger sliding windows.
Similar to the main text (Fig. 7dd,), we perform the perturbation and retraining analysis on multi- head-sliding networks trained with weights W and timescales т for these networks lies between the two extremes (Fig. S14, b). However, the retraining analysis suggests that multi-head-slidig for higher new Ns (Fig. (S14,.d) If the network is originally retrained for a small N (e.g.. N = 16), the retraining relative accuracy is similar between multi head and sliding networks but is larger than single-head networks. For networks trained for larger Ns (e.g., N = 31), sliding networks exhibit superior retraining ability compared to the other two curricula. These results suggest that the curriculum with the sliding window helps multi-head networks to better adjust to new Ns.
5. The relative accuracy after perturbation of recurrent
networks can be retrained better
G SINGLE- AND MULTI-HEAD CURRICULA FOR TRAINING GRU AND LSTM
The results presented in the main text were generated using a modified version of a vanilla RNN (leaky-RNN) with an explicit definition of the timescale parameter r. To test whether the difficulties in training for long memory tasks without curriculum would carry over to recurrent networks that were specifically designed for long memory tasks, we train two other architectures, an LSTM (long short-term memory) and a GRU (gated recurrent unit) on the N N-parity task for increasing N, with and without a curriculum. Both the GRU and L.STM have similar network sizes to the RNN with 500 neurons, though they differ in their activation functions (the RNN used single leakyReL.U whereas the GRU/LSTMs have both sigmoids and tanhs for different gates). Furthermore, in contrast with the RNNs, an Adam optimizer is used with learning rate Ir = 10 and the input signals to the models take values E 1) (to have a zero-mean input signal).
We find that for both architectures, training the networks without curriculum is extremely slow for large N and relatively unstable for small N and probably requires strict tuning, introducing the multi-head curriculum
tun-
ing (Fig. (S15a). Without additional dyper-paramee speeds up the training significantly, and both architectures can easily learn the N-parity task with large N similar the leaky-RNN (Fig. (15p).. Moreover, similar to RNNs, the multi-head curricu- lum has a higher training speed than the single-head curriculum (Fig. S16). Our results indicate that GRUs and LSTMs are subject to similar training dynamics RNNs used in the main text and the multi-head curriculum is an optimal curriculum regardless of the RNN architecture. The advan- tage of using the leaky-RNi RNN architecture is that its parameters are easier to interpret, and allows us to study better the mechanisms underlying each curriculum by explicitly studying the role of timescales.
H BACKWARD AND FORWARD RETRAINING OF NETWORKS
To understand how trained models develop their ability to create longer timescales throughout the curriculum as well as their backward compatibility and robustness to catastrophic forgetting, we measure the retrainability of models trained on a task with memory N on a different task with memory N We freeze all parameters of trained network except the final readout layer weights which are retrained on an N' task. Specifically, we load models trained for N E [2..... 19) and retrain them new N E [2. ...... N +2], independently for each N for maximum of 10 epochs or until its accuracy was above 98%. Note that we retrain both single- and multi-head networks as single-head.
17

---

Published as a conference paper at ICLR 2024
We find that the multi-head networks exhibit near-perfect backward compatibility well as better forward compatibility than the single-head models (Fig. [S17]. while single-head networks suffer from catastrophic forgetting. For the multi-head networks, the backward compatibility is enforced through the loss function (as the case in the multi-head curriculum) hence, the necessary represen- tations for N N persist. However, the multi-hear curriculum also has positive implications for forward compatibility, which is evident in the off- off-diagonal entries of the accuracy where N > N (to the right of the dotted line) when compared to the single-head values.
I EMERGENCE OF CURRICULUM DURING MULTI-HEAD TRAINING
In the multi-head curriculum, the difficulty of the task increases gradually; new head with larger N is added at each step of the curriculum. In the main text, we discussed that networks trained with such curriculum generally train well up large Ns. Here ask whether this optimal curriculum can emerge by itself if we train a network with multiple heads, but without any predefined curricula. For this analysis, we train RNNs with 19 (N [2...., 20)) and 39 heads (N E (2...., 40) solve all the available Ns simultaneously.
We find that despite the absence of explicit curriculum, these networks learn the task by generat- ing an internal multi-head curriculum. While all the heads contribute equally to the loss, heads with smaller N reach the higher accuracy faster (Fig. SS1a).. However, the speed of training strongly depends on the total number of heads in each network. For the same N, the network with 19 heads reaches the 98% accuracy faster than the network with 39 heads (Fig.[ST8p) but both networks have a slower training speed when compared to the head curriculum. These results suggest that the multi-head curriculum an optimal curriculum that can arise naturally during multi-head training and can increase the training speed when applied explicitly.
J ROLE OF STRONG INHIBITORY CONNECTIVITY IN SINGLE-HEAD NETWORKS
The main difference between single- and multi-head networks in terms of connectivity is the stronger inhibitory (negative) connectivity for large N in the single-head networks compared with the rela- tively balanced connectivity in multi-head networks (4)) We hypothesized that larger inhibition in single-head networks is required to keep the dynamics stable in the presence of slow single-neuron timescales т. To test this hypothesis, we perturb only the inhibitory connections in networks trained with both curricula as:
W Wis c Wiy,
A W.R < 0.
for a given amount of 0-0.1,0.1]. We observe that by reducing the amount of inhibition in single-head networks, the network activity explodes even before reaching the balanced point, i.e., the point when the average incoming weight of neurons becomes 0 (Fig, (I9 On the contrary, multi-head networks are significantly more robust to such perturbations and their activity remains within reasonable range for broad range of inhibitory scaling SI9p).
This difference is most likely attributed to the difference in single-neuron timescales 7 between single- and multi-head networks. The single-head networks have a larger average compared to the multi-head networks whose average T R 1 for large N. Longer 7 leads to neurons with self- sustaining activity, and thus, a stronger inhibition might be required prevent the runaway activa- tion. Such a relationship can be observed when comparing the average and inhibitory strength across networks: for single-head networks as 7 grows, the average weight becomes more negative (inhibitory) (Fig. (S19)), but such correlation does not exist in multi-head networks (Fig. (19H)..
K DEPENDENCE OF DIMENSIONALITY OF POPULATION ACTIVITY ON N
We measure the dimensionality as the number of principal components that explain 90% of the population activity variance. The dimensionality increases with N' for both tasks and curricula, but the increase follows a linear relation with N for N N-parity task but a sub; linear relation for the N-DMS task (Fig. (b).
18

---

Published as a conference paper at ICLR 2024
To demonstrate this difference, fit two separate lines for the data up N 20 and from N = 20 up to the largest N. We observe that for the N --parity task, the slope of two lines largely overlaps, indicating linear relation. However, for the N-DMS task, the second line clearly has smaller slope than the first one, indicating a sub-linear growth with N (Fig. $200.
L ABLATION DETAILS
To test whether neurons with fast or slow timescales (T) are necessary for computations in the trained RNNS perform the ablation analysis. For this analysis, compute the relative accuracy of the model (Eq. 4 in the main text) after removing a single neuron. We ablate neuron by setting all incoming and outgoing associated weights zero
W W =0 0
Vj
ijt
WOO W =0
Here Wi refers recurrent weights, W input weights and W readout weights. To measure the relative accuracy, we simulate the RNN forward using random binary inputs for 1000 time steps after 100 time steps of burn-in period (to reach the stationary state). Then, evaluate the accuracy of the network at each time step. We repeat this procedure over 10 trials and compute the average and standard deviation of the relative accuracies across trials.
M SIGNIFICANCE OF THE RESPONSES TO PERTURBATIONS OF WEIGHTS AND RETRAINING
We investigate the significance of differences between single- and multi-head networks presented in Fig. 7 using t-test (two-sided, unpaired). Perturbations are computed 10 times for 4 networks group with results being pooled across networks. Retraining accuracy is computed once per network. Table 12 and 3 indicates with stars the significance levels corresponding to p-values below 2,-2, le 2, le 3, Te 4, and 1e-5.
Weight Perturbation Strength
p-value
Significance
1.0e-02
8,8e-03
#$
2.2e-02
4.5e-01
n/s
4.6e-02
2.9c-01
n/s
1.0e-01
5.1e-15
*****
2.2e-01
2.2e-39
4.6c-01
8.3e-47
1.0e+00
8.6e-04
##8
2.2e+00
7.6e-01
n/s
4.6c+00
6.5e-01
n/s
1.0e+01
9.0c-01
n/s
Table 1: Significance the weights' perturbation for different perturbation sizes Fig. 7 Two-
sided and unpaired t-test, indicate p-values below 5e-2, le-2, le 1e-3, le-4, and 1e-5.
19

---

Published as a conference paper at ICLR 2024
Perturbation of
p-value
Significance
1.0e-03
4.3e-01
n/s
2.2e-03
8.9c-01
n/s
4.6e-03
5.4e-01
n/s
1.0e-02
1.2e-02
2.2c-02
2.3e-14
#****
4.6e-02
9.1e-29
$***$
1.0e-01
1.3e-50
#*4*0
2.2c-01
4.1e-35
4.6e-01
4.8e-01
1.0e+00
6.0e-01
n/s
2.2e+00
1.3e-01
n/s
Table 2: Significance of the r's perturbation for different perturbation sizes Fig. 7d. Two-sided and
unpaired t-test, indicate p-values below 5e-2, le-2, 1e-3, 1e-4, and le-
retraining for N
p-value
Significance
17
1.7e-03
18
9.7e-04
1.2e-07
20
3.2e-05
21
1.7e-05
22
1.6e-05
#888
23
7.4e-06
24
6.4e-05
25
3.1e-04
26
1.7e-03
27
1.7e-02
28
2.1e-03
29
7.5e-03
30
1.6e-01
11/
31
1.5e-01
32
6.5e-01
n/s
Table 3:
Significance of the retraining differences between single and multi-head, Fig. 7. Two- sided and unpaired t-test, stars indicate p-values below 5e-2, 1el2e,2, 1e-3, 1e-4, and 1e-5.
N CODE AND DATA AVAILABILITY
Codes for training and evaluating the RNNs and reproducing the experiments (c.g., measuring timescales, performing ablations, etc.) together with example trained networks available GitHub at wttttps//guhubb.com//eeeaaaalllo timescale public (more details in README).
20

---

Published as a conference paper at ICLR 2024
O
SUPPLEMENTARY FIGURES
Single-head, N-parity
Single-head, N-DMS
Conseensue belveen AIC and BIC
99
98
97
97-
96
95
5
10
15
20
25
30
20
40
60
80
Multi-head, N-parity
Multi-head, N-DMS
CsConsensuu between AIC and BIC
100.0
99.5
99.0
98
98.5
98.0
97.5
20
80
20
100
N
Figure S1: AIC and BIC choose the same model for most neurons. We fitted AC of each neuron's s activity with single- and double-exponential functions and used AIC or BIC select the best-fitting models. The results show that for above 95% of neurons, the two criteria select the same model. The colors of the dots indicate different networks (4 networks for each task, curriculum and N).
Single head
Multi head
109
109
Single-exp Double-exp
Data
Probabiiity densily
15
30-
10-1
10
10.
20
10-2
10-2
10-
10-3
10-3
Time lag
Time lag
selected
selected
Figure S2: ACs of neurons well captured with single- double-exponential fits. Note that the y-axis is in logarithmic coordinates, meaning that deviations between the fit and data AC are much smaller in the AC tail compared initial time lags. a, b. Fitting double and single exponential functions to the AC of example neurons. e. Values of coefficient of determination R² estimated for all selected fits using AIC close to 1, indicating good fit.
(a) single-timescale (Tnet T) and (b) double timescale (Tnet > T)
21

---

Published as a conference paper at ICLR 2024
1.8
No Curriculum
Mean(Trec)
No Curriculum
Mean(t)
Single Head
Single Head
Multi Head
Multi Head
1.4
5
10
15
20
5
10
15
20
Figure S3: Networks trained without curriculum have similar single-neuron (a) and network- mediated (b) timescales networks trained with the single-head curriculum in the range of N that the no-curriculum-traine networks can learn.
Single-head, N-parity
Single-head, N-DMS
Multi-head, N-parity
Multi-head, N-DMS
7.5
60
Mean(
5.0
20
40
2.5
10
20
10
0.0
10
50
100
100
Figure S4: Dependence of population activity timescales Tpop on N. For both tasks and curriculum, the timescale of population activity fluctuations increases with N, indicating general trend toward slower collective dynamics for tasks with larger memory requirements. Shade + STD across networks.
2
1.0
single-head
Mean(1) (T)
STD(t)
multi-head
0.5
10
20
30
10
20
30
N
N
e
100
training range
Accuracy (%))
30
75
50
10-2
10-1
10a
10-2
10-1
10"
10-2
10-1
10a
At
At
Figure S5: Impact of discretization time-step At on training performance. We train networks with
At [1] 1
11 while presenting each input digit for the duration of T 1. (a) Similar to 10 8 1.777,
discrete-time networks (At 1), the mean of single-neuron timescales increases with N for single-head networks and decreases towards T 1 for multi-head networks. (b) The standard deviation of indicates heterogeneous 7s for single-head networks but constrained values for multi- head networks. (c,d) Single-head (c) and multi-head (d) networks can solve the task above the chance level for At smaller than their training regime (indicated by the red rectangle) when trained with multiple At. (e) The networks are slightly more inaccurate when trained on only single At 10 Lines and the color bar indicate different N.
22

---

Published as a conference paper at ICLR 2024
Time Warped Sequence (N
1.3 4
k
Curriculum
2
multi-head
Original Sequence
1.2
single-head
1.1
tt>
0.9
Figure S6: Changing the duration of input presentation. Each input digit is presented to RNN for a duration of kåt, A = 1. Single-neuron timescales (rS) normalized by remain roughly constant in multi-head networks (i.e. - kAt), but increase with N in single-head networks (cf. Fig. 4)
Single-head
Multi-head
20
Mean(Tnel neet)
6
ReLU
Mean(T nent
Leaky ReLU
Tanh
10
2
10
20
100
Single-head
Multi-head
1.55
1.4
Mean(t)
Mean(t)
1.50
1.2
1.45
10
20
100
N
Figure S7: The development of timescales follows similar trajectories when the self-interaction is inside (nonlinear 7) or outside (linear 7) the nonlinearity (leaky-ReLU), Top: timescales, bottom: single-neuron timescales. Shades = STD.
23

---

Published as a conference paper at ICLR 2024
Single-head
Multi-head
8
15
Mean(T (Ten)
Non-linear
Mean( net)
Linear
10-
5
2
10
20
30
50
100
N
N
Single-head
Multi-head
1.5
Non-linear
Mean(1)
Mean(1)
1.4
1.6
Linear T
1.3
1.2
30
100
N
Figure S8: The development of timescales in networks with different nonlinearities follows similar trajectories. Top: network-mediated timescales, bottom: single-neuron timescales. Shades STD.
a
N 5, Single-head
Multi-head
c 30, Single-head
30, Multi-head
Relative accuracy
Short
Long
0.8
0.6
1.0
1.97
3.85 Average of ablated neurons
1.0
1.94
Average of ablated neurons
Average t ablated neurons
Average t ablated neurons
Figure S9: Impact of ablating neurons with distinct timescales on RNNs' performance when neural self-interactions are linear (cf. Fig. (b..). a, Ablating the longest and shortest timescale neurons has minimal effect on network performance when N is small for both curricula. c, d. For higher N, ablating long timescale neurons largely decreases the performance single-head networks. while multi-head networks are more affected by the ablation of short-timescale neurons. Bars mean, emror bars STD, dots 4 individual networks.
Single-head
Multi-head
Maximum trained
100
75
50
Trained
r=l 1
Trained
T=2
r
Figure S10: The maximum N solved in the -DMMS task after 1000 epochs (reaching an accuracy of 98%). Similar to the N-parity task (cf. Fig. F models trained with single-head curriculum rely more on training 7 than the multi-head curriculum networks, which prefer to have a small 7 and more agnostic to being trainable. Horizontal bars mean.
24

---

Published as a conference paper at ICLR 2024
Single-head
Multi-head
Maximum trained
Maximum trained
100
40
80
30
Trained
T
2
rsa 3
Trained
T=1 1
T 2
T
Figure S11: The maximum N solved in the N-parity task (for models with the self-interaction outside the nonlinearity) after 1000 epochs (reaching accuracy of 98%). In the single-head cur- riculum, models rely on training т, whereas the multi-head curriculum, having rs fixed at 1 value is as good training them. Results consistent with the models where the self-interaction is inside the nonlinearity (cf. Fig. 4) Horizontal bars mean.
b
SH
SL, W
SL, W W =
2.0
30
Max trained
80
Mean(z)
Mean( (Fne))
20
60
10
40
1.0
SH
SL. W= 1
=
5
MH
100
SL. W
N
SL.
Figure S12: The behavior of networks trained with nulti-head-slidingg curriculum depends on the size of the sliding window and lies in between extreme curricula. The maximal trained N (with > 98% accuracy, within 1000 training epochs) for multi- seati-hhaa--ae -eddd-iiiing (SL) lies between single-head (SH) and multi-head (MH) networks and increases with the size of sliding window (w), Dots indicate individual networks (4 networks) and the horizontal bars indicate the mean value. b. Single-neuron (T) and network-ment mediated (Tnet) timescales increase with N, but the pace of change reduces the sliding window grows. Shadings indicate + std computed across 4 trained networks.
No curriculum
100
10
Accuracy
16
80
20
23
20 Epochs
40
Figure S13: Training without curriculum the N-DMS task. For each N, 4 models are inde- pendently trained for 50 epochs or until reaching > 98% accuracy. Similar to the N-Parity task (cf. Fig. B4), the ability to solve the task decreases no longer capable finding a solution to the task even with longer training time.
increase N. From N 20, the network
25

---

Published as a conference paper at ICLR 2024
Multi-head Sliding, w=5
Relative accuracy
1.0
Single-head
0.5
0.0
10-1
10'
10-2 Perturbation of
10
16
21
26
31
36
31
36
41
46
51
Perturbation e Wa
of single-head re-training
N of single-head re-training
Figure S14: Robustness of networks trained with multi-heaad-liding curriculum. b. Multi-head- sliding networks are more robust than single-head networks but less robust than the multi-head networks against perturbations of recurrent connectivity (a) and trained timescale (b). Each line indicates one trained network (4 networks for each curriculum). Shades indicate + std computed across 10 trials. d. retraining of networks trained with different curricula single-head network on new Ns (for 20 epochs). Multi-hcad-sliding networks achieve higher relative accuracy when retrained for a higher N in comparison single-head networks. If originally trained for small Ns (c, N = 16), they have similar retraining accuracy to multi-head networks, but for larger Ns (d, 31) their accuracy suppresses the multi-head networks. Each line indicates the relative accuracy for one network (4 networks for each curriculum).
GRU
LSTM
100
80
RNN LSTM
3
Accuracy
80
80
10
20
10
20
40
Epochs
Epochs
Epochs
Figure S15: Comparing the impact of curriculum on different recurrent architectures. Two different architectures, GRU and LSTM, are trained on the N-Parity task with and without curriculum. We observe that the GRU and LSTM both exhibit instability when training without curriculum (a), but are comparable to the RNNs with the multi-head curriculum (b).
75
Single-head
Mulli-head
Nmax
25
40
80
100
apoch
Figure S16: Comparison of single- and multi-head currieula for training LSTMs on N-parity task. Networks trained with the multi-head curriculum can reach higher N faster than networks trained with the single-head curriculum.
26

---

Published as a conference paper at ICLR 2024
Single-head
Loss
Accuracy
100
0.8
90
N original
0.6
80
10
10
0.4
70
60
50 (chance)
21
Multi-head
2-
2
100
0.6
90
N original
0.4
80
10
10
70
0.2
60
50 (chance)
retrained
N retrained
Figure S17: Single-head (a) and multi-head (b) networks loaded for N [2...., 19] have new readout heads retrained on new tasks with N ,,..., N + 2]. The heat map of the loss and accuracy of these retrained networks (after maximum of 10 epochs reaching 98%++ shows the robustness of the multi-head networks to catastrophic forgetting, as well as an improvement towards forward compatibility in the N > N region. The dotted line indicates the diagonal.
accuracy of
b
19 heads
39 heads
100
100
40
Max trained N
10
Accuracy
80
19
19 heads
60
60
heads
0
20
40
0
20
40
0
20
40
Epochs
Epochs
Epochs
Figure S18: Emergence of curriculum during multi-head training. a. In the absence an explicit curriculum, multi-head networks solve smaller Ns before solving the large Ns. The color bar indi- cates the range of Ns. b. The speed of training reduces with the increasing number of heads. The network with 19 heads needs fewer epochs to solve the same N (i.e. reaching 98% accuracy) than the network with 39 heads.
27

---

Published as a conference paper at ICLR 2024
Single Head
b
Multi Head
Average Network Activity
5
Balance
Balance
3
2-
2-
-10
% Change in Strength of Inhibition
% Change in Strength of Inhibition
Average incoming
-0.25
0.10
weight (W
-0.50
0.15
-0.75
-020
-1.00
-025
-1.25
-0.30
1.6
1.8
2.0
1.2
1.3
1.4
1.5
Mean(T)
Mean(1) (T)
Figure S19: Networks trained with the single-head curriculum require strong inhibitory connectivity. We perturb the inhibitory (negative) connections of single (a) and a multi-head network (b). We see that the activity of the single-head network explodes as we approach the balanced point (the point where the average incoming weights becomes O, indicated by the horizontal blue line). On the contrary, the multi-head network is quite robust and produces activity within a normal range even after the balanced point. (c) In the single-head networks, we observe negative correlation between the average t and the average strength of incoming weights for each neuron (i.e. higher is correlated with more negative average weight). This relationship not present for multi-head networks (d) that are largely balanced and maintain small т.
a
parity
b
- DMS
Dimensionality
40
40
20
20
Multi-head Single-head
0
40
60
100
20
40
60
80
100
N
N
Figure S20: Dimensionality of activity increases approximately linearly with for the N-parity task and sub-linearly for the N-DMS task. We separately fit the data points for N [o..... 20 and N E [20.... 100] (N E [20,... 30) for the single- head N-parity network) and we observe that in the N-parity task, the two lines largely coincide, while in the N-DMS case there is clear change in the slope of the line, suggesting a sub-linear increase with N.
28


Reviews:
[{"review_id": "6yLaNM3DIX", "reviewer": "Reviewer_vRYi", "summary": "The paper trains RNNs on two tasks that require memory with two curriculum learning types of approaches, one with sequential learning of a single-head RNN and one with multi-head RNN. The former learns a single task sequentially and the latter simultaneously learns multiple closely related tasks that build upon one another simultaneously. The paper shows that these two approaches exhibit different properties in single-neuron and network-mediated timescales.", "questions": "1. Page 3 states \"In the limit case τ → ∞, the neuron’s activity is constant, and the input has no effect.\" Does this assume convergence as $\\tau$ approaches infinity? Are there cases in which this wouldn't hold?\n\n2. The paper uses AIC to select the best fitting model. \n  a. Why do you use this information criterion?\n  b. Why use this criterion instead of another or others collectively?\n  c. For support in using this criteria, consider referencing [1] that used AIC alongside other information criterion to select the best fitting model for time-series data and won the IJCAI'23 Best Paper Runner Up award.\n\n3. Figure 4 caption says \"whereas in the multi-head curriculum, having τs fixed at k value is as good as training them.\" Should this include a specific numerical value or range of values for $k$?\n\n4. On page 5, how is optimal strategy defined in the statement \"Furthermore, the multi-head curriculum emerges when training on a multi-objective task without introducing an explicit curriculum (Appendix D), supporting the optimal strategy of this curriculum.\"\n\n5. In Figure 5c, 5d, do you have insight why the mean and standard deviation for the multi-head networks go up and then back down? I.e., they peak at ~30 and then go down.\n\n6. On page 7, could you further explain the statement \"The strong negative weights in single-head networks are required to create stable dynamics in the presence of long single-neuron timescales.\"\n\n7. For the \"retraining\" (Perhaps a slight typo) experiment details on Page 8 and 9, why use 20 epochs?\n\n[1] Pasula, P. (2023). Real World Time Series Benchmark Datasets with Distribution Shifts: Global Crude Oil Price and Volatility. arXiv preprint arXiv:2308.10846.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "**Originality**\n- The paper proposes and explicitly evaluates experiments on single-neuron and network-mediated neuron timescales properties for two overarching tasks, with different value $N$ subtasks, and two curriculum learning type approaches.\n\n**Quality**\n- The paper does a good job for the most part on evaluating and showing results on means and standard deviations and some sort of error or measure of spread of each of these for various properties of interest (e.g., $\\tau_{\\text{net}}$) in Figure 5 as $N$ changes.\n- Most claims are supported by empirical results.\n- The paper identifies or is inspired by other sources to make certain principled decisions, such as identifying that autocorrelation bias is an issue, and simulating network activity for long periods of time to mitigate this issue. (Mentioned in Page 3)\n\n**Clarity**\n- The paper generally does a great job explaining the motivation, related work, preliminaries, some experimental design, and some analysis of results. For example, page 3 contains an excellent explanation of the limitation posed by having non-linear dynamics and how the proposed approach overcomes these limitations through an approximation method that the paper explains as well.\n- Figures are generally well made. Some take a little effort to decipher, but I was able to understand all of them. However, accessibility to mildly or significantly colorblind people could be improved by differentiating the single and multi-head N-DMS line plots in Figure 4 and any other similar cases.\n\n**Significance**\n- With concrete results on specific tasks, I hope this work inspires people to further explore this area, especially in ways that show that it generalizes as well as cases that deviate from expected behavior with analyses and findings as to potential causes.\n\n---\n\nNote: I've raised my ratings of Soundness from a 2 to a 3, Presentation from 3 to a 4, and of the overall paper from a 3 to a 5 due to the extensive improvements made and clarifications provided by the Authors during the Author Rebuttal phase.", "weaknesses": "For Figure S12, it would be helpful to readers to point out that the y-axis is on a log scale, so even though visually the linear fit doesn't look good towards larger $t$, the residual errors are much smaller here because this regime has much smaller values than those for earlier $t$.\n\nIn Figure 6, I can infer, but I don't believe it is explicitly stated why N=30 is used as the maximum value for linear regression line fitting. Furthermore, I don't understand why the paper uses N=30 for N-DMS if the reason to the previous question is that N=30 was about the limit for single-head networks on N-parity.\n\nFor Figure 8c, I would've liked to see a mean line and N extend to a higher number to see at what relative accuracy each converge and how these compare to one another other.\n\nThe Limitations section is lacking significantly, especially with work as general as individual and network neuronal timescales in DNNs. Future work is also lacking, both as a result of lacking limitations as well as no suggestions for future work outside of the limitations section. This section seems like an afterthought that was thrown in to satisfy having \"Limitations\" of the work mentioned.\n\nWith the large amount of related work that is close to the proposed work, it seems like the proposed work is primarily an empirical investigation into very simple tasks and limited approaches. Even equations are generally adapted from another source, and the paper merely needs to plug in its variable symbols to create the equations shown. I think there is novel concrete empirical insights here, but I'm not confident as to how they generalize based on the limited number of experiments provided. I have more confidence in how they generalize based on the related literature's work that forms the basis for which the paper proposes and evaluates the experiments it does. Though the paper is generally well written and presented, I find that it being primarily, or even about entirely, empirical that it is too limited in its experiment design quality, empirical results, and scope to recommend it for acceptance in its current state.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5l3H63jAfn", "reviewer": "Reviewer_h2ki", "summary": "This paper investigates mechanisms in recurrent neural networks for solving tasks that require long memory representation. The motivation for this work is that organisms need to process signals from a variety of timescales, and discovering mechanisms for timescale adaptation may be useful. Prior work has studied timescales in both single neurons and networks, and this paper aims to study specifically how these timescales shape RNN dynamics and performance on long-memory tasks. \n\nThe paper introduces the model: an RNN where each neuron has a trainable leak parameter tau - for higher values of tau, memory is a higher component of the input; for lower tau, the incoming input and recurrent signals dominate. The paper considers dynamics in terms of the single-neuron timescales tau and the network timescales (estimated from the decay rate of the neuron activity's autocorrelation).\n\nExperiments are conducted for two tasks: N-parity and N-DMS. Two curricula are used: single-head, with a single response head that does the task for the input N, and multi-head, which adds a head for every subsequent N during training and therefore does the task for all integers up to N. This mitigates catastrophic forgetting seen in the single-head model as higher N come up in the curriculum.\n\nResults are presented for a variety of research questions. First, performance under different curricula: results here show performance without a curriculum (N alone) - these fail for N > 10. Then, single-head vs. multi-head: multi-head does far better. Next, investigation of the mechanisms underlying long timescales, in which the paper argues that while the single-head curriculum leads to long single-neuron timescales for memory purposes (large, heterogenous tau), the multi-head curriculum leads to shorter tau (and higher tau_net and activating recurrent interconnectivity), suggesting that the memory mechanism is more in the recurrent connections. Finally, ablations are investigated, looking at ablations of neurons with especially short or long tau values, perturbations to weights and timescales, and retraining, for all of which the multi-head approach is superior.\n\nThe paper finally takes us through a more detailed version of the related work in the intro, and concludes by suggesting that 1) training on sets of related tasks may lead to more performant and robust networks than training on single tasks and 2) long timescales are necessary for long memory but these can be achieved through connectivity rather than just single neurons.", "questions": "- What other experimental settings/testbeds might be easily accessible for your setup?\n- How might this connect back to biological intelligence? I realize this isn't necessarily the focus of the paper, but it is discussed extensively in the intro. \n- Section 4.2 goes into this but it's hard to get key conclusions: it's clear that with the multi-head curriculum, single-neuron timescales are not the dominant memory mechanism. What intuition should we take away about the memory mechanism the multi-head curriculum *does* induce? We see that high, less inhibitory network connections are involved - anything more specific?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "#### Quality\n- Experiments are highly intuitive\n- The paper is framed around comparison - N-parity and N-DMS, results on which both serve the same goal and also allow comparison on difficulty; single- and multi-head; single-neuron and network-mediated timescales. The results space is therefore very accessible to reason about.\n- Results are compelling and analysis is convincing - section 4 has a lot of useful material, and each subsection makes interesting claims. Overall, the results mostly back up the claims. \n\n#### Clarity\n- Figure 7 is useful and well-annotated, and a good way of presenting the data\n- Sections 2 and 3 is well-written. The model setup and experiments are presented very clearly - I am not confused at any point. \n- More involved concepts (tau_net, autocorrelation) are made easy to understand\n- Ablation/perturbation/retraining setup is particularly useful to understand the nature of the problem.\n\n#### Originality\nBased on related work, this paper does appear to position itself as an experimental realization of theory presented in prior works, leading to a novel and unknown result\n\n#### Significance\nThe setup is compelling, and the suggested conclusions (1 and 2 from my summary above) are promising and significant if useful.", "weaknesses": "#### Quality\n- Claims in conclusion seem overstated. This is more a significance issue than a quality issue.\n- I would say more experiments are needed, but it's not a matter of specific additional experiments - for what it is, this seems to be a careful and informative set of experiments. It's more that the paper needs considerable expansion, which may also be a significance issue. \n- Figure 8: the curves do show a difference but not a drastic one, making it hard to judge how much more robust multi-head is. Significance testing, grounding according to some baselines (including visually on the figure), or comparison to similar results in other contexts may help. \n\n#### Clarity\n- While Section 4 has a lot of good info, it is difficult to understand. The writing is prose-based and somewhat meandering and disorganized. The subsection structure is useful, but I think it would help to add subsubsections, especially because the subsections are topic-based rather than claim-based. It would help to add claim-based subsections (or paragraphs) - e.g. in section 4.2, a subsubsection summarizing in a sentence how tau and tau_net change as N increases.\n- Aside from fig 7, figures are under-annotated - it takes a lot of effort to understand the message. This could be easily rectified with more visual information. Figure 4a in particular needs better explanation - the x-axes are confusing.\n- The term \"time-lag\" is introduced without clear definition, and it takes a while to get the relevant info. \n\n#### Originality\nNo weaknesses as far as I am aware.\n\n#### Significance\nAs stated in the limitations, there is only one testbed in this paper and it is very toy. Unfortunately, the claims of this paper seem to be about improving RNN performance, which means that some kind of scale or breadth is requisite. The motivation connects to biological neural networks, but that stops at the intro - the paper never connects the investigated mechanisms back to biological intelligence in any way after the prose in the intro, let alone through experimentation. It's therefore unclear how significant these results are, though they are intriguing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YFPfMvUlpV", "reviewer": "Reviewer_XC2K", "summary": "Training RNNs can adjust the time scales of their responses in order to perform memory tasks. The authors point out that this may involve adjustment of the time scales of individual neurons, or of the network-generated timescales created by interactions between neurons. They set out to tease apart the roles of either component in different memory tasks and different training settings.\n\nThey train RNNs on two memory tasks whose difficulty increases with sequence length N (N-DMS and N-parity), and under two settings (successive Ns on a single network, and all-N-so-far with different heads for different Ns).\n\n They show that the single-head setting tends to favor changes in single-neuron timescale, while the multi-head setting tends to keep single-neuron timescale low and increases network-generated timescales instead.\n\nVarious ablations and analyses confirm this observation. The multi-head networks are shown to rely more on shorter-timescale neurons, while the single-head networks rely more on the longer-timescale neurons (whose tiomescales are much longer in the first place). Multi-head networks are more robust to parameter perturbation.", "questions": "- Most glaringly: The authors themselves point out that tau=1 makes the network memoryless. Yet in Figure 4 they show the networks easily solve difficult memory tasks with tau fixed at 1? How is that even possible?\n\n- The initial text suggests that tau and tau_net are independent components (one strictly single-neuron, one strictly network-generated). However, later on it is acknowledged that longer tau causes longer tau_net. Some clarification would be in order.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I'm not aware of this question (whether RNNs learn more by adjusting single-neuron or network-generated timescales) having been addressed previously.\n\n- The analyses are convincing, though some clarifications are needed.", "weaknesses": "I did not see any obvious \"weakness\" in the demonstration. I do note that the tasks and settings are quite specific, and it's not clear how instructive the results are for more realistic domains (this is also noted by the authors in the Limitations)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ippG4AUL6m", "reviewer": "Reviewer_TkDB", "summary": "The authors explore how RNNs derive a solution during curriculum learning for working memory tasks. In particular, in pursuit of functional working memory the authors contrast the optimisation of intrinsic neuronal timescales, as seemingly favoured by single-headed RNNs which are trained on one task, with the optimisation of recurrent weights, as favoured by multi-headed RNNs which are trained on several tasks at once.", "questions": "- In the 4th paragraph of section 2 it is written tau_{net} <= tau, should it not be the other way around?\n- For fig 3a, which task is that? N-parity? The learning curves on this plot also appear to be still going up; it is not also clear whether these N will eventually be learned. \n- Section 3.1: \"N -parity: The network has to output the binary sum (XOR) of the last N digits, indicating an odd or even number of non-zero digits\". I don't understand this; is it that the model should output whether the sum of the last N digits is odd or even?\n- Same N-Parity section: \"the value of the digit presented at t − N needs to be subtracted from the running sum\". I am not such a fan of this description as the authors' are suggesting the underlying mechanism for how the RNN solves the problem is known. Perhaps the RNN is incorporating another solution; e.g. perhaps N running sums are simultaneously encoded in the RNN and are reset every N timesteps. \n- For Figs 3a, 4a, do these results also apply for the N-DMS task? Why are only \n- is k > 1 ever considered? for the multi-head case the authors claim that tau -> k, and this seems true for k = 1; was this also tested for higher k?\n- For Fig 3 was tau fixed? so only recurrent connectivity optimised? This should be made clear\n- is Fig 5a for the single-headed network?\n- For Fig 5d in each task the network timescale tau_net increases up to about N=30 then shrinks. Is this shrink surprising? It is also unclear to me how these multi-headed network can perform the task so well, when their tau_net is overall considerably smaller (about half the size) of the worse-performing single-head networks\n- Fig 6b: I was surprised at how high dimensional these network activities are. In my experience trained RNNs produce quite low dimensional dynamics (< 10 or often < 3). Is this high dimensionality surprising to the authors?\n- \n\nTypos and suggestions: \n- add (AC) after autocorrelation in the Fig 1 caption. \n- In the 4th paragraph of section 2: tnet -> tau_{net}\n- I would appreciate a more formal, mathematical formulation of how tau_{net} was obtained for each neuron in the appendix\n- End of first paragraph of section 4.1, remove comma after K\n- I'd recommend the related work prior to the results\n- I'd recommend all supplementary figures together at the very end. Their appearance amongst the different appendix sections seems quite random/distracting", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is generally well written \n- The authors discover, in my view, an interesting and convincing contrast in the solutions obtained by single vs multi-head learning with respect to intrinsinc vs network mediated timescales. \n- I appreciate the depth of analysis undertaken by the authors with respect to the performance and dynamics of the networks, and generalisability to other network architectures", "weaknesses": "- I think this paper could and should be valuable to neuroscientists, but at the moment the link between the paper results and biology is not so clear. For example, do the authors surmise that optimised, heteregenous instrinsic neuronal timescales should not be employed in the brain? Would there be any experimental predictions one could infer from this work? \n- Related to the above, it is not clear to me if other potential enablers of short-term memory have been considered. For example, there is one experimental work (Hu et al. 2021, PLOS) which comes to mind which shows that short-term plasticity (STP) in neurons, rather than recurrent connectivity, is the more consistent enabler of short term memory in a visual task. Similarly, one computational work (Salaj et al. 2021, Elife) promotes spike frequency adaptation as a powerful tool for sequence processing. I would like to know the authors' opinion on these additional neural mechanisms known in biological circuits. \n- The tasks selected are relatively simple and it is not clear if these results generalise to harder temporal tasks, though this is addressed as a limitation of the study. More generally, I would caution the authors' with the implication that their findings can generalise to \"networks on sets of related tasks instead of a single task\".The authors do demonstrate their case well for the setting of learning different N simultaneously during curriculum learning, but whether these results apply when, e.g. language processing in two different languages, is not clear to me. \n- The motivation for two working memory tasks: N-parity and N-DMS, is not very well presented. What are the differences between the tasks. The authors note that Fig 6b reflects \"distinct computational requirements for each task\" but this is not elaborated upon.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)