Brain insights improve RNNs' accuracy and robustness for hierarchical control of continually learned autonomous motor motifs
Authors: Laureline Logiaco
G Sean Escola
Venue: ICLR 2022 Submitted (2022)
Keywords: neuroscience
dynamical systems
thalamocortical architecture
motor preparation
continual learning
hierarchical continuous motor control
out-of-distribution generalization
robustness

Abstract: We study the problem of learning dynamics that can produce hierarchically organized continuous outputs consisting of the flexible chaining of re-usable motor ‘motifs’ from which complex behavior is generated. Can a motif library be efficiently and extendably learned without interference between motifs, and can these motifs be chained in arbitrary orders without first learning the corresponding motif transitions during training? This requires (i) parameter updates while learning a new motif that do not interfere with the parameters used for the previously acquired ones; and (ii) successful motif generation when starting from the network states reached at the end of any of the other motifs, even if these states were not present during training (a case of out-of-distribution generalization). We meet the first requirement by designing recurrent neural networks (RNNs) with specific architectures that segregate motif-dependent parameters (as customary in continual learning works), and try a standard method to address the second by training with random initial states. We find that these standard RNNs are very unreliable during zero-shot transfer to motif chaining. We then use insights from the motor thalamocortical circuit, featuring a specific module that shapes motif transitions. We develop a method to constrain the RNNs to function similarly to the thalamocortical circuit during motif transitions, while preserving the large expressivity afforded by gradient-based training of non-analytically tractable RNNs. We then show that this thalamocortical inductive bias not only acts in synergy with gradient-descent RNN training to improve accuracy during in-training-distribution motif production, but also leads to zero-shot transfer to new motif chains with no performance cost. Besides proposing an efficient, robust and flexible RNN architecture, our results shed new light on the function of motor preparation in the brain.

Under review as a conference paper at ICLR 2022
BRAIN INSIGHTS IMPROVE RNNS' ACCURACY AND ROBUSTNESS FOR HIERARCHICAL CONTROL OF CON- TINUALLY LEARNED AUTONOMOUS MOTOR MOTIFS
Anonymous authors Paper under double-bling blind review
ABSTRACT
We study the problem of learning dynamics that can produce hierarchically or- ganized continuous outputs consisting the flexible chaining of re-usable motor motifs from which complex behavior is generated. Can motif library be effi- ciently and extendably learned without interference between motifs, and can these motifs be chained in arbitrary orders without first learning the corresponding mo- tif transitions during training? This requires (i) parameter updates while learning new motif that do not interfere with the parameters used for the previously ac- quired ones; and (ii) successful motif generation when starting from the network states reached at the end of any of the other motifs, even if these states were not present during training (a case of cut-of-diisributionn generalization). We meet the first requirement by designing recurrent neural networks (RNNs) with specific ar- chitectures that segregate motif-depender parameters (as customary in continual learning works), and try standard method to address the second by training with random initial states. We find that these standard RNNs are very unreliable during zero-shot transfer to motif chaining. We then use insights from the motor thalam- ocortical circuit, featuring specific module that shapes motif transitions. We de- velop a method to constrain the RNNs to function similarly to the thalamocortical circuit during motif transitions, while preserving the large expressivity afforded by gradient-based training of non-analytically tractable RNNs. We then show that this thalamocortical inductive bias not only acts in synergy with gradient-descent RNN training to improve accuracy during in-training srraiiinnndistibbttinn motif produc- tion, but also leads to zero-shot transfer to new motif chains with no performance cost. Besides proposing an efficient, robust and flexible RNN architecture, our results shed new light on the function of motor preparation in the brain.
INTRODUCTION AND RELATION TO OTHER WORKS
Animals have the remarkable ability to efficiently learn and compose elaborate continuous behav- iors, which often relies the flexible chaining of 'motifs' reproducible bouts of behavior in response to hierarchical commands (Zimnik& & Churchland 2021 Geddes et al. 2018 [errel et al.) 2019b). The mechanisms behind this are however not fully understood, and engineering controllers for dynamical systems that could perform such complex and structured continuous behaviors in the real world has been long-standing challenge of robotics (Brooks 1986 Prescott et 1999; Merel
et al. 2019b). Such tasks involve two different computational operations: first, at coarse temporal intervals, the flexible selection of discrete and abstract action commands by a 'high-level controller'; and, second, the transmission of each abstract command to a 'lower-level controller' that continu- ously produces corresponding command a 'motif" for the motor effector.
Many lines of work have used artificial neural networks (ANNs) to solve the first discrete compu- tational operation (notably using tools from deep reinforcement learning, e.g. (Merel et al.) 2019a 2018 Dennis et al. 2020)) and have leveraged the ability of ANNs to perform well when using a rich training set that includes many motif sequences (Frans et al. 2018 Dennis et al. 2020 OpenAl et al. 202 Xu et al. 2020). However, to the best of our knowledge. in the context of the second computational operation, there is a gap in the literature about how to design ANN mecha- nisms that enable zero-shot transfer to performing new continuous motif sequences from a library of
Frans et al.

---

Under review as a conference paper at ICLR 2022
independently learned motifs. Here we will focus on this latter task, which notably highlights markable human skill, as we can for instance - learn to pronounce a new word and then immediately include it in arbitrary sentences. While this biological relevance has led the neuroscience community to start investigating related questions, this literature has focused on a hand-designed 'bottom-upp approach. This approach studies how networks function in spite of strong constraints introduced e.g. to mimic the brain activity patterns or connectivity, and/or to ensure analytical tractability (Kao al. 2020 Logiaco et al., 2019, (Sussillo et al. 2015 Zimnik & Churchland] 2021; [[sppert et al.) Kulvicius et al. 2012). Importantly, this approach cannot determine whether specific network
2013
features are more generally advantageous for solving a task instead, with this bottom-up approach, these chosen features could reflect constraints that arose through random evolutionary idiosyncra- cies, or could become irrelevant for networks that cannot be designed through analytical insights but that can still be successfully trained with modern machine learning algorithms. Overcoming these limitations has implications for engineering and can provide new ncuroscientific insight.
To address this knowledge gap, here, we use Recurrent Neural Networks (RNNs) whose computa- tional power is not arbitrarily constrained. An RNN is a type of ANN which a generic dynamical system, and which therefore naturally fits the desired characteristics of motor outputs. Consequently, ANNs are indeed regularly used for tasks requiring the production of continuous outputs, including
in the context of robotics (Wyffels & Schrauwen 2009. [Sussillo & Abbott 2009 [Tni] 2003 Liu
eet al. 2019 Merel et al., 2019a] Mahessarannthhan et al.] 2019)). We will examine the ability of RNNs to (i) independently learn motor motif's in order to build a continuously expandable motif library; and (ii) flexibly chain motifss in arbitrary orders (Fig. Ta and see Appendix A..] for a formal definition). In order to better dissect the mechanisms by WhichhNNG can fail or succeed at this task, we focus on the purest form of motor control through dynamics: the production of trajectories with- out needing the external anchor of time deme-depnddent sensory input. We refer to this as autonomous control, which is especially relevant when sensory input is too unreliable (Yeo et al. [0066] Shenoyy (et al. 2013 Brembss 2021), but also in other cases such as the mnooe-menniones speech production. We will show that while it possible to engineer RNNs to independently, extendably and efficiently learn to produce single motifs in response to discrete input commands, these RNNs are limited in their generalization ability during improvisation of motif sequences. We will then use insights from the mammalian thalamocortical motor system- notably the presence of a motor preparation phase before each motif (Zimnik& & Churchland] 2021 Nashef et al. [02 Logiaco et al. 2019). We will show that weaving in these insights into performance-optimizee RNNs leads to both improved mo- tif production accuracy (through positive synergy with single motif gradient-descent training) and
excellent robustness during generalization to motif sequencing.
2 TASK AND ARCHITECTURE DESIGN
Here, we study the ability of RNNs to fulfill the first requirement of our task: the ability to acquire an extendable library of autonomous motifs (Fig. la, left column). However, before describing these analyses, we want to clarify how we chose the motifs that the RNNs have to learn. First, chose motifs of long durations- on the order of a thousand timesteps - so that they strongly leverage the noove-mentioned autonomous capabilities of RNNs. Second, because we are interested in assessing the relative expressive power of the RNNs we study, we have designed two different types of mo- tor motifs so that they constitute two 'difficulty" levels for the RNNs. Following recent analytical approaches for studying RNN dynamics (Schuessler et al. 22020b a Logiaco et al. 2019). we char- acterize the difficulty of motifs as the number of certain basis functions complex exponentials that act similarly as different frequencies of a Fourier transform needed to approximate a motif well through linear combination. Therefore, we define a set of oscillatory motifs that are relatively easy to produce, and a set of more difficult 'step' motifs (Fig. f. see Appendix A.2 for the full list of motifs used in this paper). We train motifs using gradient descent - specifically, ADAM (Kingma & Ba 2015)). Our objective function is the mean square error between desired and actual output.
We will now study how RNNs can meet the requirement to learn a new motif without 'catastrophic interference' with the memory of previously learned motifs- which relates to the literature on con- tinual learning (Kirkpatrick et al. 2017 Parisi et al., 2019: Yoon et al.) 2020 Farajtabar et al. 2019). This line of work emphasizes avoiding interference between gradient updates used to train a network on many sub-tasks, while promoting the re-use of neural resources across sub-tasks (so that the network makes efficient use of its parameters). We will use "architectural" approach to
2

---

Under review as a conference paper at ICLR 2022
Learn Mbraryoffmssi
orders
Add Aidiliie archinccture
Traning all ON... fanasioons
Mininumn
N-300
notE?
Misimum 1000 musa squaee atrar
CIMAE DISN mill arciionctio TTRNAT
Leamn moot with interfeionce
Introduco in soiing with other mottss
Multipiiaatve arch arrchisacuur
N-100
Osoillaing cosy
hard
Trakein sinple
from
motth
Figure I: Task and candidate networks. a) Task. Left: learning an extendable library of motifs without interference. Right: without additional training, stringing the motifs into chains with arbi- trary orders. b) Additive and multiplicative architectures, who may succeed in the task in a because they segregate parameters into motif-specifi sets (schematized in colors) while benefiting from fixed shared recurrent and readout weights (schematized in black). c) Minimum root mean square error over training with overcomplete training set, depending on the gain hyperparameters y _ and go Dots are individual networks, the line is the average. In red, we show the mean minimum root mean square error in the control architecture (averaged over five individually trained networks for each 96"). d) Examples easy oscillatory motif and hard step motif; fitting the latter requires a larger number of basis functions (complex exponentials) of varied oscillation frequencies. e) 'Classical' strategy to promote zero-shot transfer in order to produce a chain of motifs: train ANNs to produce the motifs starting from random initial network states Xinit that emulate the variability of network states at the end of motifs x (where μ indexes the motif). f) Increased inaccuracy during zero- shot transfer chains of motifs compared to when starting from random Xini drawn from the same distribution as during training (additive architecture). g) As in f but for oscillatory motifs. Note that the target trajectory (black) is buried below the output of the network (colored lines), because the latter is almost perfectly accurate with random Xinit-
continual learning that consists in segregating tuned parameters across the different motifs, because it will later facilitate the weaving of biological insights in our networks. With this approach, in- terference is fully prevented during sequential learning; but the proposed architectures may not be efficient, which is what we investigate below. To do so, we will compare these architectures to a 'standard' RNN with no segregation of tuned parameters. This RNN ('control architecture') (i) is fully-tuned, (ii) receives a static input by to instruct the motif, μ (Maheswaranathan et al.) 2019 Sussillo& & Abboti 2009 Tani 2003), and (iii) is trained a plausible noise-robus robust regime imposed by random initialization of its state (sce A.6.3). As expected given that this RNN has no protection against catastrophic interference (Kirkpatrick et al. 2017). we indeed find that when learning motifs sequentially, this RNN immediately forgets previously learned motif's when learning new one (not shown)- even though it is possible to learn motifs by simultancously including all of them in each training batch (Fig. Tb. and see below). This 'control architecture' obeys the following dynamics:
x = - + en J tanh(x) + by
The output y = T tanh(x) = w is produced through the vector w that is initialized from a cen- tered Gaussian distribution with std 1/vNon where here Nzen 50 the number of recurrent units of this control network, so that its outputs have appropriate maximal positive or negative magnitude scaling as VNNai, Also, the recurrent interactions weights J are initialized with iid elements taken from a standard Gaussian as previous work has shown that when choosing a gain of > this leads to rich dynamical regimes appropriate for complex computations (Sompolinsky et al. 1988 Sus-
sillo & Abbott 2009; Sussiillo et al.) 2015) Schuessler et al.] 2020b). We performed hyperparameter tuning on go. The dynamics are discretized using Fuler's integration method where di = 0.17.
3

---

Under review as a conference paper at ICLR 2022
Table 1: Number of neurons and number of parameters in the different architectures
Additive
Multiplicative
Control
# of recurrent units N
300
100
50
# of learned parameters for 10 motifs
3000
3000
3050
# of motif-specific parameters per motif
300
300 (input: 100, loop: 200)
50
We now consider modifications of this control RNN that address this issue of catastrophic forgetting by segregating the parameters involved in learning different motifs while sharing common computa- tional resources across motifs (Parisi et al. 2019:] Merel et al., 2019a b). Here, we will share (i) fixed readout weights (with the above-mentioned centered Gaussian distribution with std 1/vN N as this is sufficient ensure the successful production of our motifs); and (ii) the recurrent weights as they can set rich 'baseline dynamics' that can be modulated by some motif-specine weights. We globally adjust the shared recurrent weights through tuning their above-mentioned gain hyperparameter g (a similar alternative could be to pre-tune these recurrent weights to an original set of motifs and to then freeze them, but as we will see our chosen approach leads to good results).
First, we consider 'additive" architecture (Fig. Dp. top). Here, each motif μ is produced in re- sponse to learning the input vector by. leading to the following dynamics for the network activities:
x = -x x g"d tanb(x) + byy
-
where the input acts as a motif-specific controller of the dynamics. Note that this occurs because the gradient of the loss with respect to the input weights propagates through the recurrent connections whereas if different outputs weights would be learned for different motifs (Jaeger 2007). the dynamics would not be affected by learning.
Second, we consider 'multiplicative' architecture (Fig. Tb, bottom), that is inspired by both previ- ous machine learning literature (Sutskever et al. 201 Schuessler et al. (020b) and the anatomy of the brain's 's motor system (Guo et 2017: Logiaco et 2019). Here, each motif μ is produced in response both learned input vector by and learned rank-one perturbation of the connec- tivity V. The latter is equivalent to a loop through an instantaneous 'unit' receiving input from the recurrent network through the weights V and feeding back through the weights u, Here, this motif-specific loop participates to modulating the dynamics (Logiaco et al.] 2019), in a way that yields more computational flexibility compared to networks with random feedback weights (Sus-) man et al., 2021). Interestingly, learning the full recurrent weights in randomly initialized RNNs can yield low-rank weight update (Schuessler et al. 2020b). Therefore, by imposing that the motif-specific learning is restricted to a low-rank weight perturbation, we expect to get close to full- weight learning while enabling segregation of the learned weights per motif. The dynamics are:
x
- (g + un v.) tanh(x) b,.:
where nTH and J are defined as for the additive network, and un and V are each learned and initialized iid from centered Gaussian with std g/thh/ //N (i.e. expected norm g..h)
To test the bascline relative accuracies of the additive, multiplicative and control networks, we trained them all the possible chains of motifs of length two (excluding repetitions of the same motif) for the ten step motifs (Fig. Th left). Consistent with the continual learning aspect of our task which values limiting the number of parameters that need to be tuned and stored per motif we equalized the number of tunable parameters across architectures (Collins et al., 2017). In- terestingly, after optimizing over hyperparameters (i.c.. gidd n 9% and gin), we found that all networks had similar accuracy (Fig. De. reminiscent of (Collins al. 2017)). This suggests that our strategy of segregating tunable parameters in the additive and multiplicative networks do not lead to drastic decrease in per-tuned- paraamteeee expressivity, while by construction preventing interference when learning motifs sequentially (while the control network suffers from forgetting of previously learned motifs when learning new ones as expected, not shown). On the other hand, we note that the multiplicative architecture, which is closer models constrained to mimic brain dynamics and architecture (Logiaco et al., (209)), appears to have similar accuracy as the additive network while requiring fewer neurons. This echoes recent results suggesting that more poologically-plasiin
4

---

Under review as a conference paper at ICLR 2022
abject-reconnition ANNs tend to have architectures that require fewer neurons (Naybbi al. 2021 We now turn to investigate the robustness these different architectures when improvising motif sequences after being trained single motifs a training strategy which, by construction, enables learning new motifs without interference for the additive and multiplicative architectures.
3 BRITTLENESS OF STANDARD RNNS DURING GENERALIZATION
Here, we ask whether RNNs trained on single motifs can produce arbitrary chains of motifs form of zero-shot transfer where the RNN's internal state differs between training and testing due to the RNN's memory over several timesteps (a similar network mechanism in (Lake & Baroni 2017))
We evaluate a standard technique used to promote both generalization and noise-robustness in ANNs, that relies on leveraging randomness (e.g. (Vezhnevets et al. 2017 Liu & Hodgins 2017 Merel et al. 2019a)) during training. Each motif is trained in isolation but with initial network ac- tivities selected randomly according distribution that approximates the activities at the ends of all other motifs (Fig. [ )... In this scenario, if we can approximate the end-of-motifs distribution well enough and if training succeeds, then all transitions should work with no cransiiio--peeiiie training. Unfortunately, the end-of-motifs distribution is unknown, is training dependent,,, and could be het- erogeneous across motifs. However, since we are using networks with large N, Gaussian weights, and a tanh nonlinearity, it is known that an uncorrelated Gaussian vector can in some cases well- approximate this unknown distribution (Landau & Sompolinsky 2018). Indeed, we observed that a standard normal is a good choice for the marginal statistics (Fig. 2a and b, panels (īi). Hence, tried training the RNNs to generate the step motifs one at time when starting from random initial network activities Xmni drawn from standard normal distribution. We
hyperparameters. to the optima from Fig. I (g 1.4, g 1.4 and g.thh 1.5) and otherwise trained as above. For all architectures, training was successful (Fig. 2 and b panel (i), left; Fig. 2 left).
Additive architecture
b (0)
Multiplicative architecture
From random
From other motifs
From random
From other motils
T
10
(i)
(1))
[ii)
RMSE
RMSE
Cumulativ
Curullative
Gaussoon on id Int. Stetiingg
From other
Geuadinn Fel
From offer
in coining! modss best
(iv)
Cumulatha plobi.
mean RMSSE lover motifs)
RMSE
sodiive indi La No anc cap micors
endii
32
1i motit
(35 In traning))
milher mottls Itest
Canssion Inc. DS6 in triring)
From other monls Dest
Figure 2: Motif sequencing by RNNs trained on individual motifs from random initializations. Step motifs are always used except in panel a) Additive architecture; b) multiplicative architec- ture. Panels (i): Network output (black traces, compared to the desired motif shown in color) when initial conditions Xioit are randomly drawn from standard Gaussian during training when transitioning from other motifs (with example full motif chains shown on the right). Panels (ii): Af- ter training with Xinit drawn from standard Gaussian, 'marginal' cumulative distribution of network states X at the end of example motifs. Panels (iii): for each motif, root mean square еттог taken over 9 random initializations (left) vs. when transitioning from 9 other motifs. P-values from two- sided Wilcoxon signed-rank test. Panels (iv) and (v): same as (ii) and (iii) but when training with Xmi drawn from a Gaussian with standard deviation 1.2. e) Average root mean square error over motifs during training conditions vs. during zero-shot transfer, for different architectures, network sizes, motif types, and variance of the random initialization. All comparisons are significant.
5

---

Under review as a conference paper at ICLR 2022
We then asked our networks to chain two motifs (the first one starting from random initial activities as in training), and found that the performance of the second motif was substantially degraded compared to the same motif produced when starting from random (Fig, 2ub panels (i) and (iii); see also Fig. L right). Fach transition from a given first motif leads to a relatively reproducible output for the second motif because the network variability is low at the end of the first motif. However the second motif's output can vary substantially from the target when following certain first motifs. Moreover, the inaccuracies are not restricted the start of the second motif; rather, the whole output is either clearly shifted or even more grossly wrong. Note that these issues are also present in the control architectures and are therefore not caused by our strategy of segregating the tuned parameters per motif to tackle catastrophic interference (Fig. 2 Appendix A.3.5
Interestingly, the shape of the marginal distribution of the elements of at the end of a given motif (xEM)) was not good predictor of whether transitions from this motif would lead to worse perfor- mance. For instance, in the case of the additive network, the transition from motif to motif 2 leads
to poor performance for motif 2 (Fig. 2 (i), top right), even though the marginal distribution of x at the end of motif appears extremely similar to a standard Gaussian (Fig. 2 (ii), red curve). Con- versely the distribution of x the end of motif 5 is wider than the standard Gaussian (Fig. 2 (ii), purple curve), but the transition from motif to motif 2 leads to a more accurate output (Fig. 2 (i), second chain from top right). This strongly suggests that between-unit correlations impact transition success, which complicates sampling the various x"nd when initializing motifs during training.
Figs, 2h.b panels (ii) show that the standard deviation of the marginal distribution of the values of X slightly exceeds for some motifs. To ensure that the transition failures we observed when initializing with the standard normal were not due this mismatch in scale, we retrained our additive network with initial x values sampled from a Gaussian with std of 1.2 (Fig. 2n panels (iv) and (v)). We observe that the end-of-motif distributions still have standard deviations around for all motifs (Fig. 2 panel (iv)). Despite training with a wider distribution, performance impainment after transitioning still occurred with even slightly greater errors compared to training with a standard normal (Fig. 2 panel (iv) Fig. 2 dashed cyan line).
In addition, we verified that the decreased performance during zero-shot transfer was not conse- quence of using our relatively difficult step motifs for which accuracy is limited even when starting from random. Using our oscillatory motifs instead (as in Fig. [g], we see that performance when starting from random improves significantly, but zero-shot transfer is again unsuccessful with rela- tively large second motif errors during motif chaining (the light blue line in Fig. 2b. notice that the increase of error during zero-shot transfer is of similar amount as for the step motifs).
Finally, we also investigated whether larger networks would somehow be more regularized in a way that would favor zero-shot transfer performance. However, increasing the network size from N = 300 to N = 1000 did not improve the performance during zero-shot transfer, instead, as expected, it only improved accuracy when the network was initialized from random in training (Fig. cyan line vs. dark blue line).
The failure to robustly transition between motifs when training on single motifs that are randomly initialized is particular instance of an out- -istrribbtiiinnonnnnnnn generalization limitation, a general feature of ANNs (Russin al., 2019 D' Amour et al. 2020). Not only is the shape of the true marginal distribution of the values of Xnd at the end of given motif not exactly matched to the statistics of the Gaussian distribution used during training, but we also neglect the correlations and other higher order interactions between the values of x As these complex x statistics are shaped during training, there is a self-consistency issue between initialization during training and resulting x
Note that trying to avoid this issue by including all transitions between motifs in the training set (Fig. [T., left) would scale quadratically with the number of motifs and is thus ultimately prohibitive. Further, even including few motifs transitions in the training set would disrupt the sequential learn- ing strategy that prevents the training set to grow over time in a continual learning setting. Indeed, on the one hand, sequential training would require motif parameters only be trained for transitioning from previously learned motifs which would compromise the robustness of the motifs that were learned earlier, especially if the motifs' properties change over the 'life' of the agent. On the other hand, re-training earlier-learned motifs to improve their accuracy when they are initialized from later-learned motifs causes the training set to grow with the number of learned motifs, and runs the risk of modifying the network's 's oodd-of-moif activity for these re- teairid motifs, which could in turn

---

Under review as a conference paper at ICLR 2022
affect the transitioning from these motifss to any others. Another naive attempt to solve the transition issue would be to train all motifs from a fixed arbitrary network state, and to implement a hard reset to this state before each motif. However, this strategy would introduce detours at transitions in case of discrepancy between two motif's end and start points and would more generally prevent the RNN's dynamies to use the implicit information about the previous motif's shape encoded into its internal state in order to implement a smooth transition to the next motif. As a side note, training single motif's initialized with reset from noiseless fixed state would not be noise robust (Fig. (Fig.L6h), while using a noisy reset actually corresponds to testing and training our networks on random ini- tial conditions- and we will show below that, besides introducing detours at transitions, this naive strategy leads to impaired learning and poorer performance compared to our biologically-inppiee solution. To address these issues, we will now leverage insights from the brain's motor network. This is promising because human behavior demonstrates that the brain has acquired inductive biases that enable zero-shot motor sequence improvisation with agile transitions between motifs.
BRAIN-INSPIRED INDUCTIVE BIASES IMPROVE ACCURACY & ROBUSTNESS
Recordings from the brain's 's motor cortex during flexible sequencing indicate that period of mo- tor preparation precedes every motif in a sequence (Zimnik & Churchland] (02T). This preparation leads the activity to quickly converge towards a motif-specific pattern (Lara al. 2018). This motor preparation process involves interactions between the recurrent motor cortex and thalamus (Nashef) al., 2021) brain region whose neurons quickly respond cortical input and then projects back to cortex, forming feedback loops that effectively act as perturbations of the effective cortical connectivity (Kao et al., 2020 Logiaco et al.) 2019). In a computationally limited linear regime, previous work has shown that this preparatory perturbation can be linked to the optimal control so- lution to bring the cortical activity towards the unique motif-specific stable state instructed by an external input provided to cortex (Kao et al., 2020 Logiaco et al. (019). However, the dynamics of the continuously nonlinear RNNs that we and modern ML practitioners consider (Mahes arrantann
et al.
2019: Wyffels & Schrauwen 2009; Sussillo & Abbott 2009 Tani 2003 Liu et al. [019]]
Merel et al.
2019a) are richer and qualitatively different from this linear regime, and do not allow
setting the parameters through analytics done in the former neuroscience works (Kao al.. 2020
Logiaco et al. (019)). Indeed, in this limited linear regime, (i) single motifs could not be produced in response to simple static input (Logiaco al., 2019) whereas the RNNs can (Fig. 2): (ii spon- taneous activity decays to zero, whereas units are spontaneously active in our networks (Fig. 33, 'prior learning' curves, (Sompolinsky et al., 1988) ); and (ii) single stable fixed point exists, while continuously nonlinear RNNs can exhibit multistability @Maheswaranathan et al. 2019). Therefore, it is unclear whether such a preparatory connectivity perturbation can be successfully trained with gradient descent to shape convergence phase in the more powerful cerformmanceooptimizze contin- uously nonlinear RNNs. Also, to draw conclusions about the functionality and applicability of the preparatory connectivity perturbation, it is critical to determine whether it can act in synergy with gradient descent training of the motif-specific parameters in the continuously nonlinear RNNs. To answer these questions, we first design training protocol for preparatory connectivity perturba- tion (that we call 'preparatory module', Fig. 3 left). This training is done once and for all before learning any motif; details are in A.5. and we summarize the procedure helow. We want to train the weights U Ureep Viee that can shape fast convergence to zero activity in the absence of inpur in non- linear RNNs. To do so, we initialized the network activities with standard random initial x values and trained using the cost function E. ||r(t]| (Fig. 3R), while the dynamics of our networks obey:
7 x + [ U Upeep Trop)) tanh(x)
where g" is فdd or gr for the additive and multiplicative networks respectively. We initialized the weights U Ueep and V prep with centered Gaussian with std /0.05/VPP N, with P 50 (though smaller values of P give similar results as long as P/N is more than a few percent).
Then, with the thalamic preparatory modules in hand (whose weights are then fixed), we trained additive and multiplicative networks on individual motifs with the network activities x randomly initialized with a standard Gaussian, but starting each motif with motif preparation period now also involving a before is that we included the thalamic preparatory module in the network dynamics for the first 57:
input (Fig-3)) For the additive network, our only modification from
" J + 1 1eg5r U prep VT (rep)) tanh(x) + by--
7

---

Under review as a conference paper at ICLR 2022
a
min/ hunboc
oecr 6V
T.E= + Ug
shan
rot with props 1o-100 sir -tutop
Time Inamber
Me-00h stict mourss
maan RAISE Coverr mottisi
TWI Incuts
Time (numbeer 1
Dreat
Gandanic PR:
Foom Doid
d
With proparation
Without proparation
moou Pants Lover morto
mac 8314
maw 3300
man Lul
646
Additivo
mee 2013
mac ao
me 1312
FER 0337
Aippestinosttat si-see motts
Mult.
Figure 3: Improvisation of motif sequences by RNNs with thalamocortical insights.a) Training preparatory module consisting of P loops weights forming a perturbation pepp V prep of the con- nectivity, for a strongly nonlinear network. b) Training the motif-specific parameters in the additive, multiplicative and two-input architectures. c) Average root mean square error over motifs during training conditions vs. during transitioning, in the additive and multiplicative architectures; and with or without preparatory module. Ten step motifs used. d) Example network performance when initializing the network state with the same random distribution as during training, as opposed to when transitioning from another motif. e) Same but comparing additive and two-inputs archi- tectures using more motif transitions. Stars indicate a significant error increase (signed rank test).
For the multiplicative network, to make a more direct comparison with the thalamocortical model (Appendix (..4), we the dynamics such that the input b,, and loop u, VI were only active during the preparatory and post-preparatory periods respectively:
7 = + " 1. 1sgsrUpeeq peep Veopp + 11251 V. tanh(x) + Tegsr
First, the RNNs trained with a preparatory module are much more accurate even during the pro- duction of single motifs initialized with the same distribution used during training, especially for the more difficult step motifs (=s 50%, Fig. Be and d, left). This strongly suggests that, when training with random initialization, the imposition of fast network dynamics by the preparatory mod- ule allows the tuned preparatory input to more efficiently steer the dynamics to explore and find a motif-specific nefwork state that leads to cordingly, we find that the preparatory module speeds up learning (see (sec [A.6.4). After training, the preparatory module also helps converging quickly to the correct motif-specific network state. Con- versely, without the preparatory module, the motif-specific parameters appear to struggle to support accurate motif production on their own (create fast dynamics motif start, shape the pattern towards which activity must converge, and modulate the shared RNN dynamics during motif production).
accurate motif production (Kemeth al., (02T). Ac-
Second, the RNNs trained with preparatory module are now able to tackle zero-shot transfer to chains of motifs at no performance cost, while driving smooth interpolating transitions Cc-e, Fig 14 top right). This strongly argues that the preparatory module shapes a wide attractive land- scape motif start that includes the complex correlated network states reached at the end of motifs.
Third, through nonlinear interactions between the preparatory input and the rich dynamics of the RNNs, our networks can match the motifs' shapes during preparation (Fig. 31, first two rows). This is another advantage of our RNNs over the boooe--mentionel dynamically limited models (which can only implement mootif-independent interpolation during preparation, besides having more limited expressivity (Kao et al.] (2020); Logiaco et al.] (2019) and Appendix [A.4). This enables robust, accurate and plausible motif transitions without any tuning of the duration of the preparatory period.
8

---

Under review as a conference paper at ICLR 2022
Finally, we want to stress the advantages of the preparatory connectivity perturbation U prep Vreep compared to possible alternative approaches that would only try to match the apparent timecourse of the neural activity during preparation in the brain (Sussillo et al. 2013 Zimmik & Churchland, 202I), First, the preparatory loops can modulate the dynamics in a unit-specific way to create efficient convergence with subtle modulation of the RNN connectivity (Fig. (2 This is the reason why the preparatory input can still interact with the rich RNN's s recurrent connectivity, which allows this input to shape the output into the desired motif even during preparation. Second, because these loops are not tuned for any particular motif, they can be optimized once and for all, and do not impose any additional cost when learning new motifs. Instead, when considering naive ways of setting specific dynamics at the beginnings of motifs such as using second motif-specific input at the start of each motif ("two-inputs' approach, Fig. (3,, any parameter used for preparation scales with the number of motifs. We made this 'two-inputs approach comparable to our preparatory module approach by applying the first input the network for the first 5T of the motifs and then a second input during the remainder of the motif, and by equalizing the number of parameters involved (as we had done between the additive and multiplicative network) by setting N = 150. We found that this strategy led to both decreased accuracy and robustness compared to using our preparatory module, with both effects combining to a large ( 40%) increase in RMSE during transitioning. We could precisely assess the transitions robustness in these networks this by using an extended motif transition data set Fig..3). More precisely, we considered transitions between pairs of step motifs where the transition occurs at fractions of 0.5, 0.6, 0.7, 0.8, 0.9, 1.0 of the total duration of the first motif (e.g. Fig. 3 right, bottom). This is a more challenging task- that involves larger variety of network states from which to transition than before (then, we only considered the 100% scenario where all motifs ended at the same value: zero). For these transitions, the 'two-inputs' network showed significant increase in error - including occasional very large errors for some transitions, see Appendix(A.33. compared to initializing motifs with the randomness used during training. In contrast, the additive network had equally robust performance in both conditions (Fig. 3D. Thus, our results argue that using brain insights to constrain the architectures of networks has advantages, even beyond taking inspiration from the biological activity patterns (Li et al. 2019 Sinz et al. 2019 Zador 2019).
5 DISCUSSION
We found that when trained on many randomly initialized single motifs, gradient-trainet nonlinear RNNs struggle during zero-shot transfer to new sequence orders. This is relatively surprising given that such RNNs are known to be able to learn attractive trajectories (Laje & Buonomano) [013]] Sussillo & Abbott/ 2009] Pollock & Jazayeri, 2020). This suggests that training mostly constrains the readout dimension in our setting. This is both a blessing as it is the ultimate reason why dif- ferent motifs can efficiently share parameters and thus benefit from a common dynamical baseline that is only partially modulated, without being hurt by irrelevant aspects of the dynamics (Logiaco) et al. 2019 Russo et al.) 2018)- and curse as it makes flexible motif chaining difficult. In this context, we show that using a preparatory module whose architecture brain-inspired and who im- poses a dynamics that mimics motor cortical activity during hierarchical motor sequencing (Zimnik) & Churchland] (02I) can efficiently enable zero-shot transfer to new sequence orders. Even more surprisingly, introducing the preparatory module in the nonlinear RNNs actually sizably improves single motif training such that the motifs are also more accurate during training conditions. These results reveal general function for motor preparation specifically for flexible chaining of motor motifs, and show that it is not a byproduct of arbitrary biological hardware constraints or of the restricting assumptions made by previous models to allow for analytical tractability (Logiaco al.) 2019 Kao et al., 2020). Therefore, ultimately, for the application to robotics control - at least in a context where intrinsic dynamics matter (Liu et al. [0199] Yeo al. 2016) this preparatory module could be very useful (i) to increase accuracy, (ii) to enable smooth transitions governed by dynamics that can be tuned to avoid passing through undesirable states, and (iii) to offer a mechanism for learning appropriate transition times by gradient descent - which are all current challenges for state- of-the-art networks (Merel et al., 2019a b). In addition, our approach is extendable to combining recurrent dynamics- useful in contexts where sensory feedback is not fully reliable (Yeo et al.) 2016 Liu et al. 2019) with corrections driven by sensory inputs (Guo et al. 2019). Our work therefore joins several recent calls outlining the need to infuse ANNs with more expert domain knowledge in order to achieve good generalization in real- real-world scenarios involving oft-oof-dstrrbbe
general-
ization and zero-shot transfer (Russin al. 2019 D' A'mour et al. 2020 Zador 2019).

---

Under review as a conference paper at ICLR 2022
REFERENCES
Björn Brembs. The brain as a dynamically active organ. Biochemical and Biophysical Research Communicationss 564:55-69, July 2021. doi: 10.1016j..brr.2200.1..000 URL https://
doi
borc. 2020 12. 011
R. Brooks. A robust layered control system for a mobile robot. IEEE Journal on Robotics and Automation, 2(1):14 23, 1986. doi: 00.1109/ja.1186.1187733.. URL /ttp:://doo.oo/
10.1109/3ra. 986 1087032
Jasmine Collins, Jascha Sohl-Dickstein, and David Sussillo. Capacity and trainability in recur- rent neural networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, Frunce, April 24-26, 2017, Conference Truck Proceedings. OpenReview.net, 2017. URL
https://onn /pwprr/v/eeeeeeveeeeeeee eew ew net net//oorumiid-Byddaeee.
Alexander D' Amour. Katherine Heller, Dan Moldovan, Ben Adlam. Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D. Hoffman, Farhad Hormozdi- ari, Neil Houlsby, Shaobo Hou, Ghassen Jerfel, Alan Karthikesalingam, Mario Lucic, Yian Ma, Cory McLean, Diana Mineu, Akinori Mitani, Andrea Montanari, Zachary Nado, Vivek Natarajan, Christopher Nielson, Thomas F. Osborne, Rajiv Raman, Kim Ramasamy, Rory Sayres, Jessica Schrouff, Martin Seneviratne, Shannon Sequeira, Harini Suresh, Victor Veitch, Max Vladymy- rov, Xuezhi Wang, Kellie Webster, Steve Yadlowsky, Taedong Yun, Xiaohua Zhai, and D. Sculley. Underspecification presents challenges for credibility in modern machine learning, 2020.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre M. Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised envi- ronment design. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Bal- can, and Hsuan-Tiee Lin (cds.), Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL /
Abst html
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for contin- ual learning. CoRR, 0ttss/991.07//1 104, 2019. htt::tp::// http:p::/axxvvv..og.
/ttp/////////ttt///////tt/tt//// http://gggabb//ll1/r// 07104
Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 2018, Conference Track Proceedings. OpenReview.net, 2018. net.
URL /attps://openeee hwtp:/wwww.......e..eeeeeeeeeeee
Claire Geddes, Hao Li, and Xin Jin. Optogenetic editing reveals the hierarchical organization of learned action sequences. Cell, 174(1):32-43.d)) June 2018. doi: (0.10166j.ell.2018.066. 06.012
URL 1t www.o/////.1111/jj.cojoooo
Jian-Zhong Guo, Britton Sauerbrei, Jeremy D. Cohen, Matteo Mischiati, Austin Graves, Ferruccio Pisanello, Kristin Branson, and Adam W. Hantman. Dynamics of the cortico-cerehellar loop fine- tune dexterous movement. bioRXiv, May 2019, doi: 10.1101/637447, URL :ttpss///doi..
crg/10.1101/633
Zengcai V. Guo, Hidehiko K. Inagaki, Kayvon Daie, Shaul Druckmann, Charles R. Gerfen, and Karel Svoboda. Maintenance of persistent activity in a frontal thalamocortical loop. Nature, 545(7653) May 2017. doi: 10.1038/mauree23224. URL https::/doi.orgg/
2233//nauure2321
Auke Jan Ijspeert, Jun Nakanishi, Heiko Hoffmann, Peter Pastor, and Stefan Schaal. Dynamical Movement Primitives: Learning Attractor Models for Motor Behaviors. Neural Computation, 25 (2)332-373,, 02 2013. ISSN 0899-7667. doi: 10. 16.1162/ a_t.E//CC__000970 a_00393. URL httpp://doo. org 11//111.1211... 10..112/NNEB. NECO 00393
Herbert Jaeger. Echo state network. Scholarpedia, 209) 2007. doi: 10.4249/scholapecia.
2330. URI.https:://e/t .aceiiorrg/000 wcoo4////chlac arrredia
10

---

Under review as a conference paper at ICLR 2022
Ta-Chu Kao, Mahdieh S. Sadabadi, and Guillaume Hennequin. Optimal anticipatory control theory of motor preparation: a thalamo-cortical circuit model. bioRxiv, 2020. doi: 10.1101/ 2020.02.02.933124.. URL https: /tt:://wwwwt www.::/wioxvvv.ooo http:///o/oootteeeeea///////////
06/ 26/2020.2 02.00.....222..2.....0002.....02 02. 931246
Matthew Kaufman, Mark M Churchland, Stephen Ryu, and Krishna Shenoy. Cortical activity in the null space: permitting preparation without movement. Nature Neuroscience, 17(3):440- 448, February 2014. doi: 10. 1038/nn.3643. URLhttps: //ottps://do... otg/10.......o...... 1038 nn. 3643,
Felix P. Kemeth, Tom Bertalan, Nikolaos Evangelou, Tianqi Cui, Saurabh Malani, and Ioan- nis G. Kevrekidis. Initializing Istm internal states via manifold learning. Chaos: An Inter- disciplinary Journal of Nonlinear Science, 31(9):09311,, Sep 2021. ISSN 1089-7682. doi: 0..1063/5.0055377 URL http:// doi www.o.oogg/.....
0055371
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Reprexentations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URL URLhttp: // Maaxiiv.com wwxiv.org/abe.....ooooooooa ablg://bb//44222 6980
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Baabska-Barwinska. Demis Hass- abis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forget- ting in neural networks. Proceedings of the National Academy of Sciences, 114(13)3521-3526 2017. ISSN 0027- 8424. doi: 1..1073/pnas.16111333 content/114/13/3
URL https: //wb: pnas. org/
Tomas Kulvicius, KeJun Ning, Minija Tamosiunaite, and Florentin Worgtter. Joining movement se- quences: Modified dynamic movement primitives for robotics applications exemplified on hand- writing, IEEE Transactions on Robotics, (3(1):145-157, 2012. doi: 10.1109/TRO.20 200.02011..21166 1.2163863.
Rodrigo Laje and Dean V Buonomano. Robust timing and motor patterns by taming chaos in recurrent neural networks. Nature Neuroscience, 16(7):925 933, May 2013. doi: 10.1038/mn. 3405. http:ttttp..ioooetettt mm1..//00000110mmm33330000
Brenden M. Lake and Marco Baroni. Still not systematic after all these years: On the compositional skills of sequence-to-sequueeee recurrent networks. CoRR, abs/1711..0332 2017. URL http:
00350
Itamar Daniel Landau and Haim Sompolinsky. Coherent chaos in recurrent neural net- work with structured connectivity. PLOS Computational Biology, 14(12)xe0006399 December URL
2018. doi: 10.13711/ wcb.i pcbi. 1006309
Antonio Lara, Gamaleldin F Elsayed, Andrew Zimnik, John P Cunningham, and Mark M Churchland. Conservation of preparatory neural events in monkey motor cortex regardless of how movement is initiated. eLife, 7, August 2018. doi: 10.7 URL https: 7/doi /tgg/1.. 1554/elifee 31826
Zhe Li, Wieland Brendel, Edgar Y. Walker, Erick Cobos, Taliah Muhammad, Jacob Reimer, Mathhis Bethge, Fabian H. Sinz, Xaq Pitkow, and Andreas S. Tolias. Learning from brains how to regularize machines. CoRR, abs/1911.05072, 2019, URL 1911. 05072
Libin Liu and Jessica Hodgins. Learning schedule control fragments for physics-based characters using deep q-learning. ACM Transactions on Graphics, 36(3):1-14, July 2017. doi: 10. 145/ 3083723. URL htttts://dd..:: 0ttpe/oor//11.11/303377777
Vincent Liu, Ademi Adeniji, Nathaniel Lee, Jason Zhao, and Mario Srouji. Recurrent control nets for deep reinforcement learning. CoRR, abs/ 1901.9199, 2019. URL aatp:://aaaaa orq/ abs/ 901 01994
11

---

Under review as a conference paper at ICLR 2022
Laureline Logiaco, L.F. Abbott, and Sean Escola. A model of flexible motor sequencing
through thalamic control of cortical dynamics. bioRxiv. 2019. doi: 10.1101/2019.12.17. 880153. URL https: :/wps../www... www.biioxxvv. wwtp//ioordv.ooo/oooooeeeeteee htt.://wwww.....o...eo/aa//////. 12. 17. 880151
Niru Maheswaranathan, Alex H. Williams, Matthew D. Golub, Surya Ganguli, and David Sussillo. Universality and individuality in neural dynamics across large populations of recurrent networks, 2019.
Josh Merel, Arun Ahuja, Vu Pham, Saran Tunyasuvunakool, Siqi Liu, Dhruva Tirumala, Nicolas Heess, and Greg Wayne. Hierarchical visuomotor control of humanoids. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019a. [RLLattps://ppeerrrrii http:/neeeeeeeteeeeeeeeeo Fort://orm?aaa wtt.://ooooonddddoooooooooo
Josh Merel, Matthew Botvinick, and Greg Wayne. Hierarchical motor control in mammals and machines. Nature Communications, 10(1), December 2019b. doi: 10. 1011000/1466--000 13239966.
Nima Mohajerin and Steven L.. Waslander. Multistep prediction of dynamic systems with recurrent neural networks. IEEE Transactions on Neural Networks and Learning Systems, 30(1 1):3370- 3383, 2019. doi: 10.1
Abdulraheem Nashef, Rea Mitelman, Ran Harel, Mati Joshua, and Yifat Prut. Area-specific tha- lamocortical synchronization underlies the transition from motor planning execution. Pro- ceedings of the National Academy of Sciences, 11860))20126581888 February 2021. doi: 1..0733ppas.201266333
URL pistps://6est http:////eee...01111011111100
201265811111111111
Aran Nayebi, Javier Sagastuy- Brena, Daniel M. Bear, Kohitij Kar, Jonas Kubilius. Surya Ganguli, David Sussillo, James J. DiCarlo, and Daniel L. K. Yamins. Goal-driven recurrent neural network models of the ventral visual stream. bioRXiv, February 2021. doi: 10.1 (0.1100/2021.2.10 02.222....1111 17.447777 URL httpt::://// /ttp::/doooooooo www.do...og..oooo
000001222000 02. 17.11.7. 431717
OpenAl OpenAl, Matthias Plappert, Raul Sampedro, Tao Xu, Ilge Akkaya, Vineet Kosaraju, Peter Welinder, Ruben D'Sa, Arthur Petron, Henrique Ponde de Oliveira Pinto, Alex Paino, Hyeonwoo Noh, Lilian Weng, Qiming Yuan, Casey Chu, and Wojciech Zaremba. Asymmetric self-play for automatic goal discovery in robotic manipulation. 2021.
German I. Parisi, Ronald Kemker, Jose L.. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, May 2019. doi: 10.1016/j no.101//..cuuee220199 neunet.2019.01.012. URL https:///oi.org/// 101.101161111 www.eeee.........meeeeeoeeeeee.. 2019. 01.012
Eli Pollock and Mehrdad Jazayeri. Engineering recurrent neural networks from task-relevant mani- folds and dynamics. PLOS Computational Biology, 16(8)= 128, August 2020. doi: 10.1371/ journal.pcbi. 1008128. URL https:ts:://dot
.wt.innallpb......... 1008118
Tony J. Prescott, Peter Redgrave, and Kevin Gurney. Layered control architectures in robots and vertebrates. Adaptive Behavior, 7(1):99-127, 1999. doi: 101117/10597119990000001 URL
Jacob Russin, Randall C. OReilly, and Yoshua Bengio. Deep learning needs a prefrontal cortex. In Yoshua Bengio and Yann LeCun (eds.), 8th International Conference Learning Representa- tions, ICLR 2015, San Diego, CA, USA, May 7-9, 2020, Workshop on Bridging Al and Cognitive Science, 2019.
Abigail A. Russo, Sean R. Bittner, Sean M. Perkins, Jeffrey S. Seely, Brian M. London, Antonio H. Lara, Andrew Miri, Najja J. Marshall, Adam Kohn, Thomas M. Jessell, Laurence F. Abbott, John P. Cunningham, and Mark M. Churchland. Motor cortex embeds muscle-like commands in an untangled population response. Neuron, 97(4):953 966.c8, February 2018. doi: 1016// neuron.2018 2018.00 10.01100044.....0 URL htLhttts: htappp:://doioo //oo .tw.i.org/1000 6/ neuron. 201 8.01 .0.000
12

---

Under review as a conference paper at ICLR 2022
Friedrich Schuessler, Alexis Dubreuil, Francesca Mastrogiuseppe, Srdjan Ostojic, and Omri Barak. Dynamics of random recurrent networks with correlated low-rank structure. Physical Review Research, 2(1), February 2020a. doi: 10.1 h...0///pyycerrsserch22000333 httttattsssttttt::t...... URLhttps://aol
1155/10.110311 htce///...11///psssssveeeeeeeee 2. 01 3111
Friedrich Schuessler, Francesca Mastrogiuseppe, Alexis Dubreuil, Srdjan Ostojic, and Omri Barak. The interplay between randomness and structure during learning in rnns. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33. pp. 13352- 13362. Curran Asso- ciates, Inc., 2020b. URL
neur ips pccppperr/222/fi1ee pdf
K. V. Shenoy, M. Sahani, and M. M. Churchland. Cortical control of arm movements: dynamical systems perspective. Annu. Rev. Neurosci, 36:337-359, Jul 2013.
Fabian H. Sinz, Xaq Pitkow, Jacob Reimer, Matthias Bethge, and Andreas S. Tolias. Engineering less artificial intelligence. Neuron, 103(6)967-977, September 2019. doi: 10.1016/j (0.1016/j.neuron. neuron. 2019.08.034. URL /attps::/doi..oo http://1111111....o.. neuron. 201 9.19.00 08.08..4
H. Sompolinsky, A. Crisanti, and H. Sommers. Chaos in random neural networks. Phys. Rev. Lei.. 61:259-262, Jul 1988. doi: 10. 1ww//hyseeeeeet..1.1....
URL htt.::ttt////t /ink.ii....... aps.
61. 61.259
Lee Susman, Francesca Mastrogiuseppe, Naama Brenner, and Omri Barak. Quality of inter- representation shapes learning performance in feedback neural networks. Physical Re- view Research, 3(1), February 2021. doi: 10. 10.1103/1 1000/pyysevrreearcb..00111 //doi. www/orr./.......... 11.0..113000 www.:///p/peeeeeeeear....
URL https:
3131777
D. Sussillo, M. M. Churchland, M. T. Kaufman, and K. V. Shenoy. A neural network that finds naturalistic solution for the production of muscle activity. Nat. Neurosci, 1823,,,05,, Jul 2015.
David Sussillo and L.F. Abbott. Generating coherent patterns of activity from chaotic neural networks. Neuron, 6344)5544-557, August 2009. doi: hw0.://.nuuron.2009.00
URL
https: http:s///oooooggg1111111.//.... new..o...eeee..ooooooooo. 2009. 07 07.007011 018.
Ila Sutskever, James Martens, and Geoffrey Hinton. Generating text with recurrent neural networks. In Lise Getoor and Tobias Scheffer (eds.), Proceedings of the 28th International Conference on Machine Learning (ICML-11), ICML '11, pp. 1017-1024, New York, NY, USA, June 2011.
ACM. ISBN 978-1-4503-0619555
Jun Tani. Learning to generate articulated behavior through the bottom-up and the top-down inter- action processes. Neural Networks, 16(1):11-2) January 2003. doi: 10.101 (016///893-60000000 60000(02) 6080 (C2) 00214-9
00214-9. URL http:::://o...oro///////
Alexander Sasha Vezhnevets, Simon Osindero. Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference
on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 3540-3549. PMLR, 2017. URL http:
7 p/ppceediiggs... mlr http://wwnn.......pe.ssssseeee wttpss//w//veeheeeeeeee 1.t.tmmmmm
Francis Wyffels and Benjamin Schrauwen. Design of a central pattern generator using reservoir computing for learning human motion. In 2009 Advanced Technologies for Enhanced Quality of Life, pp. 118-122, 2009. doi: B0.11000//T
Kelvin Xu, Siddharth Verma, Chelsea Finn, and Sergey Levine. Continual learning of control primitives : Skill discovery via reset-games. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 4999-5010. Curran Associates, Inc., 2020. URL htt UBL//atppee//prooeeddnng neurips cc/
13

---

Under review as a conference paper at ICLR 2022
Sang- Hoon Yeo, David W. Franklin, and Daniel M. Wolpert. When optimal feedback control is not enough: Feedforward strategies are required for optimal control with active sensing. PLOS Computational Biology, 1005110 December 2016. doi: 10.137 foi 190.. URL htt.hhttp:://ootoo. www.:oo..ogglooooo jouruna journal. papernall..b...... 10051 90
Jaehong Yoon, Saehoon Kim, Eunho Yang, and Sung Hwang. Scalable and order-robust con- tinual learning with additive parameter decomposition. In International Conference on Learning Representations, 2020. URL
Anthony M. Zador. A critique of pure learning and what artificial neural networks can learn from animal brains. Nature Communications,, 10(1), August 2019. doi: 10. 10.10088641467-01--1177883 URL [ttp://pp//....oo//////////11111
Andrew J. Zimnik and Mark M. Churchland. Independent generation of sequence elements by motor
cortex. Nature Neuroscience, 24()):112222244 February 2021. doi: 10. 10388/441599-0001000799555 URL
00798-
A APPENDIX
A.1 FORMAL DEFINITION OF OUR PROBLEM SETTING
Here is how we formally define the task solved in our paper:
Given a dynamical system D with (i) internal variables (i.e., h(t) is defined through its derivative with respect to the continuous time variable t) and (ii) the property that 'D can be in one of M different states indexed by l (with μ E I... M, and M is large and a priori unknown), each lasting fixed duration ly. such that whenever D is in state μr, function of the internal states F (h) exactly follows a fixed continuous function 9μ (of length t.)) that we term a 'motif". (Concretely, if a particular trajectory of states [$2, $1, $1, $3] is visited, F (h(t)) is formed of the concatenation of three continuous 'motifs' indexed by [82, 81, 83).)
Given sequence of indices I = [p.,..pk] that label sequence of states undergone by D, where Vi,p; E (1)
Given labeled training examples that only contain finite number Ta of discrete samples from each continuous function 9u. and given that the learning from different 9p can occur sequentially and extendably without interference
Find a function ff which, for each function gk. is parameterized by a vector Vj of size Ny < T (where Ny is small possible) such that f(1; Vi,... VM, dt) produces time series cor- responding to F (h) when D undergoes the sequence of states ordered by 1, with an adjustable sampling interval given by di (such that smaller dl leads smoother time series).
Finally, we note that is also desirable that f be noise-robust.
In our work, we solve this problem using an RNN to implement the function f- natural choice given that it models a dynamical system D. In addition, the tunable motif-specific parameters Vy either the input b, for the additive network, or the concatenation of the input b and the loop parameters u, and v in the multiplicative network. As we mention in the text, given that we work a continual learning setting where we would like learn an extendable library of many motifs, we are trying to limit the number of tuned parameters per motif Nv (table []. This is important for our networks to constitute viable solutions that can be built on for designing motor controllers, because it is then desirable to limit both the memory requirements and the time needed to update the state the network.
Note that we ensured noise-robustness in our work by injecting noise during training in the hidden state of our RNN (which, by a simple transformation of variable, is equivalent to injecting a trans- formed noise into the input b, received by the network). The larger component of the noise used consists a random initialization of the network state, and this noise then propagates to later timesteps through the recurrent dynamics (see section A.6.3
14

---

Under review as a conference paper at ICLR 2022
A.2 MOTIF GENERATION METHOD AND FULL LIST OF MOTIFS USED
For this study, we use two different types of motifs that challenge the dynamics of continuous time RNNs: oscillatory motifs, that are easier produce; and step motifs that are harder.
A.2.1 OSCILLATORY MOTIFS
To generate oscillatory motifs that would be relatively easy to produce by RNNs, we sampled the output from a random Gaussian network with tanh nonlinearity set the chaotic regime (Sompolin- sky et al. 1988; Sussillo & Abbott 2009) Sussillo et al. 2013). To make these motifs very easy and possible to connect in 'seamless' chains, we also low-pass filtered the resulting trajectories.
Fig. lists the oscillation motifs used in this paper.
moti:
mobif:
-2-
motif
moti:
motif:
mati:
metif:
mati:
-2
-2
0.6
0.4
0.5
Soconds
seconds
Figure Oscillatory motifs used in this paper.
A.2.2
STEP MOTIES
These motifs consist of series of positive and negative discrete jumps with intervening constant periods. To generate a particular motif, we follow the following steps. First, we generate a centered
CeastiinUhhlebeck process z by rumning the dynamics dz/dt - (2/vat dt x (t), where (t) is taken from a standard Gaussian independently at each timestep, dt=0.1 and y = 3 such that the steady-state standard deviation of this process 3. After 'warmup' period of 200 di that
15

---

Under review as a conference paper at ICLR 2022
discard, produce additional 1000 dt of z-values that will be the basis for generating the motif. Second, we draw random time intervals TĄ with a uniform probability between 50 dt and 500 dt and select the first few in a list [Ti, 72, T2..... T3] such that Ex=1 Th < 1000 dt. Third, we set the value of the motif between the start and time Zi the average of the z-values over the same time interval; and similarly for the subsequent intervals, the value of the motif between the times [T--1 dt, [e......... is set the average the z-values over the corresponding interval. Finally, we pad the end of the motif with zeros for 50dt and also reset the first value of the motif to O.
Fig. 5 lists the step motifs used in this paper.
motif:
motif:2
-2-
motif:3
motif
motif:
motif.
motil
mottt &
notif:
motif: 10
-2-
0.2
0.6
0.6
0.2
0.4
0.0
spconds
seconds
Figure 5: Step motifs used in this paper.
A.3 PERFORMANCE OF 'VANILLA' RNNS
Here we show examples for the motif-by-moti performance different "vanilla' RNNs (i.e., with- a preparatory module) that relate to section 3 of the main text, where network training involves producing single motifs from a random initialization of the network state. The networks are tested both on new random initialization (in dindiistributin generalization) and during motif transitioning -ittof--ditribution generalization..
A.3.1 ADDITIVE NETWORK, N=300, STEP MOTIFS
In Fig. 6 we show an additive network of size 300 trained on the step motifs. This is a different network than the show in the main text Fig. 2a (i)-(iii) but the conclusions are identical
16

---

Under review as a conference paper at ICLR 2022
despite the presence of small variability between different networks trained from different initial seeds (see e.g. main text Fig. Ic).
Whes
me: 3.46
630
3251
new: 175
330
Teu (rarbe
Tran Jonner
(rambr
Figure 6: Performance of an additive network, N=300, step motifs. Left: the network outputs for each motif when starting from different random values. The saturations (light dark) indicate different trials. Right: network outputs for each motif when starting from the x values taken at the end of the other 9 motifs. Colors indicate the identity of the prior motif (as labeled in the left panel). Numbers indicate the root mean square errors (mmse) for each motif (and their average over motifs).
A.3.2 MULTIPLICATIVE NETWORK, N=100, STEP MOTIFS
In Fig. show multiplicative network of size 100 trained on the step motifs. This is different
network than the one we show in the main text Fig. 2b but the conclusions are identical - despite the presence of small variability between different networks trained from different initial seeds (see e.g. main text Fig. Ic). Note that the number of parameters tuned for motifs is adjusted relative to the additive network of size 300 shown previously (main text table 1).
When stating fuom candor
Wiene staring foom ciler eet
136
Time Oueber
Tire Jumber et:c
Time oranber of
Tres (umber of
Performance of multiplicative network, N=100, step motifs. Conventions in Figure 6
Figure 7:
17

---

Under review as a conference paper at ICLR 2022
A.3.3 ADDITIVE NETWORK, N=1000, STEP MOTIES
Fig- S we show an additive network of size 1000 trained on the step motifs. Notice that while increasing the network size improved the performance when starting from the random initialization used during training, there are still large errors when transitioning from other motifs.
When stating form rancor
When staring foom the mst
2000
3,49
$10
3443
1.55
Tiu (farer
Tiee Juanbrr
Tee (fanler
Tran Jainber of
Figure 8: Performance of an additive network, N=1000, step motifs. Conventions as in Figure
6
A.3.4 ADDITIVE NETWORK, N=300, OSCILLATORY MOTIFS
In Fig. g we show an additive network of size 300 trained on the oscillatory motifs. Notice that while, with these easier motifs, the performance is really good when starting from the random ini- tialization used during training (as compared the performance for the step motifs), there are still large eπrors when transitioning from other motifs. Also, we want to stress that, during sequencing, even when focusing on particular first motif, the network performance can be very different de- pending on the identity of the second motif. For instance, let's focus on the light green motif in the panel 'When starting from random', bottom row on the left the sequences where this motif the first motif are shown as green traces in the panel 'When starting from other motif's, When this green light motif is the first motif in a sequence, the performance is almost flawless for following the motif in the panel 'When starting from other motifs', second line the left. In contrast, the performance is very poor for another following motif situated in the panel 'When starting from other motifs', fourth line the right.
A.3.5 CONTROL NETWORK, N=50, STEP MOTIFS
In Fig. I0, we show a control network of size 50 trained on the step motifs. Like the additive and multiplicative networks, this control network struggles when generalizing from single motif training to motif sequencing.
A.4 ANALYTICALLY TRACTABLE THALAMOCORTICAL MODEL
Here, we summarize the insights that can be gained from an analytically tractable thalamocortical model whose switching linear dynamics are constrained to stay in the linear regime during individ- ual motif preparation and exccution (Logiaco et al. 2219), see also (Kao et al. 2020 Importantly, the constraints on the dynamics of this model largely limit its computational capacity, and on- line learning method has been suggested for this model so that the approach is strongly limited to simple scenarios where analytical tractability is possible. The work set forth in the main text is therefore critical to test and expand the idcas of this model in a setting that is amenable to solving useful tasks in an engineering sense, and for determining whether the biological features included in
18

---

Under review as a conference paper at ICLR 2022
When starting from random:
when staring from other moltls: 0.211
0.107
mas
mra G323
mas 0.06
OREM 0Z21
0.122
C322
mia
0328
0.07
0228
0.00
Time biumber
Time tnumber
Time thumber of
in
thumber
Figure 9:
Performance of an additive network, N=300, oscillatory motifs. Conventions as in Figure
When starting from random:
When stariin from other motlls:
GLO
6128
0.0
0224
0.00
mas 0257
mas 0.26
1/30
mia 0.30
Joumber
Inumber
jnumber
(number
Figure 10: Performance control network, N=50, step motifs. Conventions in Figure 6
this model might have been selected as generally useful rather than reflecting random evolutionary idiosyncracies.
The model consists of a recurrent cortical module with connectivity J and activities x whose projection through the readout weights constitutes the output. This cortical module interacts with a non-recurrent thalamic module through instantancous loops consisting of corticothalamic and thalamocortical weights. The basal ganglia, which provides inhibitory input into thalamus, are modeled as selectively disinhibiting specific thalamic loops in order to cause execution of the associated motif.
Motif execution: During motif μ., a single thalamic loop is disinhibited leading to the dynamics:
7 = J,x, where J g cJ + V.
with motif-specific loop vectors u. and vJ.. We now consider how these dynamics can approx- imate desired output y,. knowing that in general good approximation for yy, can be reached through a (preliminarily numerically identified) linear combination of small number K of com-
plex exponentials: (t) (1) _ @
(see main text Fig. 1d: (Logiaco et al.
19

---

Under review as a conference paper at ICLR 2022
a
ii moit 2. ppentart matro Cencned
Gaussian lid Initialization
Transitioning from other motifs (test)
cncdle Rom wwweeradeeappee.. sppiic reakied Fravedio Josalound
192
162
02
WE
Troe (number
602
the vealde
O perturt matti
dallical
19
698
Nom of
1989
Real (M
joumber of
637
12
Time
Inumber
Inumbermerrr
mean RMSE (over motifs)
0.4
0.4
RMSSE
0.5
9.5
0.1
0.1
Gousson
Gaus9on
Figure 11: Robust transitions in thalamocortical model. a) Adjusting motif-specific loop through the thalamic unit to (i.e. motor preparation, left), leading to the control of both eigenvalues (middle) and eigenvectors the dynamics x(t) such that the readout robustly follows motif 2 (right). b) Thalamic module used for all motif transitions (which involves preparation of the cortical state to execute the next motif, left). After optimization, the thalamic module sets the eigenvalues of the dynamics to be more negative (middle) which results in fast decrease of the distance to steady- state |óx| (right). c) Example sequences. d) Example trials starting from standard Gaussian random initial conditions each motif between random initialization and zero-shot transfer to motifs transitions (not significant per signed rank test). f) Comparing the performance between the thalamocortical model and the models presented in the main text, for both random initialization and zero-shot transfer to motifs transitions.
when transitioning from other motifs. e) Change of root mean square error for
). The cortical readout can exactly match dr if the eigenvalues of Ju contain the entries of the vector d and the initial network activities xinit are set correctly. We accomplish the former (Fig. 11 by setting V, = LT dia -'Q 1, where L is the left eigenvector matrix of J and Quy = 1/ Xÿ) where Xj is eigenvalue of g (J I). Next, set the initial activities the beginning of motif μ xii" R2 diag(R..w) a,, where R,, contains right eigenvectors of
j,, (with the first K columns corresponding to the eigenvalues in Åμ), and [ox]]e< [ár,] and [α,]<>>= 0.
The preceding two steps do not specify u, and with random uy the readout will be highly sensitive to noise in xinit (pink trace in Fig. 11p right; see (Logiaco et al. 2019)). However, if u, is set minimize the analytically-computed expected readout deviation due to noise in xint by modify-
20

---

Under review as a conference paper at ICLR 2022
ing the eigenvectors of the dynamics, then robust readout is possible (cyan trace in Fig. 11a right; minimization of the cost C(u) in (Logiaco et al. 2019)).
To successfully transition to motif it is sufficient to implement a mechanism by which x approaches x which will be the case if the dynamics during so-called "preparatory period" has x as its steady-state. Additionally, it is desirable that the transition dynamics are fast and that they do not cause large transients values the readout while relaxing to steady-state (Kauf-) man et al., (014)). To achieve this, it is possible (Logiaco et al. 2019) to employ a specific thalamic subpopulation of size P which disinhibited during all motif transitions, well as a constant input Cn specific to the upcoming motif je, leading to the dynamics:
Motif transitions:
T = =prpps prep + Cus where J prep = (J U prep VT
(2)
prep
with N x P loop weights U Upeep and V prep- With these dynamics, the activity at steady-state will match xiiit if C₁ = -J pepp (Fig. Iíp). Note that the difference 6x = x - xinir between the cortical activities and their steady state decays a rate that is independent of Cy. and therefore of the upcoming motif: 7ox- = pmpp 6x for all p. This allows us to optimize the same weights Upep and V prep to favor rapid and smooth transitions between all pairs of motifs even though this approach limited: this optimization has to be motif-indeppndent, so that the shape of the readout cannot be do this, see Fig. Dd), Following ref. (Logiaco et al., 2019). we achieve fast transitions by minimizing the time-integres of the expected square norm of ox, with rates 8x, at the beginning of the transition period sampled iid. Here, we also augment our cost function with the time-integral of the expected squared derivative of the readout favor smoother transitions average. Our total cost function is therefore:
adjusted to the specific motif being prepared (while the continuously nonlinear RNNs
C(U peep= Vpccp) = Esxo [. dt [|xx|]
S1 (' w Ex
+ 3BN Esxs
(3)
x Tr (Rerep (IIprep pepp))
@ A) R prp) +BN w Reepp ((Lrrc LTep)
+ оT) R w
where Rycp and peep are the right and left eigenvectors of J paop- and its eigenvalues Aprep are used
to compute Aij - 11/(ptpp + +Xpepp AgreP) and i = X, grr X Aij. Finally, N is the number of cortical units and / is a hyperparameter which trades off the relative importance of transition speed and readout smoothness.
prop
Note that the model parameters can be adjusted through analytical and semi-analytical techniques which do not require stochastic gradient descent on the simulated dynamics. This is an advantage when strictly imposing linear dynamical regimes during each motor preparation and motor execution for the fixed type of autonomous computation that was designed for, but these restrictions can be limiting to optimize the network for more complex objectives (for instance, including correcting responses to sensory feedback).
We simulated the model on the same task as the gradient-trained RNNs, using our step motifs. The motif-specific parameters scale as in the multiplicative architecture (main text table 1), but the thalamocortical model does have a few more hyperparameters. To make sure that these did not induce an inability to compare between approaches, we reduced the cortical size to N = 99. Further, after exploring a few values, we set g/T = 0.5, K 10, P 50, 3 = 1 1/20, and the motif transition duration to 57. The readout weights w and recurrent weights J were sampled from a centered Gaussian distribution with std I/VN. The approximations ري were fit to the target motifs under the constraints that 9,. (0) O, that the elements of Xष had negative real part and were at least c = 0.05 apart from each other, and that the magnitudes of the elements of å were not exceedingly large (no larger than 3). The resulting Rn and âG were then used to optimize Up. Vy,, and Cp as described above.
We generated sequences from the thalamocortical model after initializing with iid standard Gaussian samples for the elements of x (this choice of unit standard deviation indeed leads to readout values within the same range the target motifs, Fig. I... Interestingly, the motif' were produced with the same reliability when preceded by another motif in a sequence or when starting from random initial conditions (Fig. IId.e). Also, choosing K 10 pushed the network close to its limit in terms of noise robustness, without allowing the constrained thalamocortical network to reach the performance of nonlinear RNNs with the preparatory module (Fig. 11)..
21

---

Under review as a conference paper at ICLR 2022
In conclusion, the constrained thalamocortical model is therefore limited in its expressivity, in its pacity to shape the readout during motif transitions, as well as by the rigid and unpractical procedure by which its weights are adjusted.
We will now present additional results demonstrating that insights from this constrained thalamocor- tical model can be weaved into RNNs to combine zero-shot transfer ability with large expressivity and the flexibility afforded by gradient-base training.
A.5 WEAVING IN INSIGHTS FROM THE THALAMOCORTICAL MODEL IN GRADIENT-TRAINED ANNS
In Section 4 of the main text, we presented the results of augmenting each of the additive and multiplicative networks with a 50-umit thalamic transition module that is active for the first 57 of every motif. Here we present more detail about these results.
A.5.1 DYNAMICS OF THE RNNS INTERACTING WITH A THALAMIC TRANSITION MODULE
The dynamics of our nonlinear recurrent networks with the transition module but without any input are:
- (g" + prep prou) tanh(x),
where gd is gdd or gtt for the additive and multiplicative networks respectively. The weights in Upeep
and prop were initialized with centered Gaussian with std / /0.05/ We trained the weights of the transition modules with ADAM under the cost function E, |r())2 where r(I) = tanh(x(()). Minibatches consisted of 64 trials of length 20r, each starting with random x values sampled iid from the standard normal distribution. After 1,000 minibatches, both N 300 and N 100 networks were seen to have converged.
Figs. 12a.d show the eigenvalue distributions of J prep Vrepp after training. Importantly, though the thalamic module is low rank (P < N), all eigenvalues have real part significantly less than which causes decay the network rates towards a fixed-point in the vicinity of this fixed- point. For comparison, we show the eigenspectrum of J + U peep prep for random matrix Jad which has the same statistics as In this case, there continues to be large eigenvalues that will prevent fast rate decay. These results demonstrate that the solutions U prep and V prep needed to negate the amplifying dynamics of J are specific to that particular
In Figs. 12p.e, we show the time evolution of the norm of the rate vector during three sample trajectories prior to learning and one post-learning (all post-learning samples are nearly identical). On the scale plotted, the norms after learning are indistinguishable from zero after approximately 7r.
With the thalamic transition modules in hand, we retrained the additive and multiplicative networks as described in the main text Sections 2 and 3. For the additive network, our only modification from before is that we included the thalamic transition module in the network dynamics for the first 57:
( J Li55T U prep prep) tanh(x) + b
For the multiplicative network, to make a more direct comparison with the thalamocortical model presented in the previous section of the Appendix, we set the dynamics such that the input b, and loop u V. were only active during the transition and post-tram cost-transition periods respectively:
7 = -x (g "J + 1.557 U Ppeup Viep + Less7 Un v.) tanh(x) + Legsr
Figs. 12 show that both networks learn to perform the task and show no degradation in their performance when tested sequence generation (i.e.. having initial values given by the ends of other motifs) rather than when starting them with standard Gaussian x values (which was their training regime).
Similarly, Fig. 1 shows that training on smooth motifs using a preparatory module leads to robust performance during zero-shot transfer to motif chaining in example additive network (compare to Fig. 99. Notice how the shape of the motif is also matched during the preparatory period, which is enabled by the interaction between the motif-specific input b, and the nonlinear tanh dynamics.
22

---

Under review as a conference paper at ICLR 2022
(19--117
Plstaaaiio Plat Phar-tomp Tomph
Additive
(ver 11M
yeat
4-
Multiplicative
Tax Jurr
Using thalamic transition module rescues transitioning for nonlinear RNNs trained with SGD. a,b,c Additive model. a. Eigenspectrum of gdd + U Upep Vrrpp after training of Urep and V prep (orange crosses) and when replacing J with random matrix not used during training (blue dots). Black circle has radius ged.. b. |r] versus time before (dotted lines) and after (solid line) training of prep and V prepp- c. The grey bars indicate the time during which the transition module was active. Other conventions in Figure 6 d,e,f. As a, for the multiplicative model.
Figure 12:
starting
0.067
0.040
cm
1914 0,300
Yow Fine
0.74
Time Jtumbor of ol
Time tnumber of 14)
Time trumbor of tf
Time inumbor of
Figure 13: Using a thalamic transition module rescues transitioning between oscillatory motifs in an additive ANN. Conventions as in Figure 6
23

---

Under review as a conference paper at ICLR 2022
A.5.2 COMPARING THE THALAMIC TRANSITION NETWORK TO OTHER WAYS OF IMPLEMENTING A DIFFERENT DYNAMICAL REGIME AT TRANSITIONS
In the main text, at the end of section 4, we gave statistics about a comparison between using transition module described above and an alternative approach for setting the RNN different dynamical state at the beginning of a motif. More specifically, we considered 'two-inputs' ap- proach where we use a first motif-specific input during the first 57 of each motif to try to facilitate transitions, and second motif-specifis input during the rest of the motif to facilitate the production of the motif by the network.
Following the general approach in our article which adjusts the number of parameters tuned per motif (main text table 1), we set the size of the recurrent network to = 150 so that the two inputs together lead to 300 parameters tuned per motif (and therefore 3000 parameters tuned in total when learning the ten motifs).
We used an extended data set consisting of transitions between pairs of step motifs where the tran- sition occurs at 50%, 60%, 70%, 80%, 90%, or 100% of the total duration of the first motif. This is more challenging task than before when we only considered the 100% scenario because all motifs end at the same value (zero). As stated in the main text, we found that the 'two-input" network did not perform as well as the matched additive network with transition module (as described in the previous subsection of the Appendix. this module is not tuned to particular motifs, so the size of the recurrent network is here simply N 300 to match the per-motif parameter count with the 'two-input" network).
In Fig. 14 we show examples for the single-motif performance of the 'two-input' network and the matched additive network, when transitioning from 70% vs. 100% of the duration of the previous motif (the latter one being the same situation as considered elsewhere in this work).
Examining Fig. 14 reveals that the 'two-inputs' network behaves differently from the additive net- work in two major ways. First, its overall motif accuracy is lower; and second, when transitioning from 70% of the duration of the previous motif, their accuracy consistently decreased compared to when initializing the motif at random as during training. This decrease in accuracy was seen consis- tently when transitioning in all scenarios when the previous motifs interrupted before their ending (which is zero-padded). This is probably because the larger variability of network outputs when interrupting motifs is reflected in a larger variability of network states. Importantly, the additive net- work with a transition module could handle this variability to allow the network to produce a more accurate output when chaining an interrupted motif with subsequent one.
These results strongly suggest that 'engineering' a fixed point in the dynamics at the start of motif's (at a position that adjusted with preparatory input) accurate and robust motif performance. In addition, close to fixed point (which is presumably good description of the dynamical regime close to the end of the preparatory period), the dynamics is quasi-linear, in which case classical results from control theory indicate that feedback loops such as those of the transition module are the optimal strategy to steer the network towards its target. Therefore, these results further strengthen our conclusion that the transition module we propose is particularly potent strategy for zero-shot transfer during motif chaining.
particularly potent way of ensuring
A.6 NOTES ON TRAINING
A. ADAM PARAMETERS
We tested various parameters of ADAM. We identified the following as yielding successful training in our setting: learning rate- = -1 10 31 = B2 = 0.5, and e = 10-8,
Default parameters for ADAM most often worked very well (especially for the additive networks), but the parameters above could help training for the small control networks, or occasionally for the multiplicative networks whose training was often non-monotonous (see Fig. [5 below), which could be especially limiting when training easier motifs for which the errors are small and/or the network activity varies little between different runs.
24

---

Under review as a conference paper at ICLR 2022
"Two input network
Additive network
Micesstarii fore pandon:
Whon stating fre oiker rostis yiner
wher stariig bon candom (rew))
Whet staring Fow other motil (1eme)
Trandtions 100% of prew Transtoos 70% prev.
D33
0254
Io Eat
vea um
Mi
mis Un
re Ddg
Kma $2%
0J0
120
ain
116
(narber
line (nurber
Time Jeamsor
(tamber
Fime (samber r)
(namber af
Time (rumber
[eumsor
(1mi)) 9.244
(yss)) 0.22
(ynd 0.10
[mea) - 0.367
n
n
1111
0243
rell
130
0281
mil cofe
mai $2%
trw 6:06
131
v0a 300
200
rsso 650
rmam Edht
te
Mi
020
Deea Time (famber of
new Tine (purtee of (o) Time (esmse 7M
1103
res Time (samber of
E.lt Time (nunche of TT)
Time (sarber of
Time (samber (i]
Time [eunvier
Comparing a 'two-inputs' vs. additive network. Left half: 'two-inputs' network: right half: additive network, adjusted for the number of parameters tuned per motif. Top half: comparing random initialization as experienced during training, and transitioning from 70% of the duration of other motifs. As reminder, in the pancls 'when starting from other motifs', colors indicate the identity of the prior motif (as labeled in the panel "when starting from random' where the motifs in this panel are shown in full, even though here during transitioning they were interrupted 70% of the duration shown). Notice that, during sequencing, the networks smoothly interpolate between the various output values the moment when the first motif is interrupted, and the readout value at the start of the next motif, without having to make a detour through fixed readout value (as would occur if the networks were reset to fixed i.e., non motif-specific- state before each motif). In the additive network, this is because the preparatory module directly drives the RNN from its state at the moment when the first motif is interrupted, to a state specific to the second motif (and instructed by the motif-specific preparatory input) that corresponds to the readout at the end of the preparatory period. Bottom half: comparing random initialization as experienced during training, and transitioning from 100% of the duration of other motifs. For each quadrant, the two left vs. two right columns differ in the initial state of the motif with conventions as in Figure
Figure 14:
A.6.2 DIFFERENCES IN LEARNING CURVES BETWEEN THE ADDITIVE AND THE MULTIPLICATIVE NETWORKS
Our investigation through hyperparameter search of the performance of additive and mutliplicative networks matched in number of tuned parameters (by fixing the size of the additive network to N = 300 and the size of the multiplicative network to N = 100) suggests that they lead to simi- lar errors on average when trained to produce the step motifs (main text Fig. Interestingly, we observed that these similar performances were typically reached through different types of learning curves (Fig. [5]: the multiplicative networks tend to show discontinuous steps in the learning curve, whereas the additive networks typically have smooth learning curves. This suggests that the per- formance of the multiplicative networks may be limited by the ability of gradient descent to fully optimize these networks to their best possible performance. In the future, it would be interesting to investigate ways of improving the optimization of the multiplicative networks for instance using motif-specin learning rate schedules, by increasing its size N 300 while only training one- third of the weights of the loop and input vectors (so that it has as many untuned parameters as the additive network).
25

---

Under review as a conference paper at ICLR 2022
Auutlpiliattive architecture
Additive architecture
Rootmon momm square esrar
b
Feeatre
mooif
N=300
N-100
O
Roor
# of training minibatches
of training minibatches
Representative example learning curves for the additive and multiplicative net- works (without a preparatory module). These are chosen examples from the networks shown in the main text lcg.lc. In this case, because we were trying to compare the small control network (which was harder train) with the additive and multiplicative networks, we used the non-default ADAM parameters described in the previous section with a small learning rate (learning rate 10-4) 1) Note that all transitions are included in the training set here.
Figure 15:
A.6.3 ADDING NOISE DURING TRAINING IS NECESSARY TO GET ROISE-ROBUS, NETWORKS
The RNNs we propose need to be noise robust so that they can constitute viable solutions that can be built upon for motor control in the real world.
First, a transition to new motor motif can be triggered in order to correct corrupted motor com- mand that corresponds
noisy network state.
Second, for more general applications of the current framework to motor control in the real world, the dynamics often need to modulated by sensory input - which is never perfectly reliable. In particular, the triggering of motif transitions is likely to often depend sensory input.
For these reasons, we imposed noise robustness through relatively large noise at motif start, imple- mented through random initialization of the state of the network: at each run, the initial activities r are sampled iid from Gaussian distribution. Importantly, because of the recurrent dynamics, the initial noise is also propagating at later times through the recurrent dynamics. Additionally, in most of our simulations including all the simulations with our preparatory module we added a small (Gaussian with std 0.001) amount of noise to the dynamics each timestep.
Here, we show that the inclusion of noise is indeed necessary for getting (Fig. T6), illustrating this for additive networks. When training networks on single motifs initialized from the origin without noise- as opposed to adding Gaussian noise to the origin as we did in the main text testing the networks with small amounts of noise leads to unreliable network outputs (Fig. 16 a, second column). In contrast, a network trained on motif' initialized from a standard Gaussian more reliable output across range of standard deviation of the initialization (Fig. T6 b, first three columns). Finally, we note that while training the vanilla additive network without noise yields higher accuracy the noiseless conditions compared by the requirement of noise robustness during training (Fig. [6] left vs. Fig. [6] b left), even in these noiseless initialization conditions the vanilla additive network does not perform better than an additive network with a preparatory period and noisy initialization (Fig. [6 a left Fig. 12.).
solutions
vanilla RNN handicapped
Therefore, not only do our caalloiccall-iissiie networks with a preparatory period display robust generalization to both in-distribution and out-of-distribution perturbations, they also do it at no ac- curacy cost compared to networks that are much less noise robust, and at no additional cost in terms of number of tuned parameters per motif. In addition, we will see in the following section that our preparatory module also speeds up learning.
A.6.4 THE PREPARATORY MODULE COMBINED WITH RANDOM INITIALIZATION SPEEDS UP LEARNING
In the last section of the main text results, we argued that the preparatory module which intro- duces fast convergence timescales at the beginning of motifs helps single motif training. Indeed, when initializing motifs from standard Gaussian both during training and testing, observed that the networks with a preparatory module reach much lower error than the networks without such a module (Fig. 3 left, dashed lines vs. full lines).
26

---

Under review as a conference paper at ICLR 2022
a) Trained when starting from the origin (additive network, without preparation);
When starinn tontherrggc
Whet suaring tad Gausin atfattiooo
When For saay Gauss
Whanstaiin www.ettnggoooeeeaaeeaeaaa
6.50
6287
L4W
1ilt
The shorrf
Thme (ombereff
Thes humeeref
Thee (amberff
Trained when starting from the origin with noise: standard Gaussian initialization, lee. std=1 (additiive network, without preparation)
Figure 16: Adding noise in the initial conditions during training improves noise robustness. Comparing two additive networks (without preparation) when trained single motifs that are: a) initialized at the origin without noise; b) initialized with standard random Gaussian values (standard deviation of 1), i.e. around noisy origin. Both networks also receive small additive noise at each timestep (drawn iid from Gaussian with standard deviation of 0.001). The first three columns show the testing performance, when varying the standard deviation of the random initial conditions used to initialize motif production (centered iid Gaussian, from left to right: no noise, i.e. std=0; very small noise with std=0.01; standard: std=1). Last column: testing performance when motifs are initialized from the activity of a previous motif during two-motif sequences (here, the first motif initialized from the origin without noise). Notice how training without noise leads to strong noise sensitivity (in some trials are very inaccurate even in the very small noise condition), whereas the networks trained when initializing motif's from a standard Gaussian do not lead to 'catastrophic' trials across all standard deviations of the random initialization (b, first three columns), Note that while training the vanilla additive network without noise yields a higher accuracy in the noiseless conditions compared to vanilla RNN handicapped by the requirement of noise robustness during training (a left vs. b left), even in these noiseless initialization conditions the vanilla additive network does not perform better than additive network with a preparatory period with a noisy initialization (a left vs. Fig. 12c). Color conventions as in Fig. 6 Standard ADAM parameters with a learning rate of 10 10-3).
Here, we give more evidence of this by examining the learning dynamics of the networks with or without a preparatory module. We find that networks trained with a preparatory module and standard Gaussian initialization reach a performance plateau faster (Fig. T7]. Interestingly, not only is this true compared to training without a preparatory module with the same a standard Gaussian initialization (Fig. 17 a middle and right; and Fig. 17 but is is also true when compared to training without preparatory module from noiseless origin (Fig. 17 left).
Taken together, our results strongly argue that our preparatory module helps network performance in two ways. First, when training single motifs from fixed noisy initialization, the fast dynamics timescales introduced by the preparatory module appear to help the training- induced changes in the input to drive the exploration of different possible activity states until finding motif-specinf one that leads to particularly accurate motif production (Mohajerin & Waslander) 2019) Kemeth et al.) 2021). Second, it helps filtering noise and robustly converging this motif-specific activity
27

---

Under review as a conference paper at ICLR 2022
pattern to start motif production. Therefore, the preparatory module helps the network implement computations that are shared between motifs, and then acts as an inductive bias such that the motif- specific parameters can be fully devoted to motif-specific functions (shape the motif-specific readout and final activity pattern during the preparatory period, and modulate the network dynamics during the remainder of motif production for accurate readout). This is why the preparatory module can be trained once and for all before training any motif specific parameters, which leads to improvements in accuracy and robustness no per-motif cost in terms of tuned parameters and without introducing any learning interference in continual learning scenario.
a) Additive networks:
Without properation. initiiizzed from origin (noiseless): spoch 40: mue 642
Without preparation. inttiaized from naisy origin (standard Gauzsaan std-13
With preparation, initialzed from noity origin (standard Gaussiare std=1]:
ap2dh: 40, mas 075
spect: 40, mes: 1.036
0.--
CA-
root reean square emoor
root mean square enror square
noot mean Square error square
sqquare
544
26-
35-
0.2
33
Training epach number
Training epoch number
Training epoch number
b) Multiplicative networks:
Without preparation, Initialized from noisy origin (standard Gaussian: sto=1):
With preparation, initialized from nois origin (standard Gauaslare std=1k:
cpech: 300, moc: 1,08
goihh 136, mso 024
root mean square erTOr
Foot mean SQuare error
meon squuare error
mean square enor
3.3
Enror
12-
31-
100
Training epoch number
Training epoch number
Figure 17: Using a preparatory module along with random initialization during training speeds up learning. Evolution of the error (the mean square error our objective function in blue, and its square root in orange) over training epochs (each of these consists of several training batches), when training networks on single step motifs. Standard ADAM parameters with a learning rate of 10-3 are used, and a small Gaussian iid noise (std = 0.001) is added to the dynamics at each timepoint. a) Additive networks, without preparatory module with either noiseless initializa- tion (left) of standard Gaussian initialization (middle); or with preparatory module and standard Gaussian iid initialization of motifs (right). b) Multiplicative networks, with standard Gaussian ini- tialization, either without (left) or with (right) a preparatory module. The preparatory module speeds up learning in both additive and multiplicative networks.
28


Reviews:
[{"review_id": "k8w3RN8q94m", "reviewer": "Reviewer_MvgU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors discuss two important alternatives to the approach that they ultimate pursue, which would involve attacking the problem with learning rather than with building in inductive biases. Rather than running experiments to test the efficacy of these alternatives, the authors instead tried to argue away these points (bottom of pg. 6). This is extremely unsatisfying. \n\nLearning has proven remarkably powerful, and incorporating constraints/structure/inductive biases, may indeed be necessary in some domains to overcome current challenges, as the work the authors cite suggests. But those domains are ones in which learning has been doggedly pursued for an extended period of time. These authors instead introduce a novel synthetic task and immediately jump to the need to incorporate more structure, without doing the legwork necessary to convincingly demonstrate that such structure is indeed necessary.\n\nTo directly respond to the authors arguments over why these alternatives are insufficient:\nFirst, they suggest that directly training the RNNs to generalize from end states is insufficient, because they can't train from all possible end states, particularly as the number of trained states scales. This argument is unpersuasive, as training on all of anything is often impossible in general, yet still learned systems generalize. E.g., work with ImageNet over the last decade provides a great example of this. \n\nThe authors could instead test this possibility by incorporating the end points into training as initial states for subsequent trajectories. They could hold out some transitions and test how well the system generates the motifs on these left-out transitions. They could hold out the transitions in two ways: Either exclude some transitions randomly (i.e., salt and pepper of the transition matrix) or hold out some motifs from the training at all (i.e., holding out rows or columns of the motifs). The latter would be a stronger test, but both would be important to see. Critically, for the authors to convincingly show that learning was insufficient, they would need far, far more than 10 motifs (e.g., like >>1000, most likely).\n\nSecond, the authors could train to return to a consistent point at the end of each motif. The authors assert that doing so would make the system not robust to perturbation/noise, but that would be true only if the system actually learned to arrive at an identical point each time, which I imagine would not occur (and if they did, they could add noise to the network during training). The network could return to the origin imperfectly and convey state information (e.g., on 299 dimensions return to origin, but on one axis keep information about). Who knows if these speculations are correct -- but an experimental test of these ideas would be quite useful.\n\nMy second major concern is the relevance of this work to the broader field. The networks are small and seemingly toy: 300 units, 10 motifs. This is smaller than MNIST. I'm not one to think that everything needs to be shown on ImageNet and achieve SOTA to be a useful contribution, but the scale of these experiments are so, so small that I worry that anything learned in this domain would transfer or inform work at far greater scales -- particularly when the conclusion is that in such a low data regime, greater inductive biases are required rather than learning.\n\n\nMinor issues:\n\n- The authors assert that thalamic activity may change the effective connectivity of motor cortical activity, but they don't make clear why that change would result in a bias towards the origin. In what sense is biasing the network to the origin motivated by neuroscience? Also, how is preparatory activity being the same across all motifs brain-like? Isn't that not seen in the brain (e.g., in Susillo et al. 2015)?\n\n- The authors seem to repeatedly suggest that matching the marginals of a distribution is a strong approximation to the joint -- e.g., \"Subtle variations in the state of the network at the end of different first motifs x_{end}^{\\mu} can lead to very different outcomes\". I don't understand why the authors are overstating the extent to which the marginals and the joint are the same thing...\n\n- The authors assert that \"a standard normal is a good choice\" for the marginals of the states. Please provide some quantification for this claim. As it stands, the authors visualize cumulative distributions of a subset of the states rather than performing statistics. \n\n- How did the authors pick a standard deviation of 1.2 (in Fig 2.a.iv-v)? Did they fit this number to the marginals?\n\n- Typo, top of page 5: \"require less neurons\" --> \"require fewer neurons\"", "summary_of_the_paper": "The authors seek to use a small RNN to generate extended, time-varying outputs given a cue indicating which output to generate. They focus on the problem of generating a sequence of outputs, cued in series, without the end state of one interfering with the subsequently cued trajectory. They show that small vanilla RNNs do not readily perform this task: RNNs at a random state can generate a sequence when cued, but they fail to do so when their state is at the end point of another sequence. The authors then show that by biasing the activity of the network towards the origin in the absence of an input (which they suggest is inspired by neuroscience), RNNs more readily learn sequences and more consistently generate accurate sequences in series. My enthusiasm for the work is relatively low for a variety of reasons. Primarily, the authors fail to test obvious alternative hypotheses of ways that the problem domain they design could be solved by a learned system, without the need to add the inductive bias they add. Moreover, and more fundamentally, the relevance of such a small toy system (300 units, trained to generate just 10 trajectories) for machine learning is unclear.\n", "summary_of_the_review": "The small scale of the models/data combined with the lack of evaluation of alternative approaches diminishes the potential contribution of this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cffZV65ix-F", "reviewer": "Reviewer_hBJi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n- The modification that's required to the \"control\" network in order is extremely simple, and has a clear theoretical goal of decreasing \"residual\" activity between motif transitions, without quenching activity in response to the new control signal.\n- The level of experimental detail is sufficient such that the experiments  in the main text can be replicated by an independent reader.\n\nCons\n- The methods in the main text are clear, and lead to convincing results in the main-section. However, without digging much deeper into appendix 4 & 5, there's no clear reason why the learned preparatory modules are able function", "summary_of_the_paper": "The article proposes a biologically inspired architecture which increases the degree to which simple recurrent neural networks may be able to chain together elemental output sequences. The authors show that the proposed \"predatory module\" allows networks to perform on par as if the network were starting from a set initial state, while also providing a smooth transition between motifs. ", "summary_of_the_review": "The preparatory module appears to function well, and the quenching activity is reasonable well tied to biological thalamocortical loop actions. To my knowledge, the motif-transition problem is not solved in switching recurrent networks, however I am not overly familiar with that literature. Overall, my low confidence ratings come from not knowing how interesting or novel the analytical work in the appendices is.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "VpztyguVdsK", "reviewer": "Reviewer_BooE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: The proposed architectures have been shown to work for \"zero-shot learning,\" or recombination without re-learning.\n\nWeakness: It is not convincing whether the particular resetting mechanism is the most reasonable one, either computationally or biologically. Computationally, any strong inhibitory input to bring the network state to the origin should suffice. Biologically, the most likely mechanism is the tonic inhibition and timely disinhibition by the basal ganglia to the thalamus and subcortical motor controllers like the superior colliculus.", "summary_of_the_paper": "This paper presents the problems in chaining nonlinear motor pattern generators and proposes a resetting mechanism inspired by the thalamo-cortical network.", "summary_of_the_review": "An interesting proposal worth discussion.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Vg1xZqbEOXH", "reviewer": "Reviewer_W1np", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\nContinual learning is specially relevant as a setting for the purposes of this paper, partly given that learning from non-stationary distributions is closer in many aspects to the way humans learn. As such, the problem of catastrophic forgetting and generalization in continual learning are important research topics that are quite relevant for this venue. In this sense, the study of some of the architectures presented here (e.g. inspired on Schuessler et al. 2020) are interesting. Also welcomed are explicit addressing of the continuous time problem, which is relevant for many fields including control. Finally, the attempt to connect with Neuroscience by introducing a model inspired by thalamocortical interactions with the preparatory module is also acknowledged.\n \n**Weaknesses**\n\t\n- **Technical**: At the technical level, I found a couple of issues that affects the task described in section 1. First, there are references to the continual learning setup, but I can’t find the specifics, such as figures and detailed comments about the task structure etc., apart from mentioning the inclusion of all the motifs vs a sequential presentation (for which results are not shown). If the models are to be analyzed in this setting, it is important to see a robust description of the task and the metrics (see, for example, Chaudhry et al. 2019). Also, although I recognize the value of synthetic tasks, I also would have preferred to see in addition a task that encodes a “naturalistic” dataset, specially given that the authors are interested in a general noise-robust solution.\n\n\tSecond, there is a problem that the authors seems to be well aware of as per the second to last paragraph in section 3. If I understood it correctly, the task described in section 2 admits a trivial solution, that of a hard resetting to an initialization state before processing a motif. This raises some considerations, including importantly that point (i) in that paragraph should be demonstrated empirically, regarding to different possible initialization schemes.\n\t\n\n- **Writing** In my opinion this article suffers a series of important defects in its writing that ultimately have a strong negative impact in the communication of its scientific ideas. Please note that the intended feedback here is not about writing style. We acknowledge that not all papers are meant to be written the same way. However, I found reading this paper to be somewhat arduous and the presentation confusing. It also contains many questionable decisions into what constitutes the main text versus what goes in the appendix (e.g. presenting the numeric values of the hyperparameters for Adam in the first paragraph of the task description, while no formal description of the problem is included in the main text). \n\n\tBeing constructive, I think this situation could be alleviated by introducing a better formalization of the task and the optimization problem in a simpler manner.\n\t- *Description of the problem*:\n\tThe description of the problem in section A.1 could be improved by stating the optimization problem in simple terms. This can be done collecting information that is scattered in section 2, and introducing the statements with a notation consistent with the rest of the paper (I found strange that uses other notation from the main text). Please note that in section A.1 some terms are not defined (“descriptor vectors”) or introduced but seemingly not used later (N_V is stated to be as small as possible). Also, no discretization method is specified, for example, and the information about initialization is scattered.\n\n\t\tA beautiful example in this domain that you cite is Schuessler et al. (2020), which targets a similar conference in section 2: the equation of evolution of the RNN seen as a dynamical system (e.g. what you have in your first equation), followed by a clear specification of each of the terms (input, output, parameters, target function, etc.), parameters specific for each task, details about training and initialization, etc. \n\t\n\t- *Description of the task*:\n\tSection 2 in the paper is devoted to the description of the task and introduces the three architectures that are used in the rest of the paper. However, after reading the section, it is not clear what the essential components are. Given that the introduction of the task is a core contribution of the paper, a more structured description would be really appreciated, especially the details concerning continual learning. This should also go in the main section of the paper (it could be short, but it should be complete). Also as a good example of what I mean is Schuessler et al. (2020), which opens section 2 with a very concise description of the task.\n\n\t**Minor presentation issues**\n\tAt a different level, the paper would benefit from:\n\t- less strange wording in many cases (e.g “more powerful performance-optimized continuously nonlinear RNNs”),\n\t- other small fixes (numbering equations, normalizing the references (use \\citep in latex), fix a number of typos (e.g. “gradient decent” last paragraph in Section 1), etc). \n\n**References**\n\nChaudhry, Arslan, Marc’Aurelio Ranzato, Marcus Rohrbach and Mohamed Elhoseiny (2019) Efficient Lifelong Learning with A-GEM. *ICLR 2019*\n\nSchuessler, Friedrich, Francesca Mastrogiuseppe, Alexis Dubreuil, Srdjan Ostojic, and Omri Barak (2020) The interplay between randomness and structure during learning in RNNs. *NeurIPS 2020* ", "summary_of_the_paper": "This paper considers the problem of RNNs learning arbitrary sequences of “motifs” (continuous time functions, discretized) after having learned the individual motifs separately. Inspired by some previous work, it proposes an architectural approach to deal with interference by introducing a motif-dependent low-rank perturbation to the recurrent weights of the RNN, following Schuessler et al. (2020). To this end, the paper introduces a new evaluation task (motif generation), which is used to compare that architecture and sensible baselines. It then moves to propose a new model inspired in thalamocortical interactions in the brain: an RNN equipped with a preparatory module that imposes a beneficial bias when following a specific training protocol.", "summary_of_the_review": "The general topic of this paper is relevant, but I found the presentation confusing and difficult to understand. Overall, given the current state of the presentation, some issues with the details of the continual learning experiments, and some concerns related to the existence of a trivial solution that would need to be addressed, the contributions of this paper remains unclear to me and I suggest a rejection.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript presents a method to allow RNNs to chain together sequences of behaviors. Reviewers had numerous concerns but the most important is that the problem posed here is solved by a simple method: resetting the state of the RNN before processing a motif.

Overall, reviewers noted a few key topics, although this list is not exhaustive:
1. Experiments are in a very simple but confusing setting.
2. Even though alternatives exist to solving this problem, they are not considered.
3. The networks considered are very simple.
4. The manuscript is difficult to understand.
5. The task admits a trivial solution.

In more detail:

1. The setting of learning to memorize time series and outputting them on command is very simple compared to what most modern work considers. Moreover, there is much confusion in the manuscript about what the setting is precisely.  For example, the setting is described as "independently learn motor motifs in order to build a continuously expandable motif library". But there is no continuously expandable motif library, the motif library is fixed at test time. The authors focus heavily on calling this setting "motor motifs", but these are RNNs that output an arbitrary time series. They are in no sense motor programs and this work is not connected to the extensive literature on motor control in machine learning. More broadly, there is no clear mathematical definition of what the problem being solved is anywhere in the manuscript.

2. It is unusual for manuscripts to not present other baseline models. But more importantly, many other approaches exist to this problem. As one reviewer pointed out, the manuscript essentially sets out to solve a problem that is completely solved in machine learning today. It rejects the solutions that exist for arbitrary reasons, and then adopts its own new solution.

3. The models used are very simple, but this is a consequence of 1, the problem domain being very simple.

4. Reviewers had difficulty understanding the details of the task. In particular, the task description section begins with minutia about the implementation rather than succinct definition of the task.

5. Most critically, reviewers identified that the model could be hard reset and would have the same behavior as the model presented in the manuscript. The proposed solution is essentially hard resetting the state to zero as it stands. There is no reason why a hard reset cannot be followed by a smoothing operation -- this seems to be the main objection of the authors.

Overall, the manuscript needs significant improvements. The task considered is too simple by modern ML standards and the fact that it admits a simple solution cannot be overlooked. Demonstrating the idea of the preparatory module on an existing ML task and dataset, while comparing with existing baseline models, carrying out ablations, and producing an extensive quantitative evaluation is what will get the community excited about preparatory modules.
Decision: Reject