Brain insights improve RNNs' accuracy and robustness for hierarchical control of continually learned autonomous motor motifs

1 INTRODUCTION AND RELATION TO OTHER WORKS . Animals have the remarkable ability to efficiently learn and compose elaborate continuous behaviors , which often relies on the flexible chaining of ‘ motifs ’ - reproducible bouts of behavior - in response to hierarchical commands ( Zimnik & Churchland , 2021 ; Geddes et al. , 2018 ; Merel et al. , 2019b ) . The mechanisms behind this are however not fully understood , and engineering controllers for dynamical systems that could perform such complex and structured continuous behaviors in the real world has been a long-standing challenge of robotics ( Brooks , 1986 ; Prescott et al. , 1999 ; Merel et al. , 2019b ) . Such tasks involve two different computational operations : first , at coarse temporal intervals , the flexible selection of discrete and abstract action commands by a ‘ high-level controller ’ ; and , second , the transmission of each abstract command to a ‘ lower-level controller ’ that continuously produces a corresponding command - a ‘ motif ’ - for the motor effector . Many lines of work have used artificial neural networks ( ANNs ) to solve the first - discrete - computational operation ( notably using tools from deep reinforcement learning , e.g . ( Merel et al. , 2019a ; Frans et al. , 2018 ; Dennis et al. , 2020 ) ) and have leveraged the ability of ANNs to perform well when using a rich training set that includes many motif sequences ( Frans et al. , 2018 ; Dennis et al. , 2020 ; OpenAI et al. , 2021 ; Xu et al. , 2020 ) . However , to the best of our knowledge , in the context of the second computational operation , there is a gap in the literature about how to design ANN mechanisms that enable zero-shot transfer to performing new continuous motif sequences from a library of independently learned motifs . Here we will focus on this latter task , which notably highlights a remarkable human skill , as we can - for instance - learn to pronounce a new word and then immediately include it in arbitrary sentences . While this biological relevance has led the neuroscience community to start investigating related questions , this literature has focused on a hand-designed ‘ bottom-up ’ approach . This approach studies how networks can function in spite of strong constraints introduced e.g . to mimic the brain activity patterns or connectivity , and/or to ensure analytical tractability ( Kao et al. , 2020 ; Logiaco et al. , 2019 ; Sussillo et al. , 2015 ; Zimnik & Churchland , 2021 ; Ijspeert et al. , 2013 ; Kulvicius et al. , 2012 ) . Importantly , this approach can not determine whether specific network features are more generally advantageous for solving a task - instead , with this bottom-up approach , these chosen features could reflect constraints that arose through random evolutionary idiosyncracies , or could become irrelevant for networks that can not be designed through analytical insights but that can still be successfully trained with modern machine learning algorithms . Overcoming these limitations has implications for engineering and can provide new neuroscientific insight . To address this knowledge gap , here , we use Recurrent Neural Networks ( RNNs ) whose computational power is not arbitrarily constrained . An RNN is a type of ANN which is a generic dynamical system , and which therefore naturally fits the desired characteristics of motor outputs . Consequently , ANNs are indeed regularly used for tasks requiring the production of continuous outputs , including in the context of robotics ( Wyffels & Schrauwen , 2009 ; Sussillo & Abbott , 2009 ; Tani , 2003 ; Liu et al. , 2019 ; Merel et al. , 2019a ; Maheswaranathan et al. , 2019 ) . We will examine the ability of RNNs to ( i ) independently learn motor motifs in order to to build a continuously expandable motif library ; and ( ii ) flexibly chain motifs in arbitrary orders ( Fig . 1a and see Appendix A.1 for a formal definition ) . In order to better dissect the mechanisms by which RNNs can fail or succeed at this task , we focus on the purest form of motor control through dynamics : the production of trajectories without needing the external anchor of a time-dependent sensory input . We refer to this as autonomous control , which is especially relevant when sensory input is too unreliable ( Yeo et al. , 2016 ; Shenoy et al. , 2013 ; Brembs , 2021 ) , but also in other cases such as the above-mentioned speech production . We will show that while it is possible to engineer RNNs to independently , extendably and efficiently learn to produce single motifs in response to discrete input commands , these RNNs are limited in their generalization ability during improvisation of motif sequences . We will then use insights from the mammalian thalamocortical motor system - notably the presence of a motor preparation phase before each motif ( Zimnik & Churchland , 2021 ; Nashef et al. , 2021 ; Logiaco et al. , 2019 ) . We will show that weaving in these insights into performance-optimized RNNs leads to both improved motif production accuracy ( through a positive synergy with single motif gradient-descent training ) and excellent robustness during generalization to motif sequencing . 2 TASK AND ARCHITECTURE DESIGN . Here , we study the ability of RNNs to fulfill the first requirement of our task : the ability to acquire an extendable library of autonomous motifs ( Fig . 1a , left column ) . However , before describing these analyses , we want to clarify how we chose the motifs that the RNNs have to learn . First , we chose motifs of long durations – on the order of a thousand timesteps – so that they strongly leverage the above-mentioned autonomous capabilities of RNNs . Second , because we are interested in assessing the relative expressive power of the RNNs we study , we have designed two different types of motor motifs so that they constitute two ‘ difficulty ’ levels for the RNNs . Following recent analytical approaches for studying RNN dynamics ( Schuessler et al. , 2020b ; a ; Logiaco et al. , 2019 ) , we characterize the difficulty of motifs as the number of certain basis functions - complex exponentials that act similarly as different frequencies of a Fourier transform - needed to approximate a motif well through linear combination . Therefore , we define a set of oscillatory motifs that are relatively easy to produce , and a set of more difficult ‘ step ’ motifs ( Fig . 1d , f-g , see Appendix A.2 for the full list of motifs used in this paper ) . We train motifs using gradient descent – specifically , ADAM ( Kingma & Ba , 2015 ) ) . Our objective function is the mean square error between desired and actual output . We will now study how RNNs can meet the requirement to learn a new motif without ‘ catastrophic interference ’ with the memory of previously learned motifs - which relates to the literature on continual learning ( Kirkpatrick et al. , 2017 ; Parisi et al. , 2019 ; Yoon et al. , 2020 ; Farajtabar et al. , 2019 ) . This line of work emphasizes avoiding interference between gradient updates used to train a network on many sub-tasks , while promoting the re-use of neural resources across sub-tasks ( so that the network makes efficient use of its parameters ) . We will use an ‘ architectural ’ approach to e xinit xend xinit xend xinit≈ ? xend xend f g xinit xend From random From other motifs From random From other motifs xinit 0 20 40 60 80 100 3.0 2.0 1.0 0.0 -1.0 -2.0 -3.0 Time ( neuronal timescales ) Time ( neuronal timescales ) 0 20 40 60 80 100 1.0 0.0 -1.0 target motif 2 3 4 5 10 # of basis functions to fit Oscillating motif : easy Step motif : hardd Training single motifs to test generalization a 100 timescales Learn library of motifs String in arbitrary orders Learn new motif without interference ... ... ... Introduce it in a string with other motifs ... b c gad / g0 cn gmu / g0 cn g0 ptb go ptb=1.5 M in im um ro ot m ea n sq ua re e rr or A dd it iv e ar ch it ec tu re M ul ti pl ic at iv e ar ch it ec tu re M in im um ro ot m ea n sq ua re e rr or mean rms , add . architecture mean rms , control mean rms , mult . architecture mean rms , control gmu=1.2 Training all O ( Nmotif 2 ) transitions ... ... ... ... 1 2 2 1 1 10 10 1 10 2 2 10 Figure 1 : Task and candidate networks . a ) Task . Left : learning an extendable library of motifs without interference . Right : without additional training , stringing the motifs into chains with arbitrary orders . b ) Additive and multiplicative architectures , who may succeed in the task in a because they segregate parameters into motif-specific sets ( schematized in colors ) while benefiting from fixed shared recurrent and readout weights ( schematized in black ) . c ) Minimum root mean square error over training with overcomplete training set , depending on the gain hyperparameters gad , gmu , and gptbo . Dots are individual networks , the line is the average . In red , we show the mean minimum root mean square error in the control architecture ( averaged over five individually trained networks for each gcn0 ) . d ) Examples easy oscillatory motif and hard step motif ; fitting the latter requires a larger number of basis functions ( complex exponentials ) of varied oscillation frequencies . e ) ‘ Classical ’ strategy to promote zero-shot transfer in order to produce a chain of motifs : train ANNs to produce the motifs starting from random initial network states xinit that emulate the variability of network states at the end of motifs xµend ( where µ indexes the motif ) . f ) Increased inaccuracy during zeroshot transfer to chains of motifs compared to when starting from random xinit drawn from the same distribution as during training ( additive architecture ) . g ) As in f but for oscillatory motifs . Note that the target trajectory ( black ) is buried below the output of the network ( colored lines ) , because the latter is almost perfectly accurate with random xinit . continual learning that consists in segregating tuned parameters across the different motifs , because it will later facilitate the weaving of biological insights in our networks . With this approach , interference is fully prevented during sequential learning ; but the proposed architectures may not be efficient , which is what we investigate below . To do so , we will compare these architectures to a ‘ standard ’ RNN with no segregation of tuned parameters . This RNN ( ‘ control architecture ’ ) ( i ) is fully-tuned , ( ii ) receives a static input bµ to instruct the motif µ ( Maheswaranathan et al. , 2019 ; Sussillo & Abbott , 2009 ; Tani , 2003 ) , and ( iii ) is trained in a plausible noise-robust regime imposed by a random initialization of its state ( see A.6.3 ) . As expected given that this RNN has no protection against catastrophic interference ( Kirkpatrick et al. , 2017 ) , we indeed find that when learning motifs sequentially , this RNN immediately forgets previously learned motifs when learning a new one ( not shown ) – even though it is possible to learn motifs by simultaneously including all of them in each training batch ( Fig . 1c , and see below ) . This ‘ control architecture ’ obeys the following dynamics : τ ẋ = −x + gcn0 J tanh ( x ) + bµ . The output y = wᵀ tanh ( x ) = wᵀr is produced through the vector w that is initialized from a centered Gaussian distribution with std 1/ √ N cn – where hereN cn = 50 is the number of recurrent units of this control network , so that its outputs have appropriate maximal positive or negative magnitude scaling as √ N cn . Also , the recurrent interactions weights J are initialized with iid elements taken from a standard Gaussian as previous work has shown that when choosing a gain gcn0 > 1 this leads to rich dynamical regimes appropriate for complex computations ( Sompolinsky et al. , 1988 ; Sussillo & Abbott , 2009 ; Sussillo et al. , 2015 ; Schuessler et al. , 2020b ) . We performed hyperparameter tuning on g0 . The dynamics are discretized using Euler ’ s integration method where dt = 0.1τ . We now consider modifications of this control RNN that address this issue of catastrophic forgetting by segregating the parameters involved in learning different motifs while sharing common computational resources across motifs ( Parisi et al. , 2019 ; Merel et al. , 2019a ; b ) . Here , we will share ( i ) fixed readout weights ( with the above-mentioned centered Gaussian distribution with std 1/ √ N as this is sufficient to ensure the successful production of our motifs ) ; and ( ii ) the recurrent weights as they can set rich ‘ baseline dynamics ’ that can be modulated by some motif-specific weights . We globally adjust the shared recurrent weights through tuning their above-mentioned gain hyperparameter g ( a similar alternative could be to pre-tune these recurrent weights to an original set of motifs and to then freeze them , but - as we will see - our chosen approach leads to good results ) . First , we consider an ‘ additive ’ architecture ( Fig . 1b , top ) . Here , each motif µ is produced in response to learning the input vector bµ , leading to the following dynamics for the network activities : τ ẋ = −x + gad J tanh ( x ) + bµ , where the input acts as a motif-specific controller of the dynamics . Note that this occurs because the gradient of the loss with respect to the input weights propagates through the recurrent connections - whereas if different outputs weights would be learned for different motifs ( Jaeger , 2007 ) , the dynamics would not be affected by learning . Second , we consider a ‘ multiplicative ’ architecture ( Fig . 1b , bottom ) , that is inspired by both previous machine learning literature ( Sutskever et al. , 2011 ; Schuessler et al. , 2020b ) and the anatomy of the brain ’ s motor system ( Guo et al. , 2017 ; Logiaco et al. , 2019 ) . Here , each motif µ is produced in response to both a learned input vector bµ and a learned rank-one perturbation of the connectivity uµvᵀµ . The latter is equivalent to a loop through an instantaneous ‘ unit ’ receiving input from the recurrent network through the weights vµ and feeding back through the weights uµ . Here , this motif-specific loop participates to modulating the dynamics ( Logiaco et al. , 2019 ) , in a way that yields more computational flexibility compared to networks with random feedback weights ( Susman et al. , 2021 ) . Interestingly , learning the full recurrent weights in randomly initialized RNNs can yield a low-rank weight update ( Schuessler et al. , 2020b ) . Therefore , by imposing that the motif-specific learning is restricted to a low-rank weight perturbation , we expect to get close to full-weight learning while enabling segregation of the learned weights per motif . The dynamics are : τ ẋ = −x + ( gmu J + uµv ᵀ µ ) tanh ( x ) + bµ , where gmu and J are defined as for the additive network , and uµ and vµ are each learned and initialized iid from a centered Gaussian with std gptbo / √ N ( i.e . expected norm gptbo ) . To test the baseline relative accuracies of the additive , multiplicative and control networks , we trained them on all the possible chains of motifs of length two ( excluding repetitions of the same motif ) for the ten step motifs ( Fig . 1c left ) . Consistent with the continual learning aspect of our task – which values limiting the number of parameters that need to be tuned and stored per motif – we equalized the number of tunable parameters across architectures ( Collins et al. , 2017 ) . Interestingly , after optimizing over hyperparameters ( i.e. , gad , gmu , gptbo , and gcn0 ) , we found that all networks had similar accuracy ( Fig . 1c , reminiscent of ( Collins et al. , 2017 ) ) . This suggests that our strategy of segregating tunable parameters in the additive and multiplicative networks do not lead to drastic decrease in per-tuned-parameter expressivity , while by construction preventing interference when learning motifs sequentially ( while the control network suffers from forgetting of previously learned motifs when learning new ones as expected , not shown ) . On the other hand , we note that the multiplicative architecture , which is closer to models constrained to mimic brain dynamics and architecture ( Logiaco et al. , 2019 ) , appears to have similar accuracy as the additive network while requiring fewer neurons . This echoes recent results suggesting that more biologically-plausible object-recognition ANNs tend to have architectures that require fewer neurons ( Nayebi et al. , 2021 ) . We now turn to investigate the robustness of these different architectures when improvising motif sequences after being trained on single motifs – a training strategy which , by construction , enables learning new motifs without interference for the additive and multiplicative architectures .

Summary: The authors seek to use a small RNN to generate extended, time-varying outputs given a cue indicating which output to generate. They focus on the problem of generating a sequence of outputs, cued in series, without the end state of one interfering with the subsequently cued trajectory. They show that small vanilla RNNs do not readily perform this task: RNNs at a random state can generate a sequence when cued, but they fail to do so when their state is at the end point of another sequence. The authors then show that by biasing the activity of the network towards the origin in the absence of an input (which they suggest is inspired by neuroscience), RNNs more readily learn sequences and more consistently generate accurate sequences in series. My enthusiasm for the work is relatively low for a variety of reasons. Primarily, the authors fail to test obvious alternative hypotheses of ways that the problem domain they design could be solved by a learned system, without the need to add the inductive bias they add. Moreover, and more fundamentally, the relevance of such a small toy system (300 units, trained to generate just 10 trajectories) for machine learning is unclear.