Curriculum learning as a tool to uncover learning principles in the brain 

We present a novel approach to use curricula to identify principles by which a system learns . Previous work in curriculum learning has focused on how curricula can be designed to improve learning of a model on particular tasks . We consider the inverse problem : what can a curriculum tell us about how a learning system acquired a task ? Using recurrent neural networks ( RNNs ) and models of common experimental neuroscience tasks , we demonstrate that curricula can be used to differentiate learning principles using target-based and a representation-based loss functions as use cases . In particular , we compare the performance of RNNs using a target-based learning principle versus those using a representational learning principle on three different curricula in the context of two tasks . We show that the learned state-space trajectories of RNNs trained by these two learning principles under all curricula tested are indistinguishable . However , by comparing learning times during different curricula , we can disambiguate the learning principles and challenge traditional approaches of interrogating learning systems . Although all animals in neuroscience lab settings are trained by curriculum-based procedures called shaping , almost no behavioral or neural data are collected or published on the relative successes or training times under different curricula . Our results motivate the systematic collection and curation of data during shaping by demonstrating curriculum learning in RNNs as a tool to probe and differentiate learning principles used by biological systems , over statistical analyses of learned state spaces . 1 INTRODUCTION . The biological brain is thought to be the ultimate learner as it learns from few examples , solves unstructured problems , and has an impressive task repertoire . Understanding how it achieves these learning feats could lead us to better artificial intelligence ( AI ) algorithms ( Hassabis et al. , 2017 ; Macpherson et al. , 2021 ) . Interrogating learning in the brain , however , poses significant challenges both experimentally and computationally . Experimentally , measuring the entire synaptic connectivity map or connectome–a primary substrate of learning–is only just approaching reality for small brains ( Zador et al. , 2012 ; Dorkenwald et al. , 2020 ) . Even so , the emerging connectome data provides just a snapshot in time whereas learning concerns connectome dynamics , and these temporal data are still far off . Computationally , it is unclear how a sequence of brain-wide connectivities would be registered in order to reveal how animals learn different tasks ( Babai , 2015 ) . Uncovering the key substrates linking structural ( connectome ) , dynamic ( neural activity ) , and behavioral elements in biological brains could help us develop better AI algorithms with brain-like learning properties . In this paper , we use behavioral dynamics to infer learning principles . Measuring behavior in animals is comparatively easier than neural recordings or synaptic strengths . The challenge , however , is for theory to attribute individual behaviors to specific learning principles . We suggest the use of curricula to get information from behavioral dynamics about underlying learning principles . A curriculum is a schedule for information to be presented to a student learner . In the machine learning ( ML ) framework we consider , the student learner is a neural network , specifically an RNN . Recurrence in network models brings some key advantages we exploit here–ability to produce dynamics and analogy to the biological brain ’ s ubiquitous feedback connections ( Yang et al. , 2019 ; Singer , 2021 ; Ehrlich et al. , 2021 ) . Although some benefits of curricula have been shown in ML ( Graves et al. , 2017 ; Weinshall et al. , 2018 ; Saglietti et al. , 2021 ) , curriculum learning has not been widely adopted for practical applications , with notable exceptions in robotics ( James et al. , 2019 ) and reinforcement learning ( Taylor & Stone , 2009 ) . In contrast , curriculum learning is very important to experimental neuroscience–animals in neuroscience lab settings are trained using curricula , a process called shaping ( Pinto et al. , 2018 ; Koay et al. , 2021 ; Guo et al. , 2014 ) . Yet , very little relevant neuroscience data on curricula exists , as data are most often collected from `` expert '' animals after shaping . Our goal is to classify learners–here , RNNs–based on the principles they use to learn different tasks using a set of pre-designed curricula . We build and analyze RNNs trained on two common experimental neuroscience tasks using three different curricula inspired by shaping procedures . In particular , we use the evidence accumulation task ( Pinto et al. , 2018 ; Stine et al. , 2020 ) and delayed decision task ( Romo et al. , 1999 ; Constantinidis et al. , 2018 ; Liu et al. , 2014 ) . We show , using only behavioral dynamics during the execution of different curricula , that it is possible to distinguish two learning principles solely on the basis of outcomes : target-based learning from learning through representational constraints . Importantly , we find that all RNNs , regardless of curriculum or learning rule , are indistinguishable post-shaping by standard statistical analyses applied to neural dynamics in their trained state . Our results emphasize the importance of studying animals during shaping and the value of curriculum learning in RNNs as a hypothesis test-bed for probing learning principles in the biological brain . 1.1 RELATED WORK . Curriculum learning has long been relevant to the fields of AI/ML and neuroscience ( Wang et al. , 2021 ) . Curricula have been used to learn difficult control problems in robotics and reinforcement learning ( Selfridge et al . ( 1985 ) , Schmidhuber ( 1991 ) , Sanger ( 1994 ) ) . Elman ( 1993 ) noted that humans and animals use curricula and asked how they could benefit machines ; and Bengio et al . ( 2009 ) has related curriculum learning to input complexity in a key paper on optimizing learning . Despite the presence of curricula in ML literature for several decades , our work is the first we are aware of that uses curricula to characterize learning principles used by neural network models to learn tasks , analogously to experimental animals in lab settings . Recently , the question of identifying learning rules was studied by Nayebi et al . ( 2020 ) , where the full knowledge of network activations available to artificial systems is used and related to neural data from experimental recordings . Here , we are interested in being able to glean learning principles solely from behavioral data . Ultimately , our two approaches may be used in tandem to further our understanding of learning principles . Ashwood et al . ( 2020 ) link behavioral data and learning rules using large-scale data from a mouse task in the International Brain Lab ( IBL ) . The IBL task and consequently , the Ashwood et al . ( 2020 ) model includes no time dependence , history , or state dependence . Furthermore , mice can learn the task easily without a curriculum . Our work is distinct in that we target complex tasks with temporal dependencies for which a ) curricula are appropriate , and b ) the analogous experiments require shaping procedures for the lab animal to learn . In this paper , we demonstrate that different learning principles and curricula converge to similar solutions in all cases ( Fig 5 ) . In Maheswaranathan et al . ( 2019 ) , the authors claim that there is a universal solution for a given task that all learning algorithms converge to . Their approach does not incorporate curricula ; our work posits a different , complementary invariance . Furtheremore , our goal is not to prove universality , but rather to identify learning principles that may be inaccessible by only studying the trained or post-shaping state . 2 MODELS AND METHODS . 2.1 RECURRENT NEURAL NETWORKS ( RNNS ) . We use neural networks with N = 350 recurrently connected , continuous , firing rate based , leaky integrating model neurons . A neuron with index n connects to a neuron with index m through the recurrent weight wrecnm and input channel index j through input weight w inp nj . The internal state of neuron n , xn , is determined by : τ dxn dt = wrecnmam ( t ) + w inp nj vj ( t ) − xn ( 1 ) where v ( t ) is a time-dependent , task-relevant input at time t and a ( t ) is the activation function applied to the internal state–here , the hyperbolic tangent function : an ( t ) = tanh ( xn ( t ) ) . τ is the time constant of the neuron , here , 10ms . We use the Euler method to calculate neural states with dt = 1ms . We define the linear readout of a network at time t , z ( t ) , as the weighted sum of the activations a ( t ) via weights woutn : z ( t ) = w out n an ( t ) Recurrent weights W rec are initialized i.i.d from a Gaussian with mean 0 and variance g 2 N . We set , g = 1 , the critical point above which random networks are chaotic ( Sompolinsky et al. , 1988 ) and backpropagation fails . Input- W inp and readout weights W out are each drawn from a uniform distribution from -1 to 1 . Internal states , X , are initialized from a Gaussian ( mean 0 , variance 1 ) . Figure 1 - Full Width 2.2 TASKS . Our tasks model commonly used neuroscience tasks for studying working memory and decision making in rodents , non-human primates , and humans ( Pinto et al. , 2018 ; Stine et al. , 2020 ; Romo et al. , 1999 ; Constantinidis et al. , 2018 ) . In the rodent setup , a mouse runs down a hall with cues on its left and right . The number of left or right cues inform the mouse which of two turns it should make when the corridor ends , and is rewarded if it turns toward the side with more cues . In our model tasks , we have a evidence period to mimic the hall in the mouse task and task parameters chosen to be proportional to mouse experiments in Pinto et al . ( 2018 ) . Two input channels , corresponding to left and right cues in the mouse task , send pulsatile input events into the RNN units , each with an amplitude of 0.25 and duration of 50ms . Cue start times for each channel are Poisson distributed with rates λ of 150ms−1 and 300ms−1 for the two channels throughout . On each simulated trial , λs for input channels are randomly swapped . The RNN also has a third input channel which produces a 0.25 amplitude `` go cue '' lasting 50ms in the decision period . The target function , ztar , for the linearly read out RNN-unit activations is 0 during the cue period . During the decision period , the target function is half a cosine wave starting an ending at zero with amplitude either 2 or -2 depending on which channel accumulated more cues ( Fig 1C ) . We use two task variants : The first is an evidence accumulation ( EA ) task ( Fig 1A ) to challenge primacy and recency biases . RNNs on this task , similar to lab animals , sometimes overweigh early or recent evidence at the cost of running counts and relative discrepancy of cues between the two channels . The second is a delayed decision ( DD ) task ( Fig 1B ) , with a delay period between the evidence and decision periods to challenge temporal credit assignment , which is often problematic for networks to span . Evidence Accumulation ( EA ) tasks have a 2.4s cue period followed by a 0.25s decision period . Delayed Decision ( DD ) tasks have a 0.5s cue period , a 0.5s delay period , and a 0.25s decision period . The target function during the delay is 0 with no cues during this time .

Summary: This paper simulates simple RNNs performing two classic decision-neuroscience experiments (a free choice evidence accumulation task and a delayed decision evidence accumulation task). The paper examines learning behaviour of these networks under three hand-crafted curricula, for each of two different RNN loss functions. The paper claims that one can diagnose the underlying learning rule (really the loss function) an RNN is using based upon the learning behaviour observed across the set of training curricula. They suggest a similar approach can be used to identify learning rules in animal neuroscience experiments.