Curriculum learning as a tool to uncover learning principles in the brain 
Authors: Daniel R. Kepple
Rainer Engelken
Kanaka Rajan
Venue: ICLR 2022 Poster (2022)
Keywords: curriculum learning
neuroscience

Abstract: We present a novel approach to use curricula to identify principles by which a system learns. Previous work in curriculum learning has focused on how curricula can be designed to improve learning of a model on particular tasks. We consider the inverse problem: what can a curriculum tell us about how a learning system acquired a task? Using recurrent neural networks (RNNs) and models of common experimental neuroscience tasks, we demonstrate that curricula can be used to differentiate learning principles using target-based and a representation-based loss functions as use cases. In particular, we compare the performance of RNNs using target-based learning rules versus those using representational learning rules on three different curricula in the context of two tasks. We show that the learned state-space trajectories of RNNs trained by these two learning rules under all curricula tested are indistinguishable. However, by comparing learning times during different curricula, we can disambiguate the learning rules and challenge traditional approaches of interrogating learning systems. Although all animals in neuroscience lab settings are trained by curriculum-based procedures called shaping, almost no behavioral or neural data are collected or published on the relative successes or training times under different curricula. Our results motivate the systematic collection and curation of data during shaping by demonstrating curriculum learning in RNNs as a tool to probe and differentiate learning principles used by biological systems, over conventional statistical analyses of learned state spaces.

Published as a conference paper at ICLR 2022
CURRICULUM LEARNING AS A TOOL TO UNCOVER LEARNING PRINCIPLES IN THE BRAIN
Daniel R Kepple Friedman Brain Institute Icahn School of Medicine Mount Sinai New York, NY 10029, USA
Rainer Engelken
Department of Neuroscience, Zuckerman Institute Columbia University New York, NY, 10027, USA
Kanaka Rajan
Friedman Brain Institute
Icahn School of Medicine at Mount Sinai New York, NY 10029, USA
ABSTRACT
We present novel approach to use curricula identify principles by which a system learns. Previous work in curriculum learning has focused on how curricula can be designed to improve learning the inverse problem: what can curriculum tell us about how learning system acquired a task? Using recurrent neural networks (RNNs) and models common experimental neuroscience tasks, demonstrate that curricula be used differentiate learning principles using target- based and a representation based loss functions as cases. In particular, we compare the performance of RNNs using a target-based learning principle versus those using a representational learning principle on three different curricula in the context of tasks. We show that the learned state-space trajectories of RNNs trained by these two learning principles under all curricula tested are indistinguishable. However, by comparing learning times during different curricula, we can disambiguate the learning principles and challenge traditional approaches of interrogating learning systems. Although all animals in neuroscience lab settings are trained by curriculum-based procedures called shaping, almost no behavioral or neural data are collected or published on the relative successes or training times under different curricula. Our results motivate the systematic collection and curation of data during shaping by demonstrating curriculum learning in RNNs as tool to probe and differentiate learning principles used by biological systems, over statistical analyses of learned state spaces.
model on particular tasks. We consider
INTRODUCTION
The biological brain thought to be the ultimate learner it learns from few examples, solves unstructured problems, and has an impressive task repertoire. Understanding how it achieves these learning feats could lead to better artificial intelligence (AI) algorithms (Hassabis et al. 2017] Macpherson et al., 202T). Interrogating learning in the brain, however, poses significant challenges both experimentally and computationally. Experimentally, measuring the entire synaptic connectivity map or connectome-a a primary substrate of learning-is only just approaching reality for small brains (Zador et al.] 2012. Dorkenwald et al. 2020). Even so, the emerging connectome data provides just snapshot in time whereas learning concerns connectome dynamics, and these temporal data still far off. Computationally, it is unclear how a sequence of wrde connectivities would be registered in order to reveal how animals leam different tasks (Babai 2015). Uncovering the key substrates linking structural (connectome), dynamic (neural activity), and behavioral elements in biological brains could help develop better AI algorithms with brain-like learning properties.
In this paper, we use behavioral dynamics to infer learning principles. Measuring behavior animals is comparatively easier than neural recordings or synaptic strengths. The challenge, however, is for

---

Published as a conference paper at ICLR 2022
theory attribute individual behaviors to specific learning principles. We suggest the use of curricula to get information from behavioral dynamics about underlying learning principles.
A curriculum is schedule for information to be presented learning (ML) framework we consider, the student learner is a neural network, specifically an RNN.
student learner. In the machine
produce Recurrence in network models brings some key advantages we exploit here-ability
dynamics and analogy to the biological brain's 's ubiquitous feedback connections (Yang et al.) 2019) Singer, 2021: EErlich et al.) 2021). Although some benefits of curricula have been shown ML (Graves et al. 2017| Weinshall et al., 2018 SSagletti et al. (02T). curriculum learning has not been widely adopted for practical applications, with notable exceptions in robotics (James et al., 2019) and reinforcement learning (Taylor& & Stone 2200). In contrast, curriculum learning is very important -niimall animals neuroscience lab settings are trained using curricula, a process called shaping (Pinto et al. 22188; Koay et al. [021 Guo et al., 2014). Yet, very little relevant neuroscience data on curricula exists, as data are most often collected from "expert" animals after shaping.
to experimental neuroscience
Our goal to classify hearners-here, RNNs-based on the principles they use to learn different tasks using set of pre-designed curricula. We build and analyze RNNs trained two common experimental neuroscience tasks using three different curricula inspired by shaping procedures. In particular, we use the evidence accumulation task (Pinto et al. 20188] Stine et al. (2020) and delayed decision task (Romo et al., 1999; Constantinidis et al. 2018 Liu et al. 2014). We show, using only behavioral dynamics during the execution of different curricula, that is possible to distinguish two learning principles solely on the basis of outcomes: target-based learning from learning through representational constraints. Importantly, we find that all RNNs, regardless of curriculum or learning rule, are indistinguishable post-shaping by standard statistical analyses applied to neural dynamics in their trained state. Our results emphasize the importance of studying animals during shaping and the value of curriculum learning in RNNs as a hypothesis test-bed bed for probing learning principles in the biological brain.
1.1 RELATED WORK
Curriculum learning has long been relevant to the fields of AI/ML and neuroscience (Wang et al.) 2021). Curricula have been used to learn difficult control problems in robotics and reinforcement learning (Selfridge et al. (1985), SSchiihhuber (199T), [anger] (1994)) Elman (1993) noted that humans and animals use curricula and asked how they could benefit machines; and Bengio al. (2009) has related curriculum learning to input complexity in a key paper on optimizing learning. Despite the presence of curricula in ML. literature for several decades, aware of that uses curricula to characterize learning principles used by neural network models to learn tasks, analogously experimental animals in lab settings.
work is the first
Recently, the question of identifying learning rules was studied by Nayebi et al. (2020), where the full knowledge of network activations available artificial systems is used and related to neural data from experimental recordings. Here, we are interested in being able to glean learning principles solely from behavioral data. Ultimately, our two approaches may be used in tandem to further our understanding of learning principles.
Ashwood et al. (2020) link behavioral data and learning rules using large-scale data from mouse task in the International Brain Lab (IBL). The IBL task and consequently, the Ashwood et al. (020)) model includes no time dependence, history, or state dependence. Furthermore, mice can learn the task easily without a curriculum. Our work is distinct in that we target complex tasks with temporal dependencies for which a) curricula are appropriate, and b) the analogous experiments require shaping procedures for the lab animal to Iearn.
In this paper, we demonstrate that different learning principles and curricula converge to similar solutions in all cases (Fig 5). In Maheswaranathan et al. (2019), the authors claim that there is universal solution for a given task that all learning algorithms converge to. Their approach does not incorporate curricula; our work posits different, complementary invariance. Furtheremore, our goal is not to prove universality, but rather to identify learning principles that may be inaccessible by only studying the trained or post-shaping state.
2

---

Published as a conference paper at ICLR 2022
2
MODELS AND METHODS
2.1 RECURRENT NEURAL NETWORKS (RNNs)
We use neural networks with N = 350 recurrently connected, continuous, firing rate based, leaky integrating model neurons. A neuron with index connects to neuron with index m through the recurrent weight wanh and input channel index j through input weight wpp The internal state of neuron Xx, is determined by:
T dxn ron mam amm (t) + Wnj V2 Xn
imp
(1)
dt
where vt) is time-dependent, task-relevant input time t and a(t) is the activation function applied to the internal state-here, the hyperbolic tangent function: an(!) (anh(xx.tt)) (t)). T is the time constant of the neuron, here, 10ms. We use the Euler method calculate neural states with dt = Ims. We define the linear readout of a network at time t, z(t). as the weighted sum of the activations (() via weights w 2())= = w. a.
8² We Recurrent weights We are initialized i.i.d from Gaussian with mean O and variance T. set, = 1, the critical point above which random networks are chaotic (Sompolinsky et al. 1988) and backpropagation fails. Input- Winp and readout weights Wout are each drawn from uniform distribution from -1 to I. Internal states, X, are initialized from a Gaussian (mean O, variance 1).
A Evidence Accumulation Task
B Delayed Decision Task
C
Example Implementations
Trals
Target functions
Input Channels
Go Right Lest
Evidence
period
Dediion period
Input Channels
Let
Exidence period
Daay pe
Decsion peroo
Go Right
0.0
Time (seconds)
(seconds)
Inputs
Outputs
Figure 1: Task Design. Both tasks use three input channels. Two correspond to "left" (blue line) or "right" (red line), respectively. The third (green line) sends the go cue, signaling the start of a decision period. A: Evidence accumulation (abbreviated EA) tasks contain 2.4s evidence period (gray box) and 0.3s decision period. During the evidence period, input events or "cues" occur at times generated by a Poisson process in the left/right channels. In the decision period, a target bump reflects the channel with more events during the evidence period. B: Delayed decision (abbreviated DD) tasks contains a 0.5s delay period (mauve box) and 0.5s evidence period. No events are presented during this period. C: Example implementations shown here. RNN outputs are compared to task target functions and weights change according to the RNN's learning update.
2.2 TASKS
Our tasks model commonly used neuroscience tasks for studying working memory and decision making in rodents, non-human primates, and humans (Pinto al. 2018 Stine al. 2020) Romo ct al. 1999 Constantinidis et al. 20188). In the rodent setup, a mouse runs down a hall with cues on its left and right. The number of left or right cues inform the mouse which of two turns it should make when the corridor ends, and is rewarded if it turns toward the side with more cues.
In our model tasks, we have evidence period to mimic the hall in the mouse task and task param- eters chosen to be proportional to mouse experiments in Pinto et al. (018). Two input channels, corresponding to left and right cues in the mouse task, send pulsatile input events into the RNN units, each with an amplitude of 0.25 and duration of 50ms. Cue start times for each channel are Poisson distributed with rates of 150ms and 300ms for the two channels throughout. On each simulated trial, Às for input channels are randomly swapped. The RNN also has a third input channel which produces a 0.25 amplitude "go cue" lasting 50ms in the decision period. The target function, 2tar, for the linearly read out RNN-unit activations is 0 during the period. During the decision period, the target function half a cosine wave starting an ending at zero with amplitude either 2 or -2 depending on which channel accumulated more cues (Fig TC).
3

---

Published as a conference paper at ICLR 2022
We use two task variants: The first is evidence accumulation (EA) task (Fig primacy and recency biases. RNNs on this task, similar to lab animals, sometimes overweigh early or recent evidence at the cost of running counts and relative discrepancy of cues between the two channels. The second is a delayed decision (DD) task (Fig 33), with delay period between the evidence and decision periods challenge temporal credit assignment, which is often problematic for networks to span.
challenge
Evidence Accumulation (EA) tasks have a 2.4s cue period followed by a 0.25s decision period. Delayed Decision (DD) tasks have 0.5s period, 0.5s delay period, and 0.25s decision period. The target function during the delay is 0 with no cues during this time.
2.3 DISCREPANCY
key characterization of the two task variants is a quantity we call Discrepaney. We define discrepancy as the instantancous difference between the number of cues in the left and the right input channels. Formally, the set of all left event or "cue" times is (tleft) k k J1=1 where K is the total
number of left events in the trial and doo is the time of the ith left event. Discrepancy at time / is defined: (tt) = Σ Klln
1= (t (t- -E H(t 44 right) where the H is the Heaviside step function: D(t) < 0 when there are more left cues than right. We define absolute discrepancy as IP()) and the absolute discrepancy of trial, [(tanat)] For all trials, we enforce [1imaa)) >
Kriils
Hxx) = i > and 0 otherwise.
2.4 LEARNING PRINCIPLES
The goal of this paper is to curriculum learning in RNN models of decision-makinn tasks as an analogy to shaping in animal neuroscience experiments, in order to identify learning principles employed by the brain. For clarity, we use the following working definitions of learning principles, rules, and updates. We refer to a learning rule as the function optimized by learning. Learning update defines the trajectory of the learner in this optimization. A learning principle is higher order categorization of learning rules, Examples of learning principles include target learning, maximum entropy. minimum energy, and representational learning, each of which could be implemented by various loss functions (i.e., learning rules) and updates.
We focus on two learning principles of interest neuroscience to determine whether the brain learns mostly by rewarding and punishing behavioral outputs target learning, or by enforcing internal representations on its neural dynamicc--repreeenttion 2011 Yamins et al.) 2014). Here, in RNNs trained by learning updates using backpropagation through time, we design separate learning rules to implement target learning and representational learning.
learning (Saxe et al., [022 Bhand et al.)
For barget-based backpropagation (BPT), we define an L2 error between target function for an idealized "behavior-like" output of the task, z (t) and the RNN's overall output, ztt):
(iar(tT) (lT)
Σ (z(l.) z (1.))
where ty is the weight-update time and to is the trial start time. ty = fiiall iee, weights are updated only at the ends of trials.
For representational backpropagation (BPR), we define an L2 error with an idcalized representa- tion. For our tasks, the ideal representation can read which channel has had more evidence. We therefore incorporate discrepancy into the loss function, forcing the network to directly represent the instantancous difference between the cues in the two input channels throughout a trial. While we only show results for this representational learning rule for simplicity, results in this paper are also consistent for representational triplet loss or a loss penalizing deviation from two fixed points.
We add representational weights Wrop with elements "oP that linearly read out discrepancy from the RNN units a,, (t)) w.rePa a,. Our representational loss function is then:
T
(tT) Σ (2(t.) z tar (1.)) + A(1) - +
(3)
4

---

Published as a conference paper at ICLR 2022
where X(t) weights the representational loss. X(t) is O during delay and decision periods; repre- sentational constraints are only applied during the cue period, during which X(tcue) = 0.01. This parameter was set to bring the discrepancy enror the same order of magnitude to the target error. We note a necessary limitation imposed by using a behavioral matching task is that both models must include target learning principles. Therefore we cannot test the case where the model follows a purely representational learning principle.
Gradients are calculated and accumulated every time step in parallel trials presented in batch sizes 32. 32. Weights are updated after each trial using Adam with 8o = 0.9 and 3 0.999 (Kingma & Ba 2014]. Learning rates are chosen proportionally to average weight initialization; 0.01 for readout or output weights, Wout and representational weights Wrep, and 0.0003 for recurrent weights, Wrec,
A Evidence Elongation Curriculum
B
Delay Elongation Curriculum
Discrepancy Reduction Curriculum
Evidence peried
Delay period
Events Discrepancy
ingh
Left
Left Right
Right
Right
15
Course number
Course number noressing dely
Course number
pridnct eeiduce
9
Go
Left
Right
Right
dee sorreeiig domppny
Left 13 Right
Left 15 Right
Right
Right
14
0.0
0.5
1.5 Time (seconds)
1.0
2.0
2.5
0.0
0.5
1.0
0.0
0.5
1.5 Time (seconds)
1.0
2.0
2.5
Time (seconds)
Curriculum Design. Each curriculum is presented in "courses", blocks of trials with the same task parameters. The number of trials in a given course is determined by the performance of the network. A Evidence elongation curriculum. Cue period is elongated at 0.1s per course for both DD and EA tasks (EA schematized here). B Delay elongation curriculum. Delay period is elongated by 0.1s per course in DD tasks. C Discrepancy reduction curriculum. The minimum allowable discrepancy is decreased by 1 for each new course, until all discrepancies are presented.
Figure 2:
2.5 CURRICULA
We define curricula using a hierarchical categorization: A curriculum is an ordered set of courses; a course is ordered set of batches; batch is set of trials. A trial is instantiation of the input space, here, either DD or an EA task. Parameterization of the input space as task parameters lets us conceptualize each course as a subset of the input space well. For example, if we consider all possible trials of the DD task, one subset-or course-could consist of all trials with a Is delay. Our goal is not learn the entire input space, but rather subset, which we call "desired task" When there is only one course (i.e., when the desired task is trained directly), we call this the mull curriculum.
We extend the above definition of a curriculum also include administration rules for different courses. These rules determine the number of batches in a course presented to the RNN during the curriculum as well as the transition between courses. For example, each course a curriculum could be administered for 50 batches before transitioning to the next course in the ordered set. Here, we instead include a test set for each course that allows us to evaluate performance after each batch. Our test set is designed to have balanced discrepancy, with 10 examples of each allowable discrepancy. The final course of all curricula for a given task (EA and DD) share the same test set. If the RNÑ correctly solves 75% of tasks in the test set, it "graduates" to the next course. This performance threshold is chosen to allow failure on tasks with low discrepancy but many cues which are known to be especially challenging (Dehaene et al. 1998). An RNN output is correct if the integral over the decision period is within 50% of the integral of the target function.
All curricula inspired by real world shaping procedures in experimental neuroscience, like those seen in Pinto et al. (2018): Duan et al. (213);; Stine el al. (2020); Romo et al. (199)); Constantinidis al. (2018). We use the following three curriculum types:
Evidence elongation curricula: The first course has cue period of 0.1s, which elongates by 0.1s per course until it reaches the length of the desired task's cue period. In EA tasks, this
Delay elongation curricula: The first course has delay period; delay elongates by 0.1s per course until it reaches the length of the desired task's delay. Only applies to DD tasks in which the final delay is 0.5s.
5

---

Published as a conference paper at ICLR 2022
Diserepancy reduction curricula: The first course has discrepancy threshold of 15 in EA tasks and in DD tasks, which decreases by with each course.
2.6 CURRICULUM COMPLETION TIME
We calculate curriculum completion time (CCT) by counting the number of batches (in our case, the same as the number of weight updates) a network sees during all the courses in curriculum. Given the difficulty of our tasks, not all networks learn to solve the task in reasonable time, and sometimes not at all, especially under a null curriculum (Fig.3. We set weight update limit of 500 iterations. For comparing CCTs between "partially undefined" sets, we use rank-ordered Mann- Mann-Whitney U tests (Hettmansperger & McKean 2010). RNNs that fail to learn can therefore still be included because their CCT must be greater than the CCT all that do learn within the limit.
2.7 STATE SPACE ANALYSES
In neuroscience, dimensionality reduction techniques, e.g., Principal Component Analysis (PCA) are commonly used to infer dominant features of neural dynamics and to evaluate complexity (Cunningham & Byron| 2014). We similarly characterize and compare state spaces of our RNNs under different curricula and learning rules, and measure their effective dimensionality using PCA. We use all RNN activations from the final test of each curriculum for computing the covariance matrix for PCA. The covariance matrix has trial time-steps X number of trials in the rows and neurons N 350 the columns. The resulting cigenvectors or PCs have N components, and points along this PC space represent times during a trial as "traaectory". As each time in a trial has a discrepancy (as per section 2.3). we can color PC trajectories accordingly (AB)
We fit the first PC as the hyperbolic tangent of discrepancy. Formally, our eigendecomposition gives the N N-dimensional state S { \ as a function of time (t) = Ptj ر (t), where P, is the ith PC vector with elements Pij- j iterates neurons, and (t( is the activation of neuron j at time t. We find parameters (a, b) to maximize the correlation between 81 and s + atanh( (t)). To report dimensionality, we measure the inverse participation ratio of PC eigenvalues (Rajan et al. 2010).
3
RESULTS
Evidence Accumulation Task
Delayed Decision Task
A
training lest
Output
Average Training Loss
Percentage of Curricula Completed
Percentage of Curricula Completed
BPR
Loss G
BPR
BPT
BPR BPT
Loss
Oubut
0.8
15
100 300
100
500
Weight updates
Time (s)
Waight updates
Percent
Percent
0.8
training
Output No
Average Training Loss
test
1.5
0.4
BPT
Loss
Loss
0.2
3 L a
0
0
100 200
300
2
200 Weight updates
100
500
200
400
200
400
Weight updates
Time (s)
Weight updates
Weight updates
Learning Without Curricula: "Null curricula". A, B: Performance of BPR and BPT during training with the null curriculum on the EA task. (left) Green loss curves are the mean loss on the test set, whereas gray is the mean loss the training batch. (middle) Sample network outputs on the test set. Left trials and right trials are separated by upper and lower panels respectively. (right) Average training loss over 50 RNNs trained. C: Cumulative density of curriculum completion times (CCT) of 50 RNNs with BPR (blue) and BPT (orange) in the null curriculum. D: Cumulative density of CCT with null curriculum on the DD task. All RNNs failed pass the performance threshold.
Figure 3:
3.1 NULL CURRICULA
Both learning rules are able to learn the desired EA task without curricula. BPR makes learning faster and more reliable, with median CCT of 200 and all 50 RNNs successfully solving >75% of

---

Published as a conference paper at ICLR 2022
the test set trials. BPT had a median CCT of 325 and only 32 of 50 RNNs learned within the 500 weight-updates limit, In 3 we show that training and testing losses in the decision period largely overlap, with less noise in test loss. This is representative of the balanced discrepancy in test set, whereas individual training batches are generated with random discrepancy. The decision period loss curve for BPR is steeper than that for BPT, demonstrating the benefit from representational loss.
For the desired DD task null curricula, both learning rules fail to learn the task within the 500 weight update limit. Loss functions during the decision period, Fig indicate little to improvement in performance over training. Errors from the decision period cannot propagate through the long delay the relevant directional cues.
A
Evidence Accumulation Task
Delayed Decision Task
BPR
BPT
BPR
BPT
Curriculum Type
un wn
Diserepancy
13
percent
%
Null
Notoo in
correct
00
200
50
coc
D00
00
300 2e
weight updates
weight updates
weight upditees
weight updates
percent correct
diccopand
Evidence
Elongation
Curriculum
ooc
dos
200
200
300
100
002
Meo updates
weight updates
weight updates
weight updlnees
percent connect
Delay
Curriculum
Elongation
Not applicable
diccrppaccy
-1
000
300 oo
weight updanes
weight updates
in mesr
Discrepancy
Discregaccy
90 correct
disoepancyy
Reduction
Curriculum
5o d. H.
correct
000
006
300
000
082
008
009
001
200
000
weight updates
weight updates
weight updazes
weight updases
C
Evidence Accumulation Task
Delayed Decision Task
percentage
completed weight update
Currculum Type
percentage curricula completed by weight updste
BPR
BPT
BPR
BPT
Null
percert
parcart
persant
percent
Elongation
Delay
Elongation
000
000
202
000
Discrepancy
601
000
00E
300
Reduction
weight updates
weight upoates
weight updates
Weiggtt upcates
Figure 4: Curriculum Learning Performance. A: Evidence Accumulation (EA) task performance with BPR and BPT under all curricula. Each row shows different curriculum and columns show BPT vs BPR learning rules. B: Delayed Decision (DD) task performance with BPR and BPT under all curricula. All heatmaps show average performance
function of discrepancy and weight update. Each pixel corresponds to the average performance of 50 networks on a single discrepancy. Pixels excluded if there fewer than three RNNs tested that discrepancy. As RNNs may be in different courses at any given time, if discrepancy is not yet in a particular RNN's test set, that D: Cumulative density of CCT
RNN's performance is assumed be zero that discrepancy. under all curricula. Each use 50 RNNs and show the % of RNNs which have completed a curriculum within a particular number of weight updates.
7

---

Published as a conference paper at ICLR 2022
3.2 CURRICULUM LEARNING
Curriculum learning mcreases learning speed and reliability in the EA task for BPT (Fig However, there no obvious benefit from curricula for BPR in the same task (Fig For the DD task, only the delay elongation curriculum successfully rescues BPT. In BPR, learning is significantly faster during both discrepancy reduction and delay elongation curricula (Fig ( Evidence elongation curricula are unable to rescue the performance of RNNs trained by BPR, and fail learn within 500 updates.
Discrepancy Reduction Curriculum on Evidence Accumulation Task
Delay Elongation Curriculum on Delayed Decision Task
PC2
04
02
BPR
PC1
PC1
-0.4
PC3
Discrepancy
Discrepancy
PC2
0.4
BPT
BPT
PC3
15
PC3
-044
Dicrepancy
Discrepancy
Effective Dimensionality
Correlation
D
Evidence Accumulation
Delayed
Curriculum Type
05
1.5
0.
Completon time (f upddtte))
Undelf
BPR for EA
RPT
Evidenca
EA Task
Elongation
for
DD
150
Delay Elongaiin
DD
Discrepancy
Delny Emepption
Bablema Etmppinn
will
Dainy Empaaton
follroa Caarppauurii Stmmpton
taie Sunqeon
Duers Ratintes
wa
Reduction
Curriculum Type:
E
F
Mann Whitnay "U" metric against null curriculum
Learning rule prediction accuracy
ype on deaeere
OPR
90%
BPT
transparency- number of samples (tom 2 50)
Acauracy
OCT
discrepancy
sonneyanyyyyyyyyy Perttctio Finak
Effective dimensionailyy
Estabosoe Dengatonn
Randam
00%
CA Task
Number ssmpless/cumo.uum
Figure 5: Discriminating principles by learned state versus curriculum completion time. A: (left) Low dimensional state spaces of networks trained by BPR and BPT with discrepancy curriculum on EA task. Principal component (PC) scores of each time during the test set shown on largest three PCs. Each point is colored by the discrepancy D(t), with blue being the most negative discrepancy; red, the most positive. (right) Fit between tanh(D(t)) and PC1. B: Low dimensional state spaces on delay elongation eurriculum BPR (blue) and BPT (orange) under each curriculum (row) for EA (filled bars) and DD tasks (open bars). Column 2: Correlation R between the first principal component and tanh(D(t)) for BPR (blue) and BPT (orange) under each curriculum. D: Comparison between curriculum completion times (CCT) for each task (open vs filled) and curriculum type (x axis). Significance asterisk indicates Z > 2 or .05. E: 3D scatter plot of Mann Whitney "U" metrics. Each axis shows different curriculum compared against the corresponding null curriculum, Each point is bootstrapped with the number of samples corresponding to the transparency (from 2 to 50 samples). Blue circles are from RNNs learning with BPR; orange, from BPT. F; Prediction accuracy with bootstrapped data, averaged over 1000 instantiations.
DD task. C: Column 1: Effective dimensionality of
8

---

Published as a conference paper at ICLR 2022
3.3 LEARNED STATE SPACES INVARIANT TO TASKS, CURRICULA, AND LEARNING RULES
Despite learning with different loss functions and under diverse curricula, state spaces of all success- fully trained RNNs have the same key features. In particular, all low dimensional, with two PCs explaining 2 90% of the variance. Effective dimensionality is approximately 1.5 in all cases (Fig 5.. Further, the first PC of all state spaces is strongly correlated with the hyperbolic tangent of the discrepancy, tanh( (t))). The correlation between the data and our fit is 2 0.9R in all solutions. We argue that given the similarity of the spaces in the learned state, differentiating between learning principles is non-trivial and will represent a key advance for neuroscience.
3.4 RELATIVE CURRICULUM COMPLETION TIME IS INDICATIVE OF LEARNING PRINCIPLEE
In the EA task, evidence elongation and discrepancy reduction curricula significantly change the distribution of curriculum completion time (CCT) only for BPT. Evidence elongation curricula significantly speeds up learning with a median CCT of 190 vs 325 weight updates (Z: 2.9 from U Test). Discrepancy reduction curricula make learning even faster with a median CCT of 134 (Z: 3.7). For BPR, learning is not significantly improved by any of the curricula tested.
For the DD task, observe that delay elongation curricula benefit both BPT and BPR, but only BPR benefits from discrepancy reduction curricula. For discrepancy reduction curricula, BPR has a median CCT of 226, which significantly faster than under null curricula that fail to converge in our 500 weight-updates limit (Z: 3.9). Delay clongation curricula were even faster with a median CCT of 150 (Z: 5.4) with BPR. Similarly, for BPT, the median CCT was 154 with significance of Z: 4.16.
Our results in Fig 5 Juggest that using the CCT distribution provides enough information to differenti- ate our representational learning (i.e., BPR) versus surget-based (i.e., BPT) models. We evaluated this on bootstrapped data 5E. F). As the number of RNNs for each curriculum increases above 20, we see that CCT performance improvement from curricula successfully disambiguates BPT and BPR. Using simple logistic regression on comparisons between CCT, we able to identify the learning principle in over 90% of samples with 20 RNNs. Performing an analogous test with the standard neuroscience state space features, however, was unsuccessful even using the full dataset.
3.5 DISCUSSION
We proposed novel use for curriculum learning as a method for uncovering the learning principles of a system. We demonstrated using model neuroscience tasks and thaping-inspired curricula that information about learning principles are inaccessible by studying only fully trained systems. However, we suggest that using only behavioral data we can uncover this information using curricula. This motivates the collection and curation of behavioral data (and eventually, concomitant neural data) during broad range of shaping procedures already being employed in labs.
Our approach focuses on easily accessible metric of performance-the time to complete a curriculum. is likely that other behavioral metrics such as a moving average of performance during curricula could also be valuable provide better resolution in uncovering learning principles, particularly when combined with neural findings. While we demonstrated that global features of learned neural activations were largely invariant to our selected tasks, curricula, and learning rules (Fig tracking the time evolution of activations (and weight matrices) during the execution of different curricula likely to be a rich source of information about learning principles. In Nayebi al. (2020), the authors successfully use information from activations during training without curricula. The addition of curricula their approach could further improve learning rule discrimination.
While we demonstrated the value of our approach in task variants of evidence accumulation and delayed discrimination, we suggest that using curricula to discriminate learning principles could be more general. In other words, curricula could be designed to separate lcarning principles in the context of other tasks as well. Furthermore, our tasks are qualitatively similar to those used in human studies investigating numerosity (Testolin et al.) 2020 Creatore et al., (221)). The noninvasive nature of our approach could be particularly advantageous where ethical or technical considerations limit our access to neural data, e.g., in humans. We hope that this paper will encourage other AI/ML and computational neuroscience researchers to expand our approach to more tasks and more curricula to further discriminate learning principles. We expect this work to result in more such effort to provide higher-resolution insights into learning principles in the biological brain and motivate better data collection from different behavioral experiments.

---

Published as a conference paper at ICLR 2022
REFERENCES
Zoe Ashwood, Nicholas A. Roy, Ji Hyun Bak, and Jonathan W Pillow, Inferring learning rules from animal decision-making. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin
(eds.), Advances in Neural Information Processing Systems, volume 33, pp. 3442-3453, Curran As- sociates, Inc., 2020. URL httppps///prrceeeiingg
pdf
L László Babai. Graph isomorphism in quasipooynonii time. CoRR, abs/1512.00354, 2015. URL
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In
Proceedings of the 26th Annual International Conference Machine Learning, ICML '09, pp. 41-48, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605585161. doi: 10.1145/1553744.155333
1tt...01111111555411111 155333388888188
Maneesh Bhand, Ritvik Mudur, Bipin Suresh, Andrew Saxe, and Andrew Ng. Unsupervised learning models of primary cortical receptive fields and receptive field plasticity. Advances in neural information processing systems, 24:1971-1979, 2011.
Christos Constantinidis, Shintaro Funahashi, Daeyeol Lee, John Murray, Xue-Lian Qi, Min Wang, and Amy FT Arnsten. Persistent spiking activity underlies working memory. Journal of neuroscience, 38(32):7020-70222 2018.
Celestino Creatore, Silvester Sabathiel, and Trygve Solstad. Learning exact enumeration and approx- imate estimation in deep neural network models. Cognition, 215: 104815, 2021. ISSN 0010-0277. doi: https:///oiiorg////..1 http.//g//0..1001i..oooiitooooo 2021 104815. URL wtt.hhttss/// www. sc enced rect com/ http:///nne/arrrreee//////011110
John Cunningham and M Yu Byron. Dimensionality reduction for large-scale neural recordings. Nature neuroscience,
2014.
Stanislas Dehaene, Ghislaine Dehaene-Lambertz, and Laurent Cohen. Abstract representations of numbers in the animal and human brain. Trends neurosciences, 21(3)3355-361 1998.
Sven Dorkenwald, Claire McKellar, Thomas Macrina, Nico Kemnitz, Kisuk Lee, Ran Lu, Jingpeng Wu, Sergiy Popovych, Eric Mitchell, Barak Nehoran, Zhen Jia, J. Alexander Bae, Shang Mu, Do- dam Ih, Manuel Castro, Oluwaseun Ogedengbe, Akhilesh Halageri, Zoe Ashwood, Jonathan Zung, Derrick Brittain, Forrest Collman, Casey Schneider-Miiell, Chris Jordan, William Silversmith, Christa Baker, David Deutsch, Lucas Encarnacion-Rive
Sandeep Kumar, Austin Burke, Jay Gager, James Hebditch, Selden Koolman, Merlin Moore, Sarah Morejohn, Ben Silverman, Kyle Willie, Ryan Willie, Szi-chich Yu, Mala Murthy, and H. Sebastian Seung. Flywire: Online commu- nity for whole-brain connectomics. bioRxiv, 2020. doi: 1AL.11001222000.0..2244444.. // bi biooxiivii...............i...ii. wwt.:/iwwv.rooooooooooooo wowtent
|RLhttps! http://w..w...oooooooooooo////// http:///arllr///2///////////2/2/ 08, 30 274225
Chunyu A. Duan, Jeffrey C. Erlich, and Carlos D. Brody. Requirement of prefrontal and midbrain regions for rapid executive control of behavior in the rat. Neuron, 86(6):1491-150; 2015. ISSN 0896-6273,
doi: htcpo//e.e..ttccoceee//eeeeessea
URL https: ///w: /www
Daniel B. Ehrlich, Jasmine T. Stone, David Brandfonbrener, Alexander Atanasov, and John D.
Murray. Psychrnn: An accessible and flexible python package for training recurrent neural network
models on cognitive tasks. eNeuro, 8(1), 2021. doi: D0011222EEUUR..2222000..
URL
wwwp://wwww..... www.ee.eueoooom.. http::///ww...eeeroooooo////ottt B/1/ENEERO....//. WWW.EEERRRR.022200200 0427-20270000 2020
Jeffrey L. Elman. Leamning and development in neural networks: the importance of start-
ing small. Cognition, (1)):91999 1993. ISSN 0010-0277. doi: htts://doi.org/100
1016/0000-027770399055-444444 URL http://wwwwsciieccee rect. wtt.ccommsccceee//oc/
arriciee/s1100010077733333
Alex Graves, Mare G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In international conference on machine learning, pp. 1311-1320. PMLR, 2017.
10

---

Published as a conference paper at ICLR 2022
Zengcai V Guo, S Andrew Hires, Nuo Li, Daniel H O'Connor, Takaki Komiyama, Eran Ophir, Daniel Huber, Claudia Bonardi, Karin Morandell, Diego Gutnisky, et al. Procedures for behavioral experiments in head-fixed mice. PloS one, 9()):88677,, 2014.
Demis Hassabis, Dharshan Kumaran, Christopher Summerfield, and Matthew Botvinick. Neuroscience-insse artificial intelligence. Neuron, 95(2):245 2017.
Thomas Hettmanspprrerr and Joseph McKean. Robust nonparametric statistical methods. CRC Press, 2010.
Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to-sim: Data- efficient robotic grasping via randomized-to-canooiic
adaptation networks. In CVPR, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14226694,, 2014.
Sue Ann Koay, Adam S. Charles, Stephan Y. Thiberge, Carlos D. Brody, and David W. Tank. Sequential and efficient neural-population coding of complex task information. bioRxiv, 2021. doi: 10.1101/801654, URL httpt::///// www. www.:.iioxiv..omm http://wwwww..eoeee///////////// 801654
Ding Liu, Xiaowei Gu, Jia Zhu, Xiaoxing Zhang. Zhe Han, Wenjun Yan, Qi Cheng, Jiang Hao, Hongmei Fan, Ruiqing Hou, et al. Medial prefrontal activity during delay period contributes learning of a working memory task. Science, 346(6208):458-4666 2014.
Tom Macpherson, Anne Churchland, Terry Sejnowski, James DiCarlo, Yukiyasu Kamitani, Hidehiko Takahashi, and Takatoshi Hikida. Natural and artificial intelligence: A brief introduction to the interplay between ai and neuroscience research. Neural Networks, 2021. ISSN 0893-6080. doi: https://doi.o http:///i..rr/1...111/oooooo 111.000 ccc.aacc//rriclee/////8333822222
URL / http://www.cieeeeeeee.t.ttt com/
Niru Maheswaranathan, Alex H. Williams, Matthew D. Golub, Surya Ganguli, and David Sussillo. Universality and individuality in neural dynamics across large populations recurrent networks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché- Buc, Emily B. Fox, and Roman Garnett (eds.), Advances Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancou- ver, BC, Canada, pp. 15603-15615, 2019. URL / www.:/ererdiiin................. Sapprr2019//B httpe//22111hhehh/55552222271882 bccdalles
bor.:002000001.00000oooooooo1a1a Abst:ttt...tttttttt....o. cc/
Aran Nayebi, Sanjana Srivastava, Surya Ganguli, and Daniel LK Yamins. Identifying learning rules from neural network observables. arXiv preprint arXiv:2010.1 1765, 2020.
Lucas Pinto, Sue A. Koay, Ben Engelhard, Alice M. Yoon, Ben Deverett, Stephan Y. Thiberge, Ilana B. Witten, David W. Tank, and Carlos D. Brody. An accumulation- -ww.aiiiion----v-eenneeeeeo task using visual pulses for mice navigating virtual reality. Frontiers in Behavioral Neuroscience. 12:36. 2018. ISSN 1662-5153, doi: 10.3389/fnbeh.22 2011 0.00000. URL www. frontiersin
org/article/11 9 fnbehh.. 201 8. 00036
Kanaka Rajan, L Abbott, and Haim Sompolinsky. Inferring stimulus selectivity from the spatial structure of neural network dynamics. Advances in Neural Information Processing Systems, pp. 1975-1983, 2010.
Ranulfo Romo, Carlos D Brody, Adrián Hernández, and Luis Lemus. Neuronal correlates of parametric working memory in the prefrontal cortex. Nature, 399(6735):40-473, 1999.
Luca Saglietti, Stefano Sarao Mannelli, and Andrew Saxe. An analytical theory of curriculum learning in teacher-student networks. arXiv preprint arXiv:2 106.08068 2021.
T.D. Sanger. Neural network learning control of robot manipulators using gradually increasing task difficulty. IEEE Transactions on Robotics and Automation, 10(3):323-333 1994. doi: 10.1 109/70.294207.
11

---

Published as a conference paper at ICLR 2022
Andrew Saxe, Stephanie Nelli, and Christopher Summerfield. If deep learning is the answer, what is the question? Nature Reviews Neuroscience, 22(1):55-67, 2021.
J. Schmidhuber. Curious model-building control systems. In /Proceedings] 1991 IEEE International Joint Conference Neural Networks, pp. 1458-1463 vol.2, 1991. doi: 10.1 10.110//UCN..199 170605.
Oliver G. Selfridge, Richard S. Sutton, and Andrew G. Barto. Training and tracking in robotics. In Proceedings of the 9th International Joint Conference on Artificial Intelligence Volume IJCAI'8,, pp. 670-672, San Francisco, CA, USA, 1985. Morgan Kaufmann Publishers Inc. ISBN 0934613028.
Wolf Singer. Recurrent dynamics in the cerebral cortex: Integration of sensory evidence with stored knowledge. Proceedings of the National Academy of Sciences, 118(33), 2021. ISSN 0027-8424. doi: (0.1073/pnas.210004333 118/33/e210104311
URL https:: wwwp://wwwwww..ooo pnas org/contet////
H. Sompolinsky, A. Crisanti, and J. Sommers. Chaos in random neural networks. Phys. Rev. Lett., www.nnkkkpppo...mmm.mm... org
61:259 262, Jul 1988. doi: 10.1 10110//hyyeeeeett.11.22 URL
do1/10.110./Phnsseeeieeec. 61.259
Gabriel M Stine, Ariel Zylberberg, Jochen Ditterich, and Michael Shadlen. Differentiating between integration and non-integration strategies in perceptual decision making. eLife, 9ce55365, apr 2020.
ISSN 2050-084X. doi: 10.7554//L.ife.55366. URL http:: orgp:i..rrrrio /10. 7554/eLife 70.......................
55365
Matthew Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(7), 2009.
Alberto Testolin, Serena Dolfi, Mathijs Rochus, and Marco Zorzi. Visual sense of number vs. sense of magnitude in humans and machines. Scientific reports, 10(1):1-13, 2020.
Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning, 2021.
Daphna Weinshall, Gad Cohen, and Dan Amir. Curriculum learning by transfer learning: Theory and experiments with deep networks, 2018.
Daniel L. K. Yamins, Ha Hong, Charles F. Cadieu, Ethan Solomon, Darren Seibert, and James J.
DiCarlo. Performance-optimmzzed hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 111(23):611-86222 2014. ISSN
0027-8424. doi: 10.1073/pnas. 1403112111. 1t773//maa.14411121111 111/23/8619
URL httpt:tp///.. www. pnas, wttg///o////ttooooooo/ooo/o
Guangyu Robert Yang, Michael W Cole, and Kanaka Rajan. How to study the neural mechanisms of multiple tasks. Current opinion in behavioral sciences, 29:134-143, 2019.
Anthony M. Zador, Joshua Dubnau, Hassana K. Oyibo, Huiqing Zhan, Gang Cao, and Ian D. Peikon. Sequencing the connectome. PLOS Biology, 10(10):1-7, 10 2012. doi: 10.137 1937//journll.piin 1001411. [tL_:tttps://b...o///1...
wbw....ppppp.................... 10014111
ACKNOWLEDGEMENTS
This work is supported by NIH, NSF McDonnell Fdn grants to KR.
12


Reviews:
[{"review_id": "rcrWkpa1oR", "reviewer": "Reviewer_cCVe", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths:\n\n- novel idea to use curricula to find signatures of different loss functions\n- curricula inspired by those used in neuroscience experiments - directly applicable. Recommends how (relatively) easily collected behavioural data could be used to infer loss functions being optimized during task training.\n\nweaknesses:\n\n- mostly relevant to neuroscience/psychology\n- missing final analysis to really demonstrate claim in detail by using the CCT to adjudicate between candidate cost functions. Right now the main result seems to be in subpanel E of figure 5, but this is only describes that differences exist. What about an accuracy on actually using this difference to predict cost functions in some simulation comparable to a neuroscience/psychology experiment? I'd recommend moving some of the other figures/sub panels to the supplemental to make room for such an additional analysis.\n- The motivation for the two cost functions and how they represent different learning \"principles\" could have been clearer.\n- The presented analysis only shows how the CCT distinguishes the particular two cost functions evaluated here. As far as I can tell, the authors don't propose that behaviour under different curricula can say something generic about the learning principles at play. It's not clear whether CCT would distinguish between two arbitrary cost functions or how it would help infer learning principles in the absence of clear hypotheses.\n\nminor comments:\n\n- Figure 4 is very useful. It makes the results much easier to grok than in the text. It would be better if two rows corresponded to curricula. Right now they are not matched.\n- check wording in 2nd last paragraph. not sure what you mean to communicate here: \"Furthermore, we suggest that relative predictions, such as the ones we make from curriculum completion times in Figure 5, are invariant to shifts in their absolute quantity might that arise from extending a model to a real brain\"", "summary_of_the_paper": "The paper shows how comparing network performance under different training curricula can distinguish learning principles. The curricula evaluated are inspired by those actually used in neuroscience to train animals to perform decision making tasks, and so the results motivate the analysis of behaviour during training periods which could provide evidence for/against candidate cost functions. ", "summary_of_the_review": "I recommend acceptance because the novel demonstration that behavioural differences under different curricula can be used as a signal to compare candidate loss functions is valuable. However, the demonstration seems somewhat narrow in scope and it's not clear how useful it would be in practice, so my recommendation is not a strong one.\n**********************\nEDIT: updated my score from 6 to 8 based on author response and changes to the paper", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FWyqwsxQLPS", "reviewer": "Reviewer_wnW4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe general idea of using learning behaviour under different training curricula to decipher something about the underlying learning process of a system is really interesting.\nSimulating classical neuroscience tasks was a nice starting point, as it creates  opportunities to compare behaviour of these RNN models to real rodent and neural data, however this ended up being a bit of a  missed opportunity as there was no comparison to real animal data in this paper, as far as I could see.\n\nWeaknesses:\n\nThere was a lot of confusion around terminology in the paper. Learning rules vs loss functions was a regular miscommunication. At many points in the text (including in the abstract) they say that their networks use two different learning rules (and so suggest the paper is able to diagnose different learning rules), when really the paper uses two different loss functions: one which uses the sum squared error loss on behaviour and the other which uses this same loss + a representational loss term. The learning rule is the same for both RNNs, which is batched gradient descent with an Adam optimizer.\n\nThe paper does not at any point, as far as I can see, actually use the RNN behaviour to predict the underlying loss function empirically, so claims that the paper shows they can disambiguate the loss functions (in the paper called ‘learning rules’), are not empirically substantiated. The paper could do this using the set of behavioural CCT data.\n\nOne of the two particular choices of loss function used in this paper require knowing the RNN’s (in their case, but animal’s in their suggested use case) internal representations, however one of the big motivators for this work was to gleam understanding about learning processes without neural data. To even reproduce these exact experiments in an animal you would need the neural data.\n\nGenerally it’s difficult to do well-controlled curriculum experiments and how to evaluate them is not always clear. CCT alone is a bit of a tricky metric to rely on. Looking at results from a single loss function, the different course completion times are to be expected as the courses themselves have different numbers of stages (just as completing an undergraduate degree should take longer and have different CCT times than learning about a subject by reading an article on wikipedia). How exactly do the authors plan on using this data to disambiguate learning processes in animals? One possible approach which I did not see (although I may have missed it) explained in the paper would be to simulate learning in network models of animal learning under different curricula and learning rules,  and then compare these sets of behaviour to different sets of animals' behaviour trained on each of the different curricula, to determine which set of network behaviours most closely matches the different sets of animals empirically. I think perhaps this is what the authors are proposing, as it seems sensible, but it should be made clearer. However, note that this requires several groups of animals (one group per curriculum) in a between-subject experimental design, which of course requires its own data collection pipeline and it is not the case that this data could just be recorded as a bi-product from ordinary animal studies, in which all animals should have as close as possible to the same training curriculum, in order to ensure they reach the final trained state for their main neuro task (which is not guaranteed under different curricula). \n\nAnother way to compare behaviour would be to subject different networks (animals) to different curricula, and then assess the effectiveness of each of these curricula on some later task. The learning curves can be expected to be different, but performance on the final task might be expected to be better under some curricula than others.\n\nIt’s really unclear (generally) how to design curricula that lead to qualitatively different learning trajectories under different hypotheses about animal or network learning. While I am sympathetic to the paper’s goals, in practice this really limits the experimental use case for this approach.\n\nA related paper that analysing biological learning dynamics to understand different potential underlying learning rules is Cao et al 2020. They show that the learning behaviour trajectory as well as the progression and final neural represeenation depend on the learning rule in linear networks (but dont use curricula to tease these apart): https://proceedings.neurips.cc/paper/2020/file/6275d7071d005260ab9d0766d6df1145-Paper.pdf\n", "summary_of_the_paper": "This paper simulates simple RNNs performing two classic decision-neuroscience experiments (a free choice evidence accumulation task and a delayed decision evidence accumulation task). The paper examines learning behaviour of these networks under three hand-crafted curricula, for each of two different RNN loss functions. The paper claims that one can diagnose the underlying learning rule (really the loss function) an RNN is using based upon the learning behaviour observed across the set of training curricula. They suggest a similar approach can be used to identify learning rules in animal neuroscience experiments.\n", "summary_of_the_review": "Interesting basic idea, but incomplete analysis to substantiate the paper’s claims about how this might work in practice with animal data.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "90JvauP9KPI", "reviewer": "Reviewer_Nbow", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- This research work is interesting and recent related work is adequately referenced.\n- The introduction and discussion sections are well-written and clearly motivate the research work and its potential impact. Vice versa, the methodological and result sections are very concise and often hard to read. I suggest rebalancing the text, by expanding and clarifying the presentation of results.\n- There are lot of hyper-parameters that seem to have been arbitrarily chosen (e.g., number of recurrent units, weights initialization schemes, events’ distribution, periods’ length, weight of representational loss, learning rates, accuracy percentage required to pass to next course, etc.). What was the rationale for setting all these parameters? How robust are the results to variations in these parameters?\n- I understand the idea (and the importance) of adjudicating between different “learning principles”. However, I am not fully convinced that the two regimens implemented here (BPT vs. BPR) faithfully represent the difference between target-based learning and representation learning: in both cases, the feedback provided is tightly related to the feature that needs to be discriminated (i.e., number of events), which makes both regimens quite biased toward a target-based learning condition.\n- Discrepancy is of course important for the task being simulated. However, I wonder whether the ratio between the number of cues should also be considered, since it should be easier to discriminate 5 vs. 6 compared to 15 vs. 16, though the discrepancy is the same -- see “size effect” and Weber’s law in psychophysics [1].\n- The focus here is mostly on neuroscience experiments where animals sequentially collect evidence (number of cues) during task execution. However, the approach proposed by the authors might also fit well with other recent frameworks that have been used to simulate numerical discrimination, where we could differentiate between target-based learning (i.e., the goal is to classify input numerosity [2]) or representation-based learning (i.e., the goal is to first build a generative model of the environment, and then eventually learn discrimination tasks [3]). Discussing these similarities could help in building useful cross-disciplinary bridges. \n- Readability of Fig. 3 could be improved. One solution would be to simply remove panels D, E and F, which contain “null” redundant information (learning is unsuccessful in both BPR and BPT) that can be easily embedded in the main text.\n- Also readability of Fig. 4 could be improved, along with the related discussion in Sec. 3.2. My guess is that the authors mistakenly swapped the bottom summary panels (EA vs. DD)?\n\nReferences\n1. Dehaene S, Dehaene-Lambertz G, Cohen L. Abstract representations of numbers in the animal and human brain. Trends Neurosci. 1998;21: 355–361. doi:10.1016/S0166-2236(98)01263-6\n2. Creatore C, Sabathiel S, Solstad T. Learning exact enumeration and approximate estimation in deep neural network models. Cognition. 2021;215: 104815. doi:10.1016/j.cognition.2021.104815\n3. Testolin A, Dolfi S, Rochus M, Zorzi M. Visual sense of number vs. sense of magnitude in humans and machines. Sci Rep. 2020;10: 1–13. doi:10.1038/s41598-020-66838-5", "summary_of_the_paper": "In this paper, the authors propose to use performance measures collected during curriculum learning as a way to adjudicate between different learning principles that might support (animal) behavior during a specific task. To this aim, the authors show that the learning principles governing two systems (RNNs) trained according to different objective functions can be inferred by examining the curriculum learning dynamics, despite the final “state” (effective dimensionality) of the networks is indistinguishable.", "summary_of_the_review": "Though the focus of this work is mostly on computational neuroscience, the research scope might be broad enough to be of partial interest to the audience at ICLR. However, the research has some weak points, and the conclusions are supported by a single experimental setup, thereby undermining the overall strength of the paper.\n\n------\nAfter revision, I update my score from 5/10 to 6/10.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-cWEkmjlnc-", "reviewer": "Reviewer_cUgC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n1. The paper is written very clearly, text and figures are easy to follow. \n2. The idea of using the learning time is new \n3. Experiments are well designed with 2 different but commonly used neuroscience tasks, and 3 different curriculum types.\n4.  The comparison of the learning process (loss function) on the basis of dimensionality, principal components, and state-space shows that it is difficult to disambiguate the learning process based on final representations thus serving as suitable controls to emphasize the comparison using curriculum learning time.\n\n\n### Weakness\n1. The plots in Figure 4 seem incorrect. In Figure 4A, in null curricula, the RNNs are unable to learn EA tasks while this is not the case in Figure 3 and as reported in the text. It seems that EA and DD columns are swapped. \n2. There are a few typos such as in section 3.2 unnecessary [] are present in the text. \n3. No caption for Figure 5E, what is the meaning of the asterisk (p < ?).\n4. In Figures 5 C, D, and E it is not clear what is compared to what and which values are significant or non-significant. Is evidence accumulation compared with the delayed decision or BPT compared with BPR? I encourage the authors to think about how to make the plots clearer so that the plots emphasize what is emphasized in the text. In the current state, from the Figures themselves, it is not clear. \n5. What is H in equation 4?\n\n\nIgnoring a few minor weaknesses, I believe the paper demonstrates the effectiveness of using curriculum learning time as a criterion to identify different learning processes very clearly, thus emphasizing the point to collect behavioral and neural data during the learning process to identify the learning mechanism in animals. \n", "summary_of_the_paper": "In this paper, the authors propose an approach using curricula to identify how a system has learned. Using two commonly used tasks in neuroscience: evidence accumulation and delayed decision recurrent neural networks (RNNs) are trained using two different loss functions (target based and representation based). They show that by simply comparing the learning time during different curricula , we can identify which loss function was used. On the other hand, the learned state-space trajectories of RNNs are indistinguishable thus unable to disambiguate which loss function was used for training.\n\nThe findings here put emphasis on the collection of behavioral and neural data while animals in neuroscience labs undergo curriculum-based learning on top of the data that is collected after the animal has learned.", "summary_of_the_review": "The idea of using the curriculum learning time as a criterion to disambiguate learning processes is novel and the authors have done a good job in clearly demonstrating that using a wise choice of tasks and curricula. However, the results are not conveyed to the reader correctly and can mislead the readers if presented in this form. Therefore, I encourage the authors to put some effort into improving the plots and correcting the mistakes. \n\nDue to the above reasons, I have given it a rating of 6 but I believe the ratings can be increased if the authors address the weaknesses.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In recent years, artificially trained RNNs have been used for studying systems and behavioral neuroscience in terms of their learned representations, dynamics, computation, and the learning process itself. This paper contributes to further identify learning principles that may be revealed by curricula. The proposed approach for finding signatures of different loss functions is a novel and interesting idea which is very well fit for the neuro-oriented ICLR audience and has potential impact in other fields.
Decision: Accept (Poster)