Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper simulates simple RNNs performing two classic decision-neuroscience experiments (a free choice evidence accumulation task and a delayed decision evidence accumulation task). The paper examines learning behaviour of these networks under three hand-crafted curricula, for each of two different RNN loss functions. The paper claims that one can diagnose the underlying learning rule (really the loss function) an RNN is using based upon the learning behaviour observed across the set of training curricula. They suggest a similar approach can be used to identify learning rules in animal neuroscience experiments.

main_review: Strengths:

The general idea of using learning behaviour under different training curricula to decipher something about the underlying learning process of a system is really interesting.
Simulating classical neuroscience tasks was a nice starting point, as it creates  opportunities to compare behaviour of these RNN models to real rodent and neural data, however this ended up being a bit of a  missed opportunity as there was no comparison to real animal data in this paper, as far as I could see.

Weaknesses:

There was a lot of confusion around terminology in the paper. Learning rules vs loss functions was a regular miscommunication. At many points in the text (including in the abstract) they say that their networks use two different learning rules (and so suggest the paper is able to diagnose different learning rules), when really the paper uses two different loss functions: one which uses the sum squared error loss on behaviour and the other which uses this same loss + a representational loss term. The learning rule is the same for both RNNs, which is batched gradient descent with an Adam optimizer.

The paper does not at any point, as far as I can see, actually use the RNN behaviour to predict the underlying loss function empirically, so claims that the paper shows they can disambiguate the loss functions (in the paper called ‘learning rules’), are not empirically substantiated. The paper could do this using the set of behavioural CCT data.

One of the two particular choices of loss function used in this paper require knowing the RNN’s (in their case, but animal’s in their suggested use case) internal representations, however one of the big motivators for this work was to gleam understanding about learning processes without neural data. To even reproduce these exact experiments in an animal you would need the neural data.

Generally it’s difficult to do well-controlled curriculum experiments and how to evaluate them is not always clear. CCT alone is a bit of a tricky metric to rely on. Looking at results from a single loss function, the different course completion times are to be expected as the courses themselves have different numbers of stages (just as completing an undergraduate degree should take longer and have different CCT times than learning about a subject by reading an article on wikipedia). How exactly do the authors plan on using this data to disambiguate learning processes in animals? One possible approach which I did not see (although I may have missed it) explained in the paper would be to simulate learning in network models of animal learning under different curricula and learning rules,  and then compare these sets of behaviour to different sets of animals' behaviour trained on each of the different curricula, to determine which set of network behaviours most closely matches the different sets of animals empirically. I think perhaps this is what the authors are proposing, as it seems sensible, but it should be made clearer. However, note that this requires several groups of animals (one group per curriculum) in a between-subject experimental design, which of course requires its own data collection pipeline and it is not the case that this data could just be recorded as a bi-product from ordinary animal studies, in which all animals should have as close as possible to the same training curriculum, in order to ensure they reach the final trained state for their main neuro task (which is not guaranteed under different curricula). 

Another way to compare behaviour would be to subject different networks (animals) to different curricula, and then assess the effectiveness of each of these curricula on some later task. The learning curves can be expected to be different, but performance on the final task might be expected to be better under some curricula than others.

It’s really unclear (generally) how to design curricula that lead to qualitatively different learning trajectories under different hypotheses about animal or network learning. While I am sympathetic to the paper’s goals, in practice this really limits the experimental use case for this approach.

A related paper that analysing biological learning dynamics to understand different potential underlying learning rules is Cao et al 2020. They show that the learning behaviour trajectory as well as the progression and final neural represeenation depend on the learning rule in linear networks (but dont use curricula to tease these apart): https://proceedings.neurips.cc/paper/2020/file/6275d7071d005260ab9d0766d6df1145-Paper.pdf

summary_of_the_review: Interesting basic idea, but incomplete analysis to substantiate the paper’s claims about how this might work in practice with animal data.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper shows how comparing network performance under different training curricula can distinguish learning principles. The curricula evaluated are inspired by those actually used in neuroscience to train animals to perform decision making tasks, and so the results motivate the analysis of behaviour during training periods which could provide evidence for/against candidate cost functions. 
main_review: strengths:

- novel idea to use curricula to find signatures of different loss functions
- curricula inspired by those used in neuroscience experiments - directly applicable. Recommends how (relatively) easily collected behavioural data could be used to infer loss functions being optimized during task training.

weaknesses:

- mostly relevant to neuroscience/psychology
- missing final analysis to really demonstrate claim in detail by using the CCT to adjudicate between candidate cost functions. Right now the main result seems to be in subpanel E of figure 5, but this is only describes that differences exist. What about an accuracy on actually using this difference to predict cost functions in some simulation comparable to a neuroscience/psychology experiment? I'd recommend moving some of the other figures/sub panels to the supplemental to make room for such an additional analysis.
- The motivation for the two cost functions and how they represent different learning "principles" could have been clearer.
- The presented analysis only shows how the CCT distinguishes the particular two cost functions evaluated here. As far as I can tell, the authors don't propose that behaviour under different curricula can say something generic about the learning principles at play. It's not clear whether CCT would distinguish between two arbitrary cost functions or how it would help infer learning principles in the absence of clear hypotheses.

minor comments:

- Figure 4 is very useful. It makes the results much easier to grok than in the text. It would be better if two rows corresponded to curricula. Right now they are not matched.
- check wording in 2nd last paragraph. not sure what you mean to communicate here: "Furthermore, we suggest that relative predictions, such as the ones we make from curriculum completion times in Figure 5, are invariant to shifts in their absolute quantity might that arise from extending a model to a real brain"
summary_of_the_review: I recommend acceptance because the novel demonstration that behavioural differences under different curricula can be used as a signal to compare candidate loss functions is valuable. However, the demonstration seems somewhat narrow in scope and it's not clear how useful it would be in practice, so my recommendation is not a strong one.
**********************
EDIT: updated my score from 6 to 8 based on author response and changes to the paper
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose an approach using curricula to identify how a system has learned. Using two commonly used tasks in neuroscience: evidence accumulation and delayed decision recurrent neural networks (RNNs) are trained using two different loss functions (target based and representation based). They show that by simply comparing the learning time during different curricula , we can identify which loss function was used. On the other hand, the learned state-space trajectories of RNNs are indistinguishable thus unable to disambiguate which loss function was used for training.

The findings here put emphasis on the collection of behavioral and neural data while animals in neuroscience labs undergo curriculum-based learning on top of the data that is collected after the animal has learned.
main_review: ### Strengths
1. The paper is written very clearly, text and figures are easy to follow. 
2. The idea of using the learning time is new 
3. Experiments are well designed with 2 different but commonly used neuroscience tasks, and 3 different curriculum types.
4.  The comparison of the learning process (loss function) on the basis of dimensionality, principal components, and state-space shows that it is difficult to disambiguate the learning process based on final representations thus serving as suitable controls to emphasize the comparison using curriculum learning time.


### Weakness
1. The plots in Figure 4 seem incorrect. In Figure 4A, in null curricula, the RNNs are unable to learn EA tasks while this is not the case in Figure 3 and as reported in the text. It seems that EA and DD columns are swapped. 
2. There are a few typos such as in section 3.2 unnecessary [] are present in the text. 
3. No caption for Figure 5E, what is the meaning of the asterisk (p < ?).
4. In Figures 5 C, D, and E it is not clear what is compared to what and which values are significant or non-significant. Is evidence accumulation compared with the delayed decision or BPT compared with BPR? I encourage the authors to think about how to make the plots clearer so that the plots emphasize what is emphasized in the text. In the current state, from the Figures themselves, it is not clear. 
5. What is H in equation 4?


Ignoring a few minor weaknesses, I believe the paper demonstrates the effectiveness of using curriculum learning time as a criterion to identify different learning processes very clearly, thus emphasizing the point to collect behavioral and neural data during the learning process to identify the learning mechanism in animals. 

summary_of_the_review: The idea of using the curriculum learning time as a criterion to disambiguate learning processes is novel and the authors have done a good job in clearly demonstrating that using a wise choice of tasks and curricula. However, the results are not conveyed to the reader correctly and can mislead the readers if presented in this form. Therefore, I encourage the authors to put some effort into improving the plots and correcting the mistakes. 

Due to the above reasons, I have given it a rating of 6 but I believe the ratings can be increased if the authors address the weaknesses.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose to use performance measures collected during curriculum learning as a way to adjudicate between different learning principles that might support (animal) behavior during a specific task. To this aim, the authors show that the learning principles governing two systems (RNNs) trained according to different objective functions can be inferred by examining the curriculum learning dynamics, despite the final “state” (effective dimensionality) of the networks is indistinguishable.
main_review: - This research work is interesting and recent related work is adequately referenced.
- The introduction and discussion sections are well-written and clearly motivate the research work and its potential impact. Vice versa, the methodological and result sections are very concise and often hard to read. I suggest rebalancing the text, by expanding and clarifying the presentation of results.
- There are lot of hyper-parameters that seem to have been arbitrarily chosen (e.g., number of recurrent units, weights initialization schemes, events’ distribution, periods’ length, weight of representational loss, learning rates, accuracy percentage required to pass to next course, etc.). What was the rationale for setting all these parameters? How robust are the results to variations in these parameters?
- I understand the idea (and the importance) of adjudicating between different “learning principles”. However, I am not fully convinced that the two regimens implemented here (BPT vs. BPR) faithfully represent the difference between target-based learning and representation learning: in both cases, the feedback provided is tightly related to the feature that needs to be discriminated (i.e., number of events), which makes both regimens quite biased toward a target-based learning condition.
- Discrepancy is of course important for the task being simulated. However, I wonder whether the ratio between the number of cues should also be considered, since it should be easier to discriminate 5 vs. 6 compared to 15 vs. 16, though the discrepancy is the same -- see “size effect” and Weber’s law in psychophysics [1].
- The focus here is mostly on neuroscience experiments where animals sequentially collect evidence (number of cues) during task execution. However, the approach proposed by the authors might also fit well with other recent frameworks that have been used to simulate numerical discrimination, where we could differentiate between target-based learning (i.e., the goal is to classify input numerosity [2]) or representation-based learning (i.e., the goal is to first build a generative model of the environment, and then eventually learn discrimination tasks [3]). Discussing these similarities could help in building useful cross-disciplinary bridges. 
- Readability of Fig. 3 could be improved. One solution would be to simply remove panels D, E and F, which contain “null” redundant information (learning is unsuccessful in both BPR and BPT) that can be easily embedded in the main text.
- Also readability of Fig. 4 could be improved, along with the related discussion in Sec. 3.2. My guess is that the authors mistakenly swapped the bottom summary panels (EA vs. DD)?

References
1. Dehaene S, Dehaene-Lambertz G, Cohen L. Abstract representations of numbers in the animal and human brain. Trends Neurosci. 1998;21: 355–361. doi:10.1016/S0166-2236(98)01263-6
2. Creatore C, Sabathiel S, Solstad T. Learning exact enumeration and approximate estimation in deep neural network models. Cognition. 2021;215: 104815. doi:10.1016/j.cognition.2021.104815
3. Testolin A, Dolfi S, Rochus M, Zorzi M. Visual sense of number vs. sense of magnitude in humans and machines. Sci Rep. 2020;10: 1–13. doi:10.1038/s41598-020-66838-5
summary_of_the_review: Though the focus of this work is mostly on computational neuroscience, the research scope might be broad enough to be of partial interest to the audience at ICLR. However, the research has some weak points, and the conclusions are supported by a single experimental setup, thereby undermining the overall strength of the paper.

------
After revision, I update my score from 5/10 to 6/10.
review_title: null

Meta-review: In recent years, artificially trained RNNs have been used for studying systems and behavioral neuroscience in terms of their learned representations, dynamics, computation, and the learning process itself. This paper contributes to further identify learning principles that may be revealed by curricula. The proposed approach for finding signatures of different loss functions is a novel and interesting idea which is very well fit for the neuro-oriented ICLR audience and has potential impact in other fields.
Decision: Accept (Poster)