Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents the problems in chaining nonlinear motor pattern generators and proposes a resetting mechanism inspired by the thalamo-cortical network.
main_review: Strength: The proposed architectures have been shown to work for "zero-shot learning," or recombination without re-learning.

Weakness: It is not convincing whether the particular resetting mechanism is the most reasonable one, either computationally or biologically. Computationally, any strong inhibitory input to bring the network state to the origin should suffice. Biologically, the most likely mechanism is the tonic inhibition and timely disinhibition by the basal ganglia to the thalamus and subcortical motor controllers like the superior colliculus.
summary_of_the_review: An interesting proposal worth discussion.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors seek to use a small RNN to generate extended, time-varying outputs given a cue indicating which output to generate. They focus on the problem of generating a sequence of outputs, cued in series, without the end state of one interfering with the subsequently cued trajectory. They show that small vanilla RNNs do not readily perform this task: RNNs at a random state can generate a sequence when cued, but they fail to do so when their state is at the end point of another sequence. The authors then show that by biasing the activity of the network towards the origin in the absence of an input (which they suggest is inspired by neuroscience), RNNs more readily learn sequences and more consistently generate accurate sequences in series. My enthusiasm for the work is relatively low for a variety of reasons. Primarily, the authors fail to test obvious alternative hypotheses of ways that the problem domain they design could be solved by a learned system, without the need to add the inductive bias they add. Moreover, and more fundamentally, the relevance of such a small toy system (300 units, trained to generate just 10 trajectories) for machine learning is unclear.

main_review: The authors discuss two important alternatives to the approach that they ultimate pursue, which would involve attacking the problem with learning rather than with building in inductive biases. Rather than running experiments to test the efficacy of these alternatives, the authors instead tried to argue away these points (bottom of pg. 6). This is extremely unsatisfying. 

Learning has proven remarkably powerful, and incorporating constraints/structure/inductive biases, may indeed be necessary in some domains to overcome current challenges, as the work the authors cite suggests. But those domains are ones in which learning has been doggedly pursued for an extended period of time. These authors instead introduce a novel synthetic task and immediately jump to the need to incorporate more structure, without doing the legwork necessary to convincingly demonstrate that such structure is indeed necessary.

To directly respond to the authors arguments over why these alternatives are insufficient:
First, they suggest that directly training the RNNs to generalize from end states is insufficient, because they can't train from all possible end states, particularly as the number of trained states scales. This argument is unpersuasive, as training on all of anything is often impossible in general, yet still learned systems generalize. E.g., work with ImageNet over the last decade provides a great example of this. 

The authors could instead test this possibility by incorporating the end points into training as initial states for subsequent trajectories. They could hold out some transitions and test how well the system generates the motifs on these left-out transitions. They could hold out the transitions in two ways: Either exclude some transitions randomly (i.e., salt and pepper of the transition matrix) or hold out some motifs from the training at all (i.e., holding out rows or columns of the motifs). The latter would be a stronger test, but both would be important to see. Critically, for the authors to convincingly show that learning was insufficient, they would need far, far more than 10 motifs (e.g., like >>1000, most likely).

Second, the authors could train to return to a consistent point at the end of each motif. The authors assert that doing so would make the system not robust to perturbation/noise, but that would be true only if the system actually learned to arrive at an identical point each time, which I imagine would not occur (and if they did, they could add noise to the network during training). The network could return to the origin imperfectly and convey state information (e.g., on 299 dimensions return to origin, but on one axis keep information about). Who knows if these speculations are correct -- but an experimental test of these ideas would be quite useful.

My second major concern is the relevance of this work to the broader field. The networks are small and seemingly toy: 300 units, 10 motifs. This is smaller than MNIST. I'm not one to think that everything needs to be shown on ImageNet and achieve SOTA to be a useful contribution, but the scale of these experiments are so, so small that I worry that anything learned in this domain would transfer or inform work at far greater scales -- particularly when the conclusion is that in such a low data regime, greater inductive biases are required rather than learning.


Minor issues:

- The authors assert that thalamic activity may change the effective connectivity of motor cortical activity, but they don't make clear why that change would result in a bias towards the origin. In what sense is biasing the network to the origin motivated by neuroscience? Also, how is preparatory activity being the same across all motifs brain-like? Isn't that not seen in the brain (e.g., in Susillo et al. 2015)?

- The authors seem to repeatedly suggest that matching the marginals of a distribution is a strong approximation to the joint -- e.g., "Subtle variations in the state of the network at the end of different first motifs x_{end}^{\mu} can lead to very different outcomes". I don't understand why the authors are overstating the extent to which the marginals and the joint are the same thing...

- The authors assert that "a standard normal is a good choice" for the marginals of the states. Please provide some quantification for this claim. As it stands, the authors visualize cumulative distributions of a subset of the states rather than performing statistics. 

- How did the authors pick a standard deviation of 1.2 (in Fig 2.a.iv-v)? Did they fit this number to the marginals?

- Typo, top of page 5: "require less neurons" --> "require fewer neurons"
summary_of_the_review: The small scale of the models/data combined with the lack of evaluation of alternative approaches diminishes the potential contribution of this work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The article proposes a biologically inspired architecture which increases the degree to which simple recurrent neural networks may be able to chain together elemental output sequences. The authors show that the proposed "predatory module" allows networks to perform on par as if the network were starting from a set initial state, while also providing a smooth transition between motifs. 
main_review: Pros
- The modification that's required to the "control" network in order is extremely simple, and has a clear theoretical goal of decreasing "residual" activity between motif transitions, without quenching activity in response to the new control signal.
- The level of experimental detail is sufficient such that the experiments  in the main text can be replicated by an independent reader.

Cons
- The methods in the main text are clear, and lead to convincing results in the main-section. However, without digging much deeper into appendix 4 & 5, there's no clear reason why the learned preparatory modules are able function
summary_of_the_review: The preparatory module appears to function well, and the quenching activity is reasonable well tied to biological thalamocortical loop actions. To my knowledge, the motif-transition problem is not solved in switching recurrent networks, however I am not overly familiar with that literature. Overall, my low confidence ratings come from not knowing how interesting or novel the analytical work in the appendices is.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper considers the problem of RNNs learning arbitrary sequences of “motifs” (continuous time functions, discretized) after having learned the individual motifs separately. Inspired by some previous work, it proposes an architectural approach to deal with interference by introducing a motif-dependent low-rank perturbation to the recurrent weights of the RNN, following Schuessler et al. (2020). To this end, the paper introduces a new evaluation task (motif generation), which is used to compare that architecture and sensible baselines. It then moves to propose a new model inspired in thalamocortical interactions in the brain: an RNN equipped with a preparatory module that imposes a beneficial bias when following a specific training protocol.
main_review: **Strengths**

Continual learning is specially relevant as a setting for the purposes of this paper, partly given that learning from non-stationary distributions is closer in many aspects to the way humans learn. As such, the problem of catastrophic forgetting and generalization in continual learning are important research topics that are quite relevant for this venue. In this sense, the study of some of the architectures presented here (e.g. inspired on Schuessler et al. 2020) are interesting. Also welcomed are explicit addressing of the continuous time problem, which is relevant for many fields including control. Finally, the attempt to connect with Neuroscience by introducing a model inspired by thalamocortical interactions with the preparatory module is also acknowledged.
 
**Weaknesses**
	
- **Technical**: At the technical level, I found a couple of issues that affects the task described in section 1. First, there are references to the continual learning setup, but I can’t find the specifics, such as figures and detailed comments about the task structure etc., apart from mentioning the inclusion of all the motifs vs a sequential presentation (for which results are not shown). If the models are to be analyzed in this setting, it is important to see a robust description of the task and the metrics (see, for example, Chaudhry et al. 2019). Also, although I recognize the value of synthetic tasks, I also would have preferred to see in addition a task that encodes a “naturalistic” dataset, specially given that the authors are interested in a general noise-robust solution.

	Second, there is a problem that the authors seems to be well aware of as per the second to last paragraph in section 3. If I understood it correctly, the task described in section 2 admits a trivial solution, that of a hard resetting to an initialization state before processing a motif. This raises some considerations, including importantly that point (i) in that paragraph should be demonstrated empirically, regarding to different possible initialization schemes.
	

- **Writing** In my opinion this article suffers a series of important defects in its writing that ultimately have a strong negative impact in the communication of its scientific ideas. Please note that the intended feedback here is not about writing style. We acknowledge that not all papers are meant to be written the same way. However, I found reading this paper to be somewhat arduous and the presentation confusing. It also contains many questionable decisions into what constitutes the main text versus what goes in the appendix (e.g. presenting the numeric values of the hyperparameters for Adam in the first paragraph of the task description, while no formal description of the problem is included in the main text). 

	Being constructive, I think this situation could be alleviated by introducing a better formalization of the task and the optimization problem in a simpler manner.
	- *Description of the problem*:
	The description of the problem in section A.1 could be improved by stating the optimization problem in simple terms. This can be done collecting information that is scattered in section 2, and introducing the statements with a notation consistent with the rest of the paper (I found strange that uses other notation from the main text). Please note that in section A.1 some terms are not defined (“descriptor vectors”) or introduced but seemingly not used later (N_V is stated to be as small as possible). Also, no discretization method is specified, for example, and the information about initialization is scattered.

		A beautiful example in this domain that you cite is Schuessler et al. (2020), which targets a similar conference in section 2: the equation of evolution of the RNN seen as a dynamical system (e.g. what you have in your first equation), followed by a clear specification of each of the terms (input, output, parameters, target function, etc.), parameters specific for each task, details about training and initialization, etc. 
	
	- *Description of the task*:
	Section 2 in the paper is devoted to the description of the task and introduces the three architectures that are used in the rest of the paper. However, after reading the section, it is not clear what the essential components are. Given that the introduction of the task is a core contribution of the paper, a more structured description would be really appreciated, especially the details concerning continual learning. This should also go in the main section of the paper (it could be short, but it should be complete). Also as a good example of what I mean is Schuessler et al. (2020), which opens section 2 with a very concise description of the task.

	**Minor presentation issues**
	At a different level, the paper would benefit from:
	- less strange wording in many cases (e.g “more powerful performance-optimized continuously nonlinear RNNs”),
	- other small fixes (numbering equations, normalizing the references (use \citep in latex), fix a number of typos (e.g. “gradient decent” last paragraph in Section 1), etc). 

**References**

Chaudhry, Arslan, Marc’Aurelio Ranzato, Marcus Rohrbach and Mohamed Elhoseiny (2019) Efficient Lifelong Learning with A-GEM. *ICLR 2019*

Schuessler, Friedrich, Francesca Mastrogiuseppe, Alexis Dubreuil, Srdjan Ostojic, and Omri Barak (2020) The interplay between randomness and structure during learning in RNNs. *NeurIPS 2020* 
summary_of_the_review: The general topic of this paper is relevant, but I found the presentation confusing and difficult to understand. Overall, given the current state of the presentation, some issues with the details of the continual learning experiments, and some concerns related to the existence of a trivial solution that would need to be addressed, the contributions of this paper remains unclear to me and I suggest a rejection.
review_title: null

Meta-review: This manuscript presents a method to allow RNNs to chain together sequences of behaviors. Reviewers had numerous concerns but the most important is that the problem posed here is solved by a simple method: resetting the state of the RNN before processing a motif.

Overall, reviewers noted a few key topics, although this list is not exhaustive:
1. Experiments are in a very simple but confusing setting.
2. Even though alternatives exist to solving this problem, they are not considered.
3. The networks considered are very simple.
4. The manuscript is difficult to understand.
5. The task admits a trivial solution.

In more detail:

1. The setting of learning to memorize time series and outputting them on command is very simple compared to what most modern work considers. Moreover, there is much confusion in the manuscript about what the setting is precisely.  For example, the setting is described as "independently learn motor motifs in order to build a continuously expandable motif library". But there is no continuously expandable motif library, the motif library is fixed at test time. The authors focus heavily on calling this setting "motor motifs", but these are RNNs that output an arbitrary time series. They are in no sense motor programs and this work is not connected to the extensive literature on motor control in machine learning. More broadly, there is no clear mathematical definition of what the problem being solved is anywhere in the manuscript.

2. It is unusual for manuscripts to not present other baseline models. But more importantly, many other approaches exist to this problem. As one reviewer pointed out, the manuscript essentially sets out to solve a problem that is completely solved in machine learning today. It rejects the solutions that exist for arbitrary reasons, and then adopts its own new solution.

3. The models used are very simple, but this is a consequence of 1, the problem domain being very simple.

4. Reviewers had difficulty understanding the details of the task. In particular, the task description section begins with minutia about the implementation rather than succinct definition of the task.

5. Most critically, reviewers identified that the model could be hard reset and would have the same behavior as the model presented in the manuscript. The proposed solution is essentially hard resetting the state to zero as it stands. There is no reason why a hard reset cannot be followed by a smoothing operation -- this seems to be the main objection of the authors.

Overall, the manuscript needs significant improvements. The task considered is too simple by modern ML standards and the fact that it admits a simple solution cannot be overlooked. Demonstrating the idea of the preparatory module on an existing ML task and dataset, while comparing with existing baseline models, carrying out ablations, and producing an extensive quantitative evaluation is what will get the community excited about preparatory modules.
Decision: Reject