Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the Authors design and implement recurrent neural networks with two types of plasticity: classical Hebbian and gradient-based w.r.t. an internal loss function. The Authors then test their models on a set of memory/learning tasks including copying, cue-reward association, image classification, and regression. They find that the new gradient-based approach fares better on the regression task while the classical Hebbian approach is better in copying and image classification tasks. An ablation study is provided to assess the importance of the models’ parameters.
strength_and_weaknesses: This is a thoroughly-designed and well-described study of plasticity in recurrent neural networks, which builds upon previous results by Jeff Clune, Ken Stanley, and colleagues, known to the ICLR community.

The paper introduces several interesting updates on top of the previous works. First, the “learning rate of plasticity” \eta here is time-dependent, which supposedly allows the models to turn on the synaptic plasticity during one/few-shot learning, and to turn it off to retain the learned knowledge during testing. Second, in the gradient-based method, the Authors introduce internal loss. This loss relies on an extra output of the RNN, conditioned upon the RNN’s weights trained in the outer loop, and is used to guide the network’s plasticity in the inner loop. For both these updates, the Authors provide (in the Appendix) the ablation studies showing how different values of these parameters affect the models’ performance.

The internal loss part seems especially interesting because it may link this work to a biological phenomenon of motivation. Previously. motivations in machine learning were set to reflect observable quantities such as curiosity; conversely, the internal cost functions here are defined by the model itself to reflect the important task-related quantities.

Another strength of this work lies in the comprehensive testing of the models which includes standard benchmarks and thorough ablation studies. It would be nice though to move more figures to the main text, if possible, as they show important results.

Speaking of weaknesses, the Hebbian-plasticity part of the work is highly similar to that in a series of works by Miconi et al, mentioned by the Authors (the similarity is also acknowledged by the Authors). The differences (which need to be clearly summarized somewhere in the text!) seem to be limited to 1) the time-dependent internal learning rate for synaptic plasticity, 2) not using Oja’s rule, and 3) using a (more complex) image classification task instead of the Omniglot. Furthermore, the proposed gradient-based rule seems to be only efficient in the few-shot regression task among the used benchmarks. These two factors limit the contribution of this work to the field. I also found the biological insights a bit preliminary – though they do not constitute the main result of the work.

clarity,_quality,_novelty_and_reproducibility: The text is well-written; the descriptions of the experiments contain sufficient information for reproducibility. The figures are good at conveying the results of the experiments (perhaps, they could be moved to the main text; also, little titles for individual panels within the figures may help parse them faster).
summary_of_the_review: This is a comprehensive study describing two mechanisms of synaptic plasticity in RNNs. The properties of the described models are thoroughly tested. In the future, some of the design choices made here (quite interesting!) may help build neuroscience-relevant models. The drawback is in high similarity to prior work (Hebb’s rule part) and high similarity to Hebb’s rule scores on benchmarks (gradient part).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Inspired by the evolved role of plasticity in neural circuits, this paper proposes a meta-learning approach for training RNNs. The method consists of an inner loop with a learning rule that updates the parameters of an RNN, and an outer loop that learns a function for the update rule. 
strength_and_weaknesses: The proposed meta-learning approach is interesting especially from a neuroscience perspective (though not novel, e.g. Najarro and Risi, NeurIPS, 2020). However, I’m not convinced that the proposed model has any advantages compared to the previously proposed methods. In particular, almost all the comparisons are between the proposed method and non-plastic RNNs which I found insufficient. 

$\textbf{Questions}$:

1- Almost all the comparisons are with what is called non-plastic RNNs in the paper. The motivation behind using non-plastic RNNs as the baseline model for comparisons is not explained in the paper. Specifically, the initialization of weights could place the untrained RNNs in a disadvantaged dynamical regime compared to the trained RNNs. It is hard to imagine a situation where non-plastic neural networks are considered as an alternative for how brains function, except in cases where, for example, the RNNs are initialized at the edge of chaos (criticality) which could give them specific advantages (better short-term memory, etc.) Can the authors elaborate more on the motivation behind comparing only with untrained RNNs? 

2- The update rules of the outer (meta-learning) rule are not shown in the paper. It was only mentioned that gradient descent was used for the outer loop, but the loss function and the target variables are not explained. Was the MSE loss shown in figures 2-3 used for the outer loop updates? Also, mentioned in section 3.2 “ [...] network parameters, including those that define the learning rules in the inner loop, are meta-trained with gradient descent.” A list of parameters meta-trained with gradient descent would be helpful.

3- Related to comment # 1, it would be interesting to see comparisons with RNNs trained in a supervised (or unsupervised) fashion (no meta-learning). The main advantages of the meta-learning approach in associative memory and few-shot learning would be more clear with those comparisons. 

4- One general comment: Although a key concept in the paper, the connection with meta-learning did not become clear until the end of the introduction (not even mentioned in the abstract). Concepts like “self-determined target” and “neuromodulated plasticity” are used a few times throughout the introduction before mentioning that the proposed method is a meta-learning algorithm. Because of that, It's difficult to follow the introduction and understand the main question of the paper. I suggest that the authors rephrase the introduction and highlight the connections with meta-learning at the very beginning. 

clarity,_quality,_novelty_and_reproducibility: The proposed method requires more clarification for the meta-learning update rule. The proposed method is not very novel, and the quality of the paper is negatively affected by the lack of more thorough comparisons. 
summary_of_the_review: Although the proposed meta-learning algorithm is interesting (not novel), the comparisons with untrained (or non-plastic) RNNs is not sufficient for demonstrating the strengths of the proposed method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The manuscript introduces a new way of learning task-specific unsupervised learning rule for RNNs in which the network computes not only its task output but also dynamically alters hyperparameters of its own learning process. The learning rule functional form is learned in a gradient-based outer loop based on task constraints provided via a validation set. The model is validated on a wide range of tasks and shows interesting few shot learning performance improvements relative to a static network.

strength_and_weaknesses: 
+ Strength: impressive performance
+ S: analogies to biology as a motivation for the general approach
- Weakness: the logic of the optimization procedure is not spelled out as clearly as it should be
- W: alternative model comparison is restricted to variations of the same scheme, hard to evaluate the benefits of the approach more broadly w.r.t. alternative few shot learning approaches
- W: conceptually unclear to me if the hebbian vs gradient based distinction is a technical choice or if one of the two versions is championed in the paper

clarity,_quality,_novelty_and_reproducibility: - the functional form of the internal loss was unclear (seems to aim to shrink the norm of the total output but it is unclear to me why that would be a generally useful task agnostic thing to do)
- the timescales at which various bits change in the model are hard to parse at times; some visual for the inner/outer loop, which parameters they change and based on which loss would be very useful to. navigate the model complexity
- the justification of the alternative models and the logic of their selection is critical for interpreting the results so it should be included in the main text
- the exact details of the experiments w.r.t. to losses and how things were set up is insufficient
summary_of_the_review: Overall an intriguing idea and interesting results, but the presentation of the core methodology and its justification is too sparse and needs substantial improvement before publication.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper compares two forms of meta-learning, with inner-loop updates based either on modulated Hebbian plasticity, or on gradient descent over a network-generated synthetic loss function. 

Various experiments show that both forms of episodic plasticity (Hebbian and gradient-based) improve performance over using fixed recurrent networks, as in RL^2/L2RL. Network-controlled modulation of plasticity is also shown to be important.


strength_and_weaknesses: Strengths:

- The paper tackles an interesting problem (learning to learn without necessarily having access to a supervised or even a reward  signal during each episode, i.e. the inner loop is unsupervised).

- The paper is interesting and may propose one novel method, i.e. network-generated gradient loss, though see below.

- The experiments, such as they are, offer reasonable evidence that the proposed methods are useful over the studied tasks


Weaknesses:

My main concern is about novelty. AFAICT the Hebbian portion of the method is identical to previous proposals that are cited in the paper. 

Similarly, the gradient-based  method seems to be an example of meta-learning a loss function to perform gradient descent on in the inner loop. This approach is clearly not new, see e.g. https://arxiv.org/abs/1802.04821 , https://arxiv.org/abs/1906.05374 , https://arxiv.org/abs/2110.03909 and  the many references therein.  (Disclaimer: I am not affiliated with any of these authors)

Surprisingly, no previous paper on this approach (meta-learning a loss function) seems to be cited in the paper, unless I missed it.

There *may* be some novelty in the proposed approach, because here the synthetic loss  is an output of the learner network itself (as opposed to a separate dedicated network). IIUC this means that the synthetic loss itself is subject to learning and tuning during the inner loop (as  in Baik et al. mentioned above, but in a much more flexible manner). This also gives rise to interesting loops in the learning process (though the precise working of the algorithm is a bit unclear, see below).

If this is correct, and if this is the claim to novelty of the paper, it should be stated explicitly.

Additional possible  problems:
- The paper is somewhat unclear on certain points, which is important to better understand the proposed method. See below. 
- The experiments are interesting, but perhaps a bit limited?
- Not really concerns, but suggestions for future work: the two approaches (Hebbian and gradient-based) seem fully compatible and it might be interesting to see what  happens when a network uses both together; the weight decay parameter, which is now just 1-eta, could include a separate learnable parameter as is Tyulmankov et al. and Rodriguez, Guo && Moraitis ICML 2022 (I am not affiliated with these authors).
clarity,_quality,_novelty_and_reproducibility: Two main points require clarification:

- The description of the network operation is split over several section and somewhat  confusing. Most importantly, exactly which parameters are updated during the inner  vs. outer loop? This should be  specified explicitly.

- The outer loop meta-optimization process should be described more explicitly, especially for the gradient-based case. E.g. do we compute full second-order gradient-of-gradients? Or is there some first-order approximation (like first-order MAML, or REPTL) ? 

For novelty, see  above.

summary_of_the_review: == Update after reading the authors' response ==

I thank the authors for their clarifications and I have updated my review and score towards acceptance.

== Original review ==

Please cite more existing literature about the concept of meta-learned loss functions; explicitly state the novelty of the work; and clarify the working of the algorithm.

I am willing to increase my score depending on the author's response to the above, especially regarding novelty.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents an approach to training recurrent neural networks that combines Hebbian weight updates with gradient descent. The authors show that this approach improves few-shot learning in memory and association based tasks.

The strengths of this paper are its clarity, and post rebuttal, its reproducibility, both of which will allow it to serve as the basis for future work.

The principal weakness is that the novelty is somewhat limited.

The reviewers all noted the issue of novelty, and originally, several were concerned about reproducibility. However, following the authors' replies, the reviewers agreed that with sufficient references to past work and clarification the novel contributions are sufficiently clear to warrant acceptance and scores were bumped up. The final score was still technically borderline (average score of 6), but there was a total consensus on this score after rebuttal (all reviewers gave a 6), which indicates sufficient agreement that this paper is appropriate for acceptance.
justification_for_why_not_higher_score: There was clear agreement amongst the reviewers that this is a relatively incremental contribution, which is also my assessment as an AC. Thus, although it is appropriate for acceptance, it is not sufficiently novel or cutting-edge to warrant a spotlight or oral.
justification_for_why_not_lower_score: Given that the paper is clear, most of the concerns were well addressed in the rebuttal, and that much of the concern was about novelty in a manner that was addressed in part by better positioning relative to past work, it does not seem appropriate to reject this paper.
Decision: Accept: poster