Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose MetODS: Meta Optimized Dynamical Synapses, a meta-RL algorithm that learns by updating its weights through interaction with the environment and its own current weight state. The update rule is recursively applied and allow the algorithm to learn relations between stored patterns and incoming information. MetODS notably uses an element-wise weighting, that allows for different plasticity amplitudes at every connection. The authors then experimentally compare their algorithm to RL2 and MAML and analyze the performances of their algorithms against these baselines in terms of efficiency (One-shot learning and rapid motor control), capacity (achievable level of performance for a distribution of task) and generality (how well the policy transfers to tasks unseen during training).  
strengths_and_weaknesses: The paper is well motivated by biological and existing artificial neural methods. It clearly presents a novel meta RL approach and compares its performances to Meta RL baselines (namely RL2 and MAML). While the claim on efficiency and generality seems to me well-supported by the experimental evaluation, I have difficulties to understand how the evaluation on the randomly generated grid world helps to evaluate the capacity of the learner. Are different generation patters used between training and evaluation? 
Otherwise, I think that the authors could sometimes add high level motivational details on the design choices of the algorithm (notably on the multi-step scheme).  
questions: What is the (high level) intuition behind the "read" and "write" operation of the update rule ?

writing suggestion/typos:
l 99 and l 296: broken citation
l 139: Synapses (with capital S)
l 174: s instead of S 
l 290: "to" is repeated
limitations: I haven't seen any discussion on potential negative societal impact. I, however, cannot really find non-generic remarks on this topic for this work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Post-response update
----------------

The authors have improved the paper and addressed some of my concerns; while I still find the assessment to be problematic in some areas, I think the present results, together with the interesting idea, make for a paper that could be a useful contribution to the field. I have updated my score accordingly.

Original review
--------------

This work presents an interesting and fairly novel perspective on meta-RL: having an agent adapt to a task by modifying its weights through an iterated Hebbian update process. The paper demonstrates that this approach outperforms vanilla baselines (RL^2, MAML, and sometimes PEARL) across a variety of settings that stress different aspects of adaptation. This is overall an intriguing approach that I hope to see published either here or in the future.

strengths_and_weaknesses: 
Strengths:
The idea is interesting—it’s nice to see a fresh perspective on adaptation drawing on dynamical systems perspectives from neuroscience.
The breadth of demonstrations is fairly compelling; it’s great that the authors articulate different aspects of the meta-RL problem and test each of them.
The results on impared robots and with continual adaptation (in the appendix) are particularly intriguing, because they suggest an increased robustness inherent in this approach; however, this difference would be more compelling if the comparison algorithms were trained to matching train performance (see below). 


Weaknesses (in order of importance):
* Some of the evaluation choices seem like they could unfairly handicap the baselines; therefore I am not sure how much to trust the overall conclusions of the paper. 
  - The authors limit to 10M steps for the MetaWorld ML1 and ML10 environments; but the original MetaWorld paper achieved much higher performance with RL^2 and MAML after much longer training (300M steps). So the evaluation seems to artificially handicap the baselines, by training for much less time than the baseline approaches require. Indeed, in ML1-reach and ML10 the curves for one of the baselines appear to be passing the proposed MetODS approach just where the plot ends. Would the baselines perform better than MetODS if the experiments were run for longer?
  -The authors limit their approach to a single layer of adaptable weights—did they make a similar limitation for MAML? I wasn’t entirely clear from either the main text or the supplement. If the baselines are restricted compared to the original work this limitation would be a major caveat that should be stated much more explicitly in the text. Algorithms like MAML are generally developed and tested with deep networks; it would not be appropriate to test them with a different architecture without doing full hyperparameter tuning; and even with hyperparameter tuning the conclusion should come with caveats if the authors are artificially limiting the baseline approaches. It’s possible that the authors are not doing this; I am happy to be corrected if so, but in either case I would suggest they be more explicit about precisely what the baseline approaches were in the paper when they revise it.
  - Unless I’m misunderstanding both the above points, I find statements like “producing better agents than previous meta-RL approaches” or “compares favorably with prior meta-RL algorithms” to be somewhat misleading. Ideally the baseline experiments should be run using architectures comparable to the original papers, for as long as the original paper in at least a subset of the domains. At the very least the plots should show e.g. the max performance figures achieved in the original MetaWorld paper as a baseline comparison in each case (e.g. a star or dashed line with label like “Fully trained MAML” etc.), and the statements about comparisons (e.g. in intro and conclusions) should qualify that MetODS performs better “[for adapting a single layer of weights] at the beginning of training.”
* MetODS consists of a complicated set of changes to the architecture; some ablation experiments should be performed to demonstrate the contributions of different changes. A few such changes are tested for Harlow (S=1 and no element-wise weights), but it would be ideal to see these on a few other tasks, as well as conditions like the following: 
  - It wasn’t entirely clear whether beta and kappa are element-wise parameters or scalar; assuming the latter, switching to the scalar version would be a useful ablation. 
  - What if the authors kept all aspects of MetODS but switched from the Hebbian update rule to an alternative rule. Ideally this would include something like a hypernetwork (see below), but if that is too computationally expensive there are a variety of changes that could be explored, e.g. instead of the updating based on the outer product of v with itself, what if learned linear projection weights from v to two new vectors q and p and then used a (weighted) outer product of q and p? This may not be the best experiment; the point is just that more exploration of which aspects of the update rule are important (e.g. whether the Hebbian update is somehow intrinsically useful)  would help future researchers understand what to explore.
  - Adding features to the baselines could be useful too in determining where the real benefits are; e.g. if the authors used MAML with S (smaller) updates per timestep rather than 1, how does it compare? What if they augmented MAML with parameter-specific meta-learned update weighting and smoothing constants alpha and beta? Etc.
* I found the connection to the policy transport perspective somewhat unsatisfying. A substantial portion of the paper (~1.5 pages) is spent presenting this perspective, but it seems like the efficiency, capacity, and generality can be (and have been) previously defined without adopting this perspective.
  - I believe that the policy transport perspective could be removed or substantially reduced; indeed, I think that this would perhaps make the paper clearer by focusing more directly on the algorithmic contributions.
  - The three concepts defined have prior precedents in the literature; for example the notion of cumulative regret has a long history in RL, has often been used in meta-RL (e.g. Wang et al., 2016), and is effectively a measure of efficiency. The paper should connect these concepts to the relevant prior ideas.
  - If the authors include the policy perspective in the paper, it would be ideal to *compare* their approach to the baselines with respect to policy transport. For example, if they visualized policy transport for other approaches (e.g. MAML) like they do for their own approach in Fig. 2C, and then *quantified* the advantage of their approach in terms of policy transport (e.g. showing that their method takes a more direct path in policy space than the baselines), that would be much more compelling.
  - As one additional note of interest, the policy transport perspective reminded me a bit of a recent paper I saw on how fast policies update with a gradient update to the weights (Schaul et al., 2022); it seems potentially relevant to thinking about the policy transport perspective, although it is not primarily focused on optimal changes in the policy so much as noisy ones (to my understanding; I’ve not actually read Schaul et al. yet).
* The discussion of the prior literature seemed a little skewed to me in places; while it is true that few prior approaches have pursued fast weights “as a function of the current synaptic state or external reward signals” per se, there has been work on using fast weights in Meta-RL (e.g. Sarafian et al., 2021) that probably should be discussed as relevant background.

The authors do not necessarily have to run every experiment suggested above (which might be infeasible) for me to consider the paper publication worthy. However, the paper would need to would need to compare to stronger baselines (including the original MetaWorld results, if it is computationally infeasible to train their own strong baselines), would need to more clearly describe the experiments they performed, and would need to outline the limitations of the experiments and approach in more detail.

questions: See above. Briefly:
* Which aspects of each experiment might be unfair to the baselines? Would the baselines outperform the proposed algorithm if the experiments were run for longer?
* Which aspects of the approach are necessary (justified with ablation experiments)?
* Can the authors justify the policy transport perspective further, e.g. by comparing to the baselines with this perspective?


limitations: See above. Briefly:
* Unless I am misunderstanding something, the experiments are a limitation because the baselines are handicapped (e.g. by far less training).
* The conclusions we can draw from the paper are limited by the lack of ablation experiments, to identify which aspects of the approach are essential.  
* The authors mention briefly that they leave extending their plasticity rule to multiple layers to future work. Of course, Hebbian rules don't tend to work well for training deep networks, and this point might deserve slightly more emphasis (e.g. a brief sentence in the discussion to highlight the challenge).
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a new neural network that is able to modify its own weights, known as the self-modifying network.
By incorporating self-modifying networks into the meta reinforcement learning (RL) framework, the new model Meta-Optimized Dynamical Synapses (MetODS) shows better empirical performance in terms of efficiency, capacity, and generality, compared to several meta RL baseline algorithms.
strengths_and_weaknesses: This work is well motivated. The proposed algorithm MetODS shows significantly better performance across different kinds of tasks, from maze navigation to motor control.

Nevertheless, It is hard for me to understand the reasons behind the success of self-modifying networks and MetODS.
In Section 3, it is claimed that "Our model learns to train itself by updating its weights through interaction with the environment and its own current weight state. This mechanism enables MetODS to rapidly compress experience of a task $\tau$ into a particular synaptic configuration ......" Arguably, an RL model using traditional neural networks optimized with SGD also update its weights through interactions with the environment, based on its current weights. What is the critical difference between self-modifying networks and classical neural networks? More explanations and analytical experiments are needed to show the advantage of self-modifying networks.

Small typos:
- Line 48, conjonction --> conjunction.
- Line 11 in Algorithm 1.
- Missing one citation in Line 296. 
- Missing x & y labels in Figure 6.

Finally, I do not find Figure 1 very helpful.
questions: What neural network structures are used for baseline algorithms such as MAML and RL2?
limitations: I may misunderstand something here, but in equation 5, it seems that computing $v^{(s)}$ and $W^{(s)}$ requires whole trajectories of $v^{(l)}$ and $W^{(l)}$ to be stored, which leads to a large memory requirement.
The discrete adjoint sensitivity method only helps with the backward pass but not the forward pass (i.e. computing $v^{(s)}$ and $W^{(s)}$). How about the forward pass?
review_title: null

Meta-review: metareview: This is exciting work that demonstrates the ability of self-modifying networks to solve meta-reinforcement learning problems. The reviewers all agree that this is strong work, and the authors have convincingly addressed most of the concerns the reviewers brought up during the reviewing phase. There are a few lingering questions about the applicability of the baselines, but these are quite minor. The authors have further promised to add analytical comparisons and additional details /motivation on the Hebbian update. Given this, I view this paper quite positively and encourage the authors to integrate the additional experiments and details they mentioned in the feedback stage.
confidence: Certain
Decision: Accept