Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: 
Summary:

The authors present an approach that leverages inductive biases for knowledge decomposition and relational reasoning with the aim of generalizing over out-of-distibution tasks in RL with stationary rewards.  They propose a meta-learning training framework where parameters of the network are divided into two subsets where one set is learned with fast updates and the other with slow updates capturing the slow changing dynamics of the environment. 

They also propose an architecture that includes N modular components in the form of recurrent independent mechanisms (RIMS, Goyal et al. 2019) and two multi-head dot-product attention components: input attention for selecting k modules and communication attention between the selected modules.  The attention parameters are updated slowly and model the stable dynamics of decomposing task input information as meta-learning parameters while the modules are updated more frequently while the attention dynamics are held constant.  To train the policy and value function the authors chose to use Proximal Policy Optimization (PPO).  Here the value head is updated slowly with the attention meta-parameters while the policy is updated in sync with RIMs.

This approach is evaluated against GridWorld and BabyAI suite tasks where the task complexity is parameterized and involve navigating in grid worlds with partial observability conditional on input instructions.  Generalization performance is measured on more complex out-of-distibution tasks and the authors also carry out ablation study on the meta-learning setup as well as analysis on the learned value function and module communication sparsity.
 
Strengths & Weaknesses:

Meta-learning and factored representations are important areas of study if we are to build RL agents that can generalize well and learn new tasks approaching the sample efficiency or small amounts of data required, for instance, by humans.  Therefore it is key to build meta-learning into RL agents and so this work I believe is well motivated and aims to contribute to a crucial area of study.  The work here seems to be largely inspired by Bengio et al. (2019) in learning via a meta-transfer objective among causal entities where the authors use meta-learned multi-head attention mechanisms facilitating the interaction dynamics of a network where the modular components are realized as RIMs.  While this work bears some strong similarity to this work I believe that there are some key differences in the operation of the two attention mechanisms, leveraging the  composability of RIMs and learning framework that make it a novel contribution.

I think there are a few things in the paper that could benefit from more detail, for instance, it would be helpful to provide the definition of recurrent dynamics D_j and how the attention, RIMs and RL all connect together quantitatively.  Also a bit more definition around the tasks in the main paper could be helpful.  I realize fitting things in can be tricky but even a small bit more description could help.

Is it hard to tune the loop sizes over the meta-sequence learning?  I think the impact of this work depends on this and it could be nice to see some analysis or get a high level sense of how different loop sizes affect learning. 

Overall I think the results are good and the analyses are helpful in understanding the utility of the approach over the baselines in terms of sample efficiency and generalization performance over more complex tasks.  Another baseline that is able to meta-learn some parameters but without a modular architecture might have been a good point of comparison.  Although, the ablation study with the modified LSTM is a simple version of this it seems.  The analyses of the value function and module sparsity are good and I think help the overall argument.

The tasks evaluated on are GridWorld and the BabyAI.  These tasks are fairly simple toy tasks but are well suited to demonstrate the power of the model to a limit.   It would be helpful to see the approach evaluated against more complex environments where factorizing representations is more difficult to noise and complexity in the task distribution.  This might help better illustrate the horizons of this area of work.

Finally, the paper is clearly written and well structured.

Recommendation:

I think overall this is a nice paper and would have no problem to see it accepted.  I think the work could be more ambitious but I believe that the authors have illustrated that this is a potentially useful innovation for RL.
review_title: Review of "Meta Attention Networks: Meta-Learning Attention to Modulate Information Between Recurrent Independent Mechanisms"

final_score: rating: 7: Good paper, accept
review_content: Summary of the paper:

The paper introduces a meta-learning approach to recurrent independent modules, with the goal to make RIMs adapt better to changing distributions, increasing generalization, and increasing sample efficiency. It postulates that RIMs, with their decomposition of independent information, can more easily adapt in these cases. The introduces approach wraps a "meta-learner" in form of two different update speeds around RIMs. The "inner" loop updates the RIM module parameters with large (fast) updates, while the "outer" loop, consisting of input and communication attention, does small (slow) updates. The authors then evaluate the approach on the reinforcement learning domains of "MiniGrid" and "BabyAI". Comparing their approach with an LSTM wrapped in a similar "meta-learning" layer, they show that their approach outperforms the LSTM, and is generally able to quickly adapt to changes in the environment.

Commentary on the goal of the paper:

Learning in non-stationary environements is an open area of research, with a lot of potential. Combining the intuitions between two different approaches - the modularity of a RIM, and the adaptivity of a "meta-learner" is a very appealing idea.

Strengths:
- The paper's main approach is both intuitive and simple
- The results are good, showing clear improvements over the baseline
- The paper is generally well-written, and well-structured

Weaknesses:
- The paper lacks detail in the description of the proposed approach. How exactly does the overall architecture work? How do the different attention mechanisms interact? From the explanations and descriptions given, I would not be able to reproduce the architecture.
- The paper should have focused more to outline the reasoning behind the fast-slow learning design: why does the "outer loop" learn slowly, while the "inner loop" learns slowly? This is against my intuitions: I would have expected the outer loop to learn fast, and the inner slow - after all, the appeal of modular architectures is to re-use the modules, not the composition.
- The paper overstates the novelty of their approach. Under "modular networks" on page 5, it claims that the approach of modular networks is "hampered by the amount of domain knowledge required to decide or predict (Johnson et al., 2017) how the modules should be parameterized, and how they should be connected with each other". While this is true for the early work of Andreas, this statement completely ignores later research that tries to address exactly this shortcoming ("Routing Networks" by Rosenbaum et al 2017; "Modular Networks" by Kirsch et al 2018; "Automatically Composing Representation Transformations as a Means for Generalization" by Chang et al 2018; "Recursive Routing Networks" by Cases et al, 2019). In particular "Routing Networks and the Challenges ..." by Rosenbaum, 2019 already mentions slowing the training of the outer loop.
- The paper falls short on qualitative analysis: what do the modules learn, what is transfered, and how does this help with non-stationary environments?
review_title: A decent paper with a great goal where several small issues outweigh the benefits

final_score: rating: 7: Good paper, accept
review_content: **Objective of paper**: Exploring methods that exhibit better systemic out-of-distribution generalization/adaptation properties. Specifically, exploring how modularity, combined with learning at different time scales, can help with this. 


**Central claim**: By expanding Recurrent Independent Mechanisms with a meta-learning layer to facilitate learning at different time scales, it is possible to achieve better within-distribution and out-of-distribution/systemic generalization and fast adaptation to new tasks.  


**Strong points**: 
1.	Important research direction: I believe this research direction has a lot of value in exploring and improving the way our current deep learning systems can exhibit better systemic generalization. 
2.	Novelty: While most of the individual components presented in the paper are not novel, the particular combination of these components and the empirical evaluation seem novel to me. (That being said, the idea of meta-learning gating mechanisms to combat catastrophic forgetting and interference (arguably the central contribution of this paper) has been explored in continual learning before – for example [1]). 
3.	Compelling experimental results: In terms of raw performance, the proposed method clearly outperforms its Recurrent Independent Mechanisms (RIM) and LSTM counterparts in sample complexity, better systemic generalization and faster adaptation. I also think that the tasks picked for evaluation are suitable to investigate systemic generalization. 


**Weak points**: 
1.	Lack of architectural/algorithmic benchmarks in experiments: Given that systemic generalization, as well as combatting catastrophic forgetting, are active research areas, it would have been nice if the paper compared its results with alternative, competing approaches. If you think there are no other comparable approaches in the literature, then it would be nice to discuss why in the related works section. It might also be helpful to try out simpler versions of the proposed method to see what component really made the approach work. For example, to really double down on how much the meta learning component matters, could the authors try training all of the parameters at the same time (i.e. no meta-learning step) but with a small learning rate assigned for the attention parameters (maybe ¼ of the current learning rate, since you update it 4 times less than the dynamics parameters, if I’m not mistaken )? 
2.	Claim about knowledge transfer not supported well enough: Besides the fact that the knowledge transfer results are reported on a small percentage of the environments, I’m not sure if we can conclude “knowledge transfer” is responsible for better performance when one uses models pretrained on similar tasks. Couldn’t this be due to some simpler explanation, such as better optimization landscape for pretrained models? I think the authors can address this by first pretraining the models on completely irrelevant tasks where knowledge transfer is not expected to help and showing that in this case the reported gains in performance are *not* shown. It might also help elaborate on what is meant by “knowledge”. 
3.	Hyperparameter tuning procedure not described: The hyperparameter tuning procedure was not described in the paper (I might have missed it too?). It would be particularly interesting to read more about how the authors arrived at updating the meta parameters once for every 4 inner loop updates. 
4.	Lack of “hard” environments: It seems from the results that both LSTM and RIM already perform well on most of the tasks shown in the paper (though slower and with more data). Is this a correct assessment? Are there tasks that only the proposed methods can solve? (I guess zero-shot learning results are somewhat of an example, but it would be nice to see harder environments too)
5.	(minor) The fact that number of modules (N) and number of active modules allowed at a time (k) are fixed: I guess this can be viewed as a weakness, thought this is shared in RIM as well, and not that relevant to what is being evaluated/investigated in the paper. 


**Decision**: This paper makes an algorithmic contribution to the systemic generalization literature, and many in the NeurIPS community who are interested in this literature would benefit from having this paper accepted to the conference. I'm in favour of acceptance. 


**Questions to authors**: 
1.	Is there a specific reason why you reported results on knowledge transfer (i.e. section 4.3) only on a few select environments? 
2.	As mentioned in the “weak points” section, it would be nice if you could elaborate on 
3.	Is it possible that the caption of Figure 4 is misplaced? That figure is referenced in Section 4.1 (Improved Sample Efficiency), but the caption suggests it has something to do with better knowledge transfer. 
4.	If you have the resources, I would be very interested to see how the “small learning rate for attention parameters” benchmark (described above) would compare with the proposed approach. 
5.	In Section 4, 1st paragraph, you write “Do the ingredients of the proposed method lead to […] a better curriculum learning regime[…]”. Could you elaborate on what you mean by this?


[1] Beaulieu, Shawn, et al. "Learning to continually learn." arXiv preprint arXiv:2002.09571 (2020).

review_title: Interesting Extension on Prior Art, and Compelling Results

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a meta-learning (called) method for the model based on RIMs architecture to realize fast adaptation on new unseen tasks. 

The base model is the same as RIM, which is composed of multiple independent LSTM modules. An input attention is used to determine which independent modules are activated, taking output state from them. Another attention fuses information from activated modules. This paper proposes to update the parameters of attention less frequently than LSTM modules instead of updating them with the same frequency, as they believe that attention should capture a more stable distribution of the task.

However, there are some concerns about the motivation of this paper. There is no doubt that meta-learning can achieve faster adaptation between domains. But why should we divide the parameters into two sets? How about treating all parameters as a single set and updating them under a traditional meta-learning way in which parameters are separately optimized on each distribution and the average loss among distributions is used for gradient calculation. In addition, will the distribution change much for different frames within the same task?

This paper is a bit hard to understand as there are too many long sentences with complex clauses. It is better be improved via revision.

Some questions:

1. concerns about motivation.

2. the current strategy is similar to setting a lower learning rate for parameters of attention modules. Does the paper have any results to prove that the proposed method is better than using different learning rates?

3. why setting the step length of slow update fixed as four times of fast update? Would it be better to determine it dynamically? (e.g. if the frame distribution difference has exceeded a threshold then taking outer update)

4. it is unclear about the parameter updating strategy of baselines (vanilla LSTM and RIM). All parameters in them will be updated under the same frequency. But is the frequency the same as the fast update or slow update in the method.

review_title: Concerns about motivation.

Meta-review: The paper examines an idea that knowledge and rewards are stationary and reusable across tasks. An interesting paper that combines number of related topics (meta RL, HRL, time scale in RL, and attention), improving the speed of training.

The authors have addressed the reviewer comments, strengthening the paper. The reviewers  agree, and I concur, that the paper contributes a novel model,  valuable to the ICLR community. It is well thought-out, presented, and evaluated. 
Decision: Accept (Poster)