Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents TrMRL (Transformers for Meta-Reinforcement Learning), which leverages transformer to learn hidden distributions for meta-reinforcement learning. Specifically, the authors claim that the model associates the recent past of working memories to build an episodic memory recursively through the transformer layers, which can help to adapt to new tasks. The conducted experiments in high-dimensional continuous control environments for locomotion and dexterous manipulation show that TrMRL achieves comparable results on performance, sample efficiency and out-of-distribution generalization in these environments.
main_review: Pros:
(1) This paper introduces a new variant of meta-reinforcement learning by leveraging transformer, which can capture long sequence and context dependence for fast adaption in new tasks. And the intuition to leverage transformer makes sense. 
(2) It is well written and easy to understand and follow. There are few typos here and there but nothing that affects the readability of the paper.
(3) They performed ablation study on how transformer architecture affects the performance. 

Cons: 
(1) In the idea level, it is trivial to combine transformer model with meta-reinforcement learning to learn hidden distribution over different tasks for fast adaption. As RL^2, it builds the policy from the episodic memory generated by the transformer, instead of RNN.
(2) The experimental results are not convincing. If I do not misunderstand it, I think N is the sequence length (correct me if it is wrong). The authors claim in the abstract that transformer can capture long sequence and context dependence for fast adaption, but why we set N=2 for all experiments? Is the transformer a good fit here? The experiment in appendix D.2 show it cannot get better result with long sequence. In Fig. 8, I also see it is not comparable to PEARL. 

Minor comments: 
(1)	A typo in Eq. 5? Is it a summation from n=1 to t?
(2)	N=2 for all experiments? Can you try different N in Fig. 8, in order to compare with PEARL?


summary_of_the_review: This paper introduces a new variant of meta-reinforcement learning by leveraging transformer, which can capture long sequence and context dependence for fast adaption in new tasks. Empirically, the proposed method shows some improvements over the existing ones. However, a few major concerns are as follows.
(1) The authors claim in the abstract that transformer can capture long sequence and context dependence for fast adaption, but why do we set N=2 for all experiments? Is the transformer a good fit here? The experiment in appendix D.2 show it cannot get better result with long sequence. 
(2) In Fig. 8, its performance is not comparable to PEARL. Is it possible to try different N and compare with PEARL?

Overall, it is a good paper, but not good enough for publication. More convincing experiments needed to show the advantage of transformer here.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper is proposing TrMRL (Transformers for Meta-Reinforcement Learning), a memory-based meta-Reinforcement Learner which uses the transformer architecture to formulate the learning process. 
It works as a memory associating recent working information to create an episodic memory which is used to contextualize the policy.
The paper deals with the recently discovered limitations of Transformer training that exhibit been important to consider the particular context of sequential decision making and over-task generalization given an underlying environment dynamics.
The approach is evaluated on Metaworld in various task transfer settings.
main_review: Strengths:
* The paper addresses an important topic of Transformer architecture for meta-RL
* The paper makes a valuable comparison of Transformer training settings and their respective impact on meta-learning
* The preliminary results on meta-world are rather encouraging.

Weaknesses:
* Several important baselines are missing like (https://proceedings.mlr.press/v139/dance21a.html and https://arxiv.org/abs/1801.01290 )
* The novelty of the model remains quite limited compared to the SoA Transformer current utilization in sequential decision making
* The experimental results don't seem significant in ML45 settings which is an important transfer setting of Meta-World
*  Maybe a connection to auto-prompting for zero-shot adaptation in controlled generation could have been interesting to develop (https://aclanthology.org/2020.emnlp-main.346.pdf).
summary_of_the_review: The paper is proposing an improvement of the transformer architecture, introducing a mechanism of memory reinstatement for meta-learning in sequential decision making.
The problem addressed in the paper is important and up-to-date.
Unfortunately, several state-of-the-art approaches are missing in the evaluation part.
Moreover, while several preliminary results are encouraging, the results depicted in ML45 remain limited compared to the proposed baseline approaches.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a Transformer-based approach to the meta RL problem, in which self-attention serves as a memory lookup mechanism for task adaptation. Experimental evaluation focuses on continuous control meta RL benchmarks.
main_review: This paper investigates whether the few-shot learning capabilities of Transformers make them well-suited to the type of task adaptation required in meta RL. This is an intriguing premise, and I agree that there is a connection here that is worth studying. Unfortunately, beyond the introductory premise I found this paper somewhat difficult to parse. Many components of the algorithm are stated qualitatively, without a precise enough description to be able to reproduce, such as:

> In this way, the transformer architecture recursively refines the episodic memory interacting output memories from the past layer.
As we compute each output memory in the perspective of a different timestep t, deeper layers reach
a consensus across the whole sub-trajectory.

I generally had a hard time understanding these descriptions in the paper. For this one in particular:
1. I am not sure what the recursive refinement refers to; could you write the recursion?
2. "Consensus" is not defined or used outside of this paragraph.
3. I think the "working memory" refers to feature vectors of recent transitions in the same spirit as the PEARL inference network $\Psi( z \mid s, a, s', r)$, and "episodic memory" is a function of summed working memory, as in Definition 6: $e_t = f(\mu_t^0, \ldots, \mu_t^\omega)$. However, as far as I can tell, $e_t$ is not referenced after its definition, so I do not know how it is used in the full algorithm. I also am not entirely sure what function $f$ is referring to. I assumed the memory terminology is the same as that in Fortunato et al 2019, but it would be better if these were precisely defined and motivated in this paper to make it more self-contained.

I did my best to keep pedantic issues off of that list and focus on areas where I genuinely found clarity issues. (For example, "from the perspective of" struck me as a slightly strange anthropomorphism of Equation 5, but it is clear from context what that means, so is not as important.)

While there is some focus given to the memory representation, there seem to be a number of missing details about the architecture and algorithm more generally. Most seriously, I could not find an actual objective aside from the general RL objective in the preliminaries. Since the experiments are in continuous control, presumably there needed to be some modification to the architecture described in Vaswani et al 2017?

The final stated contribution of this paper is a study of an important initialization method called T-Fixup. This ends up being a little underwhelming, with the ablation showing that it matters in one task but not in another. The explanation given is: 
> This result suggests that stabilizing learning since the beginning is very critical in the RL setting. This is often the
moment with higher exploration rates that prevent sub-optimal policies.

The first sentence seems a bit vacuously true, and I am not sure what the precise claim of the second is since the paper does not say anything about exploration.

summary_of_the_review: My main concerns about this paper are all related to clarity. It would help to have the full algorithm written precisely in one consolidated place (even if it's high-level pseudocode), rather than spread throughout the paper in informal descriptions with undefined terms.

I would be happy to revisit this if the authors would like to clear things up during the discussion period.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work studies the use of the transformer architecture to enable memory-based meta RL. The authors argue that transformer's attention is  inherently suitable to facilitate episodic memory which can be then inputed to the policy. Furthermore, the authors point out that transformer demonstrates better properties at  handling longer transition sequences. Finally, the authors suggest that training a transformer in RL is often unstable, and to this end they propose to apply T-Fixup initialization that evidently enables stable training.
The paper provides an empirical study on a several high-dimensional continuous control tasks that feature locomotion and dexterous manipulation to support the aforementioned claims.
main_review: Strengths:
1) The paper attempts to study an alternative network architecture (i.e. transformer) in RL, which is a direction that has a lot of importance and potential.
2) The authors provide an explanation on how transformers can be a natural fit for memory-based meta RL. They argue that the attention mechanism can straightforwardly implement memory reinstatement.

Weaknesses:
1) Re the transformer use: TrMRL proposes to use transformers exactly as if they were RNNs in RL^2. The authors suggest to use the current transitions' embedding to input to the policy. This is a trivial extension of the prior work and a very obvious one. Moreover it has been done in various prior works and in several different settings. This basically makes TrMRL to be a version of RL^2 where an RNN is replaced with a transformer.
2) Re PEARL results: the demonstrated learning curves of PEARL directly contradict the claims from the original paper and my intuition. I think that it is very unlikely that TRPO based MAML or PPO based RL^2 (both are on-policy) can outperform that dramatically SAC-based PEARL (off-policy). It is a common knowledge that off-policy methods are more sample efficient than the on-policy methods. This also makes me question the correctness of the whole experiment, as TrMRL also uses PPO as an RL backbone. 
3) Re T-Fixup initialization: the claim that the T-Fixup initialization helps is not supported in any experiment or an ablation study. It thus unclear how useful it is or essential.
4) Re Fig 4: Why does this compare performance during meta-training rather than meta-testing? In the end we are more interested in the latter.
5) Re Fig. 5: It would be more informative if other baselines are plotted here to server as a reference point.

summary_of_the_review: While that paper studies an important direction in RL of adopting new network architectures, such as transformers, the paper, unfortunately, brings very little novelty or insight.  The proposed method largely builds off of the existing work (such as RL^2) and essentially proposes to replace RNNs with transformers. The empirical study is also questionable and,  in my opinion, misleading. Moreover, the experimental methodology is a suspect as it doesn't provide an extensive ablation or support all the claims.

Taking this into account I suggest to reject the paper, as it in its current form doesn't meet the acceptance bar.
review_title: null

Meta-review: At a high level, the novelty of this paper is limited: RL2 with transformers instead of RNNs. The emphasis is then placed on the experimental evaluation. Unfortunately, the reviewers felt that the experimental methodology and results were not strong enough at this stage to warrant publication. During the rebuttal, the reviewers did not engage nor discuss the author response, unfortunately, so I do not know what they think of the rebuttal. However, on evaluating the concerns of the reviewers against the updated manuscript, I think the updates do not go far enough to satisfy the concerns raised (experiments + baselines). Therefore, I recommend rejection.
Decision: Reject