Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an offline meta-RL method that uses a causal transformer as a backbone model to predict actions . In particular, this method learns policies conditioned on a long context using a casual transformer where the context is built based on trajectories from multiple offline tasks. To show the effectiveness of their method, this paper runs various experiments on Adversarial Bandit, Dark Room, and DMLab Watermaze benchmarks. 
strength_and_weaknesses: While I find the result of this work interesting as it shows that it is possible to do more efficient offline meta-RL when the problem is formulated in the right way (e.g. using long context), I have a couple of concerns about this work:

- Meta-RL:
Using context/history in meta-RL has been studied in previous works [1,2,3] and all those works show that encoding past histories leads to significant improvement in performance (see results in [2]). Although the setting of those papers are online-RL, the similarity is uncanny and this work should have discussed those extensively in the paper (I'd suggest adding a section to discuss this). The only difference that I see with this paper and those works is use of transformer vs recurrent models to encode the context aside from online vs offline setting. Also, is it fair to say that the main contribution of this paper is to adopt the use of context/history in the decision transformer? 

- Choice of baseline:
This paper uses RL^2 as a baseline. However, previous works [1,2] showed that RL^2 has poor performance in comparison to other meta-RL methods (see figure 2  of [1] ). Hence, choosing a better baseline method could provide a better conclusion about this paper.
  
- Improvement over behavioral policy:
Authors claim in multiple parts of the paper that "AD learns to output an improved policy relative to the one seen in its context." I am confused about this as I don't see the results support this claim. Take figure 4 as an example, AD doesn't outperform RL^2. Am I missing something here? 

- Ablation studies:
"Can AD learn a more data-efficient RL algorithm than the one that produced the source data?" experiment shows that AD outperforms source algorithm. The setting of this experiment can be inconclusive as the source method and AD have different model sizes (?) and it appears to me AD uses larger network sizes. If that is the case, it is not a fair comparison. Moreover, "Training the Sequence Model" in page 5 can be a good experiment as long as it is a fair comparison in terms of model sizes. Finally, it would be useful if authors add a table in the appendix illustrating AD and baselines network/model sizes for each experiment in the paper to provide a better picture about the setting of the experiments. 

- Context/History:
The paper says dataset D contains a set of "learning" histories. However lines 3 and 4 of Algorithm 1 show otherwise. In particular, Algorithm 1 shows that D contains ONLY expert data as optimal policies are used to collect data. This weakens the entire discussion of policy improvement in the paper as data contains only expert data.


[1] Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables, ICML 2019 http://proceedings.mlr.press/v97/rakelly19a/rakelly19a.pdf

[2] Meta-Q-Learning, ICLR 2020 https://openreview.net/pdf?id=SJeD3CEFPH

[3] Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs, ICML 2022 https://proceedings.mlr.press/v162/ni22a/ni22a.pdf

clarity,_quality,_novelty_and_reproducibility: The paper reads well and it is clearly written. There are a good amount of experiments done in the paper except for the choice of baseline methods (see my comments in Strength And Weaknesses). Finally, as I mentioned in the previous section, the use of context for meta-RL is not something new and previous works studied that, though in online-RL. 
summary_of_the_review: This is an interesting paper and the results are promising. However, I'd make my final recommendation after rebuttal as I require some clarifications from others. In general, the proposed method is not original and based on previous meta-RL works. Using the decision transformer with context is the main contribution of this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to train a transformer to imitate offline data coming from a RL algorithm. When trained on a distribution of tasks, it can then generalise the behaviour of the RL algorithm into the new task. This is in comparison to recent papers using transformers to imitate already-trained RL policies, instead of algorithms. In this regard, it is closer to meta-RL algorithms like learning to reinforcement learn or RL^2, with the difference of being offline rather than online. The paper then evaluates on a wide variety of environments and analyses many different factors involved in the proposed approach.
strength_and_weaknesses: - Strengths
    - I think the evaluation of this paper is very very good. The diversity of environments is great, and most of all, I was impressed by the many different questions the author asked themselves about the proposed approach, from out-of-distribution, to training on partial demonstrations or the importance of the length of the context. Great job!
    - The paper reads very clearly. In part this is because it sets out to do one job and do it well, and I appreciate that.
    - The idea is insightful and potentially useful in the long run for meta-RL.
- Weaknesses
    - I think the originality of the proposed approach is good enough to pass the bar for acceptance, but I wouldn't consider it great. It is relatively close to GATO and RL2. It is also close to multiple works that are not cited but very relevant: Bootstrapped Meta-learning [Flennerhag et al., Outstanding Paper award last ICLR] as well as Upside-Down RL [Schmidhuber] and, less closely, the learning-to-optimize community [Li&Malik '16, Andrychowicz et al. '16].
    - In multiple places, the comparison to the standard (non-learned) RL algorithm needs to be described more in favor of the latter rather than the proposed approach. For instance, it is argued that the "algorithm" coming from AD generalizes out-of-distribution, but this wouldn't apply to widely different tasks: an RL algorithm applies to image-input problems and torque-based inputs just as fine, but the algorithm from AD wouldn't have nearly that level of generality (and which other methods in meta-RL have started to achieve). Furthermore, the sentence "the source algorithm produces many single-task agents [...] while AD produces a single generalist agent" while technically true is quite incomplete, as AD needs to store each individual history to be able to represent each specialist agent.
    - A point that concerns me from the evaluation was that in the main figure, RL2 is shown only in its asymptotic value, as if it was an Oracle whose performance couldn't be put as just another curve (which may surpass AD). One may argue that it could be unfair for AD, as it is online rather than offline, but the same can be said about the Source being on a non-meta setting and not exploiting meta-training data, being at a disadvantage w.r.t. AD. I would really appreciate a clarification on this.
clarity,_quality,_novelty_and_reproducibility: - Quality and novelty
    - As argued in strengths&weaknesses, I think the novelty is not huge but is relevant enough. I do think the execution of this paper is very good, except for my concern on figure 4.
- Clarity
    - The paper in general reads very well. I think the connections to meta-RL comes a bit late, as I was initially confused on not seeing the reference to the field despite it being pretty clearly an offline meta-RL method. Also, as mentioned under weaknesses, a bit more care needs to be made to claim that AD distills DQN/PPO/A3C, as it probably only imitates them properly on nearby tasks.
    - One thing that wasn't clear to me was how much of the learning history needs to go into the context, and if we "skip" parts of it on purpose to try to learn faster or in order to fit in memory. I saw figure 7, but I wasn't sure whether that only meant at test time. If also at training time, how can it learn pretty well with only 1 episode of inner-loop experience?
summary_of_the_review: The community will benefit from this paper as it is a very thorough exploration of a relevant idea, particularly in the current LLM-for-everything climate. The writing is also clear and it reads very well.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes Algorithm Distillation, a transformer-based method for imitating the learning process implied by a reinforcement learning algorithm. The learning process is emulated by simply imitating the sequence of actions produced by an algorithm given a history of observations with the environment.
strength_and_weaknesses: Strengths:
- The approach presented in this paper is at the same time very simple and intellectually stimulating;
- The empirical investigation is detailed and answers interesting questions one might have about this type of approaches;
- The presentation of the ideas is clear and concise.

Weaknesses:
- While I believe the careful investigation of such an idea is definitely worth even without an immediate implication on how to leverage it in a more standard context (i.e., when should I both to imitate my RL algorithm instead of simply running it?). It would be better to have a direct discussion on which ones would be the implication of improving this type of methods in the long run;
- Is the context length the only blocker in scaling this method to more complex tasks? From the current paper, it is not clear whether more complex transformer models (with longer context sizes) could actually be enough to scale to the usual reinforcement learning benchmarks.
clarity,_quality,_novelty_and_reproducibility: The clarity of the paper is really good.

The quality of the research work is good, both in terms of experiment design and execution.

Despite its simplicity, the method presented in this paper is novel and interesting.

There are no clear reproducibility issues.
summary_of_the_review: In short, I believe this is a well-executed investigation of a really intriguing idea. I recommend acceptance, but I encourage the authors to elaborate on the possible long-term consequences or more general implications of the development of such a class of methods.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The authors propose AD.  AD works by training a transformer (or other causal sequence model) on RL histories of agents that are learning (exhibiting improving performance during the history).  Then after this training, when allowed to run on the environment, the transformer continues to improve performance, resulting in improving learning curves without any parameter updates.

strength_and_weaknesses: Edits:
- intro: “does not improve through trial an error”, “and” missing a “d”

Strengths:
- Well written; this paper was a pleasure to read.
- To the best of my knowledge, this work is novel.
- The fact that this approach worked was quite surprising to me.  This “surprise factor” makes this an important and interesting topic in my opinion.
- I also think that this kind of “in-context” learning (in other words, “learning” only within what-can-be-viewed-as-analogous-to the model’s short-term memory, without updating parameters) is an important thing to study. I hope this will inspire future work that better studies the intersection of this kind of learning with more traditional learning.

Weaknesses:
- The paper would greatly benefit by a more clear early definition of “in-context”, instead of making the reader infer it through context.  (There is sort of a definition in the abstract, but even that is more of an implied definition than a definition.  I suggest adding a more clear definition to the early introduction.)
- One might argue that the contribution is slightly limited: the authors simply propose a new idea and show that it works.  However, given the novelty and importance of this topic, I am not very concerned about the extent of the contribution.

clarity,_quality,_novelty_and_reproducibility: This paper is clear and, to the best of my knowledge, novel.  See above for more details on quality.
summary_of_the_review: This paper explores a topic that I believe is extremely important; the result (the fact that this approach worked) was surprising to me.  It is well-written, I have no significant concerns, and I have very little to say other than to recommend an accept.  I would consider recommending a “strong accept” if other reviewers felt similarly.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a transformer-based imitation learning (offline RL) method that replaces the return-to-go in Decision Transformer by a long context. The key idea is that the long context contains the history of a policy improvement, and thus we can benefit more from this information than simple return-to-go scalar, e.g., for generalization. The experiment is performed very thoroughly on various tasks such as adversarial bandit, Dark Room, DMLab, etc and successfully demonstrated the claimed benefits. There are no clear reproducibility issues.

Strength. The idea is insightful and novel. The evaluation is very thorough and comprehensive. The paper is well and clearly written. The method is also simple (in a good way!) and insightful. "The fact that this approach worked was quite surprising"


Weakness. Relationship to the related works could be more clarified.

justification_for_why_not_lower_score: The fact that it works is quite surprising to me and (to another reviewer). It seems quite important to share with the community that this idea works.
Decision: Accept: notable-top-5%