Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new multimodal benchmark for language conditioned RL settings, where an agent must complete tasks specified by text instructions in the Atari Frostbite environment. Their benchmark provides a dataset of 5M text-labeled transitions for training. Finally, the authors propose a model for language conditioned RL settings based on Transformer architecture as a baseline.
main_review: I agree that this paper addresses an important problem of language-conditioned imitation learning. However, It seems that the contribution of the proposed model, Text Decision Transformer (Text-DT), is slightly marginal and Text-DT seems not related to Decision Transformer. The detailed comments and questions are as follows:

1. The biggest difference between standard Transformer and Decision Transformer is the return-to-go, but there is no return-to-go in Text-DT. The Text-DT, proposed in this paper, is thought to be just using the standard transformer as a conditional sequence generation method, so I am not sure if it is right to call it a decision transformer. Why is it called a (Text) decision transformer?

2. The authors use Text-DT and TDT interchangeably in their papers, is there any difference? If there is a difference, it should be clearly explained, and if there is no difference, I think the terminology should be unified.

3. (Figure 3) In easy/medium tasks, TDT(k=1) performs better than MLP baseline, but in hard tasks, MLP baseline shows better performance than TDT(k=1). What is the reason for this?

4. As an experiment in Figure 3, how does the performance of the MLP baseline change as the context length increases? It would be better if the authors could compare and show them together.

5. The authors describe the proposed Text-DT model as the main contribution, but there does not seem to be much difference from the existing works using the standard transformer architecture. Rather, it would be better to show the contribution to the point of proposing a new benchmark more intensively.


summary_of_the_review: I think that this paper presents an interesting benchmark for developing RL agents which can utilize language. However, the proposed model, Text-DT, is considered to have a small contribution because the Transformer architecture is simply applied to the conditional sequence generation problem with language inputs. In addition, the authors explain the proposed model based on the decision transformer, but it is not considered to have much related to the decision transformer. I would like to recommend a more detailed explanation of the proposed benchmark and possible future studies.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a new language conditioned imitation learning task based on the Atari Frostbite environment. In addition, the paper studies imitation learning with a transformer architecture and the effect of pre-training on game trajectories.
main_review: Pros
* Learning behavior policies from language supervision is an important problem setting
* Useful to study the applicability of the transformer architecture to sequential decision making problems
* New benchmark on language conditioned agent learning

Cons
* The new proposed benchmark needs to be contextualized in existing work. Why is there a need for this benchmark and how is it different from existing language instruction following benchmarks from the embodied learning literature (Eg. VLN benchmarks like Room to room and object interaction benchmarks like ALFRED)?
* Models and training approaches considered in the paper seem straightforward. What are the new technical contributions of the work?
* Weak experiments 

I would encourage the authors to think about what’s new and interesting about the proposed new dataset, what makes it challenging and how it is related to and different from other existing benchmarks.

Problem setting is unclear - In section 5.2, I don’t understand why the model is trained on the evaluation task. Also, the section ends abruptly without a conclusion. 

The conclusions on pretraining and comparisons against context size 1 seem obvious and expected. I didn't learn much from these experiments. The experiments needs to provide more insight into the dataset and the results. 

Presentation 
- Alg 1 can be moved to the appendix. 
- The MLP baseline wasn’t clearly described. 
summary_of_the_review: The paper lacks novel technical contributions. The experiments are weak and do not provide new, meaningful and practically useful insights. The work further needs to be contextualized better in existing work.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes a new instruction-following reinforcement learning environment based on the Atari Frostbite environment. They experiment with a modified decision transformer on the environment. The results largely show that performance improves with longer context, more pretraining, and more data.
main_review: I think this paper lacks important detail in that it proposes a new benchmark but declines to compare such benchmark to existing work. The claim that "transferring sequential knowledge is relatively unstudied" is just simply not true. There has been several works, old and recent, on instruction following. These include the large quantity of visual instruction following tasks such as R2R (https://bringmeaspoon.org/), Room-Across-Room (https://ai.google.com/research/rxr/), Touchdown (https://arxiv.org/abs/1811.12354), DRIF (https://github.com/lil-lab/drif), ALFRED (http://askforalfred.com/), as well as complex grid worlds with new language-entity associations and hard generalization requirements (https://arxiv.org/abs/1910.08210, https://arxiv.org/abs/2101.07393). In order to propose this benchmark, the paper ought to compare the proposed environment with these existing work - why is there a need for this new benchmark? In what ways is it different than previous environments? This crucial question is not addressed in this work. These related work and many others (https://arxiv.org/abs/1906.03926) are not even cited. Another point of contention I have with the way this work is posed is that it claims 5 million annotated transitions. What does this mean? It seems like there are only a few hundred labeled trajectories. Do the authors means they have a couple hundred labeled trajectories, each with 10k steps (where the task is fixed)? If so, I think this is at best misleading - no other dataset for language grounding that I am aware of lists annotation size like this.

Here are some concrete examples of claims with missing analyses:
1. "a diverse set of tasks" - how are these tasks diverse? What do the tasks even look like? The authors mention 500 tasks but there are only  8 listed in the appendix. Upon inspection of these tasks, they seem very unusual (die on the x level, try to get as close as you can to 1000 score). How many unique tasks are there? What does it even mean to "get as close as you can"? Or "spend as much time as possible"? How is the reward assigned?
2. "temporally extended strategies" - what does this mean? what is the distribution of trajectory lengths of annotated playthroughs per task? of agent playthroughs per task?

Moreover, what is the evaluation set? How is this set constructed? What skills is it evaluating? Is generalization required? If an agent just cycles through memorized action patterns from the training distribution, can it still do well on the test set?
summary_of_the_review: No comparison to prior tasks, unclear what new proposed environment tests and how it compares to prior environments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a corpus of text instructions for the language guided Atari Frostbite environment, and also presents a study of pretrained Text Decision Transformer on the corpus, shows that it can improves upon baselines, yield better results in the low-data regimes. 
main_review: Strengths:

1. This paper introduces a quite large text corpus for the text game community. 
2. This paper presents some baselines for pretrained text decision transformers. 

Weakness:

May need some interpretations for some weird cases in the Table and Figures. 

For example, 

1). In Table 4, it looks more context does not help for Medium tasks?

2). Similarly, in Figure 3, it looks MLP is quite a strong baseline on the Hard tasks? 
summary_of_the_review: This paper introduces a quite large corpus of text instructions for the language guided Atari Frostbite environment, which is good resources for the community, and it also presents baselines for pretrained text decision transformers (TDT), and shows some interesting empirical results.
review_title: null

Meta-review: The arguments the paper makes require a stronger foundation and justification. The reviewers and AC didn't find the author response sufficient. For example, in response to ZbHJ, the authors argue that their benchmark doesn't use automatically generated trajectories and therefore the language is not synthetic in some sense. It's not clear how it's related to synthetic language, but generated trajectories does create artificial regularities in the task, so an issue, but one that the author must address accurately. This argument also seems to focus on ALFRED and R2R, and ignored many other benchmarks, like the data used in DRIF (mentioned later), RxR, Touchdown, etc. There is also mis-used of technical term (e.g., Decision Transformer). Generally, the reviewers consider the work of potential, but it requires significant refinement, which the author response did not provide.
Decision: Reject