Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a simple method to solve the task of instruction-following in multimodal environments. While recent work makes use of pre-trained transformers, their
performance is limited by (i) lack of grounding (in the case where separate vision and
language models are used) and (ii) lack of ability to follow long instructions (in the case
of CLIP). To overcome these limitations, this paper proposes using M3AE, a recently
proposed multi-modal transformer, as the backbone to provide vision-language
representations. In experiments on RLBench, InstructRL, the proposed method, is able
to outperform prior work such as HiveFormer (which makes use of separate vision and
language models) and a CLIPort-inspired CLIP-RL method across all categories of
tasks. Further analyses reveal that the model is also capable of handling longer
instructions, novel instruction combinations, and scales well with model size.

strength_and_weaknesses: Pros:
Proposed method is simple and seems easy to implement with the provided details
for the task.
Strong performance improvements across all categories on the RLBench
benchmark. Additional analyses also reveal the importance of multiple design
choices that have been used by the authors in InstructRL.
Cons:
While the proposed system is simple, there is a slight novelty issue. Stronger pretraining models are expected to provide better representations and hence improve
model performance. The architecture itself seems derivative, given the HiveFormer
framework.
In the ablation studies, InstructRL performs with longer instructions than CLIP- or
BERT-based models. However, there is no mention of what these instructions look
like and what aspect of the original instructions have been expanded to generate
InstructRL 2
the longer instructions. Without such details, it is hard to judge the pure numbers
provided for this ablation.
In the ablation wrt context length, the author(s) mention that “improvement saturates
around 4”. From the numbers in Table 9b, it is hard to arrive at this conclusion as
there is a steady increase in performance from 1 → 2 → 4 → 8.
The writing of the paper is also highly reliant on readers having prior knowledge
about the RLBench benchmark, and does not provide adequate details about the
benchmark for new readers to understand and evaluate the findings. Without
appropriate discussion, it is hard to how hard the tasks are.

Minor Edits:
Figure 5b has some spelling mistakes in the x-axis.
In Section 4.1, under ‘Encoding instructions and observations’, the final representation
of the transformer blocks is mentioned to have dimensionality ‘d’. If that’s the case,
shouldn’t the dimensionality of be where L is the number of concatenated
layer representations?
clarity,_quality,_novelty_and_reproducibility: The paper was very well written and was very easy to follow. The authors mentioned that the code will be released. The paper has a novelty issue, as the architecture itself seems derivative, given the HiveFormer framework.
summary_of_the_review: Current decision: Borderline Reject
The novelty and the lack of self-contained information in this paper imply that the paper
currently stands as a reject. However, the system proposed is very simple and hence
the decision is at borderline reject. If the author(s) are able to address the issues wrt
ablations and writing, the score can be reconsidered.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the instruction-following problem, prior work uses language-only models,which lacks the grounding between language and observations, this paper proposes a method (InstructRL), which contains a vision-and-language encoder, and a policy transformer for action prediction, the vision-and-language transformer encoder is pretrained on millions of image-text pairs and natural language text; empirical experiments on a robotics benchmark show the SoTA results.
strength_and_weaknesses: Strengths:
1. This work leverages the pretrained vision-and-language transformer as encoder for instruction-following agents; 
2. This work shows promising (SoTA) results on a set of robotics benchmark.

Weakness:
1. Somehow the proposed model looks not novel in the vision-and-language navigation tasks, but may be a good attempt in the robotics tasks.
clarity,_quality,_novelty_and_reproducibility: it is a decent (or borderline) work to show the pretrained vision-and-language transformer as encoder for instruction-following agents in the robotics tasks/environments; However, it looks the proposed model may not novel in the vision-and-language navigation tasks [1, 2, 3, 4], it is a good practice to apply the similar idea in the robotics tasks/settings. 

[1]. Guhur et al., Airbert: In-domain pretraining for vision-and-language navigation, CVPR 2021

[2]. Shah et al., Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action, 2022

[3]. Majumdar et al., Improving vision-and-language navigation with image-text pairs from the web, 2022

[4]. Towards learning a generic agent for vision-and-language navigation via pre-training, CVPR 2020
summary_of_the_review: It is a borderline, empirical work to show the pretrained vision-and-language transformer as encoder for instruction-following agents in the robotics tasks/environments, and achieves the SoTA on a set of robotics benchmark; the main concern is the novelty of the proposed model/method, which was explored a lot in the vision-and-language navigation tasks.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work introduces a new model for agents to perform vision and language task. The method, named InstructRL relies on a vision and language transformer (ViLT) trained on passive datasets with language and images aligned with description. This ViLT is used to convert observations and descriptions into tokens, which are consumed my a policy transformer, trained with expert demonstrations to generate gripper actions. The method is evaluated in RLBench, showing SOTA performance amongst selected baselines, generalizing to longer instructions and zero-shot task specifications.
strength_and_weaknesses: **Strengths**

- Generally clear and well written paper. Both the contributions and the model are clearly explained, in some cases I would value a short description of the work that is cited to avoid switching to the refered paper for basic information. For instance for 4.1, it would help to explain M3AE is a transformer-based architecture via masked token prediction.

- Very simple model, reuses components trained on passive datasets which allows the model to potentially improve as these representations improve, I would assume this makes the policy train much faster than if using a transformer from scratch, but it would be great to validate that. The performance is comparable or higher than baselines in all tasks (both single and multitask settings), though as mentioned in strengths, I have some questions and suggestions regarding baselines.

- Results: 
	- Nice results on unseen instructions, even though the fact that it is only in a button pussing task is slightly underwhelming. It would be amazing to see it for novel object categories as well.

	- Very good insight in Figure 4.b, in general I value the ablations performed in section 7, see weaknesses though for caveats.


**Weaknesses**
- I would include in related work methods that convert observations into language and rely purely on language models to drive agents. Note that these are different from methods such as (Huang et al. 2022), since the former group still consumes observations from the agent. Examples of those would be (Li et al. 2022, Shridhar et al. 2021)


- Clarity and Figures
	- On the flip side on clarity, I would recommend adding some more detail in the VIT policy. For instance, h_{t}^{k} are intermediate layer representations after average pooling through the sequence length dimension? Or the average pooling is applied separately on each intermediate representation? More clarity here would be appreciated. Is the linear layer for proprioception a 4x(4d) layer or 4 (1xd) layers.

	- It would help if Figure 4 had a legend for colors representing what corresponds to instr/observation, proprioception and action.

	- Figures should be more carefully made. There are overlaps between numbers in Figure 5, and Figure 6, and I see no reason why the 2 figures should have different formats. Barplots should have standard error. 
	- For Fusion strategy, a figure in the appendix of the strategies would be useful, together with a figure of how is done for CLIP-RL.


- Model
	- The transformer policy is simple which is ok, but I would like a bit more motivation for it. Why not incorporating a reward at the input as done by the Decision Transformer?


- Baselines: 
	- Why not including Perceiver-Actor? 
	- Could authors add more detail about what is the difference between CLIP-RL and the original CLIPORT? It does seem pretty different from CLIPORT so I am not sure how it is inspired, or where is the difference with the proposed method. Is it the fact that the transformer takes separately vision and language?  
	- Why do authors change the architecture in the different baselines? It is hard to know if the gains come from architectural changes or from the fact the models are trained from scratch.
	- Did authors try fine-tuning the vision-language model with the RLBench dataset/environment? Either in a MAE way to directly with the MSE objective?

- Result:
	- Long instruction settings are a great idea but they are evaluated rather underwhelmingly here. From table 2 I get that InstructRL can support more tokens than Clip-RL, but the example of instructions do not add any extra info in the task. I think the right setting or evaluation would be to include instructions that add actually new information, and see if the task is done according to those.

- It would be good to validate effect of #iter training for different baselines. I assume one advantage of the proposed method is that it learns faster.



Clarification Questions:
- It seems like the policy here is deterministic, could authros comment on what is the source of randomness when running multiple seeds? Is the environment stochastic?

- Hiveformer is not trained on large-text corpora. What is it trained on?

Minor: 
- Typo: 4.2 Instruction --> instructions
- Typo Figure 5 Avgerage


Li, Shuang, et al. "Pre-trained language models for interactive decision-making." arXiv preprint arXiv:2202.01771 (2022).

Shridhar, Mohit, et al. "ALFWorld: Aligning Text and Embodied Environments for Interactive Learning." ICLR. 2021.
clarity,_quality,_novelty_and_reproducibility: **Clarity:**
Generally well-written paper, clear figures though could be improved with some legends. Some of the explanations need improvements though, see weakness section.

**Reproducibility:**
Definitely higher than many existing papers since authors provided the training code, as well as training details in the Appendix. I am missing a checkpoint for the trained model along with the code.

**Novelty:**
Not particularly novel, combines ideas from Vision-Language Transformers and Decision Transformers. It seems like the novelty is that they use an entangled vision-language representation. This is not necessarily a weakness, but I would like to see from the baselines whether the performance gains come from here or from the fact the model is pre-trained on vision and language. 
summary_of_the_review: I really value the use of pre-trained models and simple architectures for visual-language embodied tasks. The fact that the performance scales well with larger model size is promising, and it would also be interesting to see if it scales with pre-training size. The proposed model also obtains better performance than the proposed baselines. Despite the above strengths, I think the paper needs to clarify a few-points, particularly the omission of certain baselines, the effect of model architecture vs pre-training data, the choice of the policy network, and effect of fine-tuning on the RLBench dataset. If authors can clarify why the baselines I mention in my weaknesses are not mentioned, as well as my questions regarding some of the differences in the baselines selected, and correct some of the mistakes I pointed I would be happy to raise my score.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper considers the problem of instruction-following for tasks that require reasoning over visual inputs in addition to text (i.e., grounding). The paper proposes InstructRL, an architecture that uses a pre-trained vision-language transformer as a joint encoder, the output of which is provided to a policy transformer that predicts the agent's actions. Experiments on the RLBench benchmark demonstrate that InstructRL outperforms contemporary baselines, including HiveFormer (which employs separate vision and language encoders) and a CLIPort-inspired method. The paper additionally analyzes the method's ability to handle longer instructions as well as novel instructions in a zero-shot manner.

The use of transformer architectures for instruction-following has attracted significant interest in the machine learning community of-late. As the paper points out, instruction-following for embodied agents requires relating (grounding) textual instructions to the agent's environment. All three reviewers agree that encoding vision and language input via a transformer that is pre-trained on image-text data is sensible, and shows strong performance compared to contemporary methods in spite of the relatively simple nature of the InstructRL architecture. Because of this simplicity and the promise to release source code, the results can be readily reproduced. The primary concern shared by each reviewer is in regards to the significance of the contribution, which is primarily an empirical demonstration of the benefits of using a pre-trained vision-language encoder. The reviewers emphasize that the novelty of the architecture itself is rather limited considering the large body of work in this area. This is not to suggest that the simplicity of the InstructRL is not an advantage, but rather that the significance of this contribution needs to be made clearer. 

The AC notes that none of the three reviewers engaged in a discussion with the authors or acknowledged that they had read the authors' response, despite the efforts of the AC and the authors. However, given the consensus among the reviewers together with the fact that there are not any questions about the factual accuracy of the concerns that they raise, the AC feels comfortable with the decision.
justification_for_why_not_higher_score: There is clear consensus that the novelty is limited
Decision: Reject