Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This research develops a novel multimodal , multitask architecture for robot manipulation. Also introduces a multimodal and multitask benchmark for robot manipulation. Finally introduce a carefully thought methodology in 4 steps to quantify the capability of the developed architecture.  The results section provide relevant ablation studies that bring insights on this line of research, providing ideas to further improve this research domain.
strength_and_weaknesses: Strengths:
1) Novel transformer architecture for robot manipulation.
2) Novel multimodal interface for robot human interaction.
3) Novel insight on improved performance on using object centric approaches for this tasks.
4) New dataset for multimodal and multitask robot manipulation.
5) Novel methodology to measure generalization in robot manipulation.
6) Novel human robot interaction modality.
Weaknesses:
1) Unclear how this method will generalize to temporal extended actions : i.e, working the dough, beating eggs, writing, folding clothes, etc.
2) Finteunning Mask-RCNN seems that might not be practical if we want to extend this method to work in-the-wild. What's the accuracy for a non-finetune Mask-RCNN?
3) What will happen if the user change his mind after saying the command? 
4) How is uncertainty deal in this model? How could be resolved?
5) What would happen if you have a cluttered scene and the number of object grow and there some that irrelevant to task?
6) What would happened if there is something incomplete, or wrong in the text command?
7) It could be interesting if also the model can output text to resolved some of the aforementioned items.

clarity,_quality,_novelty_and_reproducibility: Clarity:
The paper is extremely clear and well written. The sections are well organize. The methodology is well done and the ablation studies well selected. The related work is extensive and completely relevant for this work.

Quality:
The paper quality also is superb. The content of the theory, the figures, the explanation and text provide a very high quality paper. The completenees of the work with the available code, the extent of the work on the dataset are very thorough and extensive.

Reproducibility:
The code is available and seems well done and functional. I have not run it.

Novelty: This work even if aligned to GATO as well describe in the text, is novel as pointed out on the different point on the strengths section before.
summary_of_the_review: A large effort on developing a novel human robot interaction interface on robotics manipulation tasks that generalize. The work provide a relevant datase, new methodology to measure generalization, and novel models to solved this multitask and multimodal tasks.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper describes a technique to create a robot control agent that can control a manipulator to execute a variety of tasks specified in the form of a multi-model prompt, combining text and visual tokens. 

The paper also introduces a large scale benchmark for problems focusing on the properties of the system VIMA-Bench.  

strength_and_weaknesses: + The paper reports on an impressive amount of work on the development of the agent and the benchmark. The 20+ pages of appendices are an impressive quantity of work. 
- The paper proper only spends about 2/3 of a page (section 5) describing the model. This is unsufficient for understanding what exactly the contribution is. 
- As a robot manipulation problem, the tasks are very simple. These are pick and place tasks, performed on a flat surface, in simulation, with objects that are clearly distinguishable. 
- The paper does not explain why it considers that the multi-model prompt is superior. Are the prompts supposed to be generated by humans? Humans don't insert visual snippets or texture samples in the speech. 
- The tasks in the tasks suite are specified in ways that are much simpler than what is the state of the art in this field. For instance, "novel councept grounding" appears to be just attaching a label to a definition. "one shot video imitiation" appears to be the exact reproducing of an exact trajectory - a much weaker definition that what is normally meant by imitation learning. 
clarity,_quality,_novelty_and_reproducibility: + The paper is clearly written.
- The novelty of the paper is the multimodal prompt input. However, the paper spends very little space explaining this novelty. 
summary_of_the_review: The paper describes an impressive project, with a large number of demonstrations. However, it does not clearly focuses on what the novelty of the project is. 

----

I have read the responses of the authors. They do not change my rating of the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents VIMA, a robot agent that can act conditioned on multimodal prompts consisting of natural language and image objects. The architecture consists of self-attention and cross-attention layers and train the model to minimize behavior cloning loss. The main novelty of the paper is to propose a scheme for multimodal prompting, and the paper shows that the proposed approach can work well for various tasks with a single model.
strength_and_weaknesses: ### Strengths
- Clear writing with nicely looking illustrations
- Exhaustive experiments
- Introducing an interesting benchmark

### Weaknesses
- Dependency on a separate object detection module. While it's shown to be effective as using oracle information in considered setups, it's not clear whether this can work in real-world scenarios or non-tabletop tasks as the paper promises to be a generalist robot agent. To add on this, I don't see this as a big weakness but it could be an interesting future direction to consider.
- Interesting results and analysis on a newly introduced benchmark. But comparison with the existing approaches [1,2] on other benchmarks [3] could further strengthen the claims made in the paper.
- MIssing details on baselines. From only reading the paper, it's a bit difficult to understand what's the main difference of the approach to considered baselines without reading the baseline papers in a detail.
- on Decision Transformer -- Trajectory Transformer [Janner et al., 2021] significantly differs from Decision Transformer [Chen et al., 2021] but the paper writes this as Decision Transformer (DT) (Chen et al., 2021; Janner et al., 2021). This should be corrected. Moreover, the main component of DT is to condition on desired returns and make the agent select the desired actions -- without this, the method that conditions on multimodal prompt cannot be called DT. It's just a behavior cloning agent conditioned on multimodal prompt with the GPT architecture. 
- on cross-attention -- it's worthwile to cite relevant video prediction literature [4] that also utilizes cross-attention to reduce the compute cost when conditioning on previous frames (it seems that the architecture is almost the same, or very similar).

[1] Dasari, Sudeep, and Abhinav Gupta. "Transformers for one-shot visual imitation." In Conference on Robot Learning, pp. 2071-2084. PMLR, 2021.

[2] Mandi, Zhao, Fangchen Liu, Kimin Lee, and Pieter Abbeel. "Towards more generalizable one-shot visual imitation learning." In 2022 International Conference on Robotics and Automation (ICRA), pp. 2434-2444. IEEE, 2022.

[3] Zhu, Yuke, Josiah Wong, Ajay Mandlekar, and Roberto Martín-Martín. "robosuite: A modular simulation framework and benchmark for robot learning." arXiv preprint arXiv:2009.12293 (2020).

[4] Yan, Wilson, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. "Videogpt: Video generation using vq-vae and transformers." arXiv preprint arXiv:2104.10157 (2021).
clarity,_quality,_novelty_and_reproducibility: ### Clarity and Quality
- The paper is very easy to read, clearly written.

### Novelty
- Architecture and training schemes are not novel but prompting scheme and introduced benchmarks seem novel enough

### Reproducibility
- The authors promised to release everything

summary_of_the_review: The paper is written well, provides a thorough investigation of the proposed architecture in a very wide setups. I don't see major concerns with the paper but I'm reluctant to further increase my score as all the results are based on a newly introduced benchmarks and baselines which are not exactly clear how they are implemented. I'm willing to change my score up or down after the discussion during the rebuttal phase.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents an agent architecture called VIMA (for **Vi**suo**M**otor **A**ttention model), which is a standard transformer encoder-decoder pair. Multimodal inputs are encoded as a single sequence of interleaved text tokens and visual object tokens. Transformer outputs are mapped to actions, as in the Decision Transformer (DT). VIMA is evaluated on a new multi-modal benchmark, VIMA-Bench, consisting of 17 meta-tasks, and thousands of tabletop task instantiations arranged in a 4-level protocol of increasing difficulty. Baseline models (including Gato and DT) are also evaluated on VIMA-Bench. The results show that VIMA outperforms the baselines by consistently large margins.
strength_and_weaknesses: **Strengths**

VIMA-Bench is a flexible and useful environment for evaluation of multi-modal prompting, scalability, and generalization in a robotic setting. Interleaving language and image tokens in a single input sequence has shown recent promise, as with the Gato agent. 

**Weaknesses**

VIMA’s performance is reliant on three critical design features:

- Visual **object tokens** from an off-the-shelf object detector.
- **Cross-attention** to the prompt sequence.
- A pre-defined set of **high-level actions**.

The **object tokens** are a crutch, giving VIMA effective access to the underlying state space, while baseline models are not given similar access. This disparity is more than enough to explain VIMA’s improved results over the baselines, and makes the comparisons unfair. The paper says “We hypothesize that the data efficiency can be attributed to VIMA’s object-centric representation, which is less prone to overfitting than learning directly from pixels in the low-data regime.” But it’s common knowledge that state representations allow greater sample efficiency than raw image observations. So there is nothing novel or surprising about taking advantage of object-level representations when available.

**Cross-attention** is one standard method of inputting a prompt sequence. Another standard method is to prepend the prompt to the single sequence, and rely on self-attention. Cross-attention is most appropriate when the prompt sequence (as in machine translation) contains a wealth of potentially variable information that strongly determines the correct output sequence. Recent models like Imagen have leveraged cross-attention in this way. Self-attention (over a single sequence) is most appropriate when the prompt sequence is less informative, and serves simply to identify the current task from among a trained set of tasks. Recent models like Gato have leveraged self-attention in this way. By construction, prompts in VIMA-Benchmark are highly variable, and strongly determine the correct outputs, and therefore call for cross-attention. Gato and DT were originally designed for the other type of prompt, and cannot be expected to perform as well on VIMA-Benchmark. But cross-attention could easily be added to Gato or DT. In fact, VIMA’s transformer architecture is equivalent to DT with cross-attention. There is nothing novel or surprising about taking advantage of cross-attention when it is called for.

VIMA’s **high-level actions** are severely limited to a pre-defined set including “pick-and-place” and “wipe”. The baseline agents are given access to the same high-level actions, so fair comparisions are at least possible. But the results are not indicative of how the agents would perform on more common low-level actions, like robotic joint torques. This makes the term “generalist robot agent” in the paper’s title especially inappropriate.

VIMA’s performance depends on these three components. The paper itself notes that good results “can only be achieved with both cross-attention and object token sequence representation”. So given the strong caveats explained above, the experimental results presented in the paper provide no reliable evidence for how VIMA compares to the baseline models. 

clarity,_quality,_novelty_and_reproducibility: **Clarity**

Much of the paper is clearly written, though the language can be improved, and a few parts are confusing.

For instance, in the statement of the second contribution:  “capable of multi-tasking”. This is abuse of the verb multi-tasking, which actually refers to switching rapidly from one task to another, as humans are able to do. Multi-tasking does not refer to an agent’s ability to learn to perform multiple tasks.

More detailed explanation is needed on the following topic:  “passing the resulted token sequence to a pre-trained T5”

**Quality**

VIMA-Bench appears to be a potentially valuable set of tasks. But the paper’s experimental comparisons to baselines are poor, for the reasons cited above.

**Novelty**

VIMA-Bench is somewhat novel, but the VIMA architecture is not, as explained above.

**Reproducibility**

Code is provided.

summary_of_the_review: VIMA-Bench is potentially useful. But because of the flaws in VIMA's comparisons to baseline agents, little can be concluded about VIMA's capabilities.

***** POST-DISCUSSION UPDATE *****

In their last response to me, the authors said:

**"Among these variants, we find that the recipe of object token + cross-attention is the best for multimodal-prompted robot agents through extensive experiments."**


This is a nice summary of the work’s key experimental finding. My concern is that both parts of this recipe are unremarkable:
- The object tokens come from a model (Mask R-CNN) trained on this task’s ground-truth state information, making it unlikely to scale well beyond VIMA-Bench. This limitation makes the object tokens a crutch, and readers already know that such crutches boost performance. 
- A long line of prior work, starting with the original transformer paper of 2017, has shown that cross-attention is a powerful method for processing complex input spaces. Since VIMA-Bench’s input space is complex by construction, readers shouldn’t be surprised to see that cross-attention performs well here.

I still see value in VIMA-Bench itself, with its diversity of tasks. In fact, I would argue for acceptance of a *VIMA-Bench* paper submitted to a datasets and benchmarks track, as long as the models evaluated (including VIMA itself) were presented simply as baselines that the community could compare against in further work. In such a light, VIMA’s reliance on an unscalable vision processing pipeline would not pose a problem, and its usage of cross-attention would simply provide experimental verification of performance one might expect. 

But instead of focusing on VIMA-Bench, this paper focuses on VIMA as a model for general robot manipulation, starting with the title itself:  **”VIMA: General Robot Manipulation…”** This is why I have to leave my scored unchanged, despite all the impressive work that has gone into the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
The paper proposes a new approach enabling multi-modal prompts (images + text) in a transformer-based agent. The authors also propose a novel benchmark for this setting.

Strength:
- First paper enabling this interesting setting
- Good experimental results
- Useful benchmark

Weaknesses:
- Neither fish nor meat paper
- Doubts about the method and its novelty

Additional comments after re-reading the paper:
- domain-finetuned Mask R_CNN: unclear which data/objects are used for that
- the single objects in the example prompts in the paper and appendix are always canonical views - bringing them somewhat closer to words, this doesn't seem to be explained
justification_for_why_not_higher_score: The paper stays very controversial. 2 reviewers could not be convinced that the proposed method is good enough. Having a major refocus on the benchmark would be OK for most reviewers, but I don't think is advisable in this stage.
Decision: Reject