Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a model for learning to predict end-effector keypoints to solve manipulation tasks. The model consists of attention mechanisms modeling relationships between text tokens and visual patches, including across time. It also includes a U-Net that generates a spatial probability over gripper locations. The input to the model are RGBD observations from three different views: left shoulder, right shoulder and gripper, and also a tokenized instruction where tokens are encoded with the CLIP encoder.
The paper demonstrates SOTA performance on 10 RLBench manipulation tasks in simulation, includes extensive ablation studies of the different model parts, and also includes an experiment with two simple instruction-conditional tasks. One of the main points made by the paper is that it's important to model the history of observations. The paper provides empirical proof to this claim.
strengths_and_weaknesses: Strengths:
- This paper doesn’t just take an off-the-shelf transformer and feed text and visual observations through it, but carefully considers what types of attention are required to model the task, and how it should fit together in an architecture. The result is a well-performing and lightweight model specifically targeted at fixed-base manipulation tasks.
- Technical sections very nicely written, with clear definitions to the different components, indexing that makes sense and helps understand the data types involved, and a clear data flow through the model.
- Impressive performance on 10 RLBench tasks.
- Overall strong experimental setup, with a very good set of ablations to study the effects of the different modeling choices like instruction input, history input, cross- vs self-attention, etc.

Weaknesses:
- In terms of studying instruction-driven manipulation, the sort of instructions and generalization studied is relatively extremely simple and significantly behind prior work. E.g. the paper did not test interesting compositions of attributes or spatial relations [C, D, E, F], nor did they test generalization to unseen objects or relations [G, H], nor tasks requiring multi-step planning [J]. As such, calling it an instruction-driven manipulation in the title might be a disservice to the paper, since this aspect is relatively underlooked. There is a large body of work that does instruction-following including modeling of history [B, I, K]. On the other hand, it is a fairly strong simulated table-top manipulation paper in the multi-task behavior cloning setting. Perhaps focusing on this aspect might better highlight the positive attributes of the paper.

- The method for computing position offsets (Lines 198-201) doesn’t seem like it could handle cases where the same task at the same time-step may require a different offset based on the environment configuration. This reduces the generality of the approach. Furthermore, the assumption of a fixed set of tasks (also present in the loss in Equation 8) cements this as a multi-task learning approach rather than an instruction-driven approach. In instruction-driven work, it's questionable to map to a fixed set of tasks, because the set of tasks is by definition extensible via specification of new instructions.

- No experiments on real robots, so tolerance to noise, or sim2real generalization couldn’t be assessed.

- Some design choices seem somewhat arbitrary and maybe strange (e.g. applying max-pooling to point-cloud patches, classifying instructions into a fixed set of task types).


Review references:
[A] Walter, Matthew R., et al. "Learning semantic maps from natural language descriptions." Robotics: Science and Systems, 2013.
[B] Blukis, Valts, et al. "Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight." CoRL. 2019.
[C] Arumugam, Dilip, et al. "Grounding natural language instructions to semantic goal representations for abstraction and generalization." Autonomous Robots 43.2 (2019): 449-468.
[D] Vaezipoor, Pashootan, et al. "Ltl2action: Generalizing ltl instructions for multi-task rl." International Conference on Machine Learning. PMLR, 2021.
[E] Wang, Christopher, et al. "Learning a natural-language to LTL executable semantic parser for grounded robotics." Conference on Robot Learning. PMLR, 2021.
[F] Corona, Rodolfo, et al. "Modular Networks for Compositional Instruction Following." Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.
[G] Shridhar, Mohit, Lucas Manuelli, and Dieter Fox. "CLIPort: What and Where Pathways for Robotic Manipulation." 5th Annual Conference on Robot Learning. 2021.
[H] Blukis, Valts, Ross Knepper, and Yoav Artzi. "Few-shot Object Grounding and Mapping for Natural Language Robot Instruction Following." Conference on Robot Learning. PMLR, 2021.
[I] Pashevich, Alexander, Cordelia Schmid, and Chen Sun. "Episodic transformer for vision-and-language navigation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
[J] Paxton, Chris, et al. "Prospection: Interpretable plans from language by predicting the future." 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019.
[K] Anderson, Peter, et al. "Chasing ghosts: Instruction following as bayesian state tracking." Advances in neural information processing systems 32 (2019).
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The main strength is superior performance on RLBench tasks compared to prior behavior-cloning work, along with great technical execution, good empirical work. However, it's not clear that the paper proposes anything paradigm-shifting. The method utilizes various techniques from prior work (such as the modelling keypoints, cross-modal attention, history modelling, U-Nets, multimodal transformers). None of these can be claimed as a contribution. However, the paper shows a good way of putting these pieces together in a robotics system.
issues: Questions:
- Line 159: why applying “max-pooling” to point clouds makes sense? Wouldn’t median-pooling (or even average-pooling) make more sense? Wouldn’t max-pooling select the highest coordinate along the x, y, and z axis individually, and take the resulting point to represent the location of the patch? Wouldn’t this representation be highly sensitive to an arbitrary choice of the origin? And e.g. for small objects, wouldn’t it be likely to select far away background points instead of a closer object point to represent the patch location?
- Equation 6: what is LN? The inner layer was represented by explicitly writing down a matrix multiplication. Why does the outer layer use LN and what does it mean?
- Equation 7: If the distribution is multi-modal, then averaging coordinates (e.g. points) may result in completely false predictions in-between the two modes, rather than better predictions on one of the two modes. Is this ever a problem in practice?

Suggestions:
- Line 149 - what type is the token E_{T}^{x}? At this point in reading I have not yet learned about how visual observations are represented or used downstream, so saying “to differentiate instructions from visual observations” doesn’t tell me much.
- Line 151 - maybe call A_{t}^{k} a “binary mask” instead of “attention map”? The word “attention” can have connotations of it being a soft mask, a mask that sums to 1, and a mask computed by a learned model for downstream use in the same or different learned model.
- Line 181 - would be nice to have specific section numbers and links when referring to supplementary material.
- There are multiple prior works on instruction-following (e.g. [A, B, K, L]), that build maps as a means to provide a concise spatial memory to a robot. This is often seen as an advantage over keeping in memory a complete history of observations. Could the paper discuss why keeping history might be advantageous?

Comments:
- Section 4.2: I’m not sure it is fair to call this method a “multi-modal transformer”. The attention mechanism was invented before the Transformer architecture for machine translation, and one important aspect of it is that the attention modules are layered in a deep network. The architecture in this paper consists of a single cross-attention and a single self-attention layer, followed by an MLP. Calling it something like “attention across modalities and time” model would more clearly pinpoint the contribution. (the whole time reading this paper I was trying to understand how it is fundamentally different from all the other transformer architectures, and this section finally got to it).
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents Hiveformer (History-aware instruction-condition multi-view transformer), which enables a robot to learn to perform novel manipulation tasks based upon natural language instructions and demonstrations for a set of training tasks. The approach uses Behavior Cloning to train the model, and the actions consist of gripper poses. A key aspect of the work is encoding multiple views of the scene and a history of information. Results in RLBench show that the approach outerpforms ARM [11] and Auto-\lambda [14] as well as ablations of Hiveformer. A video is included and depicts a robot demonstration.
strengths_and_weaknesses: Strengths:
-The figures are clear and useful.
-The right amount of space is allocated to the various parts of the paper.
-The presentation of the algorithm is intelligible. 
-Code is provided.
-The method shows positive results.

Weaknesses:
-It seems from Figure 1 that the paper addresses multi-step tasks; however, it isn't quite clear what examples of this task are and where the results are. I would guess those results are in Table 2 under "Long Term." However, it would be helpful to provide more detail.
-Some of the success rates are quite low (e.g., 16.9%). It would be helpful to discuss what it would take to reach > 70% on call tasks. On that note, it would be helpful if the table captions confirmed that what is show are the success rates (as opposed to rewards, etc.)
-It would have been helpful to show an ablation study evaluating the quality of the algorithm's performance as a function of the number of demonstrations available. 100 demonstrations per task could be too much for a human demonstrator. Further, how much variation existed among the demonstrations, and how were these demonstrations collected? 
-Some tables (etc.) go over the margins for the paper and should be adjusted accordingly. 
-This paper might have missed related work in language-guided robot manipulation. In particular, please see the references below. The first two address only RL, whereas the last one addresses imitation learning-based manipulation with language. As such, it would have been helpful to benchmark against the Silva et al. work.

Sodhani, S., Zhang, A. and Pineau, J., 2021, July. Multi-task reinforcement learning with context-based representations. In International Conference on Machine Learning (pp. 9767-9779). PMLR.

Stengel-Eskin, E., Hundt, A., He, Z., Murali, A., Gopalan, N., Gombolay, M. and Hager, G., 2022, January. Guiding Multi-Step Rearrangement Tasks with Natural Language Instructions. In Conference on Robot Learning (pp. 1486-1501). PMLR.

Silva, A., Moorman, N., Silva, W., Zaidi, Z., Gopalan, N. and Gombolay, M., 2021. LanCon-Learn: Learning With Language to Enable Generalization in Multi-Task Manipulation. IEEE Robotics and Automation Letters, 7(2), pp.1635-1642.

-CLIP, as noted by Abeba Birhane et al. (2021), is problematic. The authors leverage CLIP without discussing the potential harm that can come from its adoption. A recent paper exploring negative impacts of CLIP in robots was published by Hundt et al. (FAccT'22), but this was published after this paper's CoRL submission. It would be helpful if the paper addressed this issue in the limitations.

Birhane, A., Prabhu, V.U. and Kahembwe, E., 2021. Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arXiv:2110.01963.

Hundt, A., Agnew, W., Zeng, V., Kacianka, S. and Gombolay, M., 2022, June. Robots Enact Malignant Stereotypes. In 2022 ACM Conference on Fairness, Accountability, and Transparency (pp. 743-756).
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This paper is solid. The presentation is clear, the method is well-reasoned, and the results are positive. Ablations are provided, and the method outperforms baselines. It would have been helpful to include the work by Silva et al. (2022) as a direct baseline.
issues: Please address the notes on limitations, missing prior work, some minor issues of clarity, and the baseline of Silva et al. (2022).
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The work introduces a transformer-based robot manipulation method to execute language instructions. The policy input contains the history of actions and observations, which enables the execution of sequential tasks. The method also integrates a multi-camera view, in order to deal with the need for precise control and issues like occlusions.
The method was evaluated in simulated RLBench in single-task, multi-task, and multi-variation setup. The method was also demonstrated on a real robot (the results are only in the supplementary material).
strengths_and_weaknesses: Strengths:
- good empirical results on RLBench, compared to ARM and Auto-$\lambda$
- good results on the real robot. In particular, it shows the significant benefit of pretaining in simulation (and transfer from 3 to 2 cameras)
- a thorough analysis (Section 5.2) of the importance of different components of the perception and manipulation processing (not the language though).

Weaknesses:
- a significant part of the RLBench experiments is in the single-task setup, which if I understand it correctly, does not need the instructions at all (a separate model is trained for each task). Thus we get relatively little insight on the "instruction-driven" part. We can learn from Table 3 that it does help, but we don't know, how it compares to other ways of processing the language input. The "average over word tokens" presented in Table 4 seems to be very crude.
- the description of the architecture (Sect. 4.1, 4.2, Fig 2) seems to be pretty difficult to comprehend.
- The source code does not come with instructions, on how to run it.
- The source code contains the name of the author in the file `pyproject.toml`. It should be anonymized.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The presented method has many strenghts. It is a complete solution for executing multi-task, potentially multi-step instructions. The additional benefit is that it can leverage multiple cameras and a history of actions and observations. The method was also demonstrated to work on a robot (in time for the supplementary material deadline), which is very important for the relevance to the conference. Finally, the RLBench results seem strong.

On the other hand, the novelty of the method is somehow limited. Moreover, the claims about handling natural-language instructions do not seem to be sufficiently supported by the experiments.
issues: - Three setups for the RL bench experiments (single-task, multi-task, and multi-variation) should be defined or at least explained.
- Please provide detail on how macro steps (line 115) are used.
- Following the limited (in my opinion) evaluation of the language component, I'd like to learn answers to some of the following questions:
    - how well the method behaves, if we use a large-language model embedding of the instructions
    - lines 144-146 claim that CLIP embedding is better than BERT. Do you have any evidence to back this claim?
    - is the $E_T^x$ in Eqn (1) useful/required?
    - Does the model use task similarity expressed in language in the multi-task setup? How would the performance change, if we use a one-hot encoded representation for tasks, instead of the language input?
- Could you please provide examples of the human-written instructions? The description in the supplementary does not give me an insight, into how diverse they were (especially for the push-button task). It would be great if the full set of human-written instructions is provided in the supplementary material.
- I think that the limitation section should include a discussion on how well the method handles *natural* language / human-written instructions (as claimed in the abstract and conclusion line 309). Most of the experiments are using static descriptions of tasks. Using human-written instructions can reduce the performance up to 3x (from 77.4 to 21.3 in the tower task), which I'm not sure if it justifies the "generalizes well" claim (line 309).
    - Alternatively, support evidence that the model can actually handle language instructions. I could imagine an experiment in the multi-task setup, with synonyms for the task description (e.g. "Lamp on" -> "Turn the light on"). A similarly interesting experiment would be to show a transfer from a model trained on 73 tasks to a different one, not seen in the training (e.g. one of the "put/close/open" tasks, which would have similar concepts in the training set).
review_title: null

Meta-review: metareview: All reviewers agree that this paper is solid and well organized.
1. The paper shows superior performance in RLBench compared to reasonable baseline methods.
2. Although hardware experiments are not clearly mentioned in the main paper, supplemental video clearly demonstrates that the method has been successfully applied to a physical robot.

The reviewers raised some concerns:
1. In terms of instruction-driven manipulation, the vocabulary used in the task is very limited compared to other datasets (e.g. PFN-PIC). Linguistic information does not play a major role for the majority of the RLBench tasks.
2. The fact that hardware experiments are presented in the supplementary material is not clearly explained in the body of the paper.

### Post-rebuttal comment 
The authors have sufficiently addressed most of the concerns. I recommend acceptance of the paper as "oral".

confidence: 5: The area chair is absolutely certain
Decision: Accept (Oral)