Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper introduces a transformer-based policy class from learning manipulation policies for physical robots from demonstration. It shows how providing explicit supervision to the attention layers in the transformers can force each module in the network to specialize in a specific type of reasoning. The paper presents a method to introduce these supervision tasks gradually over time, starting from lowest level tasks, to build up the modular reasoning capabilities of the model. The paper demonstrates how the resulting modular architectures are better adaptable to new manipulator types with few demonstrations, compared to policies trained with traditional end-to-end task losses only.
strengths_and_weaknesses: Strengths:
- Interesting approach to constructing a robot policy. It’s nice that it’s a unified approach, where every module has the same architecture, and the distinctions between modules arise only from the objectives with which the modules are trained. This simplifies the design of robot policies.
- Convincing real-robot results on transferring policies between robots with different kinematic configurations. This is an important area of study, especially for deep learning based policies.

Weaknesses:
- The description of the method is perhaps too informal and sometimes vague about what are the inputs and outputs of the different modules, what types do these inputs and outputs take, and how is information being fed into these attention networks. Could also use more info on what are the labels for attention supervision and how these labels are obtained from demonstrations.
- Only slightly outperforms the baseline BC-Z, but with a seemingly much higher annotation burden.
- The “language-conditioned” aspect of the paper is somewhat weak. The main benefit of language over other task specification methods (e.g. joystick control, clicking to indicate navigation or manipulation goals) is ability to support free-form natural language instructions that include complex constraints, compositional task structures, temporal structures etc. A core challenge in language-conditioned policies is supporting flexible task specifications in a large enough task space to be interesting. Experiments should demonstrate at minimum generalization to unseen instructions combining seen primitives [1, 3, 6], but recently (and ideally) also unseen objects and primitives [2]. This work seems to uniquely focus on picking tasks, so language is used only to specify a target object. Presenting this work as doing referring expression grounding for goal object specification [4] would be more accurate. Even then, there are only 6 specific target objects, so the flexibility of referring expressions is limited compared to other works in referring expression grounding.
- The two contributions of “supervised attention” and “hierarchical modularity” are arguably overstated as contributions of this paper. “Supervised attention” has been studied multiple times in NLP research, starting with [5]. This paper may at most take credit for testing this idea on real robots. “Hierarchical modularity” simply refers to phasing in additional losses over time starting from low-level towards high-level reasoning. It’s roughly equivalent to the pre-training and joint fine-tuning paradigm that’s been commonly used in deep learning for years, and is also related to training policies with auxiliary losses, a common approach to encourage modularity in an end-to-end system [7, 8]. This paper can rightfully take credit for studying this approach (including with appropriate ablations) with attention-based networks in a robot manipulation task. This in itself is a meaningful contribution, but phrasing “hierarchical modularity” as an innovation might be overstating it.

Review references:
[1] Arumugam, Dilip, et al. "Grounding natural language instructions to semantic goal representations for abstraction and generalization." Autonomous Robots 43.2 (2019): 449-468.
[2] Shridhar, Mohit, Lucas Manuelli, and Dieter Fox. "Cliport: What and where pathways for robotic manipulation." Conference on Robot Learning. PMLR, 2022.
[3] Williams, Edward C., et al. "Learning to parse natural language to grounded reward functions with weak supervision." 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018.
[4] Shridhar, Mohit, Dixant Mittal, and David Hsu. "INGRESS: Interactive visual grounding of referring expressions." The International Journal of Robotics Research 39.2-3 (2020): 217-232.
[5] Liu, Lemao, et al. "Neural Machine Translation with Supervised Attention." Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. 2016.
[6] Lynch, Corey, and Pierre Sermanet. "Grounding language in play." (2020).
[7] Hermann, Karl Moritz, et al. "Grounded language learning in a simulated 3d world." arXiv preprint arXiv:1706.06551 (2017).

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper presents an interesting policy model, a new method (in robotics) to train the policy for a manipulation task from demonstrations, and convincing results on physical robot. A particularly good aspect is the ability to adapt the policy to different robot manipulators. It's great to see such multi-robot experiments that are often lacking.

That said, in it's current state the paper is a little vague on some important details, and slightly oversells certain aspects related to the innovations in the paper. I would like to see these writing and presentation issues are addressed.
issues: Suggestions:
- Line 163: Not sure what “part of the attention” is. Consider rephrasing, perhaps more formally? 
- Line 186: What are “all other input variable”? Can these be listed out or defined?
- Section 3.3 is perhaps a bit too informal and so hard to understand what’s really going on. E.g. What is the meaning of “target position”? Is the entire network simply inferring an end-effector goal in task space? If so, why does it need this many modules to do so?
- Algorithm 1 uses symbols that are undefined in the context in which the algorithm appears. It does not add much information.
- This work is highly related to a line of work in instruction-following that enforces modularity in end-to-end networks using auxiliary losses, and uses that modularity for sim2real transfer of a control policy on a drone navigation task [CITE], along with follow-up work on a household navigation task [CITE].


Questions:
- Line 233: Doesn’t the network predict target end-effector poses? Why is motion planning done towards a “target object”?
- How are supervision losses L_{k} obtained for each task from the demonstrations? How is supervised attention obtained in real-world experiments? Is it manually human-specified or automatically extracted from the demonstrations?
- Could you better explain the “w/o Hierarchical Modularity” baseline? Does this baseline only train on the task loss and not on the intermediate attention losses? Or does it train on all the same losses as the full model, but jointly all at once without gradually introducing more complex tasks once simpler tasks have converged.
- Is this method of transfer across robots better than e.g. predicting grasp points / end-effector keypoints, and running a motion planner to control the robots?
- Line 292: What are “register slots”? Were these defined anywhere in the paper? What kind of noise was added? Was this noise of a relative or absolute magnitude with respect to the distribution of values in these “register slots”?

Typos:
- Line 200: Typo “the the”
- Line 201: Typo “tasks-specific”
- Line 308: “three times estimate” -> “three times to estimate”.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents an approach for learning transferable language-conditioned pick & place policies by exploiting the inductive bias of attention to define chainable modules for performing tasks. Because there is a separation induced by the modules that are defined (e.g., identifying object to manipulate, estimating distance from end-effector to target object, mapping this distance to a control action), the proposed approach is able to show sample-efficient transfer to new robot morphologies, as well as across environments (sim2real).

In general, this paper proposes a neat, factored architecture for modular language-conditioned manipulation, and shows that adding such structure leads to better performance compared to other language-conditioned baselines, as well as meaningful transfer.
strengths_and_weaknesses: In general, I like the punchline of this paper — by formulating a policy architecture with components that are human-interpretable (e.g., attention), we can leverage extra supervision to aid in sample-efficient learning. Going further, we can embed different “modules” with different functionalities within this architecture (as functions of different types of attention), and use this to guide task solving for a single robot, as well as sample-efficient transfer when generalizing to new robot morphologies and environments.

---

While the approach is straightforward, my key concern, and the biggest weakness of this paper, is the generality of the proposed modules. Just like Neural Module Networks in the computer vision & NLP communities (the end2end Neural Module networks paper should probably be cited here), the benefit of the predefined modules only exists as long as the task specification is relatively clean, and matches the hierarchy exactly — in this case, we’re mostly looking at object relocation without meaningful occlusion or other challenges, so the policy holds up.

However, where does the existing hierarchy break down? What if the type of objects changed (e.g., deformable, fragile, requiring different grasping policies)? What if there were meaningful occlusions or obstacles (requiring moving around existing objects in order to make the grasp we want)? While traditional BC approaches (e.g., in BC-Z) are able to scale to these new scenarios in the limit of new data, it’s not clear that this approach, with a fixed hierarchy, is able to scale similarly, which is really concerning! I’d love to see experiments looking at how the proposed hierarchy (e.g., this exact set of modules) performs in various settings, and whether there’s a clean way to add new elements to the hierarchy in an efficient, elegant way.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: In general, while I really like the idea of supervised attention, I’m not sure that a single hierarchy is sufficient or general enough to tackle the messiness of real-world manipulation; I’d love to see new evaluations that show if/when the current hierarchy breaks down, and whether there is a clean way to fix such problems.

As a result, I advocate for a weak accept at this time.

EDIT (Post-Rebuttal): Changed score to Strong Accept!

issues: Mostly, I’d love to see arguments addressing the point from above; furthermore, I’m curious as to why supervised attention, without the hierarchy is performing so much worse than BC-Z (Table 2). What is this approach missing that is present in BC-Z?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: In the paper authors propose a sample efficient approach for training language-conditioned manipulation policies that can be transferred across robots with different morphology, capability, appearance or dynamics. They introduce two complementary methods: supervised attention and hierarchical modularity. Supervised attention optimizes attention map by maximizing similarity between the queries and the keys. Hierarchical modularity decomposes the task of object manipulation to a set of reusable subtasks such as object detection, end effector detection, distance calculation. Authors demonstrate in both simulated and real world robot manipulation experiments that the method outperforms the current state-of-the-art methods and can transfer policies across different robots in a sample-efficient manner. 

strengths_and_weaknesses: strengths:
* supervised attention mechanism 
* decomposition to hierarchical modules which also help to introspect the robots policy

weaknesses:
* authors used supervised data to train the manipulation policy, so in that part novelty is limited 
* collection of data used for supervised training of attention and manipulation should be described in more details. 
* no examples of introspecting the robots policy 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Describe in more details data collection process and provide examples of introspecting of the robots policy. 
issues: The paper is clearly written and easy to read. Some important details are missing such as: which policy was used to collect the data, examples of introspection of the learned policy, how key-query pairs were generated for the training of the supervised attention mechanism.
review_title: null

Meta-review: metareview: # Strengths
- The paper describes an interesting approach to constructing policies
- Human interpretable components theoretically allow for better supervision and introspection of the policy

# Weaknesses
- The novelty should be clarified, e.g. although supervised attention is a promising concept it is not a novel contribution.
- The improved performance is rather small when considering the additional annotation burden
- The paper would benefit from some clarifications and additional details wrt the proposed approach

# Post-Rebuttal Update
The authors provided valuable clarifications in their responses to the reviewers that would improve the manuscript significantly.
I expect the authors to incorporate these clarifications and the additional experiments into the camera-ready version of the paper.
While I still think that the additional labeling is not negligible, especially in real-world demonstrations, I do see the added benefit.

confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)