Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a shared autonomy framework that enables people to control a robot arm using a combination of natural language text and traditional (joystick) control input. While most language understanding algorithms for robots operate in the full autonomy regime, the ability for people to also provide lower-level inputs helps to mitigate deficiencies in the autonomy (or the language understanding) and provides an important sense of agency. The proposed framework encodes state-control-language tuples into a learned latent action space that is then decoded to generate low-level actions for the robot. The model is trained on a dataset state-action pairs that were subsequently annotated. At test time, a given utterance is mapped to the nearest utterance in the training set using a pre-trained language model, which is then provided to the decoder. The method is evaluated on different manipulation tasks and compared with end-effector and imitation learning-based language understanding baselines. The proposed framework outperforms both baselines in terms of task success. A user study further assesses the qualitative benefits of the approach.
strengths_and_weaknesses: STRENGTHS:

* The idea of coupling natural language-based control with lower-level inputs in a shared autonomy framework is interesting.
* The paper presents the results of a user study that demonstrate the qualitative differences between the proposed method and standard baselines.


WEAKNESSES:

* The primary advantage of combining language with lower-level inputs is the ability to leverage their complementary nature. While there may be many tasks for which language would be a sufficient way for a user to convey their intent, there are others for which another means of input is necessary (particularly in partially observable/unstructured settings). However, the scenarios that are considered here are fully known and quite structured. 
* The poor performance of the language-only baseline is suspicious. I imagine that it is a result of the fact that the state does not include information about the environment, in which case one can't expect a language-only version to perform the manipulation tasks well. Given the demonstrated success of language understanding for navigation and manipulation, a more standard baseline would presumably do much better.
* I like the idea of using language to influence learned latent actions as suggested by the title, however there is very little discussion and no evaluation of how the learned actions are influenced by language.
* The architecture combines existing components in an vanilla manner. The novelty is unclear.
---> The author response helps to clarify the novelty of the method.
* In addition to the work on latent actions, there is relevant work on shared autonomy in the case of unknown goals [1], including approaches that are task-agnostic [2].
---> The author response acknowledges this work and discusses the ways in which they differ
* The paper omits a large body of work in language understanding, including non-neural methods for language grounding in known and unknown environments [3]. These approaches have been shown to be effective for challenging manipulation commands despite being trained on relatively few samples. Of course. this comes at the cost of having to hand-engineer features and having a means of resolving abstract symbols to low-level actions (e.g., in the form of a policy), but that functionality exists, at least in the context of the scenarios considered here.
---> Again, the author response notes the intention to place the proposed method in the context of this body of work.
* The paper should discuss relevant work that combines natural language understanding with gestures [4--9], as complementary input modalities.
* The paper could be clearer on the amount of supervision required for the proposed approach. 
---> The author response addresses this concern.
* It isn't clear how the method can be expected to generalize without reasoning over the environment state.
---> I don't feel that this point was adequately addressed. Instead, it was left for future work.
* While hierarchy is noted in the discussion section, a key reason that existing language understanding models can be trained in a relatively efficient manner and are able to generalize is that they exploit the hierarchical nature of language. This is largely ignored in the text.
* Language retrieval in these settings is a function not only of the utterances, but also the environment, however the proposed approach performs retrieval only using language.


COMMENTS/QUESTIONS:

* Lines 54--60: The deficiencies of existing language understanding methods depends on the approach. Those that are given are certainly true of methods that ground language to low-level actions, but less so with those that employ more abstract symbols. In these cases, the advantage offered by incorporating low-level control seems more related to being able to tolerate scenarios when these abstract symbols (i.e., policies) can not be readily mapped to low-level actions.
* Line 200: It isn't clear how the proposed framework would behave given an utterance that significantly differed from those encountered during training with regards to making mistakes. What is to stop the architecture from effectively ignoring the user's commands or does it provide a mechanism that allows them to take full control. If so, how does this mechanism differ from one that you might integrate into a language-only system?
* The means by which annotators were filtered based on their similarity to the pre-trained model seems a bit unfair.
---> I mis-spoke in my initial review meant to comment on the possibility that filtering the training corpus based on self-consistency would reduce diversity.
* Line 303: The paper states that the latent actions model produces intuitive control spaces that humans can quickly grasp. Was this experimentally verified?



REFERENCES


[1] S. Reddy, A. Dragan, and S. Levine. Shared autonomy via deep reinforcement learning. RSS 2018

[2] C. Schaff and M. Walter. Residual policy learning for shared autonomy. RSS 2020

[3] F. Duvallet, K. T., and A. Stentz. Imitation learning for natural language direction following through unknown environments. ICRA 2013.

[4] C. Matuszek, L. Bo, L. Zettlemoyer, and D. Fox. Learning from unscripted deictic gesture and language for human-robot interactions. AAAI 2014

[5] T. Kollar, J. Krishnamurthy, and G.P. Strimel. Toward Interactive Grounded Language Acqusition. RSS 2013.

[6] C. Kennington, S. Kousidis, and D. Schlangen. Interpreting situated dialogue utterances: an update model that uses speech, gaze, and gesture information. SIGDIAL 2013

[7] S. Mohan, A. Mininger, J. Kirk, and J.E. Laird. Learning grounded language through situated interactive instruction. AAAI Fall Symposium 2012.

[8] D. Whitney, M. Eldon, J. Oberlin, and S Tellex. Interpreting multimodal referring expressions in real time. ICRA 2016.

[9] N. Mavridis. A review of verbal and non-verbal human–robot interactive communication. Robotics and Autonomous Systems 2015.
summary_of_recommendation: The primary contribution of the paper is the integration of language with low-level control in a shared autonomy framework. This is certainly an interesting line of research, particularly in the context of scenarios for which language and low-level control alone are insufficient. However, the significance of this contribution is unclear, in part because of questions about the experimental evaluation and the novelty of the architecture.

UPDATES BASED ON AUTHOR RESPONSE:

I thank the authors for their detailed response my comments/questions. Several of these have been resolved, but I still have concerns regarding the suitability of the language baseline and the (clarity of the) primary contributions of the paper, which is the use of language as an additional input modality to resolve ambiguity in joystick input. I do find this to be an interesting line of work, but the fact that including language outperforms the baselines that were used here for this set of arguably simple tasks isn't terribly surprising and it's not clear that the results would generalize to more challenging domains.
issues: In addition to the comments/questions above:

* Did the experiments involve a real robot arm? I assume that they did, but the text is a bit unclear.
---> Clarified in author response
* Are one-to-one state-action-utterance mappings needed (line 164) and if so, why is it reasonable to assume that there is an utterance for every state-action pair? If not, at what level do they need to be aligned?
---> Clarified in the author response
* Did the authors consider an IL baseline that only used control input (as opposed to only language)?
---> The author response largely answers my questions, but it should be clarified in the main text and not just the supplementary material.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a method to learn a mapping from low-DOF (e.g. 2-DOF joystick) control inputs to joint space (e.g. 7-DOF) commands that are otherwise inaccessible or unintuitive to human operators, conditioned on a natural language task description. Given a natural language utterance describing a task (e.g. pour milk into the bowl), the system produces a control mapping that allows a human user to more easily, smoothly and efficiently complete the stated task. This is a shared autonomy as opposed to a full autonomy setting as explored in prior instruction-following works, where ultimately the control over robot motion is with the end-user.

At test-time the system uses a BERT-based sentence similarity metric to map a natural language utterance to the closest synthetic task description in the training set (seemingly among a set of 5 tasks). Then, a control decoder is conditioned on the task description and maps 2-DOF joystick commands to 7-DOF joint space.

The paper performs a user study to compare the effectiveness and user-friendliness of their approach against a fully articulated mode-switching control scheme, and also against an imitation learning instruction-following approach. The paper shows that LILA is more effective at enabling the user to accomplish manipulation tasks, achieves smoother motion, is more likely to have the users prefer to use it again, and other positive qualities.

strengths_and_weaknesses: Strengths:
- Explores a novel, relevant, and interesting idea. Opens doors to exciting follow-up work.
- The approach of mapping a natural language utterance at test-time to the closest synthetic language utterance from the training set has been explored before in a full autonomy setting, where it significantly restricts the scalability of the system capabilities. This paper applies this method in the shared autonomy setting, where the user is ultimately in charge of the robot actions. Here, it makes a lot more sense. The scalability concern (e.g. of handling novel objects or composing primitives in novel combinations) is largely addressed by the human pilot, while the restricted synthetic syntax reduces the chance of an out-of-distribution utterance resulting in a noisy or unusable control mapping.
- Performs a real-world end-user study, which is both rare and necessary to assess performance of systems with natural language interfaces.
- Opens doors to interesting uses of shared autonomy besides end-user use, for example to improve the ease of crowdsourcing demonstrations on a dexterous robot arm corresponding to specific instructions.

Weaknesses:
- Unless I am missing something, the system does not take any visual inputs or observations of the environment as input. It is therefore incapable of actually understanding or grounding the natural language utterances to percepts or entities in the world. The experiments work solely because the training tasks and test tasks use the exact same objects in the same placements with no within-task variation. Therefore solving the task boils down to classifying the input utterance into one of the 5 task types, and then applying a fixed control mapping corresponding to the task type. This does not demonstrate any meaningful language grounding capabilities, which should include aspects like grounding noun phrases to visual percepts, responding to object placement variations, and allowing compositional specifications of object attributes. Any practical shared autonomy system would need to see the world and respond to it, dynamically interpreting the meaning of the instruction in relation to the objects in the environment. Prior instruction-following methods already demonstrate these capabilities, so I don’t see a strong reason to take such a significant step back in terms of the experimental setup.
summary_of_recommendation: The paper proposes a very exciting and relevant use-case of grounding natural language instructions to control mappings for a shared autonomy setting, and a user study of an initial prototype system showing effectiveness and user-friendliness in a shared-autonomy setting. This not only opens a new, exciting area of research, but is also relevant to assistive robotics, where it could significantly accelerate the adoption of robots that help people with disabilities, and bring value in the near term while fully-autonomous systems are still not robust enough.

However, as described in the “Strengths and Weaknesses”, the system does not ground language in observations of the environment, instead applying a fixed joint-space mapping that relies on objects being placed in previously known positions. This is a significant technical weakness. First, there is no meaningful language grounding or language understanding going on, and second, there does not appear to be a way to scale the system to a practical scenario, even a simple table-top scenario with various placements of objects. I understand that building a language grounding system on a real robot, and performing a user study is very hard, and I wouldn't want these obstacles to stand in the way of putting an excellent idea and the user study results out there. At the very least, the paper should make these limitations abundantly clear, so as to direct follow-up work (perhaps from groups with language grounding expertise) to improve upon these aspects of the system.
issues: - Lines 242-246: It is not stated whether the best annotators are typical or atypical. I could imagine each having pros and cons. Typical might give more sensible and natural language. Atypical might provide more diverse training data useful to expand the scope of natural language that the system can understand.
- I would suggest trying mean-pooled GloVe vector representations to capture sentence similarity. They seem to be potentially better on simple sentences than BERT, and have the additional advantage of being lightweight (See https://arxiv.org/pdf/2011.05864.pdf).
- I suggest that slightly more model details (e.g. high-level architecture, how the features h are fed into the networks, how FiLM is used) are provided in the main paper, perhaps at the expense of the introduction, which could easily make the same points more concisely..

Suggestion for incorporating vision:
- The pre-trained CLIP feature extractor (https://openai.com/blog/clip/) is known to be useful for grounding object references to locations in the environment. Perhaps there's a way to utilize it without requiring expensive language data collection?

Minor comments:
- It’s a bit hard to understand what this sentence means: “injects linguistic information into the core of a standard latent action decoder.”
- Unless I am missing something, the reward function R(s, a, u) does not appear to be ever used as there is no reward optimization taking place. I would suggest cleaning up the formalism section so that it better aligns with the actual system under consideration.

Questions:
- (Line 150): what guarantees that the controller exposes an intuitive action space that listens to the user? I could imagine that the controller may ignore z and instead focus on inputs s,u to directly optimize R(s,a,u), since z is not strictly necessary for this purpose.
- How does the loss in Equation 1 ensure that the learned latent space is smooth and intuitive? I could imagine it having jarring discontinuities.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents an approach to improving shared autonomy in which a user controls a high degree of freedom robot arm with a low dimensional controller and provides natural language instructions to disambiguate user intention. Building on recent work in learning latent actions for shared autonomy---for which language has not previously been used---the authors demonstrate the utility of adding language on a series of manipulation tasks. The experiments in the paper demonstrate that their approach, language-informed latent actions or LILA, allows the human-robot team to achieve good performance on the selected tasks, outperforming two baseline strategies.
strengths_and_weaknesses: Overall, the method is innovative and the paper is well written. However, it is the central claim of the paper is that the addition of language is necessary to disambiguate the robot's actions when trying to tackle a task in the presented shared autonomy scenario, yet the results (though otherwise informative and convincing) lack a comparison to a learned method that does not use language. Such a comparison is important to back up their central claim and show that the language input is indeed necessary for this situation. This omission is particularly surprising, as the primary point of comparison made in the remainder of the paper (e.g., the theory section) is to previous work on non-language-informed latent action learning. While the results are shown to work and the users are able to accomplish their tasks, it is not clear that it is the addition of the language that make this feat possible, greatly weakening the paper. The comparison to the Imitation Learning baseline, while informative, does not allow for a similarly realistic comparison. This is the central limitation of the current version of this work, and the authors should clarify why non-language latent action experiments were not included.

EDIT (post rebuttal): While I will admit that it was a bit of a strange decision to omit the no-language baseline from the user study, I am satisfied by the experiments now included in the paper. I think this is a reasonably solid paper, made whole by the addition of the new no-language baseline. I am changing my score to reflect this.

*Minor Comments and Suggestions*
- Figs. 1 & 2: After reading through the paper, these figures are helpful visualization of the proposed approach, but on their own (without the body text), the figures do not do a particularly effective job at illustrating how everything works. Figure 1 in particular is not particularly informative if one is not already familiar with learned latent actions; some additional visualization of the user input (to show that it was shared between the blue and orange instructions) would be incredibly helpful. For Figure 2, it was not clear from the schematic that the agent could . Furthermore, the figure is somewhat misleading, since using the 1DoF controller does not necessarily correspond to moving along the left-right direction in the figure. Both figures should be revisited for clarity.
- What does "success" refer to in Table 1 (and how many trials are included in each; I could not find this information in the text)? Is it the success of the retrieval or of the task associated with the specified task. Relatedly, what does a failure "look like" for the Pour Bowl task?
- There is a typo in Figure 3: The success rate is presented as a percentage, yet the y-axis only goes to 1.0%. Presumably, this is 100% success.
- A minor point: while not a style guide violation, the abstract is quite long, and it could be improved if the authors were to shorten it somewhat.
- I believe that, owing to the "retrieval" nature of the language model, all of the tasks appear in the training dataset. It would be helpful for clarity for the authors to reiterate this point in the experiments section. [I do not think additional experiments are required.]
- Under participants, it is mentioned that 5 are female: are we to presume that the other 5 are men (or non-binary or did not specify)? It's a small change, but promotes inclusion.
summary_of_recommendation: Overall, upon first reading I was quite enthusiastic about this paper (and generally remain so) because, though the technical advance is not overwhelming, it represents solid progress in a difficult area and is supported by some good experiments that show the efficacy of using LILA, the proposed approach. However, the omission of a comparison to prior work in non-language-informed latent actions is hard to overlook. As I mention above, the central claim of the paper relies on language being a key distinguishing feature versus previous work in latent action learning, yet the only other learned baseline uses imitation learning (without subsequent human intervention) that also relies on learning. I am willing to change my recommendation if the authors can provide sufficient justification for its omission.

EDIT (post rebuttal): See my comments above, but I believe that the addition of the no-language baseline satisfied my primary concern. I have updated my score to reflect this change.
issues: Much of the rationale for my decision and my suggestions for changes are included in the "Strengths and Weaknesses" section above, and so the authors should refer to that for more detail. In short, the lack of a baseline comparison algorithm that uses non-language-informed learned latent actions is conspicuous; the authors should provide justification for why this baseline was not included and why the central claims of the paper are sufficiently supported by the experiments that currently exist.

EDIT (post rebuttal): See my comments above, but I believe that the addition of the no-language baseline satisfied my primary concern. I have updated my score to reflect this change.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents a shared autonomy framework that reduces full robot arm control to 2 degrees of freedom, deciding along what control axes to align those degrees based on a natural language input command for the task to be completed, such as "pour the milk into the bowl". The method presented uses a transformer-based, massively pretrained language model to perform nearest neighbor lookup at inference time to map language commands to understood data points from the training set to modulate control. The work is a crisp step towards a larger space of language-guided shared autonomy.

*Post-response*

The authors substantially revised the paper in response to other reviewer's concerns. The main concern I had regarding data filtering was clarified as an MTurk cleanup procedure. My only comment for future work from the authors there is that a second "filtering" HIT is a cleaner way to do this. Basically, gather data D_0, run a filtering HIT where annotators select whether instances in D_0 are valid (e.g., are they spam, do they match a demonstration, etc.), then throw out bad D_0 instances *and* remove worker qualifications for the annotators who provided them, then repeat the data collection step to fill out D_1 replacing now-missing instances. Repeat to D_i full dataset.
strengths_and_weaknesses: The main strength of the paper is a clear method for langauge-informed shared autonomy with convincing human subject experiments for a physical robot agent.

The main weakness of the paper is not having a head-on discussion about exactly what kind of generalization from training data is possible under LILA's current formulation. In particular, the model can "sort of" generalize to unseen language commands, as long as they are paraphrases of commands seen at training time. To my understanding, the model has no capacity to generalize to unseen tasks and entirely unseen language specifications for those new tasks.
summary_of_recommendation: The paper makes a concrete step in RoboNLP towards shared autonomy and contains convincing, real-world experiments with humans controlling a robot arm conditioned on free form language input. When I imagine the intersection of "RoboNLP" and "CoRL", this paper is a great example.

issues: Questions by priority

Q1 In "Crowdsourcing Language Annotation", annotations whose representations fall away from a mean are discarded, which feels a bit icky because the method later uses those representations to do nearest neighbor lookup. It could be argued, then, that the data has been curated for the algorithm, keeping around only a local neighborhood of utterances to retrieve without allowing for outliers that could cause trouble at inference time. Can the authors clarify this methodology?

Q2 For Figure 4, I was surprised to see End Effector performing better than Imitation Learning in some cases. Is there a clear intuition for this result? Maybe IL just needs a lot more demonstrations before the policy isn't too frustrating?

Nits and questions not effecting evaluation:

- Line 77 "unprompted" unclear. Does this mean some users didn't use language at all?

- Line 166 "action" in quotes is a weird way to describe space Z since there is already an action space; the paper knows this since it's in scare quotes, but maybe just use a different word.

- Line 192 Grounding Language in Play also pulls this "retrieve utterances from training time" trick, if I recall correctly.

- Line 250; 5 students were female, were the other 5 male? Leaving this inference implicit erases other gender identities and saves almost no space in the paper.

- Line 289 "heralding a bright future" https://i.gifer.com/Dnxs.gif
review_title: null

Meta-review: This paper introduces a shared autonomy input for manipulation that is augmented with language understanding such that discrete language inputs (classification of predetermined phrases) can be used to improve task performance.

Strengths:

Development of multimodal inputs for manipulation is an important research area.  The paper introduces a novel approach and includes a user study validation.


Weaknesses:

The evaluation method does not appear to actually use language grounding as a baseline. The language only variant being tested has no access to actual state data, and thus does not actually perform any grounding.  This significantly undermines the reported results.

Furthermore, a central claim of the paper is that the addition of language is necessary to disambiguate the robot's actions when trying to tackle a task in the presented shared autonomy scenario.  However, the evaluation does not include a comparison to an approach that does not use language.  This is an important result to include in order to demonstrate the added benefit of this additional modality.

The work is not well situated in existing literature on language understanding, please see notes by R1 for additional works to include in the discussion.

It is not clear how well the presented technique would generalize beyond the limited explored scenario.

The attached videos did not significantly add to the clarity of the paper.  Some annotation of their content would be helpful.

Please see detailed reviewer comments for a list of additional questions and suggestions.


Post rebuttal comment:  Thank you for your detailed responses to the reviewers.  The paper represents an interesting and valuable advance in the area of situated NLP, and the authors' additions and clarifications were very valuable.  As is clear from the comments, there were a few common areas of confusion/concern that require clear clarification in the paper:
* The term "language grounding" means different things in different research contexts, which initially caused confusion for majority of reviewers with regard to how this paper was written.  The authors should carefully define this term.
* Clear discussion of limitation with respect to generalization.  The state of the art for most ML techniques is that we test generalization in new scenes/applications/domains, and so again readers are coming in with this assumption in mind.  This should be clearly addressed in the paper, as well as how this current approach, which does not generalize to unseen objects, is a helpful research stepping stone.
* The baseline discussion needs to be clearly presented in the paper.
Decision: Accept (Poster)