LILA: Language-Informed Latent Actions
Authors: Siddharth Karamcheti
Megha Srivastava
Percy Liang
Dorsa Sadigh
Venue: CoRL2021 Poster (2021)
Keywords: Language for Shared Autonomy
Language & Robotics
Learned Latent Actions
Human-Robot Interaction

Abstract: We introduce Language-Informed Latent Actions (LILA), a framework for learning natural language interfaces in the context of human-robot collaboration. LILA falls under the shared autonomy paradigm: in addition to providing discrete language inputs, humans are given a low-dimensional controller – e.g., a 2 degree-of-freedom (DoF) joystick that can move left/right and up/down – for operating the robot. LILA learns to use language to modulate this controller, providing users with a language-informed control space: given an instruction like "place the cereal bowl on the tray," LILA may learn a 2-DoF space where one dimension controls the distance from the robot's end-effector to the bowl, and the other dimension controls the robot's end-effector pose relative to the grasp point on the bowl. We evaluate LILA with real-world user studies, where users can provide a language instruction while operating a 7-DoF Franka Emika Panda Arm to complete a series of complex manipulation tasks. We show that LILA models are not only more sample efficient and performant than imitation learning and end-effector control baselines, but that they are also qualitatively preferred by users.

LILA: Language-Informed Latent Actions
Siddharth Karamcheti", Megha Srivastava', Percy Liang, Dorsa Sadigh Department of Computer Science, Stanford University {skaramcheti, megha, pliang, dorsa}êcs. stanford. edu
Abstract: We introduce Language- Informed Latent Actions (LILA), framework for learning natural language interfaces in the context of human-robot collabora- tion. LILA falls under the shared autonomy paradigm: in addition to providing discrete language inputs, humans are given low-dimensional controller e.g.. 2 degree-of-freedon (DoF) joystick that can move left/right and up/down for operating the robot. LIIA learns viding users with a language-informed control space: given instruction like "place the cereal bowl on the tray." LILA may learn 2-DoF space where one dimension controls the distance from the robot's end-effector to the bowl, and the other dimension controls the robot's end-effector pose relative to the grasp point on the bowl. We evaluate LILA with real-world user studies, where users can pro- vide a language instruction while operating 7-DoF Franka Emika Panda Arm complete series of complex manipulation tasks. We show that LILA models are not only more sample efficient and performant than imitation learning and end- effector control baselines, but that they are also qualitatively preferred by users."
Language-Informed
language modulate this controller, pro-
Keywords: Language for Shared Autonomy, Language & Robotics, Learned La- tent Actions, Robot Interaction
Introduction
Nearly a million American adults live with physical disabilities, requiring assistance for everyday tasks like taking bite of food, pouring a glass of milk assiisaaae that robots could provide. Paradigms for efficient human-robot collaboration that strike balance between robot autonomy and human control such as shared autonomy [2, 3, 4, 5] present a promising path towards building such assistive systems. Unlike full autonomy approaches that enforce a sharp separation between user intent and robot execution, falling prey to problems of sample efficiency and robustness, shared autonomy couples a human's input with automated robot assistance. Consider kitchen or dining environment where high-dimensional (high-DoF) robot such as wheelchair-mouse aids human who may be physically unable perform tasks requiring fine-graine mne-grained manipulation. While the human manually telcoperate the arm by fully controlling individual "modes", sepa- rate degrees- of-eredoom of the robot's 's end-effector, past work has shown this to be unintuitive, slow, and frustrating [3, 6]. Shared autonomy approaches such as learned latent actions [5, 7, 8] however, build intuitive www-dimensional controllers for high-DoF DoF robots via dimensionality reduction.
manipulator
Specifically, learned latent actions models learn state-conditioned auto-encoders directly from datasets of (state, action) pairs; the encoder takes the current state and action, and compresses it to latent action with the same dimensionality the human control interface (e.g.. 2-DoF). This is fed to a decoder to try to reconstruct the original high-dimensional action. While these approaches are reliable and sample efficient, they are limited by their reliance on just the current state: given tasks like "grab the milk" and "shift the milk to the side" that overlap in state space, these controllers fail as the models lack sufficient information to disambiguate behavior.
Additional visualizations and supplemental experiments can be found at the following webpage: https: //site/ tes. google. www/wwww///ccccttcottccccco...ot weeeew/llll wtwp///wwwa..eeeeeeeeeeeeeweewee coll1c Code can be found here: https: //gi www.tto....oo..e.eeooooooooeteet cww/ccocoo////ooooo som/sidd//iill
Sth Conference Robot Learning (CoRL. 2021), London, UK.

---

Learning Latent Actions
Instruction
"Grab
Language disambiguates
Encoder
Decoder
"Move banana into fruit basket"
Controlling with Latent Actions
Decodar
Figure I: [Left) Our breakfast buffet environment with several diverse manipulation tasks. By providing both natural language input and low-dimensional joystick control [Middle], users dis- ambiguate between different tasks while retaining the ability maneuver through the environ- ment. This is enabled by our [Right] language-informer latent actions (LILA) models that use auto-encoders learn language
conditioned low-DoF latent spaces for meaningful control.
To address this concern, we consider incorporating natural language within this framework to add an additional conditioning variable for structuring the control space. Prior works integrate language in robotics settings for similar purposes within the full autonomy paradigm [9, 10, 11, 12, 13). Unfor- tunately, these approaches suffer from poor sample efficiency, failure recovery, and generalization; many issues that shared autonomy methods including learned latent actions seek address. By express utterance
joining language and latent actions, obtain a control space that is both state and language conditioned (Fig. 1 [Right]).
"grab the cereal bowl" and
We introduce into learned latent actions. Key to LIIA the principle that language modulates user's low-level controller based their provided utterance; as intuition, given the utterance "grab the cereal bowl" as in Fig. 1, our assistive robot might learn semantically meaningful, control space where one dimension (one joystick axis) may control the distance from the robot's end-effector to the cereal bowl, whereas the other might control the angle of the end-effector relative to the bowl such that the robot's gripper can obtain solid grasp of the object. Other utterances can modulate the controller in similar ways "pour the milk into the cup" might result in a learned control space where one joystick dimension controls the jug's pouring angle, while the other may control its height. Language only serves natural means for human to communicate their intent the robot, but also helps disambiguate across wide variety of objectives well, by inducing language and state-conditioned control spaces.
Informed Latent Actions (LILA), a framework for incorporating language
-iwddmmensionll (2-DoF)
core part of our method its ability to handle diverse, realistic language. To this end, we collect a small, crowdsourced dataset natural language descriptions to describe each of our training demon- strations; we use this real, natural language as the only input while training our models. To allow for out-of-the-box generalization novel user utterances such as those that describe similar behaviors but with different words phrases, tap into the power of pretrained models [14, 15]. We per- form comprehensive user study across 10 users who use natural language and our learned LIIA controllers to complete variety of diverse manipulation tasks in simplified assistive "breakfast buffet" setting, Our results show that LILA models are not only more reliable, performant, and sam- ple efficient than fully autonomous imitation learning and fully human-drive end-effector control baselines, but are qualitatively preferred by users well.
Related Work
We build LILA within shared autonomy framework [2, 16), applied to assistive teleoperation [17, 18]. We additionally build off of work at the intersection of language and robotics [19, 20, 21].

---

LILA Decoder
Training Exemplars
Disambiguation
'Higher"
Ta
T2
T1
T2
banana away
'Pick up the bowl'
Distil-RoBERTa w/ Meani Pooling
FILM
Sim. Search
T4
T3
T4
T3
T4
T3
Distil-RoBERTa Meaa Poaling
DoF
Multi-Task Training Demonstrations
Standard Latent Actions
Language-Informed Latent Actions
"Grab the cereal bowt""
Figure 2: [Left] Stylized example: navigating toward four points cross with 1-DoF latent action, where disambiguation is required. Standard latent action models fail, while LILA accurately reaches the corners with the help of language. [Right] LILA decoder architecture. We embed an utterance using a pretrained language model, then identify the closest exemplar in the training set via similarity search. We feed the embedding for this exemplar through feature-wist linear modulation, or FiLM [46], layer that fuses language and state representations within the decoder.
Shared Autonomy & Assistive Teleoperation. Shared autonomy casts robot control as collabo- rative process between humans and robots [2. 4, 16, 22]. While other work focuses "blending" human inputs with possibly task-agoostic policies within the same action space [23, 24], in this work, we focus on assistive teleoperation, where user is provided now-dimensional controller (e.g., joystick, sip-and-puff device) to directly control high-diimensional robot manipulator. Using these controllers for end-effector control - e.g. via operational space control [25] is incredibly difficult, requiring frequent mode-switching to control specific robot DoFs [3, 26]. Instead, we adopt learned latent actions [5, 7, 8, 27, 28] framework that uses conditional auto-encoders [29] learn task- specific latent "action" spaces from demonstrations, These latent spaces match the dimensionality of the low-DoF interface and provide semantically meaningful control. However, existing methods fail to differentiate between tasks with overlapping states, hindering the ability to perform diverse behaviors in workspace (e.g., manipulating a jug of milk in different ways pouring, placing in the fridge, etc.). In this work, we use language for disambiguation; users naturally speak their intent, conditioning latent action models to produce intuitive control spaces that align with user objectives.
Language-Informel Robotics. A variety of methods have sought combine language and robotics, spanning approaches that map language to planning primitives [9, 30, 31, 32, 33], per- form imitation learning from demonstrations and instructions [11, 34, 35, 36], and pair language instructions with reward functions for reinforcement learning [12, 37, 38]. Other approaches use language in more nuanced ways, such as learning language-conditiona reward functions directly [39, 40, 41], J. or within adaptive frameworks, where language used to correct or define new be- havior [42, 43]. This list not exhaustive; we present further discussion including approaches that combine language with other modalities in the supplementary material. However, all these approaches fall within full autonomy: after providing instruction, human users cede control over to the robot policy, which then takes the actions necessary to perform a task.
While robots trained with these approaches can perform diverse tasks and generalize to new instruc- tions, not without cost. Paramount is sample efficiency; imitation learning approaches often require hundreds to thousands of demonstrations for learning navigate [35, 44], and reinforce- ment learning approaches
require millions of episodes of experience learn robust policies [12, 45]. Whereas coarse behaviors easy to learn, learning to recover from slight deviations from the training data, or to perform precise motions in a sequence, is incredibly difficult. By casting our approach, LILA, within a shared autonomy framework instead, we intelligently offload these parts that are harder for robots - but easier and intuitive for humans - onto the user.
3 Formalizing Language for Assistive Teleoperation
Formalism. We formulate user's objective, task, (on a per-user basis) oblly-observable, language-augmennte, Markov Decision Process (MDP) M defined by the tuple (S.U, A, T, R)

---

similar to prior work in -oniuaggeooniiionnll robotics [10, 34, 42]. Let E U denote a user's language utterance provided at the start of each episode, where 2/ is the full set of language utterances a user could provide to a robot. Let E S S R" be the robot's state, and a E A C R be the robot's action: taking action in state results a next state s' according to the transition function T(s, a). Given the language utterance u. the user implicitly defines a reward function R(s, u. a) E R; the human and robot collaboratively maximize this reward subject to discount factor 7 E [0.
Problem Statement. This MDP forms the basis of shared autonomy task wherein human is equipped with a low-dimensional control interface for the robot. Let = E 2 C R2d where d < m be the human's 's control input to the robot, such as the d = DoF controller in Fig. 2. Previous work on learning latent actions for assistive teleoperation [5, 28] learn a decoder Dec 2) S 2 A that maps low-dimensional inputs = E 2 and current state
to high-dimensional action E A. However, in situations where state-conditioning is not enough disambiguate users' intent, too low of control input dimension d may lead to failure. Recalling the milk jug example, we have multiple different behaviors we could execute if the end-effector were next to the milk. For example, one might want to pick up the jug, shift it to the side, pour it, etc; conditioning only on the state with a 2-DoF action space is not enough to recover all possible behaviors.
Instead, aim to learn Dec(s, u, =) S U x 2 A that takes the user's control input and utterance u, and predicts the high-DoF action that matches the user's objective. The utterance acts as additional conditioning information, producing control spaces that depend on both language and state; this circumvents the disambiguation problem above.
4 Lanfuage Latent Actions (LILA)
We are given dataset of demonstrations, where each demonstration contains utterance and a trajectory т {su, 4o, 01..1...7777 We split each demonstration into triples of (t, s;) and use these to learn a conditional auto-encoder, consisting of a language-conditional encoder Enc: S x 4 x A- > Z that maps to latent & and decoder Dec: S x x Z > A that attempts to reconstruct the original action a. We minimize the mean-squared error between the predicted and the original action:
N (DDec(s, ue Enc(s;, U₁ a:)) a;)²
L Lnce
N
We next discuss how we integrate language into the architecture of the encoder and decoder.
4.1 Integrating Language within the Latent Actions Architecture
We implement the encoder and decoder as multi-layer feed-forward networks, with the ReLU acti- vation as in prior work [7, 28]. We focus on the decoder here, but the encoder is symmetric. For the decoder, we first concatenate the robot state and latent action 2, then feed the corresponding vector through multiple ReLU layers (usually 2-3), upsampling to produce the high-dimensional robot action. We next discuss how to incorporate language within this simple scaffold.
Pretrained language models such as BERT, T5, and GPT-3 [47, 48, 49] have revolutionized NLP, providing powerful language representations. Inspired by their success when applied to robotics and reinforcement learning tasks [38, 50, 51], we use a distilled RoBERTa-Bann model [14], from Cectence-Transfomeee [15] encode utterances. This model fine-tuned on a corpus of para- phrases, allowing it to pick up sentence- semantics. We generate utterance embeddings by performing mean-pooling over token embeddings for utterance, in prior work [38, 50]. We incorporate these embeddings using feature-wise linear modulation (FiLM) layers [46] that fuse language information with other features h by mapping language embeddings to parameters (7, 3) of an affine transformation: = * h + (Fig. 2). Notably, this is the representation received after feeding the state and latent action (s, 2) through the first layer of the decoder as described above. Once the language transforms h h', we feed h' to the subsequent layers of the decoder."
IImplementation can be found in the open-source code repository: https:// thub. com/siddk/lil..

---

Task Name
Success
Example User Study Input
Mapped Training Data
Pick Banana
100%
yellow in purple
pick up the yellow banana and place it into the purple basket
Pick Fruit Basket
100%
bring basket to center of pan
+ place the basket onto the tray
Pick Cereal
100%
go to the left side of the cream bowl, + grab the cereal bowl and go down, grab the cereal bowl, and put it on the tray place the try
Pour Bowl
67% pick up the cup of marbles and pour them into the cereal bowl
-> pick and pour the cup of white balls into the bowl of cereal
Pour Cup
100% pick up the clear cup with marbles + pick up the cup and pour it and pour it in the black mug the contents in the mug with the coffee beans in it
Table I: Example utterances provided by study participants paired with the retrieved exemplar per $4.1. Success rate refers to the percentage of the time a user study utterance (over all utterances in the study) was grounded to the correct task via our retrieval method. As each participant only attempted 2 tasks, success rate can fluctuate significantly, as is the case with the Pour Bowl task.
Nearest-Neighbor Retrieval at Inference. A major concern for work in language-conddtioned robotics is generalizing to novel language inputs. While it may be unreasonable expect gen- eralization to completely new tasks, for user-facing systems with clear set of behaviors seen at train time as in our work, there is an expectation that any kanguage-ifformed system is capable of handling moderate variations of utterances from the training set. To do this, adding linguistically diverse data has been the gold standard [44, 51]; however, a new class of approaches have emerged that sidestep additional data requirements by tapping into the potential of pretrained language mod- els [43, 50]. These approaches frame language interpretation inference, when interfacing with real users, retrieval problem: each new user utterance u' is cmbeeded(with the same pretrained model above, then used to query nearest neighbors store containing all training exemplars; once the nearest neighbor u; has been identified, it replaces u' as an input to LILA.
The key benefit of such an approach is the minimal mismatch between train and test language inputs: all "test inputs" are drawn from the training set. This does mean, however, that user utterances that describe new tasks, or are otherwise unachievable also get mapped to language seen at training. While this limits the ability perform novel tasks, it again highlights the benefits of the shared autonomy paradigm doubly so, considering the cost of a mistake in assistive domain like the one we consider in this work: if, while providing control inputs to the robot, user feels the robot is not acting in alignment with the user's desired objective, they can always stop execution.
5 User Study
We evaluate LILA with a real-world user study on a 7-DoF Franka Emika Panda Arm, on series of 5 complex manipulation tasks. Each user is provided 2-DoF joystick for control. We compare against non-learning, end-effector (EE) control baseline where users "mode switch," controlling the velocity of 2 axes of the end-effector pose at given time- Y), (Z, Roll), (Pitch, Yaw)]. Lan- guage utterances u are typed into a text console for simplicity; future work could extend this work by using off-the-shelf speech recognition systems. We also compare against a fully autonomous imitation learning (IL) strategy where users solely provide language inputs, and the robot attempts perform task without additional input. Ostensibly missing is a no-language variant of the latent actions model, in keeping with prior work; however, upon evaluating this model, found it to be unintuitive and unable to make progress or solve any task, we omit it from our user study. However, further experiments and analysis can be found in the supplementary material.
'Experiments with the no-lauge baseline and extra analysis showing the necessity of extra conditioning information can be found here: https://sites google. htmm/viee/lllaaaaa cor121 wwt//hoo//ooooooe bang--aeennne
5

---

1.-
Jerk (Control Impur)
Success Rale
0.
1.0
150
:25
0.8 202
04-
076
627
025
0.0
Graap Object
Comtleto
000
Pitch Roll
Figure 3: Quantitative Results. We average success rate [Left]] across all sub-tasks for each control method, and find that LILA is significantly (p < 0.05) more performant. However, the steep drop performance when completing the full task shows the difficulty of fine-grained control. We also calculate jerk as an indicator of controller smoothness, for both user control inputs [Middle] and end-effector position [Right]. Averaged across tasks and users, we find LILA leads significantly smoother control for users than end-effector control.
Environment. Fig. shows our "breakfast buffet" setting, a scaled down version of an assistive feeding domain. We define tasks: 1) Place Banana: placing the banana in the purple fruit basket, 2) Place Basket: grasping the purple basket by the handles and dropping on the tray, 3) Place Bowl: grasping the green cereal bowl by its edge and moving the blue cup of marbles (a proxy for milk) into the cereal bowl positioned on the tray, and 5) Pour Cup: pouring the blue cup of marbles into the yellow coffee cup. Fig. 1 shows idcalized example trajectories for the Place Bowl (blue) and Place Banana (orange) tasks.
the tray, Pour Bowl: pouring
These tasks vary in difficulty, requiring precise grasping and dexterous manipulation. We evalu- ate partial success based on how many of the following subtasks users are able to complete: 1) Reaching: touching the desired object, Grasping: executing successful grasp, 3) Bring to Target: successfully transporting the manipulated object, and 4) Task Completion.
Demonstration Collection. Both LILA and IL models require learning from (language, demon- stration) pairs for all tasks. We collect demonstrations kinestheticallyy as in prior work [7, 28), recording joint states at a fixed frequency. We initially collected demonstrations per task for each method. However, on testing the model, we found incapable of performing even rudimentary reaching behaviors. To give II. the best chance, collected twice the number of demonstrations (30 per task; 150 total), requiring extra 2 hours of labor.
Crowdsourcing Language Annotation. To build a natural language interface for human-robot collaboration, we collect language annotations for each task by crowdsourcing utterances. Our goal was to capture the diverse ways users may refer to the objects and actions our tasks entail without any additional information, simulating a real user interacting with our environment for the first time. We recruited 30 workers Amazon Mechanical Turk, showing only video of recorded demon- stration, and asked them to provide "a short instruction that you would want to provide the robot to complete this task independently in the box below.". However, this procedure resulted in some an- notations containing "spam", or extremely out-of-domain text. To address this without introducing our own bias on what constitutes "spam", we filtered the data to identify workers who consistently provided "noisy" annotations, measured by the cosine distance between the sentence embedding (using any pretrained embeddings) of an annotator's provided text and the average sentence em- bedding aggregated over all other annotators for given video. We used annotations from the 15 least "noisy" annotators under this metric our ground- -ruth utterances. Further details, well as example "spam" annotations that were filtered out, are in the supplementary material. Table provides examples of crowdworker utterances from our final dataset (rightmost column).
Participants Procedure. We conducted our study with participant pool of 10 university stu- dents (5 female/5 male, age range 23.2 + 1.87). Four subjects had prior experience telcoperation robot arm. We conduct within-subjects study, where each participant completed tasks, chosen
'Due to the COVID-19 pandemic and university restrictions, only those with pre-authorized access could participate. See the COVID- 19 considerations document in the supplementary material for more details.

---

LILA lours)
Imitatio Learning
End Effector
Response (Likert Scale]
use again
helpful
smoothness
task ease
effortless
precision
Figure 4: Qualitative Results. Using a 7-point Likert scale. we ask users to evaluate each of the control methods for different properties. With high significance (p < 0.05), we find that LILA outperforms both imitation learning and end-effector control baselines on several metrics, including degree of helpfulness provided and ease in completing tasks.
randomly, with each of the three methods. Users were given 2 trials to complete each tasks, and an allotted 3 minutes per control strategy to practice. Users were also given sheet describing con- troller inputs and details for each control method, which we include in the supplementary material. For imitation learning and LILA controllers, which require language inputs, participants provided natural language utterance which which proctor entered into the model participants were allowed to verify the proctor entered their input accurately. This user user- pee--provdee language utterance is used as the query in the nearest-neighhor retrieval described in $4.1; the retrieval set consists of all training utterances collected via the crowdsourcing procedure above. In addition to tracking quantitative success rates (normalized, based on progress relative to each the 4 defined subtasks), time taken per task, and controller logs, we ask users to fill out qualitative survey evaluating each method at the end of each study. We present both quantitative and qualitative results below.
Quantitative Results, Fig. summarizes our objective results. We evaluate both full- and partial- task success rates for each task across all control methods, in addition to computing smoothness metries directly on the logged user inputs and robot actions. Smoothness itiveness when measured on user 2-DoF joystick inputs, ease of use when measured on the robot's end-effector pose, and implicit safety: trajectory with high discontinuity in acceleration can lead to rapid, unpredictable changes in the environment. Smoothness is negatively correlated with jerk, the time-derivative of acceleration. We compute jerk by taking the second-order derivative of velocity, and report average jerk across fixed windows. Our results show that LILA significantly (p < 0.05) outperforms both methods across all sub-tasks, and also smoother to use both in control input space (2-DoF input) and end-effector space (6-DoF). However, the relative drop in performance of LII.A for the final sub-task "Complete Task" shows the room for improvement in fine-gri fine-grained con- trol, such as pouring motions. Additionally of note the poor performance of imitation learning. To explore this fully, perform ablation, and show that sample inefficiency is a likely cause especially since we are in low-data regime. These results and arguments can be found in the supplementary m
measure for intu-
Fig. 4 summarizes our subjective results. We administered 7-point Likert scale survey after users finished performing tasks with each method; this survey included questions around the perceived helpfulness of the model in completing the tasks (helpful) and whether the participant would use the control method again (use again). The results show that LILA outperforms both imitation learning and end-effector control across most qualitative metrics, with significant results (p < 0.05) marked with an *. We additionally visualize samples of the observed end-effector trajectories by individual users collected during our study for of our 5 tasks in Fig. 5. Across all tasks, LILA results in smoother end-effector trajectories than end-effector control, while imitation learning comes close the target object but is unable complete the entire trajectory for the task.
Qualitative Results.
Additional experiment videos, and a results of the imitation learning ablation can be found here: https://sice google. http://e.c..eeee/ieooooo//////// abblttio
7

---

LILA
Imitation Learning
End Effector
pour
pick cereal
pick banana
Figure 5: Trajectories from one user comparing three different control methods 3 out of our 5 tasks Pour Cup, Pick Cereal, and Pick Banana. L.IIA provides smooth actions that immediately approach the target object for task, while end-effector control shows more rigid motions that result in users diverging from their intended paths. While imitation learning also enables smooth motions, it often fails shortly after reaching objects, hence the shortened trajectories.
Discussion
Summary. We present Language-Informed Latent Actions (LILA) a framework that marks the first step in combining the expressiveness and naturalness of language for specifying and execur- ing on a human user's objective within the context of assistive teleoperation. Our user study re- sults show that when compared fully autonomous, imitation learning approaches, LIIAA is more sample-efficient and performant, training on half the number of task demonstrations, but obtaining significantly higher success rates. Compared with no-learning end-effector control methods, again show LILA's effectiveness at obtaining high success rates, but also demonstrate its ability to produce intuitive low-dimensions control spaces from language input. Qualitatively, we find that users prefer LILA to alternative methods across the board, opening the door for additional work on language & latent actions.
Limitations and Future Work. Currently, LILA uses language as a mechanism for task disam- biguation in the current results, there is mechanism for generalizing to completely unseen tasks or language specifications. We believe that the ability to disambiguate with language, and the integration of language within the latent actions framework is strong research contribution, and hope that future work looks to dynamic states - perhaps by leveraging visual latent actions [28] and to adapting to new utterances and tasks dynamically [43, 52]. Furthermore, while users found LIIA intuitive and natural, they found themselves wanting to further modulate the robot's behavior with language instructions during the course of execution. Many users, upon seeing the robot make slight deviations from desired path would instinctively provide spoken corrections "a little the right", "no, grasp it by the handle!" indicating desire for multi-resolution language control.
Shared Autonomy and LILA. LILA fits within the shared autonomy paradigm, where the role of language has been underexplored. With LILA and shared autonomy approaches in general, humans retain agency they are responsible for robot motion, and if the robot moves in a way that is not safe, or does not align with their objectives, they stop providing control and possibly reset, give new instruction, drop into actions model, critically informed by language, produces intuitive control spaces that humans can quickly grasp. Finally, language is natural users specify their objectives as they would if speaking to another person, and the robot uses that language to shape their control space. These properties preserving agency, maintaining interpretabilityy and leveraging the expressive and natural features of language for specifying objectives are critical for widespread human-robot collaboration, and hope this work presents a concrete step towards achieving that goal.
complex control mode. Behavior is interpretable the latent
8

---

Acknowledgments
Toyota Research Institute ("TRI") provided funds assist the authors with their research but this article solely reflects the opinions and conclusions of its authors and TRI or any other Toyota entity. This project was also supported by NSF Awards 2006388 and 2132847. Siddharth Karam- cheti is grateful to be supported by the Open Philanthropy Project AI Fellowship. Megha Srivastava is supported by the NSF Graduate Research Fellowship Program under Grant No. DGE-16565 18.
We would additionally like to thank the participants of our user study. We further extend a special thank you to Madeline Liao and Raj Palleti for helping with the visual intuition for our imitation learning analysis. Finally, are grateful Mandelkar, Kaylee Burns, and Ranjay Krishna for their feedback earlier versions of this work.
anonymous reviewers, Suneel Belkhale, Ajay
References
[1] D. M. Taylor. Americans With Disabilities: 2014. US Census Bureau, 2018.
[2] A. D. Dragan and S. S. Srinivasa. A policy-blending formalism for shared control. Interna- tional Journal of Robotics Research (IJRR). 32:790-805, 2013.
[3] B. D. Argall. Autonomy in rehabilitation robotics: an intersection. Annual Review of Control, Robotics, and Autonomous Systems, 1:441-463, 2018.
[4] S. Javdani, H. Admoni, S. Pellegrinelli, S. S. Srinivasa, and J. A. Bagnell. Shared autonomy via hindsight optimization for teleoperation and teaming. International Journal of Robotics Research (IJRR), 37:717-742, 2018.
[5] H. J. Jeon, D. Losey, and D. Sadigh. Shared autonomy with learned latent actions. In Robotics: Science and Systems (RSS), 2020.
[6] L. Herlant, R. M. Holladay, and S. S. Srinivasa. Assistive teleoperation of robot arms via automatic time-optimal mode switching. In ACM/IEEE International Conference on Human Robot Interaction (HRI), pages 35-42, 2016.
[7] D. P. Losey, K. Srinivasan, A. Mandlekar, A. Garg, and D. Sadigh. Controlling assistive robots with learned latent actions. In International Conference Robotics and Automation (ICRA), pages 378-384, 2020.
[8] D. P. Losey, H. J. Jeon, M. Li, K. P. Srinivasan, A. Mandlekar, A. Garg. J. Bohg, and D. Sadigh. Learning latent actions to control assistive robots. Autonomous Robots (AURO), pages 1-33, 2021.
[9] S. Tellex, T. Kollar, S. Dickerson, M. R. Walter, A. G. Banerjee, S. J. Teller, and N. Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Association for the Advancement of Artificial Intelligence (AAAI), 2011.
[10] D. Arumugam, S. Karamcheti, N. Gopalan, L. L. S. Wong, and S. Tellex. Accurately and efficiently interpreting human-robot instructions of varying granularities. In Robotics: Science and Systems (RSS), 2017.
[11] Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Computer Vision and Pattern Recognition (CVPR), pages 6077-6086. 2018.
[12] K. M. Hermann, F. Hill, Green, F. Wang. R. Faulkner, H. Soyer, D. Szepesvari, W. Czar- necki, M. Jaderberg, D. Teplyashin, M. Wainwright, C. Apps, D. Hassabis, and P. Blunsom. Grounded language learning in simulated 3d world. arXiv preprint arXiv:1706.0653 2017.

---

[13] S. Stepputtis, J. Campbell,
M. Phielipp. S. Lee, C. Baral, and H. B. Amor. Language- conditioned imitation learning for robot manipulation tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[14] Y. Liu, M. Oit, N. Goyal, Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer and V. Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
[15] N. Reimers and I. Gurevych. Sentence-BERT: Sentence embeddings using siamese BERT- networks. Empirical Methods Natural Language Processing (EMNLP), 2019.
[16] D. Gopinath, Jain, and Argall. Human-in-ther -otta-ooo- optimization of shared autonomy in assistive roboties. IEEE Robotics and Aulomation Letters (RA-L), 2:247-254, 2016.
[17] M. T. Ciocarlie and P. K. Allen. Hand posture subspaces for dexterous robotic grasping. Inter- national Journal of Robotics Research (IJRR), 28:851-8677 2009.
[18] S. Jain and B. Argall. Probabilistic human intent recognition for shared autonomy in assistive robotics. ACM Transactions on Human-Rokee Roboot Interaction (THRI), 9:1-23, 2019.
[19] J. Luketina, N. Nardelli, G. Farquhar, Foerster, J. Andreas, E. Grefenstette, S. Whiteson, and T. Rocktäschel. A survey of reinforcement learning informed by natural language. In International Joint Conference Artificial Intelligence (IJCAI), 2019.
[20] C. Matuszek. Grounded language learning: Where robotics and NLP meet. In International Joint Conference on Artificial Intelligence (IJCAI), 2018
[21] S. Tellex, N. Gopalan, H. Kress-Gazit and C. Matuszek. Robots that use language. Anmual Review of Control, Robotics, and Autonomous Systems, 3(1):25-55 2020.
[22] B. A. Newman, R. M. Aronson, S. Srinivasa, K. Kitani, and H. Admoni. Harmonic: A multi- modal dataset of assistive human-robot collaboration. arXiv preprint arXiv: /80. // 154, 2018.
[23] Reddy, A. D. Dragan, and Levine. Shared autonomy via deep reinforcement learning. In Robotics: Science and Systems (RSS), 2018
[24] C. B. Schaff and M. R. Walter. Residual policy learning for shared autonomy. In Robotics; Science and Systems (RSS), 2020.
[25] O. Khatib. A unified approach for motion and force control of robot manipulators: The opera- tional space formulation. IEEE Journal on Robotics and Automation, 3:43-53, 1987.
[26] W. Wilson, C. W. Hulls, and G. S. Bell. Relative end-effector control using cartesian position based visual servoing. IEEE Transactions Robotics (T-RO), 12:684-696, 1996.
[27] M. Li, D. P. Losey, J. Bohg, and D. Sadigh. Learning user-preferred mappings for intuitive robot control. International Conference on Intelligent Robots and Systems (IROS), 2020.
[28] S. Karamcheti, A. Zhai, D. P. Losey, and Sadigh. Learning visually guided latent actions for assistive teleoperation. In Learning for Dynamics & Control Conference (L.DC), 2021.
[29] C. Doersch. Tutorial on variational autoencoders. arXiv preprint arXiv: 16d:::66000900 2016.
[30] T. Kollar, S. Tellex, D. Roy, and N. Roy. Toward understanding natural language directions. In Human-Robat Interaction, pages 259-266, 2010.
[31] C. Matuszek, E. Herbst, Zettlemoyer, and D. Fox. Learning to parse natural language commands robot control system. In International Symposium Experimental Robotics (ISER), 2012.
10

---

[32] Y. Artzi and L. Zettlemoyer. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics (TACL), 1:49-62, 2013.
[33] S. Karamcheti, E. C. Williams, D. Arumugam, M. Rhee, N. Gopalan, L. L. S. Wong, and S. Tellex. A tale of two draggns: A hybrid approach for interpreting action-oriented and goal- oriented instructions. In First Workshop Language Grounding for Robotics @ ACC,, 2017.
[34] V. Blukis, N. Brukhim, A. Bennett, R. A. Knepper, and Y. Artzi. Following high-level naviga- tion instructions on a simulated quadcopter with imitation learning. In Robotics: Science and Systems (RSS), 2018.
[35] X. E. Wang, Q. Huang. A. Celikyilmaz, Gao, D. Shen, Y.-F Wang, W. Y. Wang, and L. Zhang. Reinforced cross-modal matching and self-supervised imitation learning for vision- language navigation. In Computer Vision and Pattern Recognition (CVPR), 2019.
[36] C. Lynch and Sermanet. Grounding language in play. arXiv preprint arXiv:2005.07648, 2020.
[37] D. S. Chaplot, K. M. Sathyendra, R. K. Pasumarthi, D. Rajagopal, and Salakhutdinov. Gated-attention architectures for task-oriented language grounding. In Association for the Advancement of Artificial Intelligence (AAAI), 2018.
[38] Hill, S. Mokra, N. Wong, and T. Harley. Human instruction-folloiing with deep reinforce- ment learning via transfer learning from text. arXiv preprint #xXi:20055.09382 2020.
[39] MacGlashan, M. Vrommn, M. desJardins, M. Littman, S. Muresan, S. Squire, S. Tellex, D. Arumugam, and L.. Yang. Grounding English commands reward functions. In Robotics: Science and Systems (RSS), 2015.
[40] J. Fu, A. Korattikara, S. Levine, and S. Guadarrama. From language to goals: Inverse re- inforcement learning for based instruction following. In International Conference on Learning Representations (ICLR), 2019.
[41] D. Bahdanau, F. Hill, J. Leike, F. Hughes, S. A. Hosseini, P. Kohli, and E. Grefenstette. Learn- ing to understand goal specifications by modelling reward. In International Conference Learning Representations (ICLR), 2019.
[42] J. D. Co-Reyes, A. Gupta, S. Sanjeev, N. Altieri, J. DeNero, P. Abbeel, and S. Levine. Guiding policies with language via meta-learning. In International Conference on Learning Represen- tations (ICLR), 2019.
[43] S. Karamcheti, D. Sadigh, and P. Liang. Learning adaptive language interfaces through de- composition. In EMNLP Workshop for Interactive and Executable Semantic Parsing (InEEx- SemPar), 2020.
[44] Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, Reid, Gould, and A. van den Hengel. lision-and-language navigation: Interpreting visually-grount naviga- tion instructions in real environments. In Computer Vision and Pattern Recognition (CVPR), 2018.
[45] M. Chevalier-Boisvert, D. Bahdanau, S. Lahlou, L. Willems, C. Saharia, T. H. Nguyen, and Y. Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. In International Conference on Learning Representations (ICLR), 2019.
[46] E. Perez, Strub, H. D. Vries, Dumoulin, and A. C. Courville. Film: Visual reasoning with general conditioning layer. In Association for the Advancement of Artificial Intelligence (AAAI), 2018.
11

---

[47] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. Association for Computational Linguisties (ACL.), pages 4171-4186, 2019.
[48] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang. M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv preprint arXiv: 10683, 2019.
[49] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Vosse G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
[50] A. Marzoev, S. Madden, M. Kaashoek, M. J. Cafarella, and J. Andreas. Unnatural language processing: Bridging the gap between synthetic and natural language data. arXiv preprint arXiv:2004. 13645, 2020.
[51] A. Ku, P. Anderson, R. Patel, E. Ie, and Baldridge. Room-across-room Multilingual vision- and-language navigation with dense spatiotemporal grounding. In Empirical Methods in Nat- ural Language Processing (EMNLP), 2020.
[52] S. I. Wang, S. Ginn, Liang, and C. D. Manning. Naturalizing a programming language via interactive learning. In Association for Computational Linguistics (ACL), 2017.
12


Reviews:
[{"review_id": "qy9EvUtuo_o", "reviewer": "Reviewer_dhc1", "summary": "The paper proposes a shared autonomy framework that enables people to control a robot arm using a combination of natural language text and traditional (joystick) control input. While most language understanding algorithms for robots operate in the full autonomy regime, the ability for people to also provide lower-level inputs helps to mitigate deficiencies in the autonomy (or the language understanding) and provides an important sense of agency. The proposed framework encodes state-control-language tuples into a learned latent action space that is then decoded to generate low-level actions for the robot. The model is trained on a dataset state-action pairs that were subsequently annotated. At test time, a given utterance is mapped to the nearest utterance in the training set using a pre-trained language model, which is then provided to the decoder. The method is evaluated on different manipulation tasks and compared with end-effector and imitation learning-based language understanding baselines. The proposed framework outperforms both baselines in terms of task success. A user study further assesses the qualitative benefits of the approach.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS:\n\n* The idea of coupling natural language-based control with lower-level inputs in a shared autonomy framework is interesting.\n* The paper presents the results of a user study that demonstrate the qualitative differences between the proposed method and standard baselines.\n\n\nWEAKNESSES:\n\n* The primary advantage of combining language with lower-level inputs is the ability to leverage their complementary nature. While there may be many tasks for which language would be a sufficient way for a user to convey their intent, there are others for which another means of input is necessary (particularly in partially observable/unstructured settings). However, the scenarios that are considered here are fully known and quite structured. \n* The poor performance of the language-only baseline is suspicious. I imagine that it is a result of the fact that the state does not include information about the environment, in which case one can't expect a language-only version to perform the manipulation tasks well. Given the demonstrated success of language understanding for navigation and manipulation, a more standard baseline would presumably do much better.\n* I like the idea of using language to influence learned latent actions as suggested by the title, however there is very little discussion and no evaluation of how the learned actions are influenced by language.\n* The architecture combines existing components in an vanilla manner. The novelty is unclear.\n---> The author response helps to clarify the novelty of the method.\n* In addition to the work on latent actions, there is relevant work on shared autonomy in the case of unknown goals [1], including approaches that are task-agnostic [2].\n---> The author response acknowledges this work and discusses the ways in which they differ\n* The paper omits a large body of work in language understanding, including non-neural methods for language grounding in known and unknown environments [3]. These approaches have been shown to be effective for challenging manipulation commands despite being trained on relatively few samples. Of course. this comes at the cost of having to hand-engineer features and having a means of resolving abstract symbols to low-level actions (e.g., in the form of a policy), but that functionality exists, at least in the context of the scenarios considered here.\n---> Again, the author response notes the intention to place the proposed method in the context of this body of work.\n* The paper should discuss relevant work that combines natural language understanding with gestures [4--9], as complementary input modalities.\n* The paper could be clearer on the amount of supervision required for the proposed approach. \n---> The author response addresses this concern.\n* It isn't clear how the method can be expected to generalize without reasoning over the environment state.\n---> I don't feel that this point was adequately addressed. Instead, it was left for future work.\n* While hierarchy is noted in the discussion section, a key reason that existing language understanding models can be trained in a relatively efficient manner and are able to generalize is that they exploit the hierarchical nature of language. This is largely ignored in the text.\n* Language retrieval in these settings is a function not only of the utterances, but also the environment, however the proposed approach performs retrieval only using language.\n\n\nCOMMENTS/QUESTIONS:\n\n* Lines 54--60: The deficiencies of existing language understanding methods depends on the approach. Those that are given are certainly true of methods that ground language to low-level actions, but less so with those that employ more abstract symbols. In these cases, the advantage offered by incorporating low-level control seems more related to being able to tolerate scenarios when these abstract symbols (i.e., policies) can not be readily mapped to low-level actions.\n* Line 200: It isn't clear how the proposed framework would behave given an utterance that significantly differed from those encountered during training with regards to making mistakes. What is to stop the architecture from effectively ignoring the user's commands or does it provide a mechanism that allows them to take full control. If so, how does this mechanism differ from one that you might integrate into a language-only system?\n* The means by which annotators were filtered based on their similarity to the pre-trained model seems a bit unfair.\n---> I mis-spoke in my initial review meant to comment on the possibility that filtering the training corpus based on self-consistency would reduce diversity.\n* Line 303: The paper states that the latent actions model produces intuitive control spaces that humans can quickly grasp. Was this experimentally verified?\n\n\n\nREFERENCES\n\n\n[1] S. Reddy, A. Dragan, and S. Levine. Shared autonomy via deep reinforcement learning. RSS 2018\n\n[2] C. Schaff and M. Walter. Residual policy learning for shared autonomy. RSS 2020\n\n[3] F. Duvallet, K. T., and A. Stentz. Imitation learning for natural language direction following through unknown environments. ICRA 2013.\n\n[4] C. Matuszek, L. Bo, L. Zettlemoyer, and D. Fox. Learning from unscripted deictic gesture and language for human-robot interactions. AAAI 2014\n\n[5] T. Kollar, J. Krishnamurthy, and G.P. Strimel. Toward Interactive Grounded Language Acqusition. RSS 2013.\n\n[6] C. Kennington, S. Kousidis, and D. Schlangen. Interpreting situated dialogue utterances: an update model that uses speech, gaze, and gesture information. SIGDIAL 2013\n\n[7] S. Mohan, A. Mininger, J. Kirk, and J.E. Laird. Learning grounded language through situated interactive instruction. AAAI Fall Symposium 2012.\n\n[8] D. Whitney, M. Eldon, J. Oberlin, and S Tellex. Interpreting multimodal referring expressions in real time. ICRA 2016.\n\n[9] N. Mavridis. A review of verbal and non-verbal human–robot interactive communication. Robotics and Autonomous Systems 2015.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The primary contribution of the paper is the integration of language with low-level control in a shared autonomy framework. This is certainly an interesting line of research, particularly in the context of scenarios for which language and low-level control alone are insufficient. However, the significance of this contribution is unclear, in part because of questions about the experimental evaluation and the novelty of the architecture.\n\nUPDATES BASED ON AUTHOR RESPONSE:\n\nI thank the authors for their detailed response my comments/questions. Several of these have been resolved, but I still have concerns regarding the suitability of the language baseline and the (clarity of the) primary contributions of the paper, which is the use of language as an additional input modality to resolve ambiguity in joystick input. I do find this to be an interesting line of work, but the fact that including language outperforms the baselines that were used here for this set of arguably simple tasks isn't terribly surprising and it's not clear that the results would generalize to more challenging domains.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "In addition to the comments/questions above:\n\n* Did the experiments involve a real robot arm? I assume that they did, but the text is a bit unclear.\n---> Clarified in author response\n* Are one-to-one state-action-utterance mappings needed (line 164) and if so, why is it reasonable to assume that there is an utterance for every state-action pair? If not, at what level do they need to be aligned?\n---> Clarified in the author response\n* Did the authors consider an IL baseline that only used control input (as opposed to only language)?\n---> The author response largely answers my questions, but it should be clarified in the main text and not just the supplementary material.\n"}}, {"review_id": "c3aF6ZGZhhM", "reviewer": "Reviewer_TkuX", "summary": "The paper proposes a method to learn a mapping from low-DOF (e.g. 2-DOF joystick) control inputs to joint space (e.g. 7-DOF) commands that are otherwise inaccessible or unintuitive to human operators, conditioned on a natural language task description. Given a natural language utterance describing a task (e.g. pour milk into the bowl), the system produces a control mapping that allows a human user to more easily, smoothly and efficiently complete the stated task. This is a shared autonomy as opposed to a full autonomy setting as explored in prior instruction-following works, where ultimately the control over robot motion is with the end-user.\n\nAt test-time the system uses a BERT-based sentence similarity metric to map a natural language utterance to the closest synthetic task description in the training set (seemingly among a set of 5 tasks). Then, a control decoder is conditioned on the task description and maps 2-DOF joystick commands to 7-DOF joint space.\n\nThe paper performs a user study to compare the effectiveness and user-friendliness of their approach against a fully articulated mode-switching control scheme, and also against an imitation learning instruction-following approach. The paper shows that LILA is more effective at enabling the user to accomplish manipulation tasks, achieves smoother motion, is more likely to have the users prefer to use it again, and other positive qualities.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Explores a novel, relevant, and interesting idea. Opens doors to exciting follow-up work.\n- The approach of mapping a natural language utterance at test-time to the closest synthetic language utterance from the training set has been explored before in a full autonomy setting, where it significantly restricts the scalability of the system capabilities. This paper applies this method in the shared autonomy setting, where the user is ultimately in charge of the robot actions. Here, it makes a lot more sense. The scalability concern (e.g. of handling novel objects or composing primitives in novel combinations) is largely addressed by the human pilot, while the restricted synthetic syntax reduces the chance of an out-of-distribution utterance resulting in a noisy or unusable control mapping.\n- Performs a real-world end-user study, which is both rare and necessary to assess performance of systems with natural language interfaces.\n- Opens doors to interesting uses of shared autonomy besides end-user use, for example to improve the ease of crowdsourcing demonstrations on a dexterous robot arm corresponding to specific instructions.\n\nWeaknesses:\n- Unless I am missing something, the system does not take any visual inputs or observations of the environment as input. It is therefore incapable of actually understanding or grounding the natural language utterances to percepts or entities in the world. The experiments work solely because the training tasks and test tasks use the exact same objects in the same placements with no within-task variation. Therefore solving the task boils down to classifying the input utterance into one of the 5 task types, and then applying a fixed control mapping corresponding to the task type. This does not demonstrate any meaningful language grounding capabilities, which should include aspects like grounding noun phrases to visual percepts, responding to object placement variations, and allowing compositional specifications of object attributes. Any practical shared autonomy system would need to see the world and respond to it, dynamically interpreting the meaning of the instruction in relation to the objects in the environment. Prior instruction-following methods already demonstrate these capabilities, so I don’t see a strong reason to take such a significant step back in terms of the experimental setup.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes a very exciting and relevant use-case of grounding natural language instructions to control mappings for a shared autonomy setting, and a user study of an initial prototype system showing effectiveness and user-friendliness in a shared-autonomy setting. This not only opens a new, exciting area of research, but is also relevant to assistive robotics, where it could significantly accelerate the adoption of robots that help people with disabilities, and bring value in the near term while fully-autonomous systems are still not robust enough.\n\nHowever, as described in the “Strengths and Weaknesses”, the system does not ground language in observations of the environment, instead applying a fixed joint-space mapping that relies on objects being placed in previously known positions. This is a significant technical weakness. First, there is no meaningful language grounding or language understanding going on, and second, there does not appear to be a way to scale the system to a practical scenario, even a simple table-top scenario with various placements of objects. I understand that building a language grounding system on a real robot, and performing a user study is very hard, and I wouldn't want these obstacles to stand in the way of putting an excellent idea and the user study results out there. At the very least, the paper should make these limitations abundantly clear, so as to direct follow-up work (perhaps from groups with language grounding expertise) to improve upon these aspects of the system.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "- Lines 242-246: It is not stated whether the best annotators are typical or atypical. I could imagine each having pros and cons. Typical might give more sensible and natural language. Atypical might provide more diverse training data useful to expand the scope of natural language that the system can understand.\n- I would suggest trying mean-pooled GloVe vector representations to capture sentence similarity. They seem to be potentially better on simple sentences than BERT, and have the additional advantage of being lightweight (See https://arxiv.org/pdf/2011.05864.pdf).\n- I suggest that slightly more model details (e.g. high-level architecture, how the features h are fed into the networks, how FiLM is used) are provided in the main paper, perhaps at the expense of the introduction, which could easily make the same points more concisely..\n\nSuggestion for incorporating vision:\n- The pre-trained CLIP feature extractor (https://openai.com/blog/clip/) is known to be useful for grounding object references to locations in the environment. Perhaps there's a way to utilize it without requiring expensive language data collection?\n\nMinor comments:\n- It’s a bit hard to understand what this sentence means: “injects linguistic information into the core of a standard latent action decoder.”\n- Unless I am missing something, the reward function R(s, a, u) does not appear to be ever used as there is no reward optimization taking place. I would suggest cleaning up the formalism section so that it better aligns with the actual system under consideration.\n\nQuestions:\n- (Line 150): what guarantees that the controller exposes an intuitive action space that listens to the user? I could imagine that the controller may ignore z and instead focus on inputs s,u to directly optimize R(s,a,u), since z is not strictly necessary for this purpose.\n- How does the loss in Equation 1 ensure that the learned latent space is smooth and intuitive? I could imagine it having jarring discontinuities."}}, {"review_id": "RsILh-7GdPG", "reviewer": "Reviewer_Eg7K", "summary": "This paper presents a shared autonomy framework that reduces full robot arm control to 2 degrees of freedom, deciding along what control axes to align those degrees based on a natural language input command for the task to be completed, such as \"pour the milk into the bowl\". The method presented uses a transformer-based, massively pretrained language model to perform nearest neighbor lookup at inference time to map language commands to understood data points from the training set to modulate control. The work is a crisp step towards a larger space of language-guided shared autonomy.\n\n*Post-response*\n\nThe authors substantially revised the paper in response to other reviewer's concerns. The main concern I had regarding data filtering was clarified as an MTurk cleanup procedure. My only comment for future work from the authors there is that a second \"filtering\" HIT is a cleaner way to do this. Basically, gather data D_0, run a filtering HIT where annotators select whether instances in D_0 are valid (e.g., are they spam, do they match a demonstration, etc.), then throw out bad D_0 instances *and* remove worker qualifications for the annotators who provided them, then repeat the data collection step to fill out D_1 replacing now-missing instances. Repeat to D_i full dataset.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of the paper is a clear method for langauge-informed shared autonomy with convincing human subject experiments for a physical robot agent.\n\nThe main weakness of the paper is not having a head-on discussion about exactly what kind of generalization from training data is possible under LILA's current formulation. In particular, the model can \"sort of\" generalize to unseen language commands, as long as they are paraphrases of commands seen at training time. To my understanding, the model has no capacity to generalize to unseen tasks and entirely unseen language specifications for those new tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper makes a concrete step in RoboNLP towards shared autonomy and contains convincing, real-world experiments with humans controlling a robot arm conditioned on free form language input. When I imagine the intersection of \"RoboNLP\" and \"CoRL\", this paper is a great example.\n", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "Questions by priority\n\nQ1 In \"Crowdsourcing Language Annotation\", annotations whose representations fall away from a mean are discarded, which feels a bit icky because the method later uses those representations to do nearest neighbor lookup. It could be argued, then, that the data has been curated for the algorithm, keeping around only a local neighborhood of utterances to retrieve without allowing for outliers that could cause trouble at inference time. Can the authors clarify this methodology?\n\nQ2 For Figure 4, I was surprised to see End Effector performing better than Imitation Learning in some cases. Is there a clear intuition for this result? Maybe IL just needs a lot more demonstrations before the policy isn't too frustrating?\n\nNits and questions not effecting evaluation:\n\n- Line 77 \"unprompted\" unclear. Does this mean some users didn't use language at all?\n\n- Line 166 \"action\" in quotes is a weird way to describe space Z since there is already an action space; the paper knows this since it's in scare quotes, but maybe just use a different word.\n\n- Line 192 Grounding Language in Play also pulls this \"retrieve utterances from training time\" trick, if I recall correctly.\n\n- Line 250; 5 students were female, were the other 5 male? Leaving this inference implicit erases other gender identities and saves almost no space in the paper.\n\n- Line 289 \"heralding a bright future\" https://i.gifer.com/Dnxs.gif"}}, {"review_id": "5d6YW7UVM-J", "reviewer": "Reviewer_o584", "summary": "This paper presents an approach to improving shared autonomy in which a user controls a high degree of freedom robot arm with a low dimensional controller and provides natural language instructions to disambiguate user intention. Building on recent work in learning latent actions for shared autonomy---for which language has not previously been used---the authors demonstrate the utility of adding language on a series of manipulation tasks. The experiments in the paper demonstrate that their approach, language-informed latent actions or LILA, allows the human-robot team to achieve good performance on the selected tasks, outperforming two baseline strategies.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the method is innovative and the paper is well written. However, it is the central claim of the paper is that the addition of language is necessary to disambiguate the robot's actions when trying to tackle a task in the presented shared autonomy scenario, yet the results (though otherwise informative and convincing) lack a comparison to a learned method that does not use language. Such a comparison is important to back up their central claim and show that the language input is indeed necessary for this situation. This omission is particularly surprising, as the primary point of comparison made in the remainder of the paper (e.g., the theory section) is to previous work on non-language-informed latent action learning. While the results are shown to work and the users are able to accomplish their tasks, it is not clear that it is the addition of the language that make this feat possible, greatly weakening the paper. The comparison to the Imitation Learning baseline, while informative, does not allow for a similarly realistic comparison. This is the central limitation of the current version of this work, and the authors should clarify why non-language latent action experiments were not included.\n\nEDIT (post rebuttal): While I will admit that it was a bit of a strange decision to omit the no-language baseline from the user study, I am satisfied by the experiments now included in the paper. I think this is a reasonably solid paper, made whole by the addition of the new no-language baseline. I am changing my score to reflect this.\n\n*Minor Comments and Suggestions*\n- Figs. 1 & 2: After reading through the paper, these figures are helpful visualization of the proposed approach, but on their own (without the body text), the figures do not do a particularly effective job at illustrating how everything works. Figure 1 in particular is not particularly informative if one is not already familiar with learned latent actions; some additional visualization of the user input (to show that it was shared between the blue and orange instructions) would be incredibly helpful. For Figure 2, it was not clear from the schematic that the agent could . Furthermore, the figure is somewhat misleading, since using the 1DoF controller does not necessarily correspond to moving along the left-right direction in the figure. Both figures should be revisited for clarity.\n- What does \"success\" refer to in Table 1 (and how many trials are included in each; I could not find this information in the text)? Is it the success of the retrieval or of the task associated with the specified task. Relatedly, what does a failure \"look like\" for the Pour Bowl task?\n- There is a typo in Figure 3: The success rate is presented as a percentage, yet the y-axis only goes to 1.0%. Presumably, this is 100% success.\n- A minor point: while not a style guide violation, the abstract is quite long, and it could be improved if the authors were to shorten it somewhat.\n- I believe that, owing to the \"retrieval\" nature of the language model, all of the tasks appear in the training dataset. It would be helpful for clarity for the authors to reiterate this point in the experiments section. [I do not think additional experiments are required.]\n- Under participants, it is mentioned that 5 are female: are we to presume that the other 5 are men (or non-binary or did not specify)? It's a small change, but promotes inclusion.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, upon first reading I was quite enthusiastic about this paper (and generally remain so) because, though the technical advance is not overwhelming, it represents solid progress in a difficult area and is supported by some good experiments that show the efficacy of using LILA, the proposed approach. However, the omission of a comparison to prior work in non-language-informed latent actions is hard to overlook. As I mention above, the central claim of the paper relies on language being a key distinguishing feature versus previous work in latent action learning, yet the only other learned baseline uses imitation learning (without subsequent human intervention) that also relies on learning. I am willing to change my recommendation if the authors can provide sufficient justification for its omission.\n\nEDIT (post rebuttal): See my comments above, but I believe that the addition of the no-language baseline satisfied my primary concern. I have updated my score to reflect this change.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "Much of the rationale for my decision and my suggestions for changes are included in the \"Strengths and Weaknesses\" section above, and so the authors should refer to that for more detail. In short, the lack of a baseline comparison algorithm that uses non-language-informed learned latent actions is conspicuous; the authors should provide justification for why this baseline was not included and why the central claims of the paper are sufficiently supported by the experiments that currently exist.\n\nEDIT (post rebuttal): See my comments above, but I believe that the addition of the no-language baseline satisfied my primary concern. I have updated my score to reflect this change."}}]

Author rebuttal:

Meta-review: This paper introduces a shared autonomy input for manipulation that is augmented with language understanding such that discrete language inputs (classification of predetermined phrases) can be used to improve task performance.

Strengths:

Development of multimodal inputs for manipulation is an important research area.  The paper introduces a novel approach and includes a user study validation.


Weaknesses:

The evaluation method does not appear to actually use language grounding as a baseline. The language only variant being tested has no access to actual state data, and thus does not actually perform any grounding.  This significantly undermines the reported results.

Furthermore, a central claim of the paper is that the addition of language is necessary to disambiguate the robot's actions when trying to tackle a task in the presented shared autonomy scenario.  However, the evaluation does not include a comparison to an approach that does not use language.  This is an important result to include in order to demonstrate the added benefit of this additional modality.

The work is not well situated in existing literature on language understanding, please see notes by R1 for additional works to include in the discussion.

It is not clear how well the presented technique would generalize beyond the limited explored scenario.

The attached videos did not significantly add to the clarity of the paper.  Some annotation of their content would be helpful.

Please see detailed reviewer comments for a list of additional questions and suggestions.


Post rebuttal comment:  Thank you for your detailed responses to the reviewers.  The paper represents an interesting and valuable advance in the area of situated NLP, and the authors' additions and clarifications were very valuable.  As is clear from the comments, there were a few common areas of confusion/concern that require clear clarification in the paper:
* The term "language grounding" means different things in different research contexts, which initially caused confusion for majority of reviewers with regard to how this paper was written.  The authors should carefully define this term.
* Clear discussion of limitation with respect to generalization.  The state of the art for most ML techniques is that we test generalization in new scenes/applications/domains, and so again readers are coming in with this assumption in mind.  This should be clearly addressed in the paper, as well as how this current approach, which does not generalize to unseen objects, is a helpful research stepping stone.
* The baseline discussion needs to be clearly presented in the paper.
Decision: Accept (Poster)