Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Tasks
Authors: Albert Yu
Ray Mooney
Venue: ICLR 2023 poster (2023)
Keywords: natural language for robotics
instruction following
learning from demonstrations
multi-task learning
robotic manipulation

Abstract: Demonstrations and natural language instructions are two common ways to specify and teach robots novel tasks. However, for many complex tasks, a demonstration or language instruction alone contains ambiguities, preventing tasks from being specified clearly. In such cases, a combination of both a demonstration and an instruction more concisely and effectively conveys the task to the robot than either modality alone. To instantiate this problem setting, we train a single multi-task policy on a few hundred challenging robotic pick-and-place tasks and propose DeL-TaCo (Joint Demo-Language Task Conditioning), a method for conditioning a robotic policy on task embeddings comprised of two components: a visual demonstration and a language instruction. By allowing these two modalities to mutually disambiguate and clarify each other during novel task specification, DeL-TaCo (1) substantially decreases the teacher effort needed to specify a new task and (2) achieves better generalization performance on novel objects and instructions over previous task-conditioning methods. To our knowledge, this is the first work to show that simultaneously conditioning a multi-task robotic manipulation policy on both demonstration and language embeddings improves sample efficiency and generalization over conditioning on either modality alone.

Published as a conference paper at ICLR 2023
USING BOTH DEMONSTRATIONS AND LANGUAGE IN- STRUCTIONS TO EFFICIENTLY LEARN ROBOTIC TASKS
Albert Yu UT Austin L.bertyuuutxxas.ed
Raymond J. Mooney UT Austin
mooney@urexas, edu
ABSTRACT
Demonstrations and natural language instructions are two common ways to spec- ify and teach robots novel tasks. However, for many complex tasks, a demon- stration or language instruction alone contains ambiguities, preventing tasks from being specified clearly. In such cases, combination of both a demonstration and instruction more concisely and effectively conveys the task to the robot than either modality alone. To instantiate this problem setting, we train a single multi-task policy
few hundred challenging robotic pick-and-place tasks and propose DeL-TaCo (Joint Demo-Language Task Conditioning), a method for con- ditioning robotic policy on task embeddings comprised of two components: a visual demonstration and language instruction. By allowing these two modalities to mutually disambiguate and clarify each other during novel task specification. DeL--TCCo (1) substantially decreases the teacher effort needed to specify new task and (2) achieves better generalization performance on novel objects and in- structions over previous task task-conditioning methods. To our knowledge, this is the first work to show that simultaneously conditioning multi-task robotic manipu- lation policy on both demonstration and language embeddings improves sample efficiency and generalization over conditioning on either modality alone. See ad- ditional materials at htttps: http://eeetttoooeeeoo cetitaacoooooo.....ooooeeooooo.e github.
INTRODUCTION
significant barrier to deploying household robots is the inability of novice users to teach new tasks with minimal time and effort. Recent work in multi-task learning suggests that training on a wide range tasks, instead the single target task, helps the robot learn shared perceptual representations across the different tasks, improving generalization (Kalashnikov et al., 2021; Yu al. 2019 Dang et al. 2021 Shridhar et al. 2021). We study the problem of how to more efficiently specify new tasks for multi-tam robotic policies while also improving performance.
Humans often learn complex tasks through multiple concurrent modalities, such as simultancous visual and linguistic (speech/captioning) streams of video tutorial. One might reasonably expect robotic policies to also benefit from multi-modal task specification. However, previous work in mul- titask policies condition only on a single modality during evaluation: one-hot embeddings, language embeddings. Each has limitations.
embeddings, or demonstrationngoallinnn
One-hot encodings for each task (Kalashnikov al., 2021 Ebert al. 2021 suffice for learning repertoire of training tasks but perform very poorly on novel tasks where the one-hot embedding is out of the training distribution, since one-hot embedding spaces do not leverage semantic similarity between tasks to more rapidly learn additional tasks. Conditioning policies on goal-images (Nair al., 2017 2018 Nasiriany et al. 2019) or training on video demonstrations (Smith et al., 2020) Young et al. 2020) often suffer from ambiguity, especially when there are large differences between the environment of the demonstration and the environment the robot in, hindering the understand- ing of demonstration's 's true intention. In banguage-conditioned policies (Blukis et al.) 2018 2019 Mees et al. 2021 2022), issues of ambiguity are often even more pronounced, since humans specify similar tasks in very linguistically dissimilar ways and often speak different levels of granular- ity, skipping over common-senae steps and details while bringing up other impertinent information. Grounding novel nouns and verbs not during training compounds these challenges.

---

Published as a conference paper at ICLR 2023
Training
Testing
Train single multi task policy un hundreds tasks
One-shot generalization to to- new taskk (new objects, color, shapes).
Training Buffer
Validation Buffer
To Intraduor The adloo
Tack,, Inatrctioa car- Donnr
Lorky, Iatradior
Takk latoa has lay. Stngl bero
Instrccton: ctnler Terne
Dreas
SagleDoooo
Randomly selected
Test
Demo
Instruction
Demo
Emstraction "Put Erepecardaallhyee in grnen bin.
scaesshapeecfgee in backbun."
Leam demo language contextual associations during irmining...
... To leverage both modalities to resolve task ambiguities during lesting.
Damo encodar
Language
Demo
Language ancocil
Contrantive Leorning
Ztong
Z.eena
Zlang
Multitask Policy
Multitask Policy
Learning Algoetthm
Figure 1: DeL-TaCo Overview. Unlike current multitask methods that condition single task specification modality, DeL-TaCo simultaneously conditions on both language and demonstrations during training and testing resolve any ambiguities in either task specification modality, enabling better generalization to novel tasks and significantly reducing teacher effort for specifying new tasks.
We posit that in a broad category of tasks, current unimodal task representations are often too ineffi cient and ambiguous for novel task specification. In these tasks, current tas-connitioniin methods would need either a large number of diverse demonstrations to disambiguate the intended task, or long, very detailed, fine-grained language instruction. Both are difficult for novice users to provide. We argue that conditioning the policy on both a demonstration and language not only ameliorates the ambiguity issues with language-only and demonstration-onle specifications, but is much easier and more cost-effective for the end-user to provide.
We propose DeL-TaCo (Figure D new task embedding scheme comprised of component modalities that contextually complement each other: demonstrations of the target task and corre- sponding language descriptions. To our knowledge, this is the first work demonstrate that speci- fying new tasks to robotic multi-task policies simultaneously with both demonstrations and language reduces teacher effort in task specification and improves generalization performance, two important characteristics of deployable household robots. With bimodal task embeddings, ambiguity is bidi- rectionally resolved: instructions disambiguate intent demonstrations, and demonstrations help ground novel noun and verb tokens by conveying what to act and how. To learn several hundred tasks, we train single imitation learning (II.) policy, conditioned on joint caamonnstratoonlaaggaee embeddings, to predict low-level continuous- continuous-space space actions for robot given image observations. Task encoders are trained jointly with the policy, making our model fully differentiable end-to-end.
To summarize, main contributions are follows: (I) We present broad distribution of highly- randomized simulated robotic pick-and -nd-plee tasks where instructions or demonstrations alone are too ambiguous and inefficient specifying novel tasks. (2) We propose simple architecture, DeL- TaCo, for training and integrating demonstrations and language into joint task embeddings for few- shot novel task specification. This framework is flexible and learning algorithm-agnostic. (3) We show that DeL-TaCoo significantly lowers teacher effort in novel task-specification and improves generalization performance over previous unimodal task-conditioning methods.
RELATED WORK
2.1
MULTI-TASK LEARNING
The most straightforward way to condition multi-task policies is through one-hot vectors (Ebert
al. 2021 Kalashnikov et al. 2021 Walke al.] 2022 Yu al. 2021 We instead embed-
2

---

Published as a conference paper at ICLR 2023
ding spaces that are shaped with pretrained language models so that semantically similar tasks encoded in similar regions of the embedding space, which helps improve generalization. Multi-task robotic policies have also been studied in other settings and contexts that do not fall under the class of approaches we take in this paper, such as hierarchical coal-conditioned policies (Gupta et al. 2022).
probabilistic modeling techniques (Wilson al. 2007), distillation and transfer learning ( (Parisotto ct al.
2015: Teh et al., 2017]] Xu et al. 20200 Rusu et al. 2015). data sharing (Espcholt et al. 2018
Hessel et al. 2019). gradient-based techniques (Yu et al. [0200), policy modularization (Andreas
et al. 2017 7 Devin et al. 2017) and task modularization Yang et al. 2020).
2.2 LEARNING WITH LANGUAGE AND DEMONSTRATIONS
Conditioning Multitask Policies on Language or Demonstrations. Our work largely tackles the same problem as BC-Z. (Jang et al. 2021 of generalizing to novel tasks with multi-task learning- BC-Z trains video demonstration encoder to predict the pretrained embeddings of corresponding language instructions, while jointly training a multi-task imitation learning policy conditioned on either the instruction or demonstration embeddings. [Lnnch & Sermanet (2021); D; Mees et al. (2021) learn similar policy conditioned on either language or goal images. All of these approaches learn to map demonstration goal image to similar embedding space as its corresponding language instruction. During training. MMes et al. (2022) use both demonstrations and language to learn associations between demonstration embeddings and iinguae--condtioone latent plans, but during evaluation, only use the language embedding to produce a latent plan. With a slightly different ar- chitecture, Shao Shao et al./ (2020) learn a policy that maps natural language verbs and initial observations to full trajectories by training a video classifier on a large dataset of annotated human videos.
While all of these prior approaches use both demonstrations and language during training, their policies are conditioned on either a language instruction or visual image/demonnsrratio embedding during testing. By contrast, ours conditioned on both demonstration and language embeddings during training and testing. which we show improves generalization performance and reduces human teacher effort on a broad category of tasks.
Pretrained Multi-modal Models for Multitask Policies. Another recent line of work leverages pretrained vision-language language models to learn richer vision features for downstream policies. CLI- Port (Shridhar et al. D) uses pre-trained CLIP (Radford et al. (021 to learn robust Transporter- based (Zeng et al. (2200) robot policies. Our method resembles CLiPort, its 3-dimensional successor PerAct (Shridhar et al. 2022). and the previously mentioned multi-task policy methods in that train on expert trajectories associated with language task descriptions, but in CLIPort and PerAct, the policy is only conditioned language during training and testing: demonstrations used only as buffer data for imitation learning. Our method, however, learns tasks during training or testing by using both language and a demonstration to condition the policy.
ZeST (Cui et al. 2022) and Socratic Models (Zeng et al., 2022) demonstrate that pretrained vision- language models encode valuable information for robotic goal selection and task specification. R3M (Nair et al. 2022) pretrains a ResNet (He et al.) 2015) policy backbone on language-annotated videos from Ego4D (Grauman et al., (021)) boost downstream task performance. While our mo- tivation is similar to ZeST in using pretrained language model to leverage the structure of the pretrained embedding space, we assume access to both language and demonstrations for learning novel tasks and condition on task embeddings from both, which is unlike the ZeST and R3M prob- lem settings where the policies are not directly task-conditioned.
2.3 OTHER APPLICATIONS OF LANGUAGE FOR ROBOTICS
Language-shaped state representations. On the MetaWorld multitask benchmark YYu al., 2019), number of prior works have investigated using language instructions to learn better state repre- sentations. (Sodhani et al. (202I) use language instruction embeddings to compute an attention- weighted context representation over mixture of state encoders. Silva et al. (2021) learn a goal encoder that transforms language instruction embeddings to shape the state encoder representations.
Hierarchical Learning with Language. Our approach can be loosely framed as hierarchical leamn ing, where we have two high-level task encoders that output language and demonstration embed- dings, both of which the lovel policy is conditioned on output actions. Prior work has used
3

---

Published as a conference paper at ICLR 2023
language instructions in hierarchical learning for shaping high-level plan vectors (Mees al. 2022)
or skill representations (Garg et al. 2022). which are then fed to a low-level policy to output the ac- peseencoder-bseee architecture to predict digher-dimensional robot actions from lower-dimensional controller actions and language instructions, where the lan- guage is fed into both the encoder and decoder. All of these prior approaches condition on single high-level policy, whereas ours incorporates guidance from two high-level encoders for both demon- strations and language to learn novel tasks, giving the low-level policy access certain information expressible only through their combination.
tion. KKaramchei al. (2021)
Language for Rewards and Planning. Language has also been used for reward shaping in RL (Nair et al.) [202]] Goyal et al.] [019]] (020). Pretrained language models have also been leveraged their ability to propose plans long-horizon tasks (Huang et al., 2022 Ahn al. 2022 Chen et al., 2022). While we work with II. instead of RI. and mainly deal with highly variable pick- and-place tasks, we do not use language for training reward functions or for planning, though our multi-modal task specification framework is compatible with these additional uses of language.
PROBLEM SETTING
3.1 MULTI-TASK IMITATION LEARNING
We define a set n tasks {- and split them into training tasks and test tasks V, where (U, V) bipartition of [Ti For each task Tü, assume access to set of expert trajectories (Ti ) and a single language description 4, Given continuous state space S, continuous action space A, and task embedding space Z, the goal is to train a Markovian policy π : S x - (IA) that maps the current state and task embeddings probability distribution over the continuous action space.
During training, we assume access buffer Dirain of trajectories for only the tasks in U and their associated natural language descriptions. We define each trajectory as a fixed-length sequence of
(t) ...... where j is the trajectory index for task state-action pairs Ty [ (.) mo.3) (+1) (i.
T. U with task embedding 2g. We use behavioral cloning (BC) (Hussein et al. 2017 Pomerleau
1988 to update the parameters of π to maximize the log probability of क (t, (.j though our framework is agnostic to the learning algorithm and would work for RL approaches as well.
During evaluation, we assume access to a buffer D of trajectories for only the tasks in V and their associated natural language descriptions. Unlike Dirain where we have m demonstrations for each task, in Drai we have just single demonstration for each task. For all test tasks T; E V, we rollout the policy for a fixed number of timesteps by taking action o₂ [a](s;, 2;). The 2; for all test tasks is computed beforehand and held constant throughout each test trajectory.
3.2 TASK ENCODER NETWORKS
To obtain the task embedding 2,, we have two encoders (which are either trained jointly with policy or frozen from pretrained model): demonstration encoder, fdemo : Taj b Zdemo, mapping trajectories of task T, demonstration embeddings, and a language encoder, frang : 1, + Stang mapping task instruction strings L; to language embeddings. Previous work has explored using bi as one-hot task vector, language embedding Zlong.is or goal image/demonstrattion embedding id but approach DeL-TaCo uses task embedding 2 [ždemo,i: Zlang,)] based both the instruction and demonstration embedding during training and testing to learn novel tasks.
4 METHOD
4.1 ARCHITECTURE
Demonstration and Language Encoders. The encoder faemo is CNN network trained from scratch. Following Jang et al. (202T), we input the demonstration as an array of m X n frames (in raster-scan order) from the trajectory for faster processing. (We use (m,n) (1,2) or (2,2) 2) in our
4

---

Published as a conference paper at ICLR 2023
Policy
Spatiá
segled
gropert
Ree-Net- 18
Layers
Imrageobe urrboddinge
Caraal
(40 48 33)
Demo Encoder dema
[+] Concatenato
Action (, gripper)
CNN
Taok Demmsnattatioss
Asemo
Language Uncoder tumg
learning
"Put yellon--ddorad object in left Biu."
DivIBERT firezor)
Figure 2: Method Architecture. DeL-TaCo uses three main networks: the policy n, demonstra- tion encoder Idemo- and a language encoder Stang- During both training and testing, the policy is conditioned on the demonstration and language embeddings for the task.
experiments.) We freeze pretrained DistilBERT (Sanh et al. 2019) as the encoder Stang- where Zlang.i is simply the average of all DistiBBERT-emeet tokens in (we found this works better than taking the [CLS] token embedding).
Policy Network. We use a ResNet-18 (He al. 2015 as the visual backbone for the policy m followed by spatial softmax layer (Finn et al. 2016) and fully connected layers.
Task Conditioning Architecture. BC-Z Jaang et al. 202T inputs the task embedding into the ResNet backbone via FiLM (Perez et al. 2018) layers, which apply learned affine transformation to the intermediate image representations after each residual block. BC-Z's task embeddings are either from demonstrations or language. Since our policy conditions on both, the main architectural decision was finding the best way to feed task embeddings from multiple modalities into the policy.
Empirically, simple approach performed best. The demonstration embeddings 7demo fed into the policy's ResNet backbone via FiLM, while the language task embeddings Zlang and robot pro- prioceptive state (6 joint angles, end-effector xyz coordinates, and gripper open/close state) concatenated to the output of the spatial softmax layer. Our full network architecture shown in Figure hyperparameters are in Appendix B and architectural ablations are in Table
4.2 TRAINING AND LOSSES
The training procedure for DeL-TaCo is summarized in Algorithm I During each training iteration, we sample size i: subset of training tasks M (mmy..... Ti,.) Th, and corresponding natural language instruction 4, compute the demonstration embeddings Zemb.m, = fdemo (Tiz) and language embeddings Zlang.m, = flang (44). We collect the embeddings of tasks M in matrices Zaemo [demo, Zdemo, and Ztang [Zlang, ... Zlang.ma my..
C U. Given a trajectory Tış for task
To train the demonstration encoder, Jang et al. (2021) use a cosine distance loss to directly regress demonstration embeddings their associated language embeddings. However, this causes demon- stration embeddings to be essentially equivalent to the associated language embeddings for each task, undercutting the value of passing both our policy. To preserve information unique to each modality while enabling the language and demonstration embedding spaces to shape each other, we train with CLIIP-tyle (Radford al. (02T) contrastive loss for our demonstration encoder:
(g Ziang,
Liemno (Zdemor Ziang) CrossEntropy
(1)
where I is the identity matrix and is tuned temperature scalar. We use the standard BC log- likelihood loss the policy loss term for some trajectory composed of state-action pairs Tt,i.j

---

Published as a conference paper at ICLR 2023
(01) stg. (.g extracted from an expert demonstration Taj for task Tm,
poolicy (Tt))
Σ
log (L.)) 2.) Edemo, Plang,m.
Both demoo and π networks are trained jointly with the following loss, for a tuned or > 0:
L(T, fdemo: flang) = Cpoticy(Ti) + a (Zemoo Ziang)
(3)
Where Cpolicy (Tiy) is summed over all trajectories in the batch of training tasks M (we omit this double summation Equation 2 for brevity). Note that the language encoder does not have loss term because we use a frozen, pretrained language model and rely on the pretrained embedding space to shape the demonstration encoding space.
4.3 EVALUATION
During evaluation, we want the robot to perform some novel task T, E V. Recall that T, U. our set of training tasks. From our problem setup description Section 3.1 we have access to a validation task buffer Dsal with a single demonstration Te and natural language instruction 4, of task T, We encode the demonstration with fdema and the language with flang and pass both task embeddings to the policy. Details are summarized in Algorithm 2
Algorithm DeL-TaCo: Training
Algorithm 2 DeL-TaCo: Evaluation
Input: Denin
Input: Dval
1: while not done do
1:
for validation task T, in V do
2: M <- & random train tasks from U 1b=1
2: Get demo Tu and language L, from Daa
Sample X, (Te))=d ~ Dirain
3: Zdemo +- fdemo (Tu) Encode demo
4:
X + [T M)
4: Zlang + fiang // Encode language
L (L E M) Lang. instructions
5:
for time t = 0, ...y H - do
Zdemo c- Jdeem (X) Demo encoder
6: Take action ~ T(aas:, Zdemo, Zlang)
7:
Ziang < fiang (L) // Language encoder
7: end for
8:
Update IT, Jdemo [(T, Iddemo Stang) // per
end for
Eqn.
end while
EXPERIMENTS
We empirically investigate the following questions: (1) Does there exist distribution of tasks that are more clearly specified with both language and demonstrations rather than either alone? (2) Does conditioning on both language instructions and video demonstrations with DeL-TaCo improve gen- cralization performance on novel tasks? (3) so, how much teacher effort is reduced by specifying new task with both language and demonstrations than with either modality alone?
5.1 SETUP
Environment. We develop a Pybullet (Coumans)
Mesiier Type
Example Training
Example
& Bail 2007-2022) simulation environment with
WidowX 250 robot arm, 32 possible objects of di- verse colors and shapes for manipulation, and 2 different containers. The action space continu- ous, representing an (x, y, change in the robot's end effector position, plus the binary gripper state (closed/opened). We subdivide the workspace into four quadrants. Two quadrants are randomly chosen to contain the two different containers, and three of the 32 possible objects are dropped at random loca- tions in the remaining two quadrants. RGB image observations size 48 x 48 x 3 and fed into the policy. The input format of each demo for faemo x array of images extracted from the trajectory. Details are in Appendix
Name
Shape
Figure 3: Sample train and test tasks, grouped by the object identifier types (un- derlined in each language instruction). All container identifiers are seen in both training and testing.

---

Published as a conference paper at ICLR 2023
Task Objective. To explore the first question, design the following set of pick-and-place tasks where the objective is to grasp the target object and place it in the target container. Both the target object and container can be inferred from the demonstration and language instruction. In every task, the scene contains three visually distinct objects (of which exactly one of them is the target object) and two visually distinct containers (of which exactly one of them is the target container). To make the task more challenging, the distractor objects on the scene are chosen adversarially, whenever possible, match either the color or shape of the target object. Thus, robotic policy that disregards both the task demonstration and instruction and picks any random object and places it into any random container would solve the task with 1-in-6 odds.
Language Instructions for Each Task. Figure 3 shows a selection of our training and testing tasks. Each task is specified through language with a single template-based instruction of the format "put [target object identifier] in [target container identifier]."
We make this environment more challenging by having task instructions refer to containers by either their color or quadrant position and objects by either their name, color, or shape. We use six different container identifiers in the task instructions to convey which container to drop the object in: red, green, front, back, left, and right. Thus, if the robot is provided a demonstration of grasping a cup and placing it in the red container in the front left quadrant, and it encounters an initialization with the containers in different locations, it does not know whether place the cup in the red, front, or left container. This ambiguity can only be resolved with the language instruction. Conversely, aspects of the task, such as which object grasp, are most clearly expressed through the demonstration rather than the instruction, since for novel tasks, the language instruction contains object identifiers unfamiliar to the policy. The task instructions also refer to the objects through different types of identifiers: their unique names (32 strings such as "fountain vase"), color (8 strings such as "black- and-white colored object"), or shape (10 strings such as "trapezoidal prism shaped object").
The multiple identifiers help simulate ambiguity that arises from informal human instructions, where different humans may refer to the same object or container through different attributes, enabling demonstrations and instructions to complement each other when the robot learns new task. In total, there are 50 target object identifiers (32 + + 10) and target container identifiers, giving us 300 pick-and-place tasks. We train and evaluate on a bipartition of these 300 tasks. See Appendix A for a list of all our train and test tasks.
Success Metric. In calculating the success rate, a successful trajectory is defined one that (1) picks up the correct object and (2) places it in the correct container. Appendix D details the number of sceds and trials used to calculate success rates and standard deviations.
Data. Using a scripted policy (details in Appendix C). we collect roughly 130 successful demon- strations for each training task, and single successful demonstration for each test task. All demon- strations are 30 timesteps long. Depending on our experimental scenario (see Section (.2), we train on 65% to 80% of the 300 tasks, our training buffer contains roughly 26,000-31,000 trajectories. 5.2 GENERALIZATION PERFORMANCE ON NOVEL TASKS
To test generalization,
experiments under two scenarios: (A) generalization to novel objects, colors and shapes, and (B) generalization to only novel colors and shapes.
5.2.1 SCENARIO A: NOVEL OBJECTS, COLORS, AND SHAPES
Table D (plots in Figure 8) shows our results in experimental scenario A. where we train on 24/32 objects, 4/8 colors, and 5710 shapes (a total of 198 training tasks) and evaluate on the remaining 102 tasks. The overall success rates show much room for improvement on this challenging benchmark for a number of reasons. The robot must not only know how to pick-and-place the 8/32 objects it has never seen during training, but must also understand novel instructions that refer to these ob- jects by either their name, color, shape. Additionally, training multitask policy to perform well on hundreds of tasks remains an open question with current multitask robot learning algorithms- problem compounded in difficulty by the adversarial selection of distractor objects in environ- ment as mentioned earlier in Section 5.1
We lower-bound the performance of our task conditioning methods by first running one-hot con- ditioned policy, with the expectation that it performs worse than conditioning on language and/or demonstrations for the reasons mentioned in Section [ As bouer- we directly train
7

---

Published as a conference paper at ICLR 2023
Table 1: Evaluation on Novel Objects, Colors, and Shapes. (p) pretrained.
Domo Encodar
Languiee Fnoxler
Task Conditioning
Success Rote (%)
(loeer bound)
One bot Oracke (upper bound)
47.9 + 4.8
Language-oaly
13.7 = 1.9
CLI (p)
Demo-anly
Del.-TaCo (omrs)
15.8 +
DiSIBERT (p)
Langiage-only
10.4 =: 1.6
CNN
Decso-only
14.6 2.2
DisGIBERT (p)
De (ours)
19.9
CNN CNN
BC2, Jang et 201 l Demo-caly
8.8 + 2.D
DisiBERT (p)
MCIL (Lynch Sermatel ples Demo only Language only
9.4 + 1.8
Table 2: Evaluation on Novel Colors and Shapes. (p) pretrained.
Encoder
Languge Enoxder
Taxk Conditoning
Raie SD
Ose-hot (lower bound)
10.3 # 1.8
One-hot Oraele (upper bound)
50.9 4.9
DRUBERR (p)
Language only
15.8
CNN CNN
Demo-enly
17.0 3.7
DiSEBERT (p)
Del--TaCo (ours)
26.3 + 4.1
one-hot oracle on only the 102 evaluation tasks and evaluate on those same tasks. No other method in the table is trained on any evaluation tasks. (For consistency, the one-hot oracle is trained on the same total number of trajectories as the other methods.)
Next, we examine the performance of policies conditioned with only language, with only one demonstration, and with both (DeL-TaCo).. The language-only policies do not involve training fdemo. and only the language instruction embeddings are fed into the policy via FiLM during training and testing. The demo-only policies train fdemo as shown in Algorithm T but during training and testing, only the demonstration embedding Zdemo is passed into the policy via FiLM. DeL-TaCo (ours) conditions on both demonstration and language during training and testing as shown in Algorithms
| andd]
When using pretrained DistilBERT as Stang and a lightweight CNN for Sdemos DeC- achieves the highest performance, increasing the success rate of the second-bes conditioning method, demo- only, from 14.6% to 19.9%. Both methods using demonstration embeddings outperform the language-conditione
policy perhaps because a visual demonstration is important in conveying the nature of the chosen object and how the robot should manipulate it. Note that both the demo-only and DeL-TaCo policies train the femo CNN from scratch without any pretraining, so they must learn to ground object and container identifiers from training demonstrations alone.
We also compare to prior methods. BC-Z (Jang et al., 2021) performs worse than our approaches because its demo encoder is trained to direcily regress Zdemo to Zlang- hindering it from performing better than solely using Slang during testing. MCIL (Lynch & Sermanet, 2021). which trains sepa- rate encoders for each task embedding modality and averages the ımitation learning losses over the different encoders, also performs worse than DeL-TaCo because without any task encoder loss term, it is harder to learn well-shaped task embedding space, hurting generalization performance.
Finally, to evaluate the effect of pretraining, we use pretrained CLIP (Radford et al. [02TT as the task encoder (with its language transformer as siang and vision transformer Jdemo) and freeze during training. The language-only policy performs significantly better than the video-only pol- icy most likely because CLIP's visual transformer was trained mostly on real-world images and without further finctuning, does not know how to sufficiently differentiate between the simulation demonstrations of different tasks in our problem setting. Despite this, DeL--TaCo modestly outper- forms conditioning on language-only or demo-only, demonstrating the value of our method even with frozen pretrained models.
5.2.2 SCENARIO B: NOVEL COLORS AND SHAPES
In Table 2 (plots in Figure D. we train on 32/32 objects, 4/8 colors, and 5/10 shapes, and evaluate on the rest-an easier setting as all objects were seen during training. Since evaluation tasks in this scenario only refer to objects by their color or shape, we up-sample the color and shape training tasks to be 50% of each training batch (such up-sampling was not done in scenario A).
We take the on language, demonstrations, and both. All methods perform better in scenario B than A. The novel color and shape task demonstrations contain more ambiguity than the novel object demonstrations because the task with language instruction "put the blue object in the left bin" might have a demon-
fdemo and flang models in Table and again compare conditioning
8

---

Published as a conference paper at ICLR 2023
Table 3: Value of Language. Evaluation on Novel Objects, Colors, and Shapes.
Task Conditionng
Demo-coly
Del-ToCa (ours)
por
50
100
Suceoss Rate (8)
14.6
14.9
17.4
20.0
24.2
19.9
+ SD
#2.2
11.
22.7
22.4
+2.5
+1.8
stration where the robot manipulates the blue cup, but the test-time environment might contain blue table instead. This added ambiguity likely explains the increased importance of language and the wider 9.7% performance gap between Del.-TaCo and demo-only task conditioning.
Analysis. Overall, we see that on this wide range of tasks, language and demonstrations together do help disambiguate each other during task specification-answe better generalization performance on novel tasks- aasssrnngniing our second question.
our first question; this leads to
5.3
How MANY DEMONSTRATIONS IS LANGUAGE WORTH?
To answer our third question, we re-examine experimental scenario A (testing on novel objects, col- ors, and shapes) but now further finetune the demo-only policy expert demonstrations. Results are shown Table 3 (plots in Figure 9 The demo-only policy only starts to match and surpass DeL-TaCo (underlined) when it is finetuned on 50 demonstrations (underlined) per evaluation task (a total of around 5,000 demonstrations for all test tasks combined). This suggests that surprisingly, specifying a new task to Del.-TaCo with single demonstration and language instruction performs as well as specifying new task to demo-only policy with a single demonstration and finetuning
variable number of test-task
50 additional demonstrations of that task. This showcases the im- mense value of language in supplementing demonstrations for novel task specification, significantly reducing the effort involved in teaching robots novel tasks over demonstration -only methods.
5.4 ABLATIONS AND MULTIMODALITY FOR AMBIGUITY RESOLUTION
We provide extensive ablations of our model and algorithm in Appendix F and further explore the utility of using multi-modal task specification to resolve ambiguity Appendix
CONCLUSION
When specifying tasks through language or demonstrations, ambiguities can arise that hinder robot learning, especially when the demonstrations instructions were provided in an environment that does not perfectly align with the environment the robot is in. In this paper, we showed a problem setting of learning 300 highly diverse pick and place tasks and propose simple framework, DeL- TaCo, to resolve ambiguity during task specification by using both language and demonstrations during both training and testing. Two main obstacles deploying houschold robotic systems are the inability to generalize to new environments and tasks, and prohibitively high end-user effort needed to teach robots these new tasks. Our results show progress on both fronts: over previous lask-conditioning methods, DeL-TTaCo improves generalization performance to new tasks by 5 5-99 9% and reduces human effort on our set of tasks by roughly 50 expert demonstrations per task.
Limitations and Future Work. Our work leaves a number of areas for improvement. First, we experiment only with pick-and-place tasks. Future work may need more interpretable modular en- coders to handle a wider diversity of manipulation skills and temporally-extendel tasks. Second, we used rigid set of template-be based language instructions for each task, but our framework would likely benefit from more diverse instruction set of human paraphrases for each task. Third, did not find pretrained lisiaanlanguage models, such as CLIP, to increase performance in our simulation- based environment, most likely because of the domain mismatch between our simulation objects and the more real-worid-centric images CLIP was trained on. Investigating ways to better leverage pre- trained vision-language models for multimodal task specification, tandem with real-world robotic tasks and real-world human demonstrations, would be promising line of future research.
9

---

Published as a conference paper at ICLR 2023
7 REPRODUCIBILITY STATEMENT
Please see our appendix for details needed to replicate our results. In particular, Appendix A pro- vides the full list of our 300 tasks, instructions, and objects, as well train and test task splits. Appendix B contains architectural hyperparameters and details for network layers, initialization, and optimizer settings. The remaining appendices detail aspects of our training/evaluation processes and provide additional ablations that were not fully described in the main text of our paper.
We link our open-sourced codebase on our project website, github. ic
ETHICS STATEMENT
This work leverages language model embeddings for task conditioning, which leaves approach vulnerable to distributional biases of the language models. Future work should explore adding ad- ditional safeguards to DeL- reason about the safety of language instructions before utilizing these embeddings for execution by the robot policy.
ACKNOWLEDGMENTS
We would like to thank Prasoon Goyal and Vanya Cohen for their numerous suggestions and discus- sions during the course of this work. This research supported by NSF NRI Grant 192-1925082.
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do I can and not as I say: Grounding language in robotic affordances. In arXiv preprint arXiv:2204.01691, 2022.
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In International Conference on Machine Learning (ICML), 2017.
Valts Blukis, Dipendra Misra, Ross A. Knepper, and Yoav Artzi. Mapping navigation instructions to continuous control actions with position- position-visitation prediction. In Conference on Robot Learning (CoRL., 2018
Valts Blukis, Yannick Terme, Eyvind Niklasson, Ross A. Knepper, and Yoav Artzi. Learning to map natural language instructions to physical quadcopter control using simulated flight. In Conference on Robot Learning (CoRL), 2019.
Boyuan Chen, Fei Xia, Brian Ichter, Kanishka Rao, Keerthana Gopalakrishnan, Michael S. Ryoo, Austin Stone, and Daniel Kappler. Open-vocabulary queryable scene representations for real world planning. In arXiv preprint arXiv:2209.09874 2022.
Erwin Coumans and Yunfei Bai. Bullet and pybullet, physics simulation for games, visual effects, robotics and reinforcement learning. htttps://yybul wet. .rgg May 2007-2022.
Yuchen Cui, Scott Niekum, Abhinav Gupta, Vikash Kumar, and Aravind Rajeswaran. Can founda- tion models perform zero task specification for robot manipulation? Learning for Dynamics and Control Conference, 2022.
Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In International Conference on Robotics and Automation (ICRA), 2017.
10

---

Published as a conference paper at ICLR 2023
Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Danilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109.13396 2021.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with im- portance weighted actor-learner architectures. In International Conference on Machine Learning, pp. 1407-1416. PMLR, 2018.
Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. In International Conference on Robotics and Automation (ICRA), 2016.
Divyansh Garg, Skanda Vaidyanath, Kuno Kim, Jiaming Song, and Stefano Ermon. Lisa: Learn ing interpretable skill abstractions from language, 2022. URL 2203. 00054
Prasoon Goyal, Scott Niekum, and Raymond Mooney. Using natural language for reward shaping in reinforcement learning. 2019. URL WRRLhhtts:///a/xiii
abaaabss1990/ 1903. 1900/000000000
Prasoon Goyal, Scott Nickum, and Raymond Mooney. Pix12r: Guiding reinforcement learning using natural language by mapping pixels to rewards. 2020. URL [tll:ttt/://///
22//2000..1544
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, et al. Ego4d: Around the world in 3,000 hours of egocentric video. arXiv preprint crXiv:2110.0005. 2021.
Abhishek Gupta, Corey Lynch, Brandon Kinman, Garrett Peake, Sergey Levine, and Karol Haus- man. ainoonstration-boottrrapppe autonomous practicing via multi-task reinforcement learning. arXiv preprint arXiv:2203. /5755, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. arXiv preprint 112.::15 2015.
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojeiech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Confer- on Artificial Intelligence, volume 33, pp. 3796-3803, 2019.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models. In arXiv preprint arXiv:2207.05608 2022.
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. In ACM Computing Surveys, 2017.
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. BC-z: Zero-shot task generalization with robotic imitation learning. In 5th
Annual Conference on Robot Learning, 2021. URL http:httt//// pttttp:://openreviee www.:eee/oorroo id=8kbp23tsGYv
Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski,
Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic inforcement learning at scale. arXiv preprint 10Xiv:2104.08211 2021.
Siddharth Karamcheti, Megha Srivastava, Percy Liang, and Dorsa Sadigh. Lila: Language-informed latent actions. In 5th Annual Conference on Robot Learning, 2021. http:::tt/////xii.
org/pdf/ / 03205
Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. Robotics: Science and Systems, 2021. htll.htts:://ee http:///ra....ooraa////000000 2000 07648

---

Published as a conference paper at ICLR 2023
Oier Mees, Lukas Hermann, Erick Rosete-Bestt Beas, and Wolfram Burgard. Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks, 2021. URL https://aaxxy: org/ abs 2112.1112100 03227
Oier Mees, Lukas Hermann, and Wolfram Burgard. What matters language conditioned imitation learning. arXiv preprini arXiv:2204.062252 2022.
Ashvin Nair, Dian Chen, Pulkit Agrawal, Phillip Isola, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Combining supe--see learning and imitation for vision-based rope manipulation. In International Conference on Roboties and Automation (ICRA). 2017. ltttp://aaxxxx
org/ ht5//bb//1033 02018
Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. In Advances in Neural Information Processing Sys- fems (NeurIPS), 2018. URL
7807 04742
Suraj Nair, Eric Mitchell, Kevin Chen, Brian Ichter, Silvio Savarese, and Chelsea Finn. Learning Iinguage-conittion robot behavior from offline data and couredsoureed annotation. In 5th An- mual Conference on Robot Learning, 2021. /
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A univer- sal visual representation for robot manipulation. arXiv preprint arXiv:2203. /2601, 2022.
Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goal-conditioned policies. 2019.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. arXiv preprint
2015.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with general conditioning layer. In AAAI, 2018.
Dean Pomerleau. Alvinn: An autonomous land vehicle in neural network. In Conference on Neural Information Processing Systems (NeurIPS), 1988.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv: 203.:2003300020, 00020, 2021.
Andrei Rusu, Sergio Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Krikpatrick, Raz- van Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. arXiv preprint arXiv:1511.062955 2015.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. In Conference on Neural Information Processing Systems (NeurIPS). 2019.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilroberta, 2020. URL
-ase
Lin Shao, Toki Migimatsu, Qiang Zhang, Karen Yang, and Jeannette Bohg. Concept2robot: Learn- ing manipulation concepts from instructions and human demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2020.
Mohit Shridhar, Lueas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Proceedings of the 5th Conference on Robot Learning (CoRI.), 2021.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for
robotic manipulation. Conference on Robot Learning, 2022.
Andrew Silva, Nina Moorman, William Silva, Zulfiqar Zaidi, Nakul Gopalan, and Matthew Gom- bolay. lear:: Learning with language to enable generalization in multi-tase manipulation. In IEEE Robotics and Automation Letters, 2021.
12

---

Published as a conference paper at ICLR 2023
Laura Smith, Nikita Dhawan, Marvin Zhang, Pieter Abbeel, and Sergey Levine. Avid: Learning multi-stage tasks via pixel-level translation of human videos. In Robotics Science and Systems (RSS), 2020.
Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context- based representations. arXiv preprint arXiv:2102.06117 2021.
Yee Whye Teh, Victor Bapst, Wojciech Marian Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. arXiv preprint arXiv: 10X7::1770.04755 2017.
Homer Walke, Jonathan Yang, Albert Yu, Aviral Kumar, Jedrzej Orbik, Avi Singh, and Sergey Levine. Don't start from scratch: Leveraging prior data to automate robotic reinforcement learn- ing. In Proceedings of the 6th Conference on Robot Learning (CORL), 2022.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers, 2020.
Aaron Wilson, Alan Femn Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: hierarchical bayesian approach. In Proceedings of the 24th international conference on Machine learning, pp. 1015-1022, 2007.
Zhiyuan Xu, Kun Wu, Zhengping Che, Jian Tang, and Jieping Ye. Knowledge transfer in multi-task deep reinforcement learning for continuous control. arXiv preprint arXiv:20/0.07494, 2020.
Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. arXiv preprint arXiv:2003.13661 2020.
Sarah Young, Dhiraj Gandhi, Shubham Tulsiani, Abhinav Gupta, Pieter Abbeel, and Lerrel Pinto. Visual imitation made easy, 2020.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinorrrreeeeeeeeeeeeeaane learning. In Conference Robot Learning (CoRL), 2019. URL http:/aaa///aall.oooo///////////
10897
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. arXiv preprint arXiv:200J.0677 2020.
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, and Chelsea Finn. Conservative data sharing for multi-task offline reinforcement learning. Advances in Neural In- formation Processing Systems, 34, 2021.
Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, et al. Transporter networks: Rear- ranging the visual world for robotic manipulation. In Proceedings of the 4th Conference on Robot Learning (CoRL), 2020.
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Van- houcke, and Pete Florence. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv, 2022.
13

---

Published as a conference paper at ICLR 2023
Appendices
A ALL TASKS, INSTRUCTIONS, AND TRAIN-TEST SPLIT
A.1 LIST OF TASKS
All 300 tasks are shown below, by object identifier (rows) and container identifier (columns). The colors denote groups of tasks which guide train and test task splits, and the cell numbers denote the task indices.
Scenario A (novel objects, colors, and shapes) trains on all gray tasks and tests on yellow blue and green tasks.
Scenario (novel colors and shapes) trains on all gray and yellow tasks and tests on blue and green tasks.
Consainer Idenitiee
Object Mentifier Type
Object Mdenifieer
frant
left
conic cup fountain vase circular table
51
101
151
201
251
52
102
152
202
252
deep smushed dumbbell square prism bin
104
204
25d
55
105
155
205
255
56
106
156
206
255
colunnade top
stalageite chunk
108
208
258
bongo drum bowl
59
109
159
209
259
pacilier
60
110
210
260
bechive
crocked lid trub can
62
112
162
212
262
wollet bus!
13
63
113
163
213
263
popsi buttle
114
164
214
264
tongue
modern canoe
66
116
216
266
pear naged vase
67
E17
167
217
267
short handle
118
218
268
glass half galion
70
120
170
220
270
flat bottem sack vase
21
71
121
171
221
271
trapeconial bin
72
122
172
222
272
vintage
123
225
bubbub
74
124
174
226
274
Dowery half donut
25
75
125
175
225
275
box sofa
28
78
128
178
228
278
two lay layered lampshade
29
79
129
179
229
279
conic bin
80
130
180
230
280
281
black and white
82
132
182
232
782
brown
83
133
183
233
283
gay
235
color
while
36
85
136
188
236
266
87
137
187
237
287
orange
yellow
R9
1.39
189
2.39
280
vase
SU
1400
190
2.00
250
chalice
291
292
bottle
93
143
253
293
shape
cance
44
94
144
194
246
294
trapezoidal priam
97
147
197
2477
207
cylinder
48
98
148
198
248
298
hole
A.2 TASK INSTRUCTION FORMAT
the following template as the language instruction for each task: "Put [target object identifier string] [target container identifier string]." For each object identifier, we build string referring to the target obj in specific format shown in Table
As mentioned in Section 5.1
14

---

Published as a conference paper at ICLR 2023
Table 4: Object Identifier String Format for each Object Identifier Type.
Object Identifier Type
Target Object Identifier String
Name
"[object color] colored, [object shape] shaped [object name]"
Color
"[object color] colored object"
Shape
"[object shape] shaped object"
Example task instructions (with target object identifier and target container strings underlined):
Task 4: "Put black and white colored, chalice shaped smushed dumbbell in green bin."
Task 292: "Put cup shaped object in right bin."
A.3 TRAIN AND TEST SPLIT VISUALIZATIONS
We visually show our train-test splits on objects (Figure colors (Figure [}] and shapes (Figure
Objects
Test Train Objects
Figure 4: Train-Test Object Split. Objects are shown in raster-scar sean task-index order, so the object
in the second row from top, second column from left, is the "bongo drum bowl", which is associated with task index 9.
15

---

Published as a conference paper at ICLR 2023
Train Colors
Test Colors
Black and White
White
Blue
Red
Brown
Orange
Yellow
Gray
Figure 5: Train-Test Color Split.
Train Shapes
Test Shapes
Vase
Bowl
Canoe
Cup
Freeform
Bottle
Trapezoidal Prism
Chalice
Cylinder
Round Hole
Figure 6: Train-Test Shape Split.
B DETAILED ARCHITECTURE AND HYPERPARAMETERS
B. POLICY т. AND DEMONSTRATION ENCODER ftemo ARCHITECTURE
See Figure 7 for a detailed diagram of the policy and demonstration encoder (for a higher-level overview, see Figure 2). For the policy backbone, we use a ResNet- 18 architecture but made changes to the strides and number channels to adapt the network to our small image size. Hyperparameters shown in Tables and 6
B.2 TRAINING HYPERPARA COPPERPARMMTEERE
Table 7 shows II. training hyperparameters.
16

---

Published as a conference paper at ICLR 2023
(esNetBlockkCin Cour)
(Ga.
Zoeno
Imuge als: (3A4,48)
Demo: (3,49,96
conv3x3 stride
conv3x3 fetride MaxPoel2k22
Goucoh
(1622448)
BatchNorm, ReLU
consax3 fstride 1: MaxPael de 2), ReLJ
conv7x7 (stride 1)
conv3x3 (stride 1)
(16,12,24)
BatchNorm, ReLU
NorchNorm
comviká sstride 1; Rel Flelher
(16,4848)
FiLM (Ean)
MaxPool3x3 (stride 21
(4608,) )
(1624,24)
Layers 024, 512 256]
ResNetBlockk16.16
Xdrmol (768.)
ReLU
(19Net)llock(6616)
Demo Encoder f
demo
out: (Coat A,
Policy π
(65NetBlock(16322)
ResNethllock(32.32
ResNetBlocck32264)
FiLMBlock(c)
(esNetbblock(64464)
x: (c.h,w)
ResNetBlockk64.1229
Y
learned projection
ResNetRlock
(128,24,24)
Admmo
Soatial Softmax
learned projection
Pobtu (256.)
Zuang (768,)
robor state: (10.)
out: (c.ww)
Concat
FC Layers 0024, 512 255
action: [x, gripper)
Figure 7: Detailed Architecture of the Policy and Demonstration Encoder.
Table 7: Imitation learning hyperparameters. In each training iteration, we sample 16 random tasks from training buffer and get 64 samples for each task, for total batch size of 1024.
Attribute
Value
Number of Tasks per Batch
16
Batch Size per Task
64
Learning Rate
x 10-4
Task Encoder weight (a in L)
10.0
Contrastive Learning Temperature (3 in Ldemo)
0.1
17

---

Published as a conference paper at ICLR 2023
Table 5: Policy x hyperparameters.
Attribute
Value
Input Height
48
Input Width
48
Input Channels
3
Number of Kernels
[16, 32, 64,
128]
Kernel Sizes
[7. 3,
Conv Strides
1,11,
Maxpool Stride
2
Fully Connected Layers
[1024, 512, 256]
Hidden Activations
ReLU
FiLM imput size
768
FiLM hidden layers
0
Spatial Softmax Temperature
1.0
Learning Rate
x 10
Policy Action Distritution
Multivariate Isotropic Gaussian (a M σ)
Policy Outputs
(14, a)
Image Augmentation
Random Crops
Image Augmentation Padding
Table fdemo CNN hyperparameters.
Attribute
Value
Demonstration frames
First and last timesteps
Demonstration image array size (m,n) Input Height (m- Image height)
(1,2)
48
Input Width (n- Image width)
96
Input Channels
Output Size
768
Kernel Sizes
[3, 3, 3]
Number of Kernels
[16, 16, 16]
Strides
[1, 1,
Fully Connected Layers
[1024, 512, 256]
Hidden Activations
ReLU
Paddings
[1, 1, 1]
Pool Type
Max 2D
Pool Sizes
[2, 1]
Pool Strides
[2, 2, 1]
Pool Paddings
[0, 0, 0]
Image Augmentation
Random Crops
Image Augmentation Padding
C SCRIPTED POLICY DETAILS
We collect 26, 000 - 31, 000 training demonstrations using a scripted policy with Gaussian noise added the action of each timestep. Details shown in Algorithm 3 "eePos" stands for end- effector position. All variables ending in "Pos" are xyz positions. Our training buffer contains only successful demonstrations.
18

---

Published as a conference paper at ICLR 2023
Algorithm 3 Scripted Pick and Place
I: pickPos target object position
dropPos + target container position
3: distThresh -- 0.02
4: numTimesteps t- 30
5: placeAttempiexd < False
6: for in [0, mummmmmmmteepep
do
cePos + end effector position
8: dropPosDist +- |JeePos dropPos|p |
9: pickPosDist <- |JeePss pickPos|22
10: if placeAttempled then
action +
12:
else if object not grasped AND pickPosDist > distThresh then
13: // Move toward target object
14:
action pickPos cePos
15:
else if object not grasped then
gripper is very close object
17:
action +- PickkPss cePos
18:
close gripper // Object is in gripper
19:
else if object not lifted then
20:
Move gripper upward to avoid hitting other objects/containers
21:
action + [0, 0, 1]
22:
else if dropPosDist >> distThresh then
23:
// Move toward target container
24:
action dropPos eePos
25:
else
26:
open gripper // Object falls into container placeAttempted + True
27:
28: end if
29: noise (0, 0.1)
30: action +- action + noise
31: 8' +- inw.step(action)
32:
end for
D SUCCESS RATE CALCULATION DETAILS
To avoid reporting cherry-picked results, we detail our success rate calculation methodology here.
We each setting with three random seeds for 800k-900k training steps. An evaluation set, which we define as rolling out the policy for 2 trials per task for all of the test tasks, total of 80-90 evaluation sets that occur
run every 10k training steps. Thus, there throughout training. Let seed attain the success rate r(i,j) on evaluation set j. Let J top 10 evaluation set indices for the quantity mi.n,r(i.j). Our reported success rate and standard deviation in the tables are calculated as the following equations:
Reported Success Rate = mean (mman, Reported Standard Deviation meanjeJ (stddev=r(i.j))
Scenario A: Since there are 102 test tasks, each success rate Tables Tand is computed from:
2 trials 102 test tasks 10 evaluation sets
3 seeds 6120 evaluation trials
test task evaluation
seed
Scenario B: Applying the same calculation for the 54 tasks in Scenario B, each success rate in Table is computed from 3240 evaluation trials.
For Table the best final checkpoint of the three demo-only policy seeds from Table D was taken for finetuning.
19

---

Published as a conference paper at ICLR 2023
E LEARNING CURVES FOR EXPERIMENTS
Objects, Colors, Shapes
Objects, Shapes
Success Rate
1100
29k
C0oc Timesteps
19Bc
WBC
Timesteps
Iigaageong
PHLL Junch.a.aaa wean
CLT baggagaan
Figure 8: Table T learning curves, where methods are evaluated on novel objects, colors, and shapes. Left: Idemo is fdemo and Faang are from pretrained CLIP. The same upper and lower one-hot bounds (dotted) are shown in both the left and right plots.
memeed-foo-sccatee CNN and Stang is pretrained DistiBBERT. Right:
Novel Colors and Shapes
Demo-Only Palicy Finetuning (by demes per test tack)
3.5
5.6-
24-
Rate
Rete
SuCCIrS
Succes
6.38
6.0
636
08
1006
2004
100g
4bx Timestops
700c
190
53K
200c
250x
30%
Timesteps
100
Figure 9: Table 2 and learning curves. Left: Evaluation only on novel colors and shapes. Right: Evaluation novel objects, colors, and shapes, using trained-from-scratee CNN fdemo and pretrained DistiBBERT as flang. The performance of the demo-only policy and DeL-TaCo pol- icy from Table (also depicted in the left plot of Figure 8 shown lower and upper dotted lines, respectively. The solid lines indicate performance during 300k finetuning steps when given x demonstrations per test-task, where indicated in the legend.
20

---

Published as a conference paper at ICLR 2023
Table 8: Ablations, Evaluation on Novel Objects, Colors and Shapes. (p)
pretrained.
Demo Encoder
Language Encoder
Tank Encuder Loss
Tank Conditoning Arch.
Task Condituming
Success Rate I (D)
Best non-orack resllt from Tañee
DistiBERT (p)
FILM (demo), Concal (lang)
(ouas)
19.9.1
Langunge Encdder Ablatoons
DESiBEKT (iinetueed
FILM
12.6
1..61.0 5.0
CNN
DaniBEKT (Rnctuned)
Contrastive
FIMM (deaso), Concus (lang)
Del--TaCo (ours)
12.1
DiviRRoBERRa (p)
FiLM
Langiage-only
12.4 1
CNN
Do
Contrastive
FILM (demo), Concal (lang)
Del/ToCoo (our))
16.6
1.5
minLM (p)
Language-only
13.7
CNN
MiniLM (p)
Contrastive
FILM (demo), Concal (lang)
DeL (ouns)
20.9 +: 2.3
(PI
CLP MLP bead
Contrastive
FILM (lmn)) Cuncaii (demo)
Del-T5000
15,8.120
Demo Eneeeer AMlrrios
R3M (pr+ MLP head
DEt (p)
Contrastive
FiLM Commo,, Concal (lang)
DeL DeL-TaCo (ours)
13.2 1.8
Tisk Encoder Loss Ablitios
CNN
DistiBLXI (p)
Cosine distance
FiLM (demo), Concal (lang)
DeL TaCo (ouns)
12.2 2.5
CNN
DistilEERT (p)
None
FiLM (demo), Concal (ling)
Del-TaaO (ours)
12.5 1: 2.6
Task Canditioning Architecture Ablations
CNN
DistiBERT (p)
Contrastive
Concal (demo + lang
DoL-TaCo
10.6 1..61.00 4.2
CNN
DistiBBRRT (p)
Contrastive
FiLM (demo + hing
Del-TaCo (ours)
17.3 1 2.7
F ABLATION ANALYSIS
In Table g we compare the performance of different language encoders, demonstration encoders, task encoder loss types, and task conditioning architectures in Experimental Scenario A to our best model from Table (top row).
Language Encoder Ablations. Finetuning the language model did not improve performance, most likely because the model becomes significantly harder to train jointly with a relatively deep language encoder. Perhaps training the policy and finetuning the language model in separate stages may yield better results. We also compare with other language models such as oostil (Sanh et al, 2020) and miniLM (Wang al. 2020). These were chosen because they have (1) a relatively small number of parameters for computational efficiency, and (2) were shown to achieve high performance on language-condittoned robotic policies when compared other common language models (Mees) et al. 2022). When using CLIP as the demo and language encoder, we did not notice much of performance difference from adding a finetuneable MLP head on top of the frozen CLIP language encoder.
Demonstration Encoder Ablations. Using pretrained demonstration encoders, such R3M, did not perform better than training a small CNN from scratch-a similar takeaway from our experi- ments with pretrained CLIP.
Loss Ablations. Our approach outperforms using a BC---styyle cosine-distance loss term
(attemoZetema Alang) = Zetemo Stang (where || Stangll = 1). Our contrastive task encoder loss term is crucial; without it (setting a in Eqn. performance drops substantially.
Task Conditioning Architecture Ablations. Concatenating 2demo and Siang before feeding into FiLM layers causes a slight drop in performance compared to our architecture in Figure and sim- ply concatenating 7demo and Stang to the image observation embeddings without FiLM dramatically decreases performance.
G ADDITIONAL AMBIGUITY EXPERIMENTS
In Tables T and 2 we examined the effects of conditioning the multitask policy on demonstration that is ambiguous about the target container, plus an unambiguous language instruction that clearly specifies both the target object and container for each hick-and-place task. We label this as ambiguity scheme (i).
To further examine the utility of multi-modal task conditioning, we experimented with a different ambiguity scheme (ii) in which the language instruction is ambiguous about the target object but unambiguous about the target container, and the demonstration is unambiguous about the target object but ambiguous about the target container. This scheme allows to test how well two task- conditioning modalities complement each other if each modality unambiguously conveys only a single aspect of the task.
21

---

Published as a conference paper at ICLR 2023
Table 9: Ambiguity Experiments
Ambiguty Schemne
Demo Ambiguny
Language Ambiguit
Task Condiooing
Success Kate 1 SD
None
Langiage-oely
10.4
1.6
(Frm Table
Demo--nly
container
None
Del--TiCo (ours)
19.9 1.8
object
Language only
15.8 4 2.3
container
Demo-oni
22.7 2.2.4
object
(ours)
To introduce ambiguity the target objects, in scheme (ii), place two identical objects the scene in different parts of the tray, plus third visually distinct distractor object. Exactly one of these identical objects the target object. The language instruction is ambiguous because it does not convey the positional identifier that describes which side of the tray the target object is at. However, the demonstration unambiguously conveys the target object by showing the robot picking up the target object from the correct part of the tray.
The results are shown in Table 9 By displaying two identical objects we do in scheme (ii,, visually reveal that the target object is one of the two identical objects (and not the third, distinct object on the scene), causing scheme (ii) to have higher overall success rates than scheme (i). We see that the gap between Del-TTCCo and the demo-only policy slightly decreases from more than 5% in scheme (i) to 4% in scheme (ii) because the language instruction unambiguously specifies the task in scheme (i), complementing the demonstration which is ambiguous both the target object and container, whereas in scheme (ii), both the language instruction and demonstration are ambiguous in aspect (either in specifying the container or the object), narrowing the performance gap between DeL-TaCo and the demo-only policies.
H DEMONSTRATION FORMATTING FOR fdmoo
We represent each demonstration as an m X n image array consisting of observations from the first timestep, the last timestep, and ranged in raster-scan order. For our CL.IP and R3M experiments, we use (m, n) (2, 2) because CLIP and R3M perform center square crop on each input image, so we made the demonstration ray square. When using our wrained-from-scrale efficiency. This sufficed for our tasks because pick-and-place was not a particularly long horizon task, so including more frames did not improve performance.
other randomly selected timesteps from the trajectory,
fdemo. we used (m, n) (1,2) for computational
22


Reviews:
[{"review_id": "uxh5b0YZHI", "reviewer": "Reviewer_p2aG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper develops an approach, Joint Demo-Language Task Conditioning (DeL-TaCo), for behavior cloning in a multi-task setting where the task is specified by both language and a demonstration. The approach includes a contrastive loss to guide the learning of the language and demonstration embeddings, which are then fed into a policy head. Results are conducted in simulation and show that virtual robot manipulation tasks can be learned by DeL-TaCo and that this approach generalizes better to novel tasks.", "summary_of_the_review": "This paper provides positive results for conditioning on language and demonstrations in multi-task learning for zero-shot generalization. However, the paper would be improved by more baselines and a more compelling case that the results are novel and significant.", "strength_and_weaknesses": "Strengths:\n\nThe paper clearly describes a supervised learning approach to multi-task learning with tasks conditioned on both a natural language instruction and a demonstration of the task. \n\nThe figures are relatively informative.\n\nPseudocode is provided.\n\nThe paper reports positive, empirical results on a relatively large dataset of multiple tasks.\n\nTable 3 shows how much data is required to enable a demo-only policy to match the performance of the proposed method, DeL-TaCo, showing the proposed method is better up until reaching ~50 demonstrations for the dataset for the novel task.\n\nA limitations and future work section is included.\n\nWeaknesses:\n\nThis approach requires both language and instructions to function. As such, one would want to show superior performance to methods that require only one. For example, recent work by Wen et al. (2022) required only a single demonstration on a novel task:\n\nWen, B., Lian, W., Bekris, K. and Schaal, S., 2022. You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration. arXiv preprint arXiv:2201.12716.\n\nOn the language side, there are quite a few recent papers show strong performance conditioning on only language:\n\nSodhani, S., Zhang, A. and Pineau, J., 2021, July. Multi-task reinforcement learning with context-based representations. In International Conference on Machine Learning (pp. 9767-9779). PMLR.\n\nSilva, A., Moorman, N., Silva, W., Zaidi, Z., Gopalan, N. and Gombolay, M., 2021. LanCon-Learn: Learning With Language to Enable Generalization in Multi-Task Manipulation. IEEE Robotics and Automation Letters, 7(2), pp.1635-1642.\n\nNair, S., Mitchell, E., Chen, K., Savarese, S. and Finn, C., 2022, January. Learning language-conditioned robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning (pp. 1303-1315). PMLR.\n\nShridhar, M., Manuelli, L. and Fox, D., 2022, January. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning (pp. 894-906). PMLR.\n\nThis reviewer notes that the authors discussed CLIPort and the Nair et al. (2022) paper; however, these works were not benchmarked against. The paper does not present a clear argument against benchmarking against these approaches (other than the implicit argument that they only condition on language, which is actually an advantageous quality).\n\nThis paper is a relatively straightforward design of a neural network-based architecture where embeddings are learned for language instructions and demos (which are forced to be the same through a contrastive loss) and then fed into a policy head for supervised learning. It is unclear whether new theories or insights are gleaned (other than adding more input data can enhance performance by 5-9%).\n\nMore insight into why DistilBert was used could be helpful. What would other language embeddings look like?\n\nAn ablation study showing the efficacy of the contrastive loss and other components would be helpful.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cLrWqjecaL", "reviewer": "Reviewer_Ch3U", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper argues that solely using language or video demonstrations to learn robotic manipulation tasks is inadequate as it creates ambiguities in learning. To alleviate this confusion, the authors propose to train a multi-task policy that condition on *both* the video demonstrations as well as the language.\n\nThey work in the paradigm of imitation learning, wherein they train a single policy for all the robotic pick-and-place tasks.", "summary_of_the_review": "Even though the paper's contribution in methodology is significant enough for the community, due to the lack of extensive experiments and proper baselines as mentioned in the Weaknesses section, I would vote 5 : *marginally below the acceptance threshold*. However, my final decision is subject to change based on the authors' rebuttal and comments of other reviewers.", "strength_and_weaknesses": "**Strengths**\n\n1. The paper is mostly well-written, and the methodology section is easy to understand in one go.\n\n2. Section 5.3 which compared how much was a language worth was a good discussion -- which drives the point that having language as input is indeed beneficial (as corroborated by several other works in the field)!\n\n\n**Weaknesses + Questions:**\n\n**Claims**\n1. The authors claim the following in Sec 2.2\n\n> \"We argue that conditioning the policy on both a demonstration and language not only ameliorates the ambiguity issues with language-only and demonstration-only specifications but is much easier and more cost-effective for the end-user to provide.\"\n\nI'm not sure what the authors mean by \"more cost-effective for the end-user to provide\" -- I agree that providing numerous demonstrations (at least 50 according to the experiments) has a high cost on the user, but having only language would have a lower cost (as done in [1])! \n\n\n**Methodology**\n\n2. The rationale behind the Task Conditioning Architecture: How was the architecture selected? Was it based on previous works of [2], and [3]? \n\n    A simple variant (baseline) would be to concatenate the $z_{demo}$ and $z_{lang}$ and use that as input to the policy. Was this baseline tried? If so, what were the observations?\n\n**Experiments**\n\n3. Success Rate: I am not sure why this complicated way of computing the Success rate was adopted. Specifically, I'd like the authors to address the following questions:\n\n   a) Why are the rollouts *during* the training being considered? What is the problem with computing the success rate after training (800-900k steps)?\n\n   b) How does the current definition of success rate avoid *cherry-picking* when compared to (a)?\n\n4. Training + Test Sets (a Scenario C): In all the experiments, the three objects in the scene are *visually distinct*. Having the same objects with a) the same colors, as well as b) different colors in the test set would be a crucial experiment to test the generalization of the method. I would like to see the results of such an experiment. Since it is only testing, I believe it will not take a lot of time.\n\n5. Baseline Comparison: I would expect a comparison with [1] to be present in the paper. I understand that the code for [1] is unfortunately not open-sourced (which I don't hold against you, and blame the authors of [1]), but the method in [1] can be adapted to your codebase wherein you have a shared latent space -- which is essentially the crux of [1].\n\n\n-------\n**References**\n\n[1] Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. In RSS 2021.\n\n[2] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. BC-z: Zero-shot task generalization with robotic imitation learning. In CoRL 2021.\n\n[3] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, 2018.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VEpsTaG-0o9", "reviewer": "Reviewer_4Ugt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Interesting paper, it uses language and demonstration conditioning to improve learning in an behaviour cloning set up for robot manipulation. The paper proposes:\n1. a new architecture for training an eval conditioned on language and demonstrations together\n2. some level of generalization with this", "summary_of_the_review": "The paper proposes a method to condition on language and demo to solve robotic behaviour cloning. This is a decent set up in small-scale robot learning set ups. The idea is novel and simple. The claims of the paper are reasonably well supported within the data distribution used. ", "strength_and_weaknesses": "Strengths:\n- pretraining demonstration and language using contrastive loss is an interesting idea\n- it's intuitive that conditioning on both improves performance, is a simple and elegant idea\n- generalization results look decent\n- the conclusion that language is worth 50 demos is useful, but this breaks down once you've 1000s of tasks\n\nWeaknesses:\n- tested on only sim, where data is very clean\n- concerned that conditioning on one demo would make a policy rely on the spurious correlations of that demo : specific action situations, background, etc. \n- given several demos of the same task, how would you sample the one to condition on? how do you scale this for large datasets\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9AC6eLN2HN", "reviewer": "Reviewer_5Uyd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes Joint Demo-Language Task Conditioning, DeL-TaCo, as a new way of training and specifying goals for goal-conditioned task-learning agents. DeL-TaCo learns a demonstration encoder and uses a frozen, pre-trained language encoder to embed task demonstrations and language-specifications for simulated robots to perform pick-and-place tasks with various objects/colors/shapes. DeL-TaCo is compared to language-only and demonstration-only ablations, showing that both demonstrations and language are important to achieve the highest success rates, and two different encoders are compared, a pre-trained CLIP vs. a pre-trained DistilBERT + trained demonstration encoder.", "summary_of_the_review": "The proposed method, DeL-TaCo, is explained clearly, is intuitive and straightforward, and appears to work well. The paper suffers from incomplete reporting in the results, a relative lack of novelty, and unanswered questions about training details or design decisions. With simple additional experiments around fine-tuned language encoders, clearer results for demonstration-vs-language goal-specification and performance upper-bounds, and position to recent related works, I would advocate for acceptance. As the paper reads currently, it does not seem publication-ready.", "strength_and_weaknesses": "Strengths:\n* The method is clearly described and intuitive, and is a natural next-step after prior work.\n* Combination of demonstrations and language appears to provide significant improvements over either modality individually.\n* The work is positioned clearly with respect to BC-Z.\n* The authors developed a simulator/simulated task for their work.\n\nWeaknesses:\n* The proposed method feels like a solid first-step, but many natural follow-up questions are left unexplored (what if the language encoder is fine-tuned? Why not pass the CLIP embeddings through a small, fine-tuned MLP? How does a pre-trained vision module work for the demo encoder? etc)\n* Training the demonstration encoder seems to require prior knowledge on the number of possible tasks, limiting the generality of DeL-TaCo  to situations with fixed numbers of tasks, despite the goal-encoding being quite general.\n* Experimental reporting feels disjointed and incomplete. While the discussion seems to suggest that language is superior to demonstrations (e.g., saying that language is worth 50 demonstrations), in fact the demo-only method performs better than the language-only method in both experiments. There are also no upper-bound numbers for Table 2, so we can see that DeL-TaCo is better than demo/language-only, but we don't know how far it is from \"perfect\" performance.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster