MUTEX: Learning Unified Policies from Multimodal Task Specifications
Authors: Rutav Shah
Roberto Martín-Martín
Yuke Zhu
Venue: CoRL 2023 Poster (2023)
Keywords: Multimodal Learning
Task Specification
Manipulation

Abstract: Humans use different modalities, such as speech, text, images, videos, etc., to communicate their intent and goals with teammates. For robots to become better assistants, we aim to endow them with the ability to follow instructions and understand tasks specified by their human partners. Most robotic policy learning methods have focused on one single modality of task specification while ignoring the rich cross-modal information. We present MUTEX, a unified approach to policy learning from multimodal task specifications. It trains a transformer-based architecture to facilitate cross-modal reasoning, combining masked modeling and cross-modal matching objectives in a two-stage training procedure. After training, MUTEX can follow a task specification in any of the six learned modalities (video demonstrations, goal images, text goal descriptions, text instructions, speech goal descriptions, and speech instructions) or a combination of them. We systematically evaluate the benefits of MUTEX in a newly designed dataset with 100 tasks in simulation and 50 tasks in the real world, annotated with multiple instances of task specifications in different modalities, and observe improved performance over methods trained specifically for any single modality. More information at https://ut-austin-rpl.github.io/MUTEX/

MUTEX: Learning Unified Policies from Multimodal Task Specifications
Rutav Shah rutavms@cs. utexas.ede..ee
Roberto Martín Martín robertonm@cs. utexas.... edu
Yuke Zhu
yukezâcs utexas.
Abstract:
Humans use different modalities, such speech, text, images, videos, etc., to communicate their intent and goals with teammates. For robots to become better assistants, we aim to endow them with the ability to follow instructions and understand tasks specified by their human partners. Most robotic policy learning methods have focused on one single modality of task specification while ignor- ing the rich cross-mi moosll information. We present MUTEX, a unified approach policy learning from multimodal task specifications. trains a transformer- basoemeer---aeddddd architecture facilitate cross moosll reasoning. combining masked modeling and cross-modal matching objectives in two-stage training procedure. After training, MUTEX can follow task specification in any of the six learned modalities (video demonstrations, goal images, text goal descriptions, text instructions, speech goal descriptions, and speech instructions) or a combination of them. We systemati- cally evaluate the benefits of MUTEX in simulation and 50 tasks the real world, annotated with multiple instances of task specifications in different modalities, and observe improved performance over methods trained specifically for any single modality. More information http::://ttuuuutiii
newly designed dataset with 100 tasks
github 1o/MUTEX/
Keywords: Multimodal Learning, Task Specification, Robot Manipulation
Robot Observations
Text Instructions
1. Move towards the bread hand. they Slide the oven try inside.
Speech Instructions
"Please, move your gripper towards the greep it gently
move above white place and gantly release It."
5. Ler of the try
Mutex
Video Demonstration
Transformer Policy
Fusion Module
Spectficttion
Speech Coal
Image Goal
Text Goal
The bread placed
'Please, place bread on on the oven tray."
tray.
Robot Action
Figure 1: Overview. We introduce MUTEX, a unified policy that learns to perform tasks conditioned task specifications from multiple modalities (image, video, text, and speech) in the forms of instructions and goal descriptions. MUTEX takes advantage of the complementary information across modalities become more capable of completing tasks specified by any single modality than methods trained specifically for each one.
Introduction
When working in team, humans regularly make use of different modalities to specify tasks and improve communications, e.g., sharing high-level task goals ("Let's cook meal!"), verbal instruc- tions ("We will go to the kitchen, get the pot from the cabinet, and then put it on the stove."), or
"Equal advising. 'The University of Texas at Austin
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

fine-grainer visual demonstrations (showing a cooking video). Human-robor teams should aspire to a similar level of understanding. While recent research in robot learning has studied various modal- ities for specifying robotic tasks, including text, images, speech, and videos, most previous studies have treated these individual modalities as separate problems, such as language-condirioned policy learning [1, 2. 3, 4], instruction following [5], visual goal-reaching [6, 7. 8, 9]. and imitation from video demonstrations [10, 11, 12]. Consequently, these approaches lead siloed systems tailored to individual task specification modalities.
A burgeoning body of intertisciplinary Al research has suggested that joint learning across multi- ple modalities, such as image and text [13, 14, 15, 16], video and text [17, 18, 19] and vision and touch [20, 21, 22], gives rise to richer and more effective representations that improve understand- ing of individual modalities. These results align with findings in cognitive science and psychology, which suggest that incorporating multimodal cues (e.g., visual and verbal) into human learning processes enhances learning quality over using individual modalities alone [23, 24). Drawing inspi- ration from the effectiveness of cross-modal modal learning in prior work, our goal is to develop unified policies capable of reasoning about multimodal task specifications for diverse manipulation tasks, where each task will be defined in a single modality that changes from task to task. We seek to harness the complementary strengths of different modalities - some providing compact and high-level information like goal descriptions, while others providing fine-grained information like step-by-step video demonstrations thereby improving the model's ability to execute tasks from task specifications of different modalities.
Prior works primarily focus on enhancing language understanding using visual data [25, 26, 27, 28] or on a subset of modalities such as language and robot demonstration [29], language and image [30]. These approaches typically cover one or two modalities, failing to encompass the diverse ways in which humans express their goals and intent. The key challenge associated with learning across var- ied modalities is effectively leveraging cross- modal information reinforce each other and having a highly versatile architecture to encapsulate the variability introduced by multiple modalities.
To effectively learn from different task specification modalities, we improve upon two representation learning techniques, masked modeling [13, 18, 31] and cross-modal matching [14, 32], to foster cross-modal interaction through shared embedding space. Firstly, we exploit the complementary strengths of each modality text and speech specifications provide guidance for the model to extract task-relevant features from visual specifications (image goals and video demonstration), and visual specifications, in turn, help ground the text and speech to real-world observations. The model is trained on the representation learning objectives in tandem with the policy learning objective (i.e., behavior cloning) such that the representation of the task specification also captures action-relevant information. After we build richer, more informed representations for each modality, we bring them to a common space [13, 14, 18, 31]. Unlike prior work that maps visual specifications to language embeddings [33], we exploit the fact that human video demonstrations contain more fine-grained information about the task. We enrich the representations of other modalities by matching them with the information-dense video representations. This cross-modal matching leads to compact yet informative multimodal representations that can be used to execute tasks specified by any modality.
For the model to execute task specified by any modality, it must handle the variable input length of task specification tokens. Meanwhile, it has to predict robot actions alongside variable number of masked signals for masked modeling of different modalities. To achieve this, design a Perceiver- style encoder [34] where a variable number of task specification tokens are attended with a fixed history of robot observations using the cross-attention mechanisms. The embeddings obtained from the encoder are then passed through the Perceiver-style decoder [35] to predict robot actions and masked signals for individual modalities. This architecture design allows the model to learn a policy that can execute tasks specified by any single modality or arbitrary aggregation of several.
In summary, we introduce MUTEX (MUltimoda Task specification for robot EXecution), uni- fied policy capable of executing tasks specified as goal descriptions more detailed instructions in text, speech, or visual modalities. MUTEX is versatile and performant. It can not only understand multimodal task specifications but also improve the robustness of task execution for every single
2

---

modality, We demonstrated this with a comprehensive evaluation benchmark, including 100 di- verse manipulation tasks in simulation and 50 in the real world, leading to 6000 evaluated trials (per method) in simulation and 600 evaluation trials the real world. Remarkably, real-world evalua- tion indicates that MUTEX can effectively interpret human video demonstrations and perform tasks successfully with the robot, albeit with morphology differences. As part of this effort, we provide large real-world dataset of 50 tasks for complete list of real-world tasks with 30 trajectories for each task containing tasks like "putting bread on microwave tray and closing it", "opening air fryer and putting hot dogs in it", or "placing the book in the front compartment of the caddy", spec- ified with multiple rich multimodal task specifications: text goal description, text instruction, goal image, video specification, speech goal description, and speech instruction [Figure 3], supporting future research in multimodal task specification.
2 Related Work
Task Specification in Robot Manipulation. One way to communicate tasks to a multi-task policy through one-hot encoding vectors [36]. However, this approach is limited to a predefined set of tasks and cannot be extended to new ones. On the other hand, methods that use language to specify tasks [1, 33, 4, 37, 38] have shown improved multi-task generalization due to a richer semantic task representation. However, learning with language specification can be ineffective as it requires grounding language to the robot's 's observation and action spaces [39, 40]. Moreover, the compact nature of language poses a challenge for tasks that need more detailed or accurate descriptions [30]. Visual task specifications (images [7. 9, 8, 41] or videos [10, 11, 33, 12I) offer dense information which makes the policies falsely depend on task-irrelevan information (e.g., a visual demonstration of moving an object also shows the locations and motions of background objects in the scene [42]), causing poor generalization behaviors. The peculiarity of individual modalities limits methods that focus on unimodal specifications and fail to leverage complementary information across modalities.
Cross-modal Representation Learning. In the past years, there has been a large body of lit- erature learning rich representation from multimodal data with cross-modal learning objec- tives [16, 43, 44]. These works have provided convineing evidence that learning across multiple modalities can substantially boost model performances on conventional visual recognition tasks (such as image classification [15, 45, 46, 47], object detection [48, 49, 50], segmentation [51, 46), activity recognition [52]). language understanding tasks (such as sentiment analysis, paraphras- ing [53, 54]), and multimodal reasoning tasks (such as visual QA [55, 19, 56] and cross-modal retrieval [13, 14)). In robotics, multi-sensory observations have been shown to improve the perfor- mance of manipulation tasks [20, 21, 22]. Inspired by these successes, MUTEX learns a cross-modal representation of task specifications for multi-task imitation learning for robot manipulation.
Multi-Task Imitation Learning in Manipulation: Multitask imitation learning for robot manipu- lation has been extensively studied with language task specifications [1, 33, 4] and visual demon- stration [12, 9], respectively. A closer line of work to ours consists of methods that hamness multi- modal task specifications. Some leverage multimodal data for model training, but the final policies are deployed to only operate on a single modality type [25, 26, 27, 33]. Others demonstrate that policies consuming multimodal specifications can generalize to novel task instances in one shot, but the new task must be specified all modalities [29]. More recent works have explored specifying a task with multimodal prompts (text and image tokens) defined in a combination of modalities [30], providing more flexible interface and partially alleviating grounding problems. Nevertheless, none of these prior works has offered the flexibility specify the task using any individual modality, nor support as many different modalities as MUTEX.
3 MUTEX Model and Dataset
Our goal is learn a unified policy that performs diverse tasks based on dataset of demonstra- tions annotated with multimodal task specifications, including language, speech, and visual spec- ifications. We assume that, during test time, the task to perform will be specified in a single or
3

---

Video Demonstrwiion
Observations
Robot
Legend
Prediction tokknns
Matching Loss
Text Instructions
Diservali Encoder
Action Queries
Meskee Token Quories
Stop Gradient
Speech Goal
Sample Modalitios
Behawer Cloning Loss
Image
Self Attention
Cross Attention
Salt Attent
Cross Attention
Text Goal
Masked Moce
Speech Instructions
Policy Encader
Decoder
Modalit-Specific
Projection
Encoders
Layers
Figure 2: MUTEX's Model Architecture and Training Losses. Task specifications each modality are encoded with pretrained mudaalty-sseeciffe encoders, CLIP, and Whisper models 57J. During the first stage of training, one or more of these modalitics are randomly selected and masked before being passed to projection layers. The resultant tokens obtained from projection layers combined with observation tokens through N blocks of self- cross-attention layers. The encoder's hidden state passed MUTEX's transformer decoder queried actions (behavior learning loss) the masked features and tokens (masked modeling loss), promoting action-specific cross-modal learning. In the second stage of training, all modalities are enriched with information from video features through cross-modal matching loss. MUTEX predicts closed-loop actions to achieve the task based on the provided observations and task specification modality (one more) at test time.
subset of modalities that can vary from task task. For each task, T; (TT,T2, ,,.,.,,T,,,,, Tr assume that the agent learns from of human demonstrations obtained through teleoperation, D, {d] ..................... ..... .f.....,d). df") where is the number of demonstrations for task T,, forming entire dataset of demonstrations, D. Each demonstration de presents the form of sequence of observa- tions and expert actions, de = [(o), an), (OT, a.)
The goal of each task T; is specified by i alternative task descriptions (t) in six different forms within three modalities: text instructions / E {L ........ L text goal description / E 1 1 video demonstration t E {S!, S the modalities with the letters L/1, V/u, and S/s, and make use of capital letters to denote detailed instructions and small letters to denote goal state specifications. Our goal is to learn a unified policy, (alt, o), that outputs continuous actions, E A, given current observations, task specification in
{V,3, V.2..... V V.*,, and speech goal description t E {s} &s,,,?
goal image E {ut of....,et of speech instructions
S.
.?..... . Note that we characterize
O, conditioned more of the possible modalities, E L[[V/sSS) We aim to create policy that not only performs the tasks in the training dataset D (seen tasks) in initial conditions (e.g. positions of the objects) but also generalizes to previously unseen task descriptions.
3.1 MUTEX Training Procedure
Our goal is not only to obtain a unified policy that understands task specifications in different modal- ities but also to improve the policy performance when task is specified on every single modality by exploiting cross-modal interactions during training. To that end, we leverage two representation learning techniques that we integrate sequentially MUTEX's training procedure: 1) Mask Mod- eling to promote pross-modal interactions of all modalities into a shared learned latent space, and 2) Cross-Modal Matching to enrich each modality with information of the informmtioo-deensee one. Both stages of procedure are combined with behavior cloning objective for policy learning to ensure that the learned representation contains relevant information for the agent to perform the manipulation task. The overview of the proposed approach is shown in Fig. 2.
Masked Modeling for Cross-Modal Learning: In the first stage [Step 1 in PseudoCode 1]. MU- TEX promotes cross-modal interactions between the model components interpreting the different

---

modalities: text instructions (L), text goal (!), video demonstration (V), image goal (u), speech instructions (5), and speech goal (s). Inspired by the success in other cross-modal learning tasks [53, 19, 43, 44], we mask certain tokens or features of each modality and learn to predict them with the help of other modalities. This enforces the model to use the information from other modalities to enhance the representation of each one of them. Intuitively, text- and speech-masked prediction helps in focusing relevant information from visual modalities, while image- and video- masked prediction helps in grounding other modalities. During testing, task specification in only one or subset of the modalities is used. Thereby, to obtain robust single-modality representations [Table 1, 2], we recreate these conditions in our training procedure by randomly sampling modalities in each iteration.
Specifically, at each iteration of the training process, we randomly select task specifications of one or subset of modalities. If more than modality is selected, we mask certain parts of each modality. We require MUTEX to predict the masked parts alongside the action values the expert demonstrated at each step. Different modalities would mask either input or intermediate features and use a different metric loss to measure prediction error (&1 or f2). For masked text modeling (masking elements of a text goal description or text instructions), we mask out words [58] that are then passed through pretrained CLIP language model [15] to extract the features. We use the standard cross-entropy loss between the predicted (z) and ground truth (y) tokens, i.e., () Ei=1 y/4 log(d:). where N is the number of tokens in the vocabulary. For masked visual modeling (masking elements of an image goal or a video instruction), we mask out intermediate regions of the features obtained from pre-trained CLIP model [15] and, following prior works in vision-language modeling [59), we employ simple و rregression loss between the predicted and ground truth features, Le, = ly - øl- For masked speech modeling (masking elements of a speech goal description or speech instructions), we use similar approach to visual modeling but use features from pre-trained Whisper model [57] instead with an lrrreression loss (Refer to Appendix 6.2).
N
the second stage of MUTEX'S training procedure [Step in PseudoCode 1]. we enrich the common embedding space for each task modal- ity by associating it with the features of the information richer one. To that end, in contrast to prior works that use cross-moda contrastive loss to learn common embedding space [14, 16), simple l2 loss to pull the representations of all modalities towards the one with more information and better performance. Video specifications contain the most information, leading to more elucida- tive and stronger features; therefore, we enrich other modalities with information from the video representation space obtained after cross-modal learning.
Cross-Modal Matching for Richer Representations:
Concretely, let ft. ft. for fa. Iss be the feature representations of the text instructions, text goal, image goal, speech goal, and speech instructions, and fv the feature representation of the video demonstration for the same task. Our cross-modal matching loss is given by:
f.matcch (e. (ei Le, Sv) Leg (Js, + (S.(Vs.Vv)
where Lig is the fz-regression loss. Gradients from this loss are not backpropagated to the part of MUTEX that encodes the video modality, leaving them unchanged.
3.2 MUTEX Architecture
The training process delineated above requires model architecture capable of propagating and encoding the cross-modal information from multiple modalities. MUTEX'S model consists of three main components (see Fig. 2): 1) Modality- specific tokens, 2) a Policy Encoder that takes in the task task-specific tokens and robot observations and outputs hidden states, and 3) a Policy Decoder that takes in the hidden states along with decoder queries and outputs features corresponding to the queries.
Encoders that map input modalities to task-
MUTEX'S modality-specifie encoders extract tokens from input task specification using a fixed, pre-trained large model, which helps us to extract semantically meaningful representation from the input modality. To learn representations that are grounded to the observation and action space, these features passed through projection layer consisting of simple MLP single attention block
5

---

before passing it to the policy encoder. MUTEX's policy encoder effectively fuses information ob- tained from multiple task specification modalities and robot observations, employing a transformer- based architecture with stacked cross- and self-attention layers. the cross-attention layers, queries are derived from robot observations, whereas the keys and values are from task specification tokens. The encoder's 's output is then passed to the policy decoder. Although the policy encoder's output is enriched with information obtained from different task specification modalities, MUTEX requires the policy output features for predicting action values and a variable number of masked tokens. This motivates us to adopt a Perceiver Decoder [35] architecture MUTEX's policy decoder to lever- age learnable queries and output only the information corresponding to input queries. The decoder features for action prediction are passed through an MLP to estimate a Gaussian Mixture Model for continuous action values. Similarly, separate MLPs are used to predict token values or features for the masked token queries. More details can be found Appendix 6.3.
3.3
Multimodal Task Specification Dataset
Video
Demonstration
Image Goal
Text
Kindl hole control and twist
1. Orip hansle edge of the drawe:
Instructions
in was that sets the bumner "on"" Alin the frying pen above the stove surface, owerit steadlly
2. Flowly and caretally pull drower open.
Guide your groppe toward the loat.
4. Place the granora ber insiibe the opien drawer
3. Pootion the grtpper over the whitte dish Gendly place the the while dast.
trying
Text Goal
frying pan on the stove, and the heat is turned up.
The granola bar opened top drawer of the cabinet.
The bread placed on the white dish.
Speech Instructions
"Sefty adjustmenc kmab, align the on actting, and hoed the flying pan. from nastion above dlove, gently aet down Retsase the handle when It's aecure.
"Woid handle
drower Slowly "Kinaly move your grippor noar the broad. Gently gro and hover over the
and carctily open tne crower Look for and grab numiion bar the avalable fnside the crower Let of nutsiton ber"
gandl
onto the plate"
Speech Goal
"At the moment, the cooktop is on frying pan
"In the cabinet's opened top drawer, granola has carefully piaced."
"The bread arranged the plato."
Figure 3: MUTEX Multimodal Task Specification Dataset. We provide dataset comprising 100 simulated tasks (example the first column) based LIBERO- 100 [60] and real- world tasks (examples the second and third columns), annotated with 50 and 30 demonstrations per task, respectively. We annotate each task with 11 alternative task specifications in each of the six following modalities (rows from top to bottom): vidco demonstration, image goal, text instructions, text goal, speech instructions, and speech goal.
As part of our efforts to develop a unified multi task imitation learning policy, we construct new dataset of tasks with multiple task specifications per modality in both simulated and real-world tasks (see Fig. 3). the simulation, we extend the LIBERO-100 benchmark [60] entailing diverse object interactions and versatile (n 100) tasks like "turn on the stove and put the frying pan on it." Each task is annotated with m 50 human trajectories (provided by the authors) and k 11 task specifications per modality. Due to the inevitable sim2real gap, video demonstrations in the simulation are collected by teleoperating the simulated robot instead of directly by a human performing the tasks. In the real world, we collect a novel dataset with = 50 tasks (Figure 5) ranging from pick and place tasks such as "put the bread on the white plate", "pick up the bowl at the back of the scene and place inside the top drawer" to more contact-rich tasks such as "open the air fryer and put the bowl with dogs in it", "take out the tray of the oven, and put the bread on it." All tasks are collected the same environment but involve different objects from set of 17 objects. Each task is demonstrated with spacemouse telcoperation device. Each task is annotated with & 11 different task specifications
30 cuman-collected trajectories [61] using 3D

---

Method
Text Goal
Text Instructions
Image Goal
Video Demonstration
Speech Goal
Speech Instructions
Modality-Speee Specifiic
41.7 8.0
39.9 2.8
58.7 = 4.9
62.0 = 5.2
22.3 1.2
28.4 = 5.5
MUTEX (joint training)
39.2 9.1
38.3 7.4
48.6 14.8
50.7 + 16.0
32.1
9.9
38.2 13.6
MUTEX (no masked modeling)
34.8 +: 6.0
38.7 + 6.5
43.8 + 6.0
46.0 + 8.0
24.6 + 4.2
29.1 + 6.1
MUTEX (no cross-moda matching)
43.5 8.9
39.4 2.2
60.1 6.4
63.2 : 6.3
36.7 4.3
46.8 5.5
MUTEX
50.1
53.0
61.6
63.2 6.3
8.1
Table 1: Success Rate on the Multimodal Task Specification Dataset in Simulation.
Method
Text Goal
Text Instructions
Image Goal
Video Demonstration
Speech Goal
Speech
Instructions
Speelity-Seeciffe
52
48
42
52
32
46
MUTEX
64
58
62
64
50
60
Table 2: Success Rate on the Multimodal Task Specification Dataset in Soal-W.rld.
for each modality. A human performs video demonstrations specifying the tasks with their hand. To generate 11 diverse text goal descriptions and instructions, we make use of ChatGPT with prompts to generate alternative descriptions that we manually filter to avoid using synonyms that do not match the right task-relevant objects (e.g., using plate as a synonym for bowl when there are other plates in the scene). We request speech signals from several characters of the Amazon Polly service to generate diverse speech descriptions.
Experimental Evaluation
Experimental Setup:
We conduct evaluations our newly constructed dataset of multimodal task specifications, with a split of 80% (8/3) tasks for training and testing. For evaluation, we subject the robot to both unseen task specifications (from the test split) and new initial conditions (i.e., positions of objects). For each tested task, we evaluated 20 trials in simulation and 10 trials in the real world. Reported results for each evaluation modality are average for all 20 trials x 100 tasks 3 seeds in the simulation and 10 trials x tasks the real world. our evaluation, we compare MUTEX to models trained specifically for each modality using only task specifications in that modality. This resembles most existing prior works in video-based imitation learning [11, 12], and goal- coaliconditioned imitation learning [6, 9, 33, 4J. Please refer to Appendix 6.5 for details.
Experiments: In our experimental evaluation, we aim answer the following questions:
1) Does unified policy capable of executing tasks across multiple modalities outperform methods trained specifically from and for each individual modality? Table and Table 2 summarize the results of our evaluation in simulation and the real world, respectively. In both cases, we observe a significant improvement (+10.3% + +1.33% in simulation and real-world respectively) from using our unified policy MUTEX compared to podality-specifii models, indicating that the cross-modal learning procedure from MUTEX is able leverage more information from other modalities.
Additionally, we analyze the errors in our real-world evaluation and find that representations learned using MUTEX generalize better to new task specifications than unimodal representations. Specif- ically, when dealing with unseen task specifications, failed trials that were not attributed to BC compounding error, where the policy correctly completed a semantically meaningful task the en- vironment, albeit not the intended task (e.g.. picking and placing another object). We find that in the unimodal baselines, 85 of 240 trials (35.4%) are due to failed understanding of tasks specified, whereas it reduces to 40 out of 240 (16.7%) with MUTEX.
2) What the importance of the two stages of the MUTEX training procedure? it important to perform the stages consecutively? Table includes a comparison the results obtained with the two-staged training procedure consecutively (MUTEX) compared to training with both stages simultaneously (joint training), training without the first stage (no masked modeling) or without the second stage (no cross-modal matching). We observe the largest drop in performance comes from training without masked modeling, indicating that this step is critical to learning cross-modal
7

---

information from different task specifications. Cross-modas matching provides boost except for video demonstrations (we match to that modality) and observes a small drop in speech specifications.
3) Does the performance increase significantly when tasks are specified with multiple modalities? One of the advantages of MUTEX is that it can execute tasks with a single specification in any of the learned modalities or with multiple specifications in several of the modalities. We evaluated using combinations of Text Goal + Speech Goal, Text Goal + Image Goal and Speech Instructions + Video Demonstration and obtain 50.1, 59.2, and 59.6 success rates, respectively. These values are close to the performance using one single specification in the best of the two modalities, indicating that the additional modality is not providing any extra information. We hypothesize is because all possible cross- cross-modal information has already been learned by MUTEX. Interestingly, when using specifications in all modalities, the success rate is 60.1, lower than when using Image goal or Video Demonstration, possibly due the increased complexity of interpreting multiple task specifications.
4) Are the task specification representations learned by MUTEX better than state-of-the-ar lask- specification models? To further evaluate the efficacy of the MUTEX representations, we compare
Method
| Text Goal
Instruction
Image Goal
Demonstration Image Goal + Goal
TS [62]
40.0
44.0
R3M [63]
59.5
41.7
VIMA [30]
47.0
MUTEX
50.1
53.0
61.6
63.2
59.2
Table
Success Rate on the Multimodal Task Specification Dataset in Simulation.
it with other task specification models in Table 3. Although the unimodal models, T5 and R3M, achieve better results than CLIP (Table 1) in Text Instructions (+4.1%% and Image Goals (+22),, re- spectively, MUTEX consistently outperforms these models across all modalities. The consistent im- provement across modalities demonstrates the value of leveraging multiple modalities during train- ing. Moreover, MUTEX significantly outperforms VIMA, a recent method that employs both text goals and object images for task specification, highlighting that MUTEX is not only more performant than its unimodal counterparts but also effectively use multiple modalities during inference.
Conclusion, Limitations, and Future Work
We demonstrate with comprehensive experiments in simulation and the real world that multi-task learning, when trained on task specifications across multiple modalities, produces more robust and versatile policy in each modality. We are highly encouraged by the empirical results and the potential of MUTEX for designing more versatile multimodal human-robot communication interface.
We aim to improve MUTEX in future work by addressing several limitations. MUTEX assumes paired access to all the task specification modalities, which may be difficult to obtain scalable fashion. MUTEX uses of clean speech signals synthesized by Amazon Polly may not accurately represent real-world speech, which is often noisier and more difficult to understand. MUTEX uses video and image goals are provided from the same workspace as the task to be executed. Having policy that can execute task specified "in the wild" visual goal or demonstration will invite additional challenges. We also plan to explore how to foster stronger generalization by training across diverse environments, which could open the door to the use of larger human video datasets. Lastly, MUTEX uses vanilla behavior cloning to learn policies, which possesses problems like covariate shifts and compounding errors. To mitigate this limitation, incorporating interactive imitation learning and reinforcement learning techniques is an exciting direction for future research.
Acknowledgments
We thank Yifeng Zhu and Huihan Liu for real robot system infrastructure development. We thank Zhenyu Jiang, Jake Grigsby, and Hanwen Jiang for providing helpful feedback for this manuscript. We acknowledge the support of the National Science Foundation (1955523, 2145283), the Office of Naval Research 1000014-2-11220060 UT Good Systems, and the Machine Learning Laboratory.
8

---

References
[1] Stepputtis, J. Campbell, M. Phielipp, Lee, C. Baral, and H. Ben Amor. Language- conditioned imitation learning for robot manipulation tasks, Advances in Neural Information Processing Systems, 33:13139-13150, 2020.
[2] Nair, E. Mitchell, K. Chen, S. Savarese, C. Finn, et al. Learning language-conditioned robot behavior from offline data and crorddsourced annotation. In Conference on Robot Learning, pages 1303-1315, PMLR, 2022.
[3] L. Shao, T. Migimatsu, Q. Zhang, K. Yang, and J. Bohg. Concept2robot: Learning manip- ulation concepts from instructions and human demonstrations. The International Journal of Robotics Research, 40( (112-144:1419-1434, 2021.
[4] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.008117 066111777. 2022.
[5] C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, T. Armstrong, and Florence. Interactive language: Talking to robots real time. arXiv preprint vrXiv:2210.06477 2022.
[6] D. Pathak, P. Mahmoudich, G. Luo, P. Agrawal, D. Chen, Y. Shentu, E. Shelhamer, J. Malik, A. A. Efros, and T. Darrell. Zero-shot visual imitation. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 2050-2055 2018.
[7] S. Nasiriany, V. Pong, S. Lin, and S. Levine. Planning with goal-conditioned policies. Ad- vances in Neural Information Processing Systems, 32, 2019.
[8] Y. Cui, S. Nickum, A. Gupta, V. Kumar, and A. Rajeswaran. Can foundation models perform zero-shot task specification for robot manipulation? In Learming for Dynamics and Control Conference, pages 893-905. PMLR, 2022.
[9] Z. J. Cui, Y. Wang, N. Muhammad, L. Pinto, et al. From play to policy: Conditional behavior generation from uncurated robot data. arXiv preprint arXiv:22 10040 2022.
[10] T. Yu, C. Fino, A. Xie, S. Dasari, T. Zhang, P. Abbeel, and S. Levine. One-shot imitation from observing humans via domain-adapplon meta-learning, arXiv preprint arXiv: 1802:00.0020155,, 2018.
[11] A. Bonardi, S. James, and A. J. Davison. Learning one-shot imitation from humans without humans. IEEE Robotics and Automation Letters, 5(2):3533 3539, 2020.
[12] S. James, M. Bloesch, and A. J. Davison. Task-embedded control networks for few-shot imi- tation learning. In Conference robot learning, pages 783-795. PMLR, 2018.
[13] Y.-C. Chen, L. Li, Yu, A. EI Kholy, Ahmed, Z. Gan, Y. Cheng, and J. Liu. Uniter: Universal image-text representation learning. In Computer Vision-ECCV 2020: 16th Euro- pean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX, pages 104-120. Springer, 2020.
[14] H. Tan and M. Bansal. Lxmert: Learning mross-modality encoder representations from trans- formers. arXiv preprint 1rXiv:190..074900 2019.
[15] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, al. Learning transferable visual models from natural language supervi- sion. In International conference machine learning, pages 9748-8763 PMLR, 2021.
[16] A. Singh, R. Hu, V. Goswami, G. Couairon, W. Galuba, M. Rohrbach, and D. Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Confer- ence Computer Vision and Pattern Recognition, pages
2022.

---

[17] H. Xu, G. Ghosh, P.-Y. Huang, P. Arora, M. Aminzadeh, C. Feichtenhofer, F. Metze, and L. Zettlemoyer. Vlm: Task-agnostic laneo-angguage model pre-training for video understand- ing. arXiv preprint wxxi:2205.099990 2021.
[18] H. Luo, L. Ji, B. Shi, H. Huang, N. Duan, T. Li, J. Li, T. Bharti, and M. Zhou. Univl: unified video and language pre-training model for multimodal understanding and generation. arXiv preprint arXiv:2002.06335, 2020.
[19] R. Zellers, J. Lu, X. Lu, Y. Yu, Y. Zhao, M. Salehi, A. Kusupati, J. Hessel, A. Farhadi, and Y. Choi. Merlot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16375-16387, 2022.
[20] M. A. Lee, Y. Zhu, K. Srinivasan, Shah, Savarese, L. Fei-Fei, A. Garg, and Bohg- Making sense vision and touch: Self-supervised learning multimodal representations for contact-rich tasks. In 2019 International Conference on Robotics and Automation (ICRA), pages 3943-8950. IEEE, 2019.
[21] Y. Li, J.-Y. Zhu, R. Tedrake, and A. Torralba. Connecting touch and vision via cross-modal prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 10609-10618, 2019.
[22] I. Guzey, B. Evans, S. Chintala, and I.. Pinto. Dexterity from touch: Self-supervised pre- training of tactile representations with robotic play, arXiv preprint arXiv:2303.12070 2023.
[23] M. Clark and Paivio. Dual coding theory and education. Educational psychology review, 3:149-210 210, 1991.
[24] L. E. Bahrick and R. Lickliter. Intersensory redundancy guides attentional selectivity and perceptual learning in infancy. Developmental psychology, 36(2):190, 2000.
[25] C. Lynch and P. Sermanet. Language conditioned imitation learning unstructured data. arXiv preprint arXiv:2005.076445 2020.
[26] O. Mees, L.. Hermann, and Burgard. What matters in language conditioned robotic imitation learning. arXiv preprint arXiv:2204. 06252, 2022.
[27] V. Myers, A. He, K. Fang, H. Walke, P. Hansen-Estruch, C.-A. Cheng, M. Jalobcanu,
A. Kolobov, A. Dragan, and S. Levine. Goal representations for instruction following: A semi-supervised language interface to control. arXiv preprins arXiv:2307.001111 2023.
[28] H. Liu, L. Lee, K. Lee, and P. Abbeel. mnstruction-followiin agents with jointly pre- trained vision-language models. arXiv preprint arXiv:2210. 13431,, 2022.
[29] A. Yu and R. Mooney, Using both demonstrations and language instructions to efficiently learn robotic tasks. arXiv preprint arXiv:2210.04446, 2022.
[30] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arX.v:2210.03094, 2022.
[31] G. Kwon, Z. Cai, A. Ravichandran, E. Bas, R. Bhotika, and S. Soatto. Masked vision and language modeling for multi-modas representation learning, arXiv preprint arX0::2208.02211 02131, 2022.
[32] I. Xue, M. Gao, C. Xing, R. Martin-Martin, Wu, C. Xiong, R. Xu, J. C. Niebles, and S. Savarese. Ulip: Learning unified representation of language, image and point cloud for 3d understanding, In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
10

---

[33] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Be-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learn- ing, pages 991-1002. PMLR, 2022.
[34] A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pages 4651-4664. PMLR, 2021.
[35] A. Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. Ionescu, D. Ding. S. Koppula, D. Zoran, A. Brock, E. Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14799 14795,, 2021.
[36] R. Rahmatizadeh, P. Abolghasemi, L. Bölöni, and S. Levine. Vision-based multi-task manip- ulation for inexpensive robots using end-to-end learning from demonstration. 2018 IEEE international conference on robotics and automation (ICRA), pages 3758-3765 IEEE, 2018.
[37] P.-L. Guhur, S. Chen, R. G. Pinel, M. Tapaswi, I. Laptev, and C. Schmid. Instruction-driven histery-aware policies for robotic manipulations. In Conference on Robot Learning, pages 175-187. PMLR, 2023.
[38] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al. Do as can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0169) 2022.
[39] S. Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3):335-
346, 1990.
[40] Y. Bisk, A. Holtzman, J. Thomason, J. Andreas, Y. Bengio, J. Chai, M. Lapata, A. Lazaridou, J. May, A. Nisnevich, et al. Experience grounds language. arXiv preprint arXiv:2004.100 10151, 2020.
[41] Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang. Vip: Towards universal visual reward and representation via value-implcch pre-training. arXiv preprint 10000222000000300 2022.
[42] P. Agrawal. The task specification problem. In Conference on Robot Learning, pages 1745- 1751. PMLR, 2022.
[43] L. Yuan, D. Chen, Chen, N. Codella, X. Dai, J. Gao, H. Hu, X. Huang, B. Li. C. Ci, al. Florence: A new foundation model for computer vision. arXiv preprint a1X1v:2111.11432, 2021.
[44] R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra. Image- bind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11180-15990, 2023.
[45] C. Jia, Y. Yang, Xia. Y.-T. Chen, Z. Parekh, H. Pham, Le, Y.-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference machine learning, pages 4904-491 PMLR, 2021.
[46] R. Bachmann, D. Mizrahi, A. Atanov, and A. Zamir. Multimae: Multi-modal multi-task masked autoencoders. In European Conference Computer Vision, pages 348-367. Springer, 2022.
[47] Z. Lin, S. Yu, Z. Kuang, D. Pathak, and D. Ramanan. Multimodality helps unimodality: Cross-modal few-shot learning with multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19325-19337, 2023.
[48] Y. Zhong, J. Yang, P. Zhang, C. Li, N. Codella, L. H. Li, L. Zhou, X. Dai, L. Yuan, Y. Li, and
J. Gao. Regionclip: Region- based language-iman image pretraining, 2021.
11

---

[49] L. H. Li, P. Zhang, H. Zhang, J. Yang, C. Li, Y. Zhong, L. Wang, L. Yuan, L. Zhang, J.- N. Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10965-10975, 2022.
[50] X. Geng, H. Liu, L. Lee, Schuurmans, Levine, and P. Abbeel. Multimodal masked autoencoders learn transferable representations. arXiv preprint 1rXiv:2205.14204 2022.
[51] J. Xu, S. De Mello, S. Liu, W. Byeon, T. Breuel, J. Kautz, and X. Wang- Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 18134 11134-11444 18144, 2022.
[52] M. Li, L. Chen, Y. Duan, Z. Hu, J. Feng. J. Zhou, and J. Lu. Bridge-prompt: Towards ordinal action understanding in instructional videos. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19880-19889, 2022.
[53] H. Tan and M. Bansal. Vokenization: Improving language understanding with contextualized, visual-gros groundg supervision. arXiv preprint arXiv:2010.06775, 2020.
[54] Z. Tang, J. Cho, H. Tan, and M. Bansal. Vidlankd: Improving language understanding via video-distilled knowledge transfer. Advances in Neural Information Processing Systems, 34: 24468-24481, 2021.
[55] Wang. Y. Ge, R. Yan, Y. Ge, K. Q. Lin, Tsutsui, X. one: Exploring unified video-language pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6598-6608, 2023.
Lin, G. Cai, J. Wu, Y. Shan, et al.
[56] L. Li, Z. Gan, K. Lin, C.-C. Lin, Z. Liu, C. Liu, and L. Wang. Lavender: Unifying video- language understanding as masked language modeling. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 23119223129 2023.
[57] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356, 2022.
[58] Devlin, M.-W. Chang, K. Lee, and Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 2018.
[59] Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu. Simmim: A simple frame- work for masked image modeling. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9653 (653-9663, 2022.
[60] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone. Libero: Benchmarking knowl- edge transfer for lifelong robot learning. arXiv preprint wxXiv:2306.03300 2023.
[61] Y. Zhu, A. Joshi, P. Stone, and Y. Zhu. Viola: Imitation learning for vision-based manipulation with object proposal priors. 6th Annual Conference on Robot Learning, 2022.
[62] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, (1(1):545555511 2020.
[63] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation. arXiv preprint arXiv:2203.12601 2022.
[64] Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05101, 2017.
[65] I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint (atp::1680.38.33 2016.
12

---

[66] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L.. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[67] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattem recognition, pages 770-778, 2016.
[68] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cmn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
6 Appendix
6.1 Model Training Ablations
Method
Text Goal
Text Instructions
Image Goal
Video Demonstration
Speech Goal
Speech Instructions
MUTEX (w/ pretraining representations)
42.3
46.8
46.0
48.0
34.4
32.0
MUTEX (w/ finetuning full model)
37.1
40.2
44.7
46.3
31.6
34.2
MUTEX (w/ finetuning video encoder)
49.6
53.4
57.6
61.3
38.4
46.0
MUTEX
50.1
53.0
61.6
63.2
40.9
46.0
Table Model Training Ablations of MUTEX variants.
In MUTEX training, the task specification representations are learned in tandem with the policy learning objective, i.c., behavior cloning. The policy learning objective helps capture the neces- sary information for robot action prediction, missing in the task-specification dataset. To verify this empirically, we pre-train the task specification representations using the two-stage training process without utilizing the BC loss. Subsequently, we learned the policy using the frozen task specifica- tion representations (Referred as MUTEX (w/ pretraining representations) in Table 4). MUTEX significantly performs better (+10.9% than MUTEX (w/ pretraining representations).
Cross-modal matching in MUTEX aims to enrich other modalities with the more-informed video specification obtained after masked modeling. To avoid local optima, where the video specifica- tion loses information match representations other modalities, we freeze the MUTEX Policy and Video Encoder layers. To validate the hypothesis, we evaluate the performance of two vari- ants of MUTEX with finetuning full model (including video encoder)- MUTEX (w/ finetuning full model) and finetuning video encoder- MUTEX (w/ finetuning video encoder). The results are highlighted in Table 4. We observe that the training strategy adopted by MUTEX consistently out- performs other variants, corroborating our initial hypothesis that backpropagating gradients through video encoder in modal matching may lead to sub-optimal performance.
6.2 Training Details
For training MUTEX 3.2, following the standard practice, we use the AdamW optimizer [64] along with a Cosine Annealing LR Scheduler [65]. To facilitate learning robust model, we apply data augmentation like ColorJitter and translation augmentation on the RGB observation images. The hyperparameter details are adopted from LIBERO's transformer baseline [60], and the details are provided in Table 5. For masked modeling, train MUTEX for 50 epochs end-to-end, whereas for the cross- modal matching, the projection layers [Figure 2] are finetuned for 20 epochs. The model is trained using the 2 NVIDIA RTX A5000 (24 GB) GPUs with a batch size of 64 using the Py PyTorch framework [66] for all the baselines, and MUTEX with an approximate total of five days for each experiment.
Masked Modeling: We utilize masked modeling facilitate intermodal interaction between differ- ent modalities [Section 3.1]. For the text modality, mask out specific words from the input and predict the corresponding word tokens. Regarding text goals, we randomly select one word from the
13

---

input specification. For text instructions, we repetitively mask out two random words. It is worth noting that stop-words (e.g., 'a', "the") are not masked, as predicting stop-words does not provide helpful task-related information. Regarding visual specifications, we mask out features extracted from CLIP instead directly predicting noisy pixel values. We employ regression loss for train- ing purpose.. We mask the intermediate region feature obtained after the 22nd transformer block in CLIP for image goals. For video demonstrations, we mask out individual frame features. Simi- larly, for speech specifications, we mask and predict features directly obtained from the pretrained Whisper-Small encoder model for both speech goals and speech instructions.
Training Details
Batch Size
64
Gradient Clipping
100
Epochs (Masked Modeling)
50
Epochs (Cross-Modal Matching)
20
Initial Learning Rate
le-4
Betas Weight Decay
(0.9, 0.999)
le-4
Minimum Learning Rate
le-5
Masked Modeling Loss Weight
0.5
Cross-Modal Matching Loss Weight
0.5
Data Augmentation
Brightness
0.3
Contrast
0.3
Saturation
0.3
Hue
0.3
Translation
pixels
Table 5: Hyperparameter details training MUTEX
6.3 Model Architecture
MUTEX leverages multiple modalities during training effectively learn rich, well-informed rep- resentations of each modality, allowing specify the task using single any combination of task specification modalities during evaluation. To allow such flexibility, we adopt transformer architec- ture (Details in main text 3.2).
Modality Specific Encoders extract a single embedding for each input task specification modality. To extract semantically meaningful, grounded representations, each modality is passed through pre-trained large model, and then through a projection layer [Figure 2). For (a) pre-trained large model, MUTEX the CLIP Large ViT-L/14 [15] model for text and visual modalities whereas Whisper-Smale [57] encoder for speech modalities providing semantically meaningful embeddings. The (b) projection layer layers for each modality enable MUTEX to learn a grounded, common embedding for all modalities. For modalities with single embeddings from the pre-trained model, we use an MLP projection layer [Refer 1]. whereas for modalities with multiple embeddings from the pra-trdined model, we use single transformer block with mean pooling [Refer 2] to aggregate information into one embedding. The details of the modality-specifie encoders summarized in Table 6.
Listing Model Architecture for the MLP in Projection Layers
MLPB1ock
Ln.Linear (768, 512), nn.ReLU()
Dn.poo (0.1)
Lineir (512, 768)
nn. mropout (0. 1)
14

---

Pretrained Large Model
Pretrained Embedding Shape
Modality
Projection Layer
ML.PBlock[1] +
Text Goal
CLIP
(1, 768)
Cross-modal Matching MLPPBlcc(1]]
Shared MLPBIlck[[]
Text Instructions
TranstormerPooBllcck]]]
(L, 768)
CLIP
Cross-modal Matching MLPBlock(1] +
L num iunttustrustions
Shared MLPBlock[1]
MLPBBlcc([]
Image Goal
CLIP
(1, 768)
Cross-modal Matching MI MI.PBlock[1] +
Shared MLPBlock(1]
Video Demonstration
TransformerPooBlockk]] Shared MLPBlock(1]
CLIP
(16, 768)
TransformerPoolBlockk
Speech Goal
Whisper
(4, 768)
Cross-modal Matching MLPBBlcc[1]]
Shared MLPBlock(1]
Speech Instructions
TransformerPoolBlocck/
Whisper
(4, 768)
Cross-modal Matching MLPBlock[1]] Shared MLPBlock[1]
Table MUTEX's Modality Specific Encoder consists of pretrained large model each modality. The embeddings extracted from the pretrained model are transformed into single embeddings using the projection layers. The projection layers consist of modality-specift, MLP or Transformer block, an MLP block added in the cross- cross-modal matching, and shared MLP block between all modalities.
projection layer
Listing 2: Model Architecture for the Transformer Block with Mean Pooling in Projection Layers TransformmerPolBBlock nn. om.LayerNormm7683 (768),
nn. ....uuiheaatttentiont
embed_dit dim 768,
num_ heads 4, kdim 256,
vdim 256
dropout 0. 1,
Lm..LyyrrNorm(765))
Transf SeanfformmsrPeeFoovvrrlt
nn. Lii.neeerr 256), nn. GELU(), nn. Dropout (0. 1),
Linear (256, 768), nn. Dropout (0.
ww..gooooliddeerreeeeccom _eize num_embeddings)
Policy Encoder combines the robot observations with the task specification embeddings received from the modality specific encoders. The robot observations consist of a history (T = 10) RGB Images from two camera viewpoints (agentview, eye eyn_in_hand) and proprioceptive information. The RGB images are converted token embeddings using ResNet encoder [67] whereas the proprio- ceptive information is encoded using MLP [Refer [60] for more details]. The robot observation embeddings and task specification embeddings combined using stacked cross-attention and self- attention layers shown in Figure 4, and the hyperparameters summarized in Table 7.
15

---

Action
Masked Token
Robot Observation Embeddings
Queries
Queries
N
XN
Task Specification Embeddings
Cross Attention
Cross Attention
K
Cross Attention
Self Attention
Self Attention
V
Predicted
Policy Encoder
Policy Decoder
Tokens
Figure 4: MUTEX Policy Encoder and Decoder Architecture
Policy Encoder
Input Embedding Size
768
Input Query Embedding Size
64
Transformer Head Size
64
# Transformer Heads
6
Transformer Output Size
64
Transformer MLP Ratio
4.0
Dropout
0.1
Attention Dropout
0.1
Self-Attention Layers
# Cross-Attention Layers
4
Policy Decoder
Input Embedding Size
64
Input Query Embedding Size
64
Transformer Head Size
16
# Transformer Heads
Transformer Output Size
64
Transformer MLP Ratio
4.0
Dropout
0.1
Attention Dropout
0.1
Self-Attention Layers
# Cross-Attenties Layers
4
Table 7: Hyperparameter details of MUTEX's Policy Encoder and Decoder architecture.
Policy Decoder takes in the input from the policy encoder and the learnable queries, enabling Mu- TEX to output variable number of tokens. These outputs are used to predict the masked token values during masked modeling and action values. The learnable query vectors indicate the type corre- sponding output- {Action, Text Goal, Text Instructions, Image Goal, Video Demonstration, Speech Goal, Speech Instructions). These vectors are added with sinusoidal positional vectors indicating the action index and position of the masked tokens in action queries and masked modeling queries, respectively. Similar to policy encoder, it consists of stacked cross-attentin and self-attention layers shown in Figure 4 and the hyperparameters summarized in Table 7.
16

---

Prediction Heads [Not shown in Figure 2 for simplicity] using the output of the policy decoder is used to predict the masked token values during masked modeling and action values. For text modeling, the embeddings are mapped to the vocabulary size using a two-layered MLP and trained using cross-entropy loss. For visual and speech modeling, intermediate masked features are pre- dicted using a single-layer MLP and trained using LI loss. For action modeling, the action values predicted using two-layered MLP. The action distribution is modeled Gaussian Mixture Model with five components to incorporate the multimodality in human-collected demonstrations and trained using negative log-likelihood loss.
6.4 Multimodal Dataset Details
MUTEX Robot Data: MUTEX uses behavior cloning to learn from expert datasets collected by humans using teleoperation. We benchmark on two environments: simulation with 100 tasks and real-world manipulation tasks [Refer to 3.3]. For simulation, we use LIBERO 100 [60] manipu- lation benchmarking suite consisting of tasks better representative of the real world. We use the dataset provided by the original authors, which consists of 50 trajectories per task collected us- ing SpaceMouse. For real world evaluation, we designed 50 tasks in the real world consisting of pick and place tasks like "Put the bread on the white plate" to more complex tasks like "Open the air fryer and put the bowl with hot dogs in it". The 50 tasks are divided across eight different scenes, with an average horizon length of 242 for each task. A complete list of tasks, along with scene visualization, is provided in Figure 5. We use a Franka Panda robot with a Kinect Azure workspace camera and an Intel RealSense camera for eye-in-hand view. We use SpaceMouse to col- lect 30 trajectories per task, allowing us to collect 1,500 trajectories. The dataset is publicly available https: / //tt austin-rppa rpll gi thub. 1o/ /MUTEX/, and we hope that it will serve as resource for future research in real-world manipulation tasks. The real robot infrastructure is adopted from [61].
Listing 3: GPT4 Prompts for generating the annotations of the task
GPT4 Prompt for generating single text specification
To generate
for text instructions
"I will give you instructione; you should break it down into smaller instructions for a robot. Please do it without adding any extra objects in the environments in the instructions. Break it down into 3-4 instructions, for example, 'pick up the apple': 'Move your gripper towards the apply. Hold the apple gently with your gripper. Lift the apple from the surface." Some points to consider while generating instructions: The robot can only grasp one object at a time. All the objects are in front of the robot, and the robot does not need to move.
Give one such example for every instruction. Each example must complete the task. Here the task instruction (task_instruction))
Prompt for generating multiple text apecifications using the afore generated single text specification
"Replace the words with synonyma or rephrase the sentence. Use words that English -speaking person will use in day-to-day routine and simple English. You can also add words to make it more polite, such as please, carefully, with caution, etc. Give me ten such examples of rephrased paragraphs. Here is the paragraph to rephrase: phtamppl__praggaphh
Prompt for generating speech annotations using text annotations
"You are a person talking to a robot. Rewrite each paragraph with simple English, natural, and polite. Make sure the input is assertive sentence; keep it assertive. If the input is imperative, keep it imperative. Here { text_annotations) such paragraphs, rewrite each paragraph separately:\n"
MUTEX Task Specification Annotations: MUTEX leverages multiple modalities of specifying the task, including text goal, text instructions, video demonstration, image goal, speech goal, and speech instructions. During training, we assume access to all modalities of task specifications for each task, where each modality helps reinforce other modalities of specification. Thus, we can specify the task using any individual modality during evaluation more effectively. We annotate with 11 task specifi- cations per modality per task [Refer to Figure 3 for examples of task specifications). We follow the
17

---

Speaker Code
Speaker Name
Training Speakers
en-IN
Aditi, Raveena
en en-AU
Nicole, Russell
en-US
Joanna, Matthew
en-GB
Amy, Brian
Evaluation Speakers
en-US
Salli
en-GB
Emma
Table 8: Amazon Polly Codes for the speakers in the speech annotations.
standard protocol of 80% train/eval split for the task specification annotations. All the results are reported on the unseen task specification evaluation set. For simulation, to avoid the real2sim gap, we collected the video demonstration using the teleoperation of the robot in simulation [only RGB information is used to specify the task]. For real-world tasks, human demonstrations are used specify the task, allowing a more intuitive way of specifying the task for users than teleoperation. For text specifications, we leverage GPT4 to generate initial task specifications for different tasks with diverse variations. The generated task specifications are corrected by humans to ensure the quality of task specification annotations. For speech specifications, we follow a similar procedure to text annotations and, finally, use the Amazon Polly service to generate speech specifications. The prompt used to create annotations for text and speech specifications provided in Appendix 3, and the details of Amazon Polly characters are provided in Table 8.
6.5 Baseline Details
We compare MUTEX against modality-speciff baselines for each task specification modality, i.e.. Text Goals, Text Instructions, Image Goals, Video Demonstrations, Speech Goals, and Speech In- structions. To highlight the significance of multimodal training proposed in work, we do not change the policy architecture [Refer 6.3], BC training hyper-parameters [Refer 6.2] in baselines. Moreover, to avoid experimental bias due to model size, we iterate over three different model sizes for each modality-speciff baseline with increasing self-attention and tross-attention layers and use the model size with the best performance for the respective modality baseline. The ablated base- lines are highlighted Table The Modality-Speciie SSpecffi baselines in Table use the same pretrained model that of corresponding modality MUTEX. i.e., CLIP [15] for Text Goals, Text Instructions, Image Goals, Video Demonstrations and WHISPER [57] for Speech Goals, Speech Instructions. In Table 3, for T5 (w/ Text Goals) and T5 (w/ Text Instructions).
T5-small [62] model to encode the text specifications similar to language conditioned imitation learning works [2, 33). Sim- ilarly, we use R3M features [63] in R3M (w/ Image Goals) and R3M (w/ Video Demonstratoins). We also compare with VIMA [30], which uses object images and text goals to specify the task. For the VIMA baseline, we append the text goal features with the top six object crop (pad with zeros otherwise) locations obtained from Image Goal using Mask-RCNN [68]. We observe that one single model, MUTEX, consistently outperforms all the baselines in each evaluation modality. This demonstrates that MUTEX training on multimodal task specifications performs better than training with individual or a subset of modalities.
18

---

Modality Specific Baseline
Model
Text Goal
Text Instructions
Image Goal
Video Demonstration
Speech Goal
Speech
size
Instructions
S (13M)
47
47
54
66
34
32
M (14M)
30
30
62
67
20
25
(15M)
33
56
Table Model size ablation baselines: We iterate over different model sizes for each poodallit-sppccilie baseline to avoid experimental bias model size selection and report results using the best one in Table 1. Note that the results are obtained using one seed and averaged over 100 tasks x 20 trials.
Algorithm PseudoCode for MUTEX training.
input: Training data: expert robot trajectories, task specifications tę {L,, 4, V4, 24 St, 8t) for each task T, where
L : Text Instructions
Text Goals
V Video Demonstrations
Image Goals
5 Speech Instructions
Speech Goals
Policy Weights
0. Projection layer weights of modality
C
output: Policy network with updated weights
Step
for epoch = to n do
Observation (o), Expert Actions (az), Task Specifications {L.4, v,
for each batch (o, ae, do
Sample & task specification modalities: t' C
t' apply masking(t") if k >
Predict: (action, masked tokens)
à, f. = (o))
Calculate losses and update policy weights
LBC = - -, Ge. log(ū;)
[L1 regression loss for {V, v, S, cross entropy loss for (L, 1}
mmasked.modellnng L(f,t)ifks if
end end
Freeze all the policy weights except for the projection layer [Refer figure 2] of L, 1, v, S.
Step
for epoch = 1 to n' do
# Observation (o), Expert Actions (a.), Task Specifications
for each batch (o, ae, t) do
Sample k task specification modalities: t' C
# Predict: (action)
TT (o, (0, t')
# Calculate losses and update policy weights OL. 0₁, 0u, 0s, 0, Σ; o,,s log(ā;)
LBc
Laussmooaa matching
L1 (Tor (V), To,, (t'))
end end
19

---

Scene Visualization
Task Description
Average Horizon Length
open the bottom drawer
171
open the top drower
180
open the top drawer and put the bowl in the back of the scene inside
415
open the top drower and put the granols ber inside it put the granole bar in the back compariment of the caddy put the grancla bar in the front compartment of the caddy
426
185
197
tray oven put blue cup pull tray oven put
put blue cup
put the mac cheese
pull tray put the
put the granola bar inside the basket put the granola bar inside the top drawer
135
163
put the pink mug inside the badiect put the red bowl inside the basket
176
150
the pink mug inside the top drawer and close the drawer put the red bowl inside the drawer and close the drawer close the drawer and open the bottom drawer
375
410
277
pull out the tray the oven
182
pull out the tray of oven and put the bowl with hot dogs on the tray
436
pull out the trey of the oven and put the red cup on the tray put the book the back compartment of the caddy put the book the front compartment of the caddy put the red cup the back compartment of the caddy
403
193
199
161
push
the the blue the mug
174
put the bread tray and the the bread white plate
bowl tray white plate put yellow
open the air fryer
208
open the eir fryer and put the bowl with hot dogs in
410
open the air fryer and put the bread
426
put the bowl with hot dogs on the white plate
155
put the bread on the white plate
145
put the pink mug on the white plate
156
open fryer put
open fryer
compartment caddy
put the front compartment caddy put red front compartment of caddy put bowl the back compartment caddy
open the bottom drower open the top drower
157
172
open the top drawer and put the blue mug inside It the top drawer and put the pink mug inside it put the blue
404
432
the white plate
153
the bread the while plate
158
Total number tasks 50
241.9
Figure 5: MUTEX real-world task visualization: MUTEX real-world tasks consist of diverse tasks like "opening the top drawer". "placing the bread oven tray". "putting a bowl consisting of hot dogs inside an air fryer after opening it" representing the complexity of tasks in real home-kkiched environment.
20


Reviews:
[{"review_id": "eS794M1uNyz", "reviewer": "Reviewer_G6C8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a unified architecture for conditioning on task specifications in multiple modalities. \nThe modalities covered are video, audio, text, and images. \nBroadly, the method involves learning a joint representation space for the multiple modalities that can be used for guiding manipulation tasks. \nThe method also suggests matching less information-dense modalities to video demonstrations, which are more information dense. \nThe paper also introduces a real-world dataset with multiple input modalities. \nThe method is evaluated in simulation and on hardware.\nThe authors find that multimodal pre-training improves their performance on instructions given in a single modality, and that both the multimodal pre-training and matching to video data result in performance increases across almost all modalities.", "summary_of_recommendation": "Based on the weakness section in my review, I'm recommending Weak Accept. I think the technical and data contributions of the paper are very strong, but I have qualms about the current presentation, especially about the lack of details which I think would hinder the reproducibility of the method. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "\n# Questions \n- L53 was very hard for me to parse because it uses \"visual specification\" twice in a row; what do you mean here by visual specification? Is that a blanket term for images and video? \n\n- L183: why are gradients not back-propped into the video encoder? Is this because of memory considerations or performance considerations? \n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths\n\n1. **Expanding beyond two modalities**: Most multimodal work focuses on images and text. By expanding out to video and audio and supporting instructions in *any* modality, the proposed method is much more flexible than most multimodal specifications. The authors devote a lot of effort to describing and exploiting the ways in which modalities can be complementary to each other. \n\n2. **Hardware Evaluation**: The hardware evaluation is not just a proof-of-concept demonstration, and the paper includes quantitative results on how the method performed on the hardware. \n\n3. **Dataset**: The real-world data the authors introduce will likely be useful for future work on multimodal task-specifications. \n\n4. **Relevant Ablations**: The authors ablate out the relevant components of their model to support their claim that multimodal training improves performance on the tasks tested. \n\n\n\n## Weaknesses \n1. **Lack of baselines**: the model is evaluated against an ablated version of itself, but there are no external baselines. While this does not detract from the pitch of the paper as a multimodal training method (since the relative changes are important, not the absolute performance) it would be helpful to contextualize the results with other unimodal baselines, at least for the text goals/instructions and image goals, for which baseline systems exist. \nAlong the same lines, it wasn't clear to me how the unimodal baselines were trained. \n\n2. **Lack of details**: There are a number of details which are omitted from the paper, without which I think reproducing the results would be difficult. These include hyperparameter settings like the masking ratio for different modalities, the loss mixture ratios, the specifications of the different encoders, the projection layers, policy encoder, decoder, etc.. This information would be fine to include as supplementary material, but should be provided with the paper somehow. \n\n3. **Framing issues**: There is a bit of divergence between the framing of the paper and the experiments. One goal stated on L131-132 is to use multimodal training to generalize to novel tasks, but there are no experiments to this end. \n\n\n## Comments \n- The loss equation on L158-159 seems unneccessary, since it doesn't actually tell me what the loss between $y_i$ and $\\hat{y}_i$ is. The equation also uses overloaded notation. All this equation tells me is that you averaged the loss across modalities, and you could save a lot of space by just writing that out. \n\n- L226 has a typo or missing sentence in \"...whereas a Kinect Azaure workspace camera.\"", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "c5ANNcIz0J", "reviewer": "Reviewer_JffC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work presents an approach for learning a unified representation from a multimodal task representation and a collection of expert demonstrations for each task. The main premise of the this work is to allow cross-modal information to further enhance the representation required for learning robust policies to solve the downstream tasks from expert demonstrations. The approach, consists of two stages: (1) A mask modeling approach is used to allow for cross-modal information to be encoded in the learned representation; (2) A policy is learned using this representation to solve a task from any or a combination of task representation modalities. The paper then presents experimental results in both real world simulation and real world domains (100 tasks 14 in simulation and 40 tasks in the real world) and demonstrates improved performance when such representations are used.", "summary_of_recommendation": "I understand the technical material well however I do not have a broad knowledge of the relevant work except for a few. Independent of that, I think the claims are broad and the experiments do not provide enough evidence to back the claims. I recommend weak reject.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "(1) \"This loss is optimized in tandem with the policy learning objective (i.e., behavior cloning) so that the representation of the task specification also captures action-relevant information\". Also the pathway exists for this through he combined losses, are there any more results to demonstrate this? For example, video modality for task description might be capturing this by large.\n\n(2) Why not doing \"voice to text, then only use text\". I am wondering how much a raw audio signal would contribute to the unified representation. That is the part I mentioned in the review part where it lacks enough evidence or ablations in terms of representation learning.\n\n(2) \"MUTEX is not only more versatile thanks to its multi76 modal understanding, but it is also more performant, improving the robustness of the task execution on every single modality.\", this is only partially true in a behavior cloning set up. I am not convinced the learned representation is more robust in face of compounding error effect.\n\n(4) \"Our goal is to learn a unified policy that performs a diverse set of tasks ...\". It is more like a unified representation, I do not understand when it is called a unified policy.\n\n(5) \"The goal of each training task Ti is specified by k alternative task descriptions,\". It is not clear how these alternatives are generated and monitored.\n\n(6) \"we recreate these conditions in our training procedure to obtain robust single-modality representations\", where are the results showing this robustness?\n\nEditorial suggestions:\n\n(1) Remove the \"...\"in abstract.\n(2) Need polishing of the terms and phrases such as: \"the information denser one\", ...\n(3) Please : \"we mask out intermediate regions of the features obtained from a pre-trained CLIP model\", please describe what do you precisely mean by \"intermediate regions of the features \"", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is an ambitious effort to bring in multiple modalities of task description for both representation learning and control which is an important idea in Robotics. I also like the fact that it has been utilized in real world robotics experiments. However I have the following issues with this wok: \n\nTechnical: While the concept behind the work is important, it lacks sufficient supporting evidence in the form of experiments and studies to substantiate its claims and they are tested at the surface:\n-- I have some issues with the main premise of the paper. It feels like the paper aims both perception and control which requires a lot more experiments and evidence to back these claims: which one of these is the man claim (it feels like both)?: (1) The unified representation is more rich compared to learning representations based on individual modalities. If this is the claim, then this work lacks ablation experiments and results that demonstrates for example masked training performance in each modality is improved when combined with the other modalities independent of the second stage (control); (2) If the claim is that the unified representation is superior for control, my primary concern is that this work fails to adequately address crucial control issues, such as the compounding error factor present in vanilla behavior cloning.\n\n-- \"novel architecture based on transformers\": the architecture is not that novel, adding and moving around self-attention and cross attention blocks do not necessarily makes it novel. It is essentially a variant of transformer encoder-decoder architecture.\n\n-- Lack of other baselines in experiments: the authors only compare to a baseline of using individual modalities. They could have at least compared it to approaches such as VIMA although different.\n\nPaper organization and presentation: \n(1) The paper's readability and coherence are compromised as it becomes challenging to maintain a consistent context throughout. The inclusion of plain text (e.g., sections 3.1 and 3.2) without accompanying figures, algorithmic view, or visualizations makes it more difficult to comprehend and follow the research. \n(2) Additionally, there are a few readability and grammatical issues which need to be fixed.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "_9gp2XSFb7S", "reviewer": "Reviewer_n91Z", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a novel policy learning framework that allows flexible task specifications, which is an important problem for learning from different modalities. The method proposes a two-stage training method that provide supervision on cross-modality masking and distillations to enable better generalization performance. The method is evaluated on both simulation and real world experiments and demonstrated better performance compared to modality-specific training.", "summary_of_recommendation": "This paper presents a well-thought and clearly written method for solving policy learning with different modalitiy. The protocol and code (if released) are also very useful contribution for the community. There are still several weaknesses regarding experiments and algorithms and would love to hear about the response from the authors. Overall I recommend acceptance of this paper.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Based on the discussions I assume the policy is a multitask policy with 80/20 split on the task-level, and most of the primitives are pick and place? What about human tasks? Are they used in joint training? Are there any sim-to-real gaps in train/test separation? More clarification in the paper would be helpful.\n\n\nFrom the results, it seemed that the “path” instructions for languages, i.e. text instructions, does not provide much improvement gains compared to text goal. This is expected if we do not have enough diversity at the task and language level, but it’s also a bit surprising as the low-level language instructions seem to provide more details. I wonder about the specific details on this experiment, how to rephrase the languages, what encoder is used, and authors’ thoughts on why longer and more detailed instructions do not help much. Moreover, it seems that it’d be hard to provide language labels for simulation data, which often are similar task family and changing objects. I wonder how scaling is done on that direction (40x30 demos seem a lot in the real world). For example, figure 3’s left column seems to have mismatched language and images (it does not have the granola bar). \n\n\nI am still trying to wrap my head around the stage 2, although it’s shown to be important in the experiments. I feel like this process can also be placed before stage 1 to make the encoder of “sparse information” better, and then apply the joint training? The way it’s put seems a bit ad-hoc. The policy (head) is not trained anymore in the second stage, and only the modality feature encoder are trained to match each other in this stage.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strength:\n\n1. The motivation is sound that as humans, we can specify tasks with different methods and can generalize well. Recent vision results such as segment-anything also show the importance of considering multiple ways of prompting to learn a generalizable representation space.\n2. The evaluation is thorough, 100s of tasks with 1000s of trials each.\n3. The cross-masked modeling procedure and the transformer architecture (with pretrained encoders for speech and language instructions) makes sense and seemed to work well.\n\nWeakness:\n1. The current method relies on paired labels for each modality, which is expensive and prohibitive. Would be interested to see if the framework can still work with in-the-wild human demonstration data, or larger train-test gaps.\n2. The overall results seem to indicate a small improvement over only video-conditioned policy, which is, from my understanding, trained from scratch and less practical for test-time.\n3. The overall writing and figures an be improved.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers unanimously vote for accepting the paper and I concur. Some reviewers raised concerns about writing and presentation. G6C8 remarks, "but I have qualms about the current presentation". A closer look at the draft reveals that even the first sentence in the abstract might be incomplete. There are also concerns about reproducibility and comparisons to the baselines that authors presented in the rebuttal. I encourage the authors to address the presentation concerns and include the baseline comparisons in the final version of their paper.
Decision: Accept (Poster)