Any2Policy: Learning Visuomotor Policy with Any-Modality
Authors: Yichen Zhu
Zhicai Ou
Feifei Feng
Jian Tang
Venue: NeurIPS 2024 poster (2024)
Keywords: multi-modal
robot learning
Primary area: robotics

Abstract: Humans can communicate and observe media with different modalities, such as texts, sounds, and images. For robots to be more generalizable embodied agents, they should be capable of following instructions and perceiving the world with adaptation to diverse modalities. Current robotic learning methodologies often focus on single-modal task specification and observation, thereby limiting their ability to process rich multi-modal information. Addressing this limitation, we present an end-to-end general-purpose multi-modal system named Any-to-Policy Embodied Agents. This system empowers robots to handle tasks using various modalities, whether in combinations like text-image, audio-image, text-point cloud, or in isolation. Our innovative approach involves training a versatile modality network that adapts to various inputs and connects with policy networks for effective control. Because of the lack of existing multi-modal robotics datasets for evaluation, we assembled a comprehensive real-world dataset encompassing 30 robotic tasks. Each task in this dataset is richly annotated across multiple modalities, providing a robust foundation for assessment. We conducted extensive validation of our proposed unified modality embodied agent using several simulation benchmarks, including Franka Kitchen, Meta-World, and Maniskill2, as well as in our real-world settings. Our experiments showcase the promising capability of building embodied agents that can adapt to diverse multi-modal in a unified framework.

Any2Policy: Learning Visuomotor Policy with Any-Modality
Yichen Zhu, Zhicai Ou, Feifei Feng, Jian Tang" Midea Group
Abstract
Humans can communicate and observe media with different modalities, such as texts, sounds, and images. For robots to be more generalizable embodied agents, they should be capable of following instructions and perceiving the world with adaptation to diverse modalities. Current robotic learning methodologies often focus on single-modal task specification and observation, thereby limiting their ability to process rich multi-modal information. Addressing this limitation, we present an end-to-end general-pun ceneral-purpose multi-modal system named Any-to-Policy Embodied Agents. This system empowers robots to handle tasks using various modalities, whether in combinations like text-image, audio-image, text-point cloud, in isolation. Our innovative approach involves training a versatile modality net- work that adapts to various inputs and connects with policy networks for effective control. Because of the lack of existing multi-mode robotics datasets for evaluation, we assembled a comprehensive real- world dataset encompassing 30 robotic tasks. Each task in this dataset is richly annotated across multiple modalities, providing a robust foundation for assessment. We conducted extensive validation of our proposed unified modality embodied agent using several simulation benchmarks, including Franka Kitchen and Maniskill2, as well as in our real-world settings. Our experiments showcase the promising capability of building embodied agents that can adapt to diverse multi-modal in a unified framework.
1 Introduction
What is the ultimate form of robots? It should possess the ability to both listen and read, learn from demonstrations, and perceive the three-dimensional world with an understanding of time. While recent advancements in robot learning have explored various modalities for task specification and environmental perception such as text, speech, images, videos, and point clouds most prior research has approached these modalities distinct challenges.
An expanding field of research within artificial intelligence, spanning various disciplines, indicates that simultaneous learning across different modalities, such vision- lision-language [1] 2 55, video-text (6] and vision-touch [12 13] [4 [5]. leads to the development of more comprehensive and effective representations. The enriched representations derived from this approach lead to more profound comprehension of each modality on standalone basis. This multi-faceted concept finds backing in number of cognitive science and psychology studies [16] [17,, which propose that human learning is significantly enhanced when it integrates multiple cues, compared to relying on a single modality in isolation. By adopting this multi-modal approach, which leverages the collective strength of various senses or inputs, we emulate the natural human method of information acquisition and processing, indicating a path toward more integrative methodology in robotics.
7
CCoreesponiigg Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Projection
Image Projection
Image Video
image
Audio Enooddr
Audiio Projection
Emibodied Aligmment
Vicioo Projection
video
Image Eriodder
Image Projection
end-gial
Groud Projection
Point Cloud
point dood
video demonstration
Video Brooder
Viceo Projection
Policy Network
Instruction Modality Encoding
Embodied Alignment & Policy Learning
Observation Modality Encoding
Figure 1: The overall framework of Any2Policy. The Any2Policy framework is structured to handle multi-modal inputs, accommodating them either separately or in tandem at the levels of instruction and observation. We design embodied alignment modules, which are engineered to synchronize features between different modalities, as well as between instructions and observations, ensuring seamless and effective integration of diverse input rypes.
Building on the proven success of multi-modal learning in various disciplines, our goal is develop multi-modal embodied agent capable of processing wide array of instruction and observation modalities. Our method, referred as Any-to-Policy Robot Model (abbreviated as Any2Policy), is illustrated in Figure [T] The Any2Policy architecture is structured in two primary components. Initially, we harness multi-modal encoders to extract relevant features from the input. These features are then transformed into standardized token representations, ensuring uniformity at the neural network level via projection layers. the second component, we introduce embodied alignment, which processes modtll signals, enriched with specific instructions. These signals routed to encoders following projection, ultimately leading to the generation actions for manipulation.
In support of this project, we are releasing a substantial real-worr worldw dataset consisting of 30 tasks, where each task includes 30 trajectories, all annotated with multi modal instructions and observations, mirroring the setup used in our experiments. The purpose of this dataset is to foster and encourage future research in the area of multi-modal embodied agents.
We have demonstrated the efficacy of our Any2Policy approach through rigorous testing against this benchmark. The results clearly indicate that Any2Policy is highly adept at interpreting and responding to a variety of modalities. We further conduct experiments on two simulated robotics benchmarks to reinforce the strong generalizability of our approach compared to existing methods. Our project is at
In summary, contributions are the follows:
We introduce any-to- any-to-policy models that enable a unified embodied agent to process various combinations of modalities, effectively facilitating instruction and perception of the world.
We present novel embodied alignment learning techniques designed to seamlessly align instructions and observations, enhancing both the effectiveness and efficiency of policy learning.
We offer a multi-modal dataset tailored for robotics, encompassing distinct tasks. This dataset covers wide spectrum of modalities in both instruction and observation.
2 Related Work
Multi-Modal Learning. Recent large vision language models (VLMs) use pretrained image encoders as part of the larger model; some pretrain it with supervised classification, some use the pretrained CLIP encoders [18] and some with custom multi-modal pretraining. Inspired by GPT-4, which has been released and demonstrates many advanced multi-modal abilities, recent models present more

---

lightweight versions of VLM that align the image inputs with a large language model using proper
instructional tuning 19 20 22. These vision-language models also showcase many advanced
multi-modal capabilities after the alignment. Another line of research 23
24
25
26
27 28
seeks to demonstrate the transformer's capability to deal with multiple modalities a single neural network and then perform image generation, video-question answering, and multi-turn dialogue, graph learning [29 30, 31 according to the input modalities. Different from these prior works, we focus on robotics. Our design multi-modal input includes both instruction and observation, which potentially makes the robot more generalizable to diverse scenarios.
Embodied agent with Diverse Modalities. The study of embodied Al has been extensively explored through instruction and observation methods. A series of studies concentrate on learning using varied task specifications. This includes conduiiinnodd policy learning [32 [3
(38 visual goal- 58 or combination with
34
35
36
37
32
39 40), instruction following agents 41 42 with 3D observation 443
44
reaching 48 49 [50 [1], video demonstrations [52
53 54
55
56
57
multiple modalities [59] 60 61]. Notably, most systems are specialized for certain task specifications,
with few like VIMA and MUTEX engaging with multiple modalities, albeit focusing a single observation type.
Representation learning is critical in high-dimensional control settings, particularly when managing visual observation spaces. Recent advancements in this field have explored different modalities in feature learning for robotics, especially for images with the unsupervised pretrained visual repre- sentation 662 63 64
65
66
67
68
69 or using visual encoders from pretrained vision-languag
models 77 71 72 7
74
75
76
77 for robotics [78 79 80 33
82 59
84 61]. These
approaches often negle instruction or rely solely on text-based instruction, lacking exploration of diverse instructional modalities.
Our proposed Any2Policy framework differs from previous work by focusing on both instruction and observation levels and utilizing various modalities. This approach aims to enhance learning and inference for embodied agents, offering more holistic and integrated perspective compared to earlier, more modality-specine research.
Robot Manipulation and Datasets. A diverse array of robot manipulation tasks necessitates various skills and task specification formats, including instruction following 332. one-shot imitation (85), rearrangement [86), reasoning 87 88 89] and planning 990 91]. Most existing datasets predomi- nantly feature 2D images for observations and, in some cases, include language instructions to prompt specific robotic actions,
in datasets like Franka-Kitchen [92] and CALVIN [B) VIMA-
Bench [59] introduces a dataset that combines language instructions with images in multimodal prompts. Meanwhile, Maniskill (94) provides 3D visual information, enabling models to interpret point clouds as observations. Our dataset, RoboAny, stands out as the first to support a comprehensive range of modalities in robotics. It encompasses both instructions and observations across images, videos, audio, language. and point clouds, offering unparalleled diversity in robotic task specification and learning.
3 Methodology
Our objective is to develop a comprehensive policy capable of executing a variety of tasks, utilizing a dataset of demonstrations that are annotated with multi-modal instructions and observational data from various sources such as images, videos, point clouds, text, and speech. We acknowledge that the modality of task specifications and observations may differ depending on the specific scenario, and our approach is tailored to adapt flexibly to these varying modalities.
We start with formal description of our settings and provide mathematical annotations. For each , Tn ). we assume that the embodied agents learn from demonstrations obtained through teleoperation, D;
task E (Ti, T2, ..,
(d.....,dd dm). where m is the number of demonstrations for task
T. Each demonstration di presents the form of sequence of observations and expert actions, de = [(s, o, ]. where a E denotes the actions, s E represent instruction, and o E O is the observation.
The instruction of task T; is specified with four modalities: text instructions E (L] L2 Lt}, audio instruction s {A), A3. .......44 AA image goal instruction E {M. Mf. .,, video demonstration E {V. V.,..., V.J. Similarly, the observation of task T., is detailed by
M,} and
3

---

Observation
Imoge
video
point cloud
same
mage Encoddr
Video Encoder
Form: Cloud Erender
Embodied Alignment
structure
Embodied Alignment
Text Eroodder
Teat Projection
Instruction
Ausho Broader
Audio Projection
audio
key
quiry
Image Encoder
Image Pro oction
"olen
Image
Cross- Attentitnn
x N
T Phufro
Video
Self- Attention
modality token
MLP
Figure The architecture of embodied alignment and policy network.
m alternative perceiver, each with three modalities: image E {m mf ... m }. point cloud }, and video o {ul of. "e,,..,,ff o Capital letters are used to denote instructions,
o E while lowercase letters represent observations.
Our objective to develop a multi-task policy, denoted as T(als, which generates continuous actions in response to the current observation and given instruction The policy aims to profi- ciently execute n tasks included the training dataset under new initial conditions. such as varying positions of objects. Moreover, critical aspect of this policy is its capacity to generalize effectively to environments, entities, and tasks that have not been previously encountered or represented in the training data.
3.1 Overall Architecture
pivotal characteristic of an Any2Policy agent is its capacity to leverage multi-modal information throughout its training phase, exhibiting the ability to operate competently when restricted single modality. adapts its comprehension and actions to suit the available sensory input. This versatility brings complexities in crafting multi-moda models, requiring careful consideration in their design to ensure seamless adaptability and robust performance. To address these, we introduce two modules: 1) Multimodal encoders promote cross-modal interactions of all modalities into shared latent space for instruction and observation, individually, and 2) Embodied alignment to take the instruction token and merged with observation tokens via cotssatteenion. We provide overview in Figure
Multimodal Encoder. There are many choices the multi-modal encoders, for instance, CLIP for language and vision transformer for images. Specifically, we denote the encoder mapping function P(.), and for each modality, we need a particular model to extract useful modality-agnoste representation to prepare for policy learning. To make framework more consistent, would like to share the encoder if the instruction and observation have the same modality. Then, we would have five modality encoders for text, audio, image, video, and point cloud, respectively.
Using different backbone models for each modality can be cumbersome and hard maintain. Thus, encoder
we leverage existing well-established models, ImageBind [25] a unified across five modalities, encode inputs of various modalities. With the help ImageBind, we are spared from managing many numbers of heterogeneous modal encoders. Notably, the multi-modal encoders extract semantically meaningful representations from the input modality, which aims to provide essential knowledge to the policy learning networks for accurate manipulation.

---

Embodied Alignment. We leverage projection layer to map different modalities' feature represen- tations from P(z) into unified representations. In particular, we want avoid heavy computational costs during training, so that we freeze all the layers in ImageBind, and only keep the projection layers learnable. This operation kept the same for both the instruction segment and the observation segment.
As result, the projection layers ensure that the tensor of different modalities are in the same form. Nevertheless, the process is fraught with challenges, 1) aligning distinct modalities within a segment, particularly when diverse types such as images and point clouds coexist, difficult. Simultaneously, it's 's vital to control computational costs, which can escalate with the addition of modalities, and 2) the integration of instructions with observations complex. This complexity arises partly because instructions and observations don't It always correspond one-to-one; often, single instruction relates to multiple observation frames, and for most time steps, observations exist without accompanying instructions. Additionally, the modality gap between instruction and observation can be an obstacle in creating effective feature representations for manipulation tasks.
To address the first challenge, we employ a transformer architecture. Each modality undergoes tokenization, with the token count varying between modalities. For example, images and videos typically require more tokens to encapsulate visual information. We use 81 tokens for images, 81 x t for videos (where represents the number of frames), and 256 tokens for point clouds. For text and audio, a single token is generally sufficient due to their shorter length. It's important to note that the computational demands of self-attention layers increase quadratically with the number of tokens, leading to potentially high inference costs for visual modalities. Therefore, it is crucial to use model
compression methods 995 96 97
98 99
100
101
102
103 104
to reduce the computational cost.
We follow RT-1 [35] to utilize TokenLearner [05] for pruning redundant visual tokens. We apply TokenLearner individually to each visual modality, such as images, videos, and point clouds. After token pruning, the token count for images and videos is reduced to 8, while for point clouds, it's brought down to 16. These tokens are then amalgamated, using a modality token to distinguish between different modalities. We insert a modality token between every two modalities. We also use absolute position embedding maintain the token order. Consequently, the maximum token count, when integrating all three visual modalities, is significantly reduced to as few as 34 (8+8+16+2), substantially enhancing computational efficiency. For the transformer model, we use one Transformer networks.
block with self-attention and
The second major challenge is how to align instruction with observation. We seek the help of cross- attention. Specifically, when we are presented with an observation sequence, Po. and an instruction sequence, Pa. we project P. into the key (K) and P, into the value (V). We create a set number of learnable query (Q) embedding as input for the cross-attention. This learnable query embedding aims adapt different modalities for efficient fusion between instruction and observation. This setup enables us to align the two tensors through a cross artss-attention mechanism, mathematically represented as:
QiKT Vdh
h, = Softmax
Vi)
In this equation, d, denotes the dimensionality, and the construction of the query, key, and value components aligns with the standard framework utilized in BERT [106). The elegance of the method lies in its ability to effectively incorporate relevant representations from the instruction into the observation, thus ensuring a more coherent and informative interaction between the two. In our experiments, we use 16 queries, where the dimension of queries corresponds to the cross-attention layers.
Policy Networks. Designing a multi-task policy comes with its share of challenges. and one critical aspect involves choosing an appropriate conditioning mechanism. In our framework, the policy networks are conditioned based on the observation sequence and occasionally an instruction sequence. As aforementioned, this conditioning is achieved through a series of tross-attention layers, which establish connections between the policy network and the input modalities. Each Transformer block comprises self-attention, coss-attentionn and a feed-forward network. Additionally, we incorporate residual connections that link the higher layers with the input rollout trajectory sequence. We append one history action token to the model, similar to VIMA approach (59] It is important to note that the instruction aligns with the observation at every time step. To optimize efficiency, we store the instruction's 's representation to avoid repetitive computations. The decoder is constructed using L
5

---

Workspace Setup
"Pick cube place to left box"
"Pick up place right
"Pick up side
red the middle"
"Open top
"Plick toy polar bear place to
"Pick up toy zebra and place box"
'Pick cube"
Figure 3: This the setup of our Franka real robot. We have compiled examples of several tasks. To facilitate better understanding for our readers, we provide only the language-based versions of these task descriptions.
Transformer blocks. Finally, we append Multi-layer Perceptron (MLP) layer to generate continuous actions. We use behavior cloning mean squared loss the optimization objective.
3.2 Dataset Construction
In our pursuit to develop robot model that allows input with any modality, we have established new dataset featuring range tasks, each accompanied by multiple instructions and observations tailored to real-world scenarios. We developed two workstations, one of which is depicted in Figure 3 This dataset comprises actions, such as "pick up the yellow cube," and "place the toy bear in the blue box to the right," to more complex contact-rich tasks like "open the drawer" as well as tasks demanding to reason, for instance, "sort the cube with the same color." Although all tasks are executed within the same environment, they involve variety of objects selected from a set 70. Each task exemplified through m 30 human-collected trajectories. Further, every task is annotated with A- distinct instructions. To generate these diverse instructions, we utilized GPT-4, creating prompts formulate alternative descriptions. We have carefully filtered these descriptions to ensure they accurately represent the specific task-relevant objects, avoiding any potential confusion due to synonyms. Additionally, we have incorporated speech signals from various characters of the Amazon Polly service to enrich our dataset with diverse speech descriptions. In terms of observation, we recorded the video during data collection process. The depth information captured in these recordings into point cloud data, enabling detailed analysis and application of the collected data. All data are collected via teleoperation. For long-horizon tasks, operated the robot move back the initial state and then start the next move. In the appendix, we provide detailed descriptions our hardware for workstations and more qualitative examples of collected data.
distinct tasks. These tasks vary from straightforward pick-and- and-place
then converted
4 Experiments
This section aims demonstrate the generalizability of our proposed Any2Policy agent. We evaluate our method two setups. Initially, we conduct experiments a real-world setting using our own collected dataset, which comprises various modalities. This is aimed at demonstrating the efficacy

---

Table I: Comparison with modality-ppeciffe models in real-world experiments.
Instruction Observation
Text -3 Image
Text
Video
Text Point Cloud
Audio Image
Audio
Video
Audio
Point Cloud
AnylPel
55
57
Huedalit-Sppeciia
48
50
Instruction Observation
Inage ++ Image
Image
Video
Inage + Point Cloud
Video + Inage
Video -+ Video
Video - Point Cloud
Any2Policy
57
36
ModalitySSpeelln
46
28
Table Comparison with Any2Policy framework without embodied alignment in real-world experi- ments
Instruction
Image
Video
Text Point Cloud
Image
Audio
Video
Audio
Cloud
Any2Pulicy
57
Embodied Alignment
Instrution Observation
Image -> Image
Image -> Video
Image
Point Cloud
Video -1 Image
Video - Video
Video -> Point Clood
Payley
56
63
57
36
Embodied Alignment
15
29
28
11
of our approach across a wide range of instruction and observation modalities. Subsequently, we assess the performance of our method on three simulated benchmarks. Each benchmark is limited to one or small number of modalities. This phase aims to establish that, by leveraging training on cross-modal information, our method can still deliver competitive or superior results compared to standard benchmarks, even in the presence of single modality.
4.1 Real-World Evaluation
Evaluation Sctup. We conduct our evaluations using a newly constructed dataset of multimodal instruction and observation. The dataset is divided into training. validation, and testing subsets, with a split of 7/1/2, respectively. We position the objects randomly, allowing tasks to be performed on unseen objects.
Implementation details. We use an initial learning rate of 3e-5 with the AdamW W [007] optimizer, weight decay of le-6, and a linearly decaying learning rate scheduler with warm-up covering the initial 2% of the total training time [088]. We apply gradient clipping of 1.0. All experiments are evaluated over 10 trials to obtain the mean success rate. The action space is the absolute joint position.
Experiment Results. We aim to answer the following questions with our experiments.
1. Does Any2Policy outperform single-modality trained methods? We compare Any2Policy with modality-speciif approaches in Table I In particular, all implementations and network architecture remain constant across both settings, except that Any2Policy is trained variety of modalities while the modality-speciffi method is limited to particular ones. We observe that Any2Policy consistently achieves superior performance across all instruction-observation pairs compared to the modality- specific approach. Given that the total volume of training data is identical due to fixed training steps, this enhanced performance is attributed to learning from multiple modalities, which aids in better model generalization.
2. The significance of embodied alignment. The impact of embodied alignment is specifically explored in our experiments presented in Table We experiment by removing the embodied alignment module and simply concatenating the outputs of projection layers from different modalities. TokenLearner is still utilized to manage the memory burden of numerous tokens. The alignment between instruction tokens and observation tokens is then achieved using MLP layer, followed by the policy network. Notably, there a significant performance drop when embodied alignment is omitted.
3. Does Incorporating More Modalities Enhance Performance? A key advantage of the Any2Policy framework its ability to outperform modality-specific models by training on multiple modalities for instraction-observatte inference stage further improve performance? Table 4 presents our experimental findings. We used the text-image pair as the baseline, common setup in robotics. Adding additional instructional
pairs. It raises the question: can including more modalities at the
7

---

Table 3: Comparison with state-of-the-art robotic models in real-world experiments.
Instruction - Observation
Image
Image
Text -> Image
Video - Image
Image + Text Image
VIMA 59
R3M 109
42
46
T5 110
39
Any2Policy
44+2
51
59+13
62+13
Table 4: Ablation study on the effect of using different modalities in real-world experiments.
Method
Instruction
Observation
Success Rate
Text
Image
Audio
Image
52
Any2Policy
Auxi-Auei End-Goal
Image
60
Text+Audio+Inage End-Goal + Video Demonstration
Image
62
Text
Image
51
Any2Policy
Text
Image+Video
57
Text
Image+Video+Point Cloud
66
Text
Image
51
Any2Policy
Video Demonstration
Image+Video
63
Text+Image End-Goal
Image+Point Cloud
69
modalities, such as audio, image-based end-goals, and video demonstrations, consistently increased the success rate. Interestingly, although text and audio convey similar instructions, their combination still yielded slight performance improvement of 1%. For observations, significant performance gains were observed when adding video and point cloud data, integrating temporal and 3D spatial information. Combining multiple instructional and observational modalities also improved performance; for example, incorporating video demonstrations in instructions and video in observations led to a 12% increase in the success rate.
Comparison of Any2Policy with state-of-the-ar rohot models. To further assess the ef- fectiveness of Any2Policy representations, compare them with other robot models, shown in Table 3 Any2Policy consistently outperforms both T5 [110) and R3M [109] across various modalities, highlighting the advantage of incorporating multiple modalities during training. Furthermore, Any2Policy significantly surpasses VIMA [59], recent method that uses text goals and object images for task specification. This demonstrates that Any2Policy not only outperforms modale counterparts, but is also highly effective in employing multiple modalities during inference.
Other results. We observe that the performance of image-point cloud and video-point cloud pairings achieves relatively lower scores compared to other modality pairs. However, when we use text or audio instruction, the performance of point cloud is typically better than image or video due to its accurate position in three-dimensional space. This indicates that there is still room improve model performance with point cloud inputs. These results highlight the inherent complexity of integrating visual modalities like images and videos with point cloud data, which often requires sophisticated fusion techniques to effectively combine spatial information with traditional 2D features.
4.2 Simulation Evaluation
Our experiments are based on the following three datasets in our simulation. These three datasets represent three different instraction-observatioss modalities. Specifically, Franka Kitchen [92] uses text-image and ManiSkil12 [94] uses text-image and text- point cloud).
Dataset.

---

110
BR
2
EmbodeedGP
a Any2Policy (Ours)
Success
70
50
Cabinet Left
Cabinet Right
Micro Left
Micro Right
Knoha Left
Knobs Right
Light Left
Light Right
Sdoor Left
Sdoor Right
Performance comparison in Franka Kitchen with 10 demonstrations
110
RBM BLIP-2
EmbocledGPP
Any2Polley [Ours]
Siccees (%)
70
30
Cabinet Left Cabinet Right
Micro Left
Micro
Knobs
Knobs
Light Left
Light Right
Sdoor Left
Sdoor Right
Performance comparison
Franka Kitchen with 25 demonstrations
Figure 4: Performance of Any2Policy in Franka Kitchen with 10 or 25 demonstration demos. Comparison with R3M, BLIP-2, and EmbodiedGPT. On all tasks except for Knobs-left with 25 demonstrations, we obtained superior performance over SOTA methods.
The objective of our simulated environments is to evaluate the efficacy of our proposed method under conditions where modalities are limited.
Implementation Details. All models are trained A100 GPUs, implemented in PyTTorch |11 The Franka-Ko Kitchen are trained for 40K steps. We use weight decay of 1e-6, cosine learning rate scheduler with warmup steps of 2% total steps. The gradient clip of 1.0 is also applied. We use Adam optimizer with initial learning of le-3 and 3e-4 for Franka Kitchen and Maniskill-2. Note that Maniskill-2 does not have text instructions. We augment the task description into instruction via GPT-4. then tokenized and prepend with image backbone via FiLM 112].
Evaluation: We evaluate our approach using 30 rollouts from the BC learned policy. We the mean success rate of the final policy average the mean success rate across camera configurations.
metrie. When providing task suite metrics
Experimental results on Franka Kitchen, In our experiments, we benchmarked our model against two offte-ff-the-rt methods: R3M [109], and BLIP-2 (22). Embodied-GPT [13], a multi-modal model designed for robotics. Our policy network was trained using few-shot learning, utilizing either ten or twenty-five demonstrations. We assessed the success rate of these models in 100 randomized evaluations across five different tasks in each benchmark. These evaluations were conducted under settings, each with five separate runs and from two different camera perspectives, using only visual observations. The results, illustrated in Figures 4 for the Franka Kitchen, clearly show that Any2Policy outperforms the baseline methods, on both 10 and 25 demonstrations, except for Knobs-left with 25 demonstrations.
vision-language model, and
5 Conclusion
Humans interact with and perceive the world through information from multiple modalities. Existing research in robotics predominantly concentrates policy learning using single modality. In contrast, this paper introduces the Any2Policy framework, which adeptly demonstrates the integration of various sensory modalities into cohesive model. This framework efficiently processes and responds to modtil data for robotic tasks. The overall framework, coupled with its multi-modal dataset, represents significant advancement in the field of embodied AI. Our findings underscore the potential of Any2Policy to augment robotic adaptability and performance, thereby highlighting the importance of multi-modal methodologies in the development of generalized and versatile robotic systems.

---

References
[1] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed EI Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104 120. Springer, 2020.
[2] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint
2019.
[3] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark. al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 7748-8762 PMLR, 2021.
[4] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision align- ment model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15638-15650, 2022.
[5] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-languan representation learning with noisy text supervision. In International conference on machine learning, pages 4904-4916. PMLR, 2021.
[6] Hu Xu, Gargi Ghosh, Po- Po-Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feicht- enhofer, Florian Metze, and Luke Zettlemoyer. Vlm: Task-agnostic video-language model pre-training for video understanding. arXiv preprint arXiv: 105.005509999 2021.
[7] Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. Univl: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint arXiv:2002.06353 2020.
[8] Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu. Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowl- edge through vision and language and sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16375-16387, 2022.
[9] Linchao Zhu and Yang. Actbert:
Learning global-local video-text representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8746-8755, 8755, 2020.
[10] Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint arXiv:2106. 11097, 2021.
[11] Hu Xu, Gargi Ghosh, Po- Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre- pre-training for zero- shot video-text understanding. In Proceedings of the 202 Conference on Empirical Methods in Natural Language Processing, pages 6787-6800, 2021.
[12] Michelle A Lee, Yuke Zhu, Krishnan Srinivasan, Parth Shah, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In 2019 International Conference on Robotics and Automation (ICRA), pages 8943-8950. IEEE, 2019.
[13] Yunzhu Li, Jun-Yunnn Yan Zhu, Russ Tedrake, and Antonio Torralba. Connecting touch and vision via cross-modal prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 10609-10618, 2019.
[14] Irmak Guzey, Ben Evans, Soumith Chintala, and Lerrel Pinto. Dexterity from touch: Self-supervised pre-training of tactile representations with robotic play. arXiv preprint arXiv:2303.12076 2023.
10

---

[15] Edward Smith, David Meger, Luis Pineda, Roberto Calandra, Jitendra Malik, Adriana Romero Soriano, and Michal Drozdzal. Active 3d shape reconstruction from vision and touch. Advances in Neural Information Processing Systems, 34: 16064-16078, 2021.
[16] James M Clark and Allan Paivio. Dual coding theory and education. Educational psychology review, 3:149-210, 1991.
[17] Lorraine E Bahrick and Robert Lickliter. Intersensory redundancy guides attentional selectivity and perceptual learning in infancy. Developmental psychology, 1602))190, 2000.
[18] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pages 8748-8763. PMLR. 2021.
[19] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprini arXiv:2304. 10592, 2023.
[20] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lec. Improved baselines with visual instruction tuning. arXiv preprint arXIv:2310.03744, 2023.
[21] Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023.
[22] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint urXiv:2301.12597 2023.
[23] Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction audio-visual language model for video understanding. arXiv preprint arXiv:2306.0288 028588 2023.
[24] Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo. et al. Imagebind-llm: Multi-modality instruction tuning. arXiv preprint arXiv:2309 203900000000 2023.
[25] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180-15190, 2023.
[26] Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. Meta-trane ceta-trrasformer: unified framework for multimodal learning. arXiv preprint urXiv:2307.10802 2023.
[27] Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et Point-bind point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint Xiv:233009 00615, 2023.
[28] Shenggiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any 2023.
multimodal 1lm. arXiv preprint
[29] Xinke Jiang, Rihong Qiu, Yongxin Xu, Wentao Zhang, Yichen Zhu, Ruizhe Zhang, Yuchen Fang, Xu Chu, Junfeng Zhao, and Yasha Wang. Ragraph: A general retrieval-augmmened graph learning framework. NeurIPS, 2024.
[30] Xinke Jiang, Zidi Qin, Jiarong Xu, and Xiang Ao. Incomplete graph learning via attribute- structure decoupled variational auto-encoder. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 304-312, 2024.
[31] Xinke Jiang, Ruizhe Zhang, Yongxin Xu, Rihong Qiu, Yue Fang, Zhiyuan Wang, Jinyi Tang, Hongxin Ding, Xu Chu, Junfeng Zhao, et al. Hykge: A hypothesis knowledge graph enhanced framework for accurate and reliable medical llms responses. arXiv preprint arXiv:2312. 15883, 2024.
11

---

[32] Simon Stepputtis, Joseph Campbell, Mariano Phielipp, Stefan Lee, Chitta Baral, and Heni Ben Amor. language-conditonn imitation learning for robot manipulation tasks. Advances in Neural Information Processing Systems, 33:13139-13153 2020.
[33] Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, et al. Learning language- conditioned robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning, pages 1303-1315. PMLR, 2022.
[34] Lin Shao, Toki Migimatsu, Qiang Zhang, Karen Yang, and Jeannette Bohg. Concept2robot: Learning manipulation concepts from instructions and human demonstrations, The Interna- 2021.
tional Journal of Robotics Research, 40(12-14):1419-444
[35] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Roboties transformer for real-world control at scale. arXiv preprint w2xiv:2212.0666 2022.
[36] Pierre-Louis Guhur, Shizhe Chen, Ricardo Garcia Pinel, Makarand Tapaswi. Ivan Laptev, and Cordelia Schmid. Instruction-driven history-award policies for robotic manipulations, In Conference on Robot Learning. pages 175- 787 PMLR, 2023.
[37] Stephen James, Zicong Ma. David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019-3026, 2020.
[38] Oier Mees, Lukas Hermann, and Wolfram Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters, 7(4): 1al:11225-112222 2022.
[39] Minjie Zhu, Yichen Zhu, Jinming Li, Junjie Wen, Zhiyuan Xu, Ning Liu, Ran Cheng, Chaomin Shen, Yaxin Peng, Feifei Feng, et al. Scaling diffusion policy in transformer to I billion parameters for robotic manipulation. arXiv preprint arXiv:2409. 14411, 2024.
[40] Lucy Xiaoyang Shi, Zheyuan Hu, Tony Z Zhao, Archit Sharma, Karl Pertsch, Jianlan Luo, Sergey Levine, and Chelsea Finn. Yell your robot: Improving on-the-fly from language corrections. arXiv preprint arXiv:2403. 12910, 2024.
[41] Yixuan Wang, Zhuoran Li, Mingtong Zhang, Katherine Driggs- Triggs-Campbell, Jiajun Wu, Li Fei- Fei, and Yunzhu Li. D 3 fields: Dynamic 3d descriptor fields for zero-shot generalizable robotic manipulation. arXiv preprint arXiv:2309.1611 2023.
[42] Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.00 00233077.099..... 2023.
[43] Shizhe Chen, Ricardo Garcia, Cordelia Schmid, and Ivan Laptev. Polarnet: 3d point clouds for language-guided robotic manipulation. arXiv preprint arXiv:2309. 15596, 2023.
[44] Theophile Gervet, Zhou Xian, Nikolaos Gkanatsios, and Katerina Fragkiadaki. Act3d: 3d feature field transformers for multi-task robotic manipulation. In 7th Annual Conference on Robot Learning, 2023.
[45] Yanjie Ze, Ge Yan, Yueh-Hua Wu, Annabella Macaluso, Yuying Ge, Jianglong Ye, Nicklas Hansen, Li Erran Li, and Xiaolong Wang. Gnfactor: Multi-task real robot learning with generalizable neural feature fields. In Conference Robot Learning, pages 284-301. PMLR, 2023.
[46] Yanjie Ze. Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy, arXiv preprint M.Xi::243300355. 2024.
[47] Ge Yan, Yueh-Hua Wu, and Xiaolong Wang. Dnact: Diffusion guided multi-task 3d policy learning. arXiv preprint ArXi:22403.04115. 2024.
12

---

[48] Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A Efros, and Trevor Darrell. Zero-shot visual imitation. In Proceedings of the IEEF conference computer vision and pattern recognition workshops, pages 2053, 2018.
[49] Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goal- conditioned policies. Advances in Neural Information Processing Systems, 32, 2019.
[50] Zichen Jeff Cui, Yibin Wang, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. From play to policy: Conditional behavior generation from uncurated robot data. arXiv preprint arXiv:2210. 10047, 2022.
[51] Yuchen Cui, Scott Niekum, Abhinav Gupta, Vikash Kumar, and Aravind Rajeswaran. Can foundation models perform zero-shot task specification for robot manipulation? In Learning for Dynamics and Control Conference, pages 893-905. PMLR, 2022.
[52] Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak. Affordances from human videos as versatile representation for robotics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13778-13790, 2023.
[53] Gedas Bertasius. Hyun Soo Park, Stella X Yu, and Jianbo Shi. First person action-object detection with egonet. arXiv preprint arXiv: /6030040004 2016.
[54] Shikhar Bahl, Abhinav Gupta, and Deepak Pathak. Human-to-robot imitation in the wild. arXiv preprint 000v:2200.009550, 2022.
[55] Annie Chen, Suraj Nair, and Chelsea Finn. Learning generalizable robotic reward functions from" tethe--wid" human videos. arXiv preprint (ttp::2013..61111.1 2021.
[56) Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. I'ime-contaastive networks: Self-supervised learning from video. In 20/8 IEEE international conference robotics and automation (ICRA), pages 1134 1 141. IEEE, 2018
[57] Pratyusha Sharma, Deepak Pathak, and Abhinav Gupta. Third- visual imitation learning via decoupled hierarchical controller. Advances in Neural Information Processing Systems, 32, 2019.
[58] Laura Smith, Nikita Dhawan, Marvin Zhang, Pieter Abbeel, and Sergey Levine. Avid: Learning multi-stage tasks via pixel-level translation of human videos. arXiv preprint arXiv:1912.044 2019.
[59] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:22 10.03094, 2022.
[60] Rutav Shah, Roberto Martín-Martin, and Yuke Zhu. Mutex: Learning unified policies from multimodal task specifications. arXiv preprint arXiv:2.30. 2023.
[61] Yichen Zhu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. Retrieval-augmentee embodied agents. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages I 17985-17995, 2024.
[62] Zhecheng Yuan, Zhengrong Xue, Bo Yuan, Xueqian Wang, Yi Wu, Yang Gao, and Huazhe Xu. Pre-trained image encoder for generalizable visual reinforcement learning. Advances in Neural Information Processing Systems, 35:13022-13037 2022.
[63] Nicklas Hansen, Zhecheng Yuan, Yanjie Ze. Tongzhou Mu. Aravind Rajeswaran, Hao Su, Huazhe Xu, and Xiaolong Wang. On pre-training for visuo-motor control: Revisiting a farming-from-scratel baseline. arXiv preprint arXiv:22/2.057445 2022.
[64] Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsur- prising effectiveness pre-trained vision models for control. In International Conference on Machine Learning, pages 17359-17371, PMLR, 2022.
13

---

[65] Kaylee Burns, Tianhe Yu, Chelsea Finn, and Karol Hausman. Pre-training for manipulation: The case for shape biased vision transformers.
[66] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self- self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages
2021.
[67] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez. Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07198, 2023.
[68] Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Danilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109. 3396, 2021.
[69] Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du Chongyi Zheng, Tony Zhao, Philippe Hansen-Esstruch, Quan Vuong, Andre He, et al. Bridgedata v2: A dataset for robot learning at scale. arXiv preprint arXiv:2308. / /2952, 2023.
[70] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[71] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lec. Improved baselines with visual instruction tuning, 2023.
[72] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[73] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. beepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403. 005225, 2024.
[74] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308. /2966, 1(2):3, 2023.
[75] Yichen Zhu, Minjie Zhu, Ning Liu, Zhicai Ou. Xiaofeng Mou, and Jian Tang. Llava-o: Efficient multi-modal assistant with small language model. arXiv preprint arXhv:4401.02330, 2024.
[76] Minjie Zhu, Yichen Zhu, Xin Liu, Ning Liu, Zhiyuan Xu, Chaomin Shen, Yaxin Peng, Zhicai Ou, Feifei Feng, and Jian Tang. A comprehensive overhaul of multimodal assistant with small language models. arXiv preprint 1XX::22433306199, 2024.
[77] X Liu, Zhu, J Gu, Y Lan, C Yang, and Y Qiao. Mm-safetybench: A benchmark for safety evaluation of multimodal large language models. arXiv preprint arXiv:2311.17600 4(1):44, 2023.
[78] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785-799. PMLR, 2023.
[79] Felix Hill, Sona Mokra, Nathaniel Wong, and Tim Harley. Human ceruuctioo-folloiiin with deep reinforcement learning via transfer- transfer-learnin from text. arXiv preprint arX1v:2005.09382 2020.
[80] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Be-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pages 991-1002. PMLR, 2022.
[81] Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint arXiv:2005.07645 2020.
[82] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06115. 2022.
14

---

[83] Jinming Li, Yichen Zhu, Zhiyuan Xu, Jindong Gu, Minjie Zhu, Xin Liu, Ning Liu, Yaxin Peng. Feifei Feng, and Jian Tang. Mmro: Are multimodal Ilms eligible as the brain for in-home robotics? arXiv preprint arXiv:2400 19693, 2024.
[84] Junjie Wen, Yichen Zhu, Jinming Li, Minjie Zhu, Kun Wu, Zhiyuan Xu, Ran Cheng, Chaomin Shen, Yaxin Peng, Feifei Feng, et al. Tinyvla: Towards fast, data-efficient vision-language- action models for robotic manipulation. arXiv preprint arXiv: /2514, 2024.
[85] Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAl Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. Advances in neural information processing systems, 30, 2017.
[86] Dhruv Batra, Angel X Chang, Sonia Chernova, Andrew J Davison, Jia Deng, Vladlen Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, et al. Rearrangement: A challenge for embodied ai. arXiv preprint arXiv:2011.0197. 2020.
[87] Minjie Zhu, Yichen Zhu, Jimming Li, Junjie Wen, Zhiyuan Xu, Zhengping Che, Chaomin Shen, Yaxin Peng, Dong Liu, Feifei Feng, and Jian Tang. manipulation with fast and slow thinking. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 4333-4339 2024.
robotic
[88] Junjie Wen, Yichen Zhu, Minjie Zhu, Jinming Li. Zhiyuan Xu, Zhengping Che, Chaomin Shen, Yaxin Peng, Dong Liu, Feifei Feng, and Jian Tang. Object-centric instruction augmentation for robotic manipulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 4318 4325, 2024.
[89] Wenlong Huang. Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207. 056688, 2022.
[90] Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra Ian Reid, and Niko Suender- hauf. Sayplan: Grounding large language models using 3d scene graphs for scalable task planning. In 7th Annual Conference on Robot Learning, 2023.
[91] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.00 204.01691, 2022.
[92] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072111 072219,, 2020.
[93] Oier Mees, Lukas Hermann, Erick Beas, and Wolfram Burgard. Calvin: A benchmark for language-conditiong policy learning for loog-hooizze robot manipulation tasks. IEEE Robotics and Automation Letters, 7(3):7327-7334, 2022.
[94] Jiayuan Gu, Fanbo Xiang, Xuanlin Li. Zhan Ling, Xigiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, et al. Maniskill2: A unified benchmark for generalizable manipulation skills. arXiv preprint
2tt.:233020446533 2023.
[95] Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10809-10818 2022.
[96] Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. Learned token pruning for transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 784-794, 2022.
[97] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynam- ievit: Efficient vision transformers with dynamic token sparsification. Advances neural information processing systems, 34: 34:13337-13949, 2021.
[98] Siyuan Wei, Tianzhu Ye, Shen Zhang, Yao Tang, and Jiajun Liang. Joint token pruning and squeezing towards more aggressive compression of vision transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2092-2101, 2023.
15

---

[99] Yulin Wang, Rui Huang, Shiji Song, Zeyi Huang, and Gao Huang. Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition. Advances neural information processing systems, 34:1 1960-191900 1973, 2021.
[100] Youwei Liang, Chongjian Ge, Zhan Tong. Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations. arXiv preprint arXiv:2202. 07800, 2022.
[101] Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint urXiv:2210.09461, 2022.
[102] Qiqi Zhou and Yichen Zhu. Make a long image short: Adaptive token length for vision transformers. Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 69 85. Springer, 2023.
[103] Yichen Zhu, Ning Liu, Zhiyuan Xu, Xin Liu, Weibin Meng, Louis Wang, Zhicai Ou, and Jian Tang. Teach less, learn more: On the undistillable classes in knowledge distillation. Advances in Neural Information Processing Systems, 15:32011-32024 2022.
[104] Yichen Zhu, Qiqi Zhou, Ning Liu, Zhiyuan Xu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. Scalekd: Distilling scale-aware knowledge in small object detector. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 19723-19733, 2023.
[105] Michael Ryoo, AJ Piergiovanni, Anurag Arna,, Mostafa Dehghani, and Anelia Angelova. Tokenlearner: What can learned tokens do for images and videos? arXiv preprint arXiv:2106. (tti::2006.129997 11297, 2021.
[106] Jacob Devlin, Ming- Wing-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rX.v:1810004805, 2018.
[107] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
[108] Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. A closer look deep learning heuristies: Learning rate restarts, warmup and distillation. arXiv preprint arXiv:1810.1322 2018.
[109] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601, 2022.
[110] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1)55885-551, 2020.
[111] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, pirf-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[112] Ethan Perez, Florian Strub. Harn De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
[113] Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. Embodiedgpt: Vision-language pre-training via embodied chain of thought. arXiv preprint arXiv:2305. /5021, 2023.
16

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction concisely outline our framework for retrieving information from video enhance visuomotor learning. Our experiments support that our framework improves the generalization of the robot model.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
17

---

Justification: Our method contains strong assumptions that affordance in human video is similar to manipulation. Therefore, our proposed method may not benefit non- non-human like embodiment such as locomotion learning.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well vell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
We have not theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the necessary information to reproduce our method, including the training details, the dataset we used, the implementations, and the setup of our environment for the real robot.
18

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel urchitecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justfication:
The data will be attached in the webpage.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
hhipss.c//ublic/uuddds wttpee///oooeSeeeeeoeoeoooo
www.obiiiiiiioooooooiiooo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
19

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Because the evaluation of robot manipulation is extremely time-consee tome-consuming. we report the average success rate over 20 trails on our real-world experiments. For the simulation environment, we follow previous methods which report the average success rate over 30 trails. Details are specified in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the computer resources.
20

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We follow the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We specify the potential positive/negative societal impacts in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
21

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We properly cite the assets and codes that we used in our work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
22

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "gJBi5TFVVj", "reviewer": "Reviewer_h8z5", "summary": "The paper aims to enhance the generalizability of robotic agents by enabling them to handle tasks using diverse modalities such as text, audio, images, and point clouds. The authors introduce a multi-modal system named Any-to-Policy, which utilizes a versatile modality network to adapt various inputs and connects with policy networks for effective control. To support this approach, they assembled a comprehensive dataset covering 30 robotic tasks annotated across multiple modalities. Extensive experiments demonstrate the system's capability to adapt to diverse multi-modal inputs in both simulated and real-world settings, showcasing its potential to improve the generalizability and performance of embodied agents in various scenarios when comparing various baselines.", "questions": "See the section above.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper introduces a novel approach to multi-modal robotic learning with the Any2Policy framework. This framework allows for the integration and processing of diverse modalities such as text, audio, images, and point clouds in a unified manner. In my opinion, the originality seems to lie in the seamless combination of these modalities, enabling robots to handle a wide variety of tasks with greater adaptability. The assembly of a comprehensive real-world dataset annotated across multiple modalities is also a significant and innovative contribution, addressing the scarcity of such datasets in the field.\n\nMoreover, the authors provide detailed descriptions of their framework, including the use of multi-modal encoders and embodied alignment techniques. The experimental setup is relatively comprehensive, encompassing both simulated benchmarks and real-world scenarios, which validates the effectiveness of the proposed approach.\n\nFinally, the paper addresses an important challenge in robotic learning: the ability to process and integrate information from multiple modalities. By demonstrating the capability of the Any2Policy framework to generalize across various tasks and modalities, the paper opens up new possibilities for the development of more generalizable and adaptive robotic systems. The release of the multi-modal dataset will likely push further research in this area, which is important for future work along this direction", "weaknesses": "1. While the assembly of a multi-modal dataset is a significant contribution, the paper could improve by providing more detailed information on the diversity and representativeness of the tasks and scenarios included in the dataset. For instance, it would be beneficial to know more about the variation in objects, environments, and task complexities. This information would help to assess whether the dataset sufficiently covers the wide range of scenarios the framework is intended to handle.\n\n2. Although the paper compares the Any2Policy framework with several state-of-the-art models, it could benefit from a more extensive comparative analysis. For instance, including a wider variety of baseline models e.g. RT-1, RT-2, RT-X, Octo, OpenVLA etc. and providing a detailed discussion on the differences in performance could help to better highlight the strengths and weaknesses of the proposed approach. Additionally, a qualitative comparison, showing example outputs or behaviors from different models, could provide valuable insights.\n\n3. The paper includes some ablation studies to assess the impact of different components of the framework. However, these studies could be expanded to provide a deeper understanding of the contribution of each component given that Any2Policy is a fairly complex system. For example, the authors could investigate the impact of different encoder architectures or the role of specific modalities in greater detail. This would help to identify the most critical elements of the framework and guide future improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer for the comments and reference reviewer with identifier h8z5 as R1. Comment n of reviewr m is denoted as RmCn.\n\n**[R1C1]**\n\n> More details on the datasets. \nWe concur with the reviewer's suggestion to provide additional details in our paper. To address this, we have included in the Appendix examples of 8 tasks from our real-world dataset. This dataset comprises 20 short-horizon tasks, such as pick-and-place [item], and 10 medium-to-long-horizon tasks, which involve multiple steps to complete (e.g., opening a drawer, retrieving an item, placing it back, and closing the drawer). Our setup features a variety of real-world objects, including microwaves, drawers, laptops, boxes, cubes, bottles, etc. These objects range from deformable to rigid and articulated, encompassing a wide array of real-world scenarios. We will provide a detailed summary of our dataset in the revised manuscript.\n\n**[R1C2]**\n\n> Extensive comparative analysis would be beneficial, such as RT-1/RT-2/RT-X, Octo, OpenVLA. \n\n\nWe appreciate the reviewer for highlighting these issues. Since RT-2/RT-X are not open-sourced, and OpenVLA is released recently which we do not have time to implement in our environment, we provide additional results of Octo and RT-1 in our real world experiments. Given the limited time available for the rebuttal, we conducted experiments on four selected tasks. The same training data were used for all methods. For Octo, we present two versions: one that uses pretrained weights from the Open-X framework, and another that is trained from scratch using our dataset. Both versions were evaluated 10 times to calculate the average success rate for each method. Consistent training hyperparameters, including learning rate, learning rate scheduler, and training epochs, were maintained across all methods.\n\n\n| Method | PlaceBread | CloseLaptop | InsertPlug | PlaceCan | \n| ------- | ------- | ------- | ------- | ------- |\n| Octo (pretrained) |   **100** | **100** |  40 | 20 | \n| Octo (train-from-scratch) |  50 | 70 |  0 | 0 | \n| RT-1  |  60 | 80 |  0 | 0 | \n| **Ours** | **100** | **100**| **50** | **50** |\n\n\nWe demonstrate that Octo, pre-trained on the Open-X framework, achieves comparable results to our method on two tasks. However, for more challenging tasks, our method outperforms the pre-trained Octo. When Octo is trained with the same amount of data as our method, our approach achieves a significantly higher success rate.\n\n**[R1C3]**\n\n> The author could investigate the impact of different encoder architectures or the role of specific modalities in greater detail.\n\nWe thank the reviewer for raising these points. Unfortunately, due to the limited time available during the rebuttal period, we were unable to complete these experiments. However, we plan to conduct additional experiments in the future, focusing on using different encoders and replacing the policy head.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "anay6U9mQ5", "reviewer": "Reviewer_QY6x", "summary": "This submission develops a new model that can handle many different modalities as input for instruction and observations (video, text, image, pointcloud etc.). Different modalities are encoded via different frozen encoders (with projection layers kept trainable) to a shared representation to then be passed into a policy that then generates actions. The submission constructs a multi-modal dataset for several evaluation benchmarks (3 simulated, 1 real), and trains a policy on this multi-modal data and show it out-performs policies trained on a specific modality of data. While work in the past has built multi-modal models in language/vision, this is the first to do so in robotics. Cross attention is used where the observation sequence of tokens are keys and the instruction sequence of tokens are values.", "questions": "Questions:\n- Are there meta-world results? I can't find any figures/tables on it, only see results on ManiSkill2 and Franka Kitchen.\n- The name of the dataset this paper curated, RoboAny, is mentioned once in the paper in passing and never again. Is there a reason for that?\n- Can there be more clarification around how the ablation of no embodied alignment module works? My understanding is if you remove it you simply directly use the encoded tokens and put them all in one big sequence as input to the transformer. However the original encoders might output different shapes for the tokens so I am not sure how this is consolidated.\n- Is AnyPolicy trained on data from all benchmarks or is a separate policy trained per benchmark. In other words, is the multi-task policy multi-task over all tasks in a benchmark, or all tasks in all benchmarks?\n- How does the modality-specific model work, what is its architecture compared to AnyPolicy?\n- How come StackCube has such a high success rate but PickCube has such a low success rate? My understanding is in ManiSkill2 StackCube is much harder than PickCube because it requires careful placement on a cube and releasing the cube, whereas in PickCube you simply grab the cube and move to a random goal.\n- How many demonstrations are used for ManiSkill2? This detail could not be found anywhere.\n- How is R3M used in the Video-Image result in table 3? How does it process video input and produce video tokens? \n\nTypo: \n- \"We further conduct experiments on two simulated robotics benchmarks to reinforce the strong generalizability of our approach compared to existing methods\" in section 1. I think three benchmarks were tested not two.\n- Figure 4 shows 110% on the y-axis. It should be clamped to [0, 100].\n\nUnfortunately given the significant lack of details on data, and how some ablations work (see questions below for more) I recommend a reject in the current state. Moreover the performance of the model after combining several new modalities does not appear to out perform other baselines from other papers that much to warrant the amount of extra work on a simple task like FrankaKitchen. I am happy to raise my score to the accept range if these weaknesses and questions are addressed.", "limitations": "Limitations are provided in the appendix and discuss limitations from the perspective of additional things that could be done (more modalities, more robots etc.). I think limitations around data efficiency could be important to bring up. Just how many demonstrations are needed? Real-world imitation learning suffers greatly due to the expensive cost of teleoperation (especially if high success rates are desired).", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This appears to be the first work that combines video, image, pointcloud, and audio together into one multi-task model for robotics, demonstrating some improvements in success rates when multiple modalities are used in training instead of just one modality.\n- The release of a highly multi-modal annotated dataset is great to see. However more details about the data would be greatly appreciated.", "weaknesses": "- The performance of AnyPolicy looks very similar to EmbodiedGPT in the franka kitchen setting in Figure 4. The figure 4 caption appears to also be wrong as it seems EmbodiedGPT outperforms AnyPolicy on both Knobs Left and Light Left, not just Knobs left. Moreover, no error bars are shown and with results so similar over just 100 evaluations in simulation, it is hard to say if AnyPolicy out performs embodied GPT.\n- FrankaKitchen, while a useable benchmark, has almost no initial randomization and is incredibly easy given objects in the environment are not randomized, making results reported on it not mean that much. It would be great to see comparisons on harder manipulation benchmarks although I realize it might not be possible to run experiments now.\n- Very few details about the real world dataset are provided. What are the 30 tasks? What are some example instructions? What are some example audio?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer for the comments and reference reviewer with identifier QY6x as R2. Comment n of reviewr m is denoted as QY6x.\n\n**[R2C1]**\n\n> The performance of AnyPolicy looks very similar to EmbodiedGPT in the Franka kitchen setting in Figure 4. The Figure 4 caption appears to be wrong\n\nThank you for pointing out our mistake. The caption is indeed incorrect, as Any2Policy performs slightly worse than (or on par with) EmbodiedGPT on two tasks. However, we respectfully disagree that the overall performance of Any2Policy is similar to EmbodiedGPT. For the Franka Kitchen experiments, we demonstrate that:\n\n1. For challenging tasks, such as Micro Right and Cabinet Right, our method significantly outperforms EmbodiedGPT.\n2. In low-data scenarios (e.g., 10 demonstrations), Any2Policy achieves much better performance than EmbodiedGPT and other baselines.\n3. Despite being developed to handle multiple modalities, our method outperforms those trained specifically for a single modality.\n\n\n**[R2C2]**\n\n> The experiments of FrankaKitchen is an easy benchmark.\n\nWe agree that FrankaKitchen is an easy benchmark and may introduce bias in evaluation. Therefore, we have provided further evaluation on Meta-World (results are included in the rebuttal), ManiSkill-2, and real robots. To further address the reviewers' concerns, we conducted additional experiments on RLBench and Adroit, which include more challenging tasks with a 24 DoF ShadowHand and a 4 DoF arm. All experiments were conducted with 200 demonstrations.\n\nWe provide multiple baselines, including Diffusion Policy, IBC, EmbodiedGPT, and also present error bars. We ran each experiment with 3 seeds and evaluated 200 episodes every 200 training epochs. We computed the average of the highest 5 success rates, which is a typical setting in this simulation. Below are the experimental results.\n\n\n| Method | Hammer | Door | Pen | \n| ------- | ------- | ------- | ------- |\n| Diffusion Policy | 52 $\\pm$ 15| 56 $\\pm$ 7 |  17 $\\pm$ 5 |\n| IBC | 0 $\\pm$ 0 | 1 $\\pm$ 1 | 0 $\\pm$ 0 | \n| EmbodiedGPT | 35 $\\pm$ 4 | 49 $\\pm$ 10 |  21 $\\pm$ 7 |\n| **Ours**|  **64 $\\pm$ 13** | **65 $\\pm$ 8** | **29 $\\pm$ 6** |\n\n\nWe demonstrate that even on challenging benchmarks, our method consistently outperforms state-of-the-art baselines.\n\n\n**[R2C3]**\n\n> Metaworld Results are missing.\n\nWe thank Reviewer 4 for the careful review and for pointing out the missing experimental results. We sincerely apologize for this oversight. Due to limited space, please see **[R4C3]** for details.\n\n\n**[R2C4]**\n\n> Name Roboany is not used anywhere else.\n\nWe thank R4 for careful reviews and for pointing out this **typo**. We will include this in the revision. \n\n**[R2C5]**\n\n> Can there be more clarification around how the ablation of no embodied alignment module works?\n\nWe appreciate the reviewer's concern regarding the details of our ablation study. We have provided a **detailed illustration of our methodology in lines 254-258**. Specifically, we use MLP layers to align the observation tokens with the instruction tokens, ensuring the dimensions are correct. Additionally, we employ TokenLearner to reduce the number of tokens and maintain consistency, even when some modalities are missing.\n\n\n**[R2C6]**\n\n> Is AnyPolicy trained on data from all benchmarks or is a separate policy trained per benchmark. In other words, is the multi-task policy multi-task over all tasks in a benchmark, or all tasks in all benchmarks?\n\nIn our experiments, for each benchmark, including real-world scenarios, our model is trained on all collected tasks to perform multi-tasking. This is a typical setting in robot learning, as seen in works like Diffusion Policy (RSS '23) and EmbodiedGPT (NeurIPS '23).\n\n\n**[R2C7]**\n> How does the modality-specific model work, what is its architecture compared to AnyPolicy?\n\nDue to our systematic design, our method can naturally handle missing modalities. The architecture of our modality-specific model is identical to AnyPolicy. During inference, we pass empty tokens to account for the missing modalities. \n\n\n**[R2C8]**\n> Since **PickCube only needs to place a cube on a random place**, which is much easier than StackCube, why does StackCube have such a higher success rate than PickCube? \n\nWe thank Reviewer 4 for their careful review and for pointing out this observation. To explain, first of all, PickCube requires picking up a cube and placing it at a **specific goal position**. This task is not easier than StackCube. For instance, as shown in Table 3 of the ManiSkill-2 paper, **the success rate for StackCube is higher than for PickCube when using point cloud observations**. Secondly, since our method is trained in a multi-task fashion (as opposed to the single-task training in ManiSkill-2), the learning behavior could differ from that of a single-task policy. \n\n**[R2C9]**\n> How many demonstrations are used for Maniskill2?\n\nWe used 1K demonstrations for all tasks for Maniskill2. \n\n**[R2C10]**\n> How is R3M used in the Video-Image results?\n\nFor R3M, we select the keyframe with images of resolution $224 \\times 224$ as input and pass to R3M. All other settings are kept the same as the default R3M. \n\nWe thank you for the precious review time and valuable comments. We have provided corresponding responses, which we believe have covered your concerns. We hope to further discuss with you whether or not your concerns have been addressed. Please let us know if you still have any unclear parts of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ERutEeoLUT", "reviewer": "Reviewer_RRf1", "summary": "This paper proposes the simultaneous fusion of image, text, point cloud, video, and audio—four modalities—in robotic manipulation tasks, while also considering the integration of information from both instruction and observation. Through the transformer architecture and cross-attention mechanism, embodied alignment of multiple modalities is achieved. Furthermore, this paper constructs a real multimodal dataset and conducts a detailed experimental analysis of the impact of additional modality information on performance.", "questions": "Considering the excellent development of current multimodal pre-trained models, and that the models in this paper are trained from scratch, is it feasible to incorporate existing pre-trained models such as text-image, audio-image, image-point cloud, and text-point cloud models into the current training framework?", "limitations": "The authors have adequately addressed the limitations andsocietal impact of the work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper investigates the impact of simultaneously fusing five modalities: image, text, point cloud, and video, on robotic manipulation tasks. Although there is already a considerable amount of work that fuses two or three of these modalities, this paper argues that it is still meaningful to consider the simultaneous fusion of different modalities from both instruction and observation.\n\n2. This paper constructs a real robotic dataset that includes information from all the aforementioned modalities.\n\n3. The experimental section of this paper provides a detailed analysis of the impact of multimodal information fusion on robotic manipulation, confirming that the involvement of additional modalities in training or inference can lead to performance improvements.", "weaknesses": "1. The technical contribution of this paper is limited. Although the paper analyzes some difficulties in the process of fusing multimodal information and embodied alignment, the overall model architecture is still relatively straightforward and simple.\n\n2. The scale of the dataset is relatively small compared to the widely used robotic operation datasets currently available. Considering that the model architecture and training methods for multimodal information fusion typically require a large amount of data, it may be necessary to have a larger dataset to verify stronger generalizability or to draw more robust conclusions on multimodal information fusion. Additionally, some existing datasets are already capable of including modalities such as text, image, video, and point cloud simultaneously. The main contribution of this paper seems to lie in the addition of rich paired audio data. If one considers expanding the scale of the dataset as well as costs, would it be a better choice to augment the existing datasets with information from modalities such as audio?\n\n3. The experimental section does not yield particularly new conclusions. Generally speaking, it is not surprising that the addition of new modalities, especially for robotic manipulation tasks, typically leads to better performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Rebuttal: \n\n\nWe thank reviewer for the comments and reference reviewer with identifier RRf1 as R3. Comment n of reviewr m is denoted as RmCn.\n\n**[R3C1]**\n> The technical contribution of this paper is limited. Although the paper analyzes some difficulties in the process of fusing multimodal information and embodied alignment, the overall model architecture is still relatively straightforward and simple.\n\nWe thank the reviewer for raising this point. While the technical novelty may seem limited, our primary contribution lies in providing a straightforward yet effective solution that, for the first time, successfully integrates multiple modalities for both task specification and observation, enabling seamless robotic manipulation. Our method demonstrates significant performance gains over the baselines in new settings, underscoring the necessity of our approach\n\n\n\n**[R3C2]**\n> The proposed dataset seems to lie in the addition of rich paired audio data? Would it be a better choice to augment the existing datasets with information from modalities such as audio?\n\nWe appreciate the reviewer's concern about our dataset. Notably, our dataset goes beyond merely adding rich paired audio data. We offer a comparison with existing datasets in the following table. We focus on the types of modalities in existing data on both observation and task specification. \n\n| Dataset | Observation | Task Specification |\n| ------- | -----------  | ------------------| \n| Open-X Embodiment | point cloud, image | text |\n| Droid | point cloud, image | text |\n| RH20T | point cloud, image | video, text |\n| **Ours** | point cloud, image, video | text, image, video, audio | \n\nMost previous datasets provide point clouds and images for observation, and text for task specification. There are other modalities, such as image goals and audio, for task specification. Despite being smaller than Open-X and other datasets, our dataset is more comprehensive in the domain we aim to study, specifically aligning multiple modalities for both instruction and observation to facilitate the training of robotic manipulation. It highlights the unique value our dataset brings to the community.\n\nSecondly, while it is possible to expand existing datasets with this additional information, it would require significant effort and time. This presents a valuable future research direction for expanding datasets to include richer multimodal information.\n\n\n\n**[R3C3]**\n\n> The experimental section does not yield particularly new conclusions. Generally speaking, it is not surprising that the addition of new modalities, especially for robotic manipulation tasks, typically leads to better performance.\n\nWe agree with the reviewer's point. Intuitively, adding new modalities should improve model performance for manipulation tasks. However, prior works have not fully explored this area. The main contribution of our work lies in building a robust system for multimodal settings, providing a benchmark to evaluate methods in this context, and demonstrating the value and potential of this approach for future research.\n\n\n\n**[R3C4]**\n> Considering the excellent development of current multimodal pre-trained models, and that the models in this paper are trained from scratch, is it feasible to incorporate existing pre-trained models such as text-image, audio-image, image-point cloud, and text-point cloud models into the current training framework?\n\nThe challenge here is that an embodied agent needs to _observe_ and _interact_ with the world using different modalities. Not only do we need to manage diverse modalities for tasks such as observation and task specification, but we also need to align these two components together. Currently, no work addresses this comprehensive challenge, and it is infeasible to incorporate existing pre-trained models to resolve it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qZRfT4wqjx", "reviewer": "Reviewer_5TLy", "summary": "This paper aims to enable robots to understand tasks and their environments using multiple modalities such as text, audio, images, video, and point clouds. To accomplish this, the authors introduce Any2Policy, a versatile framework designed to process multi-modal inputs at the instruction and observation levels. Within this framework, they developed an embodied alignment module that effectively integrates multi-modal features. Additionally, the use of ImageBind and TokenLearner helps to mitigate tedious encoder design and enhance training efficiency, respectively, which appears to be a practical approach. The paper's thorough evaluations, conducted on 30 real-world tasks and three distinct simulation benchmarks, convincingly demonstrate the effectiveness of the proposed method.", "questions": "1. I'm curious to know whether the video instructions were recorded using the same embodiment or if they include cross-embodiment examples, such as demonstrations by humans?", "limitations": "The authors have properly discussed the limiations and societal impacts in the Appendix.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper addresses a pivotal topic in robot learning, focusing on enabling robots to understand tasks and interpret the environment through multi-modal inputs, which is essential for developing generalist agents.\n   \n2. The manuscript is well-written and well-motivated, presenting its ideas and research goals effectively.\n\n3. The methods introduced in this paper effectively integrate multi-modal inputs to substantially improve policy learning. Comprehensive evaluations conducted across both real-world tasks and simulated environments demonstrate the effectiveness of the Any2Policy framework.", "weaknesses": "However, there are still some concerns regarding the experiments.\n\n1. While the authors have conducted extensive ablations on real-world experiments, replicating these results in practice may be challenging for follow-up researchers. To facilitate better benchmarking, it would be advantageous if the effectiveness of each proposed method could also be demonstrated in simulated experiments.\n\n2. Due to the high cost associated with evaluating real-world tasks across different random seeds, reporting variance is difficult. Nonetheless, it would be beneficial if the authors could assess the impact of multiple random seeds in simulation tasks to provide more robust statistical insights.\n\n3. The authors appear to have omitted the results from the meta-world evaluations. These results were neither found in the main text of the paper nor in the appendix, despite the authors claims that they were reported.\n\n4. MUTEX [1] is another significant work that explores task specifications using multi-modal inputs. While Any2Policy incorporates a broader range of observation modalities (image, point cloud), including the comparison with MUTEX could enrich the analysis, especially given the limited range of baseline methods evaluated in this paper.\n\n\n[1]  Mutex: Learning unified policies from multimodal task specifications", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer for the comments and reference reviewer with identifier 5TLy as R4. Comment n of reviewr m is denoted as RmCn.\n\n**[R4C1]**\n> While the authors have conducted extensive ablations on real-world experiments, replicating these results in practice may be challenging for follow-up researchers. To facilitate better benchmarking, it would be advantageous if the effectiveness of each proposed method could also be demonstrated in simulated experiments.\n\nWe share the reviewer's sentiment that building a simulation environment is a paramount topic that could improve productivity and enhance future research. We are currently developing a simulation based on Isaac Sim, which we hope will facilitate further research in this domain.\n\n**[R4C2]**\n\n> Due to the high cost associated with evaluating real-world tasks across different random seeds, reporting variance is difficult. Nonetheless, it would be beneficial if the authors could assess the impact of multiple random seeds in simulation tasks to **provide more robust statistical insights**.\n\nSince our simulation environments were not ready, we performed a robustness study on random seeds in real-world experiments. We evaluated four tasks in the real world, running 3 seeds for each experiment with seed numbers 0, 1, and 2. For each seed, we conducted 10 evaluations across the four real-world tasks. Both methods were evaluated 10 times, and we report the average success rate for each method.\n\n| Seed | PlaceBread | CloseLaptop | InsertPlug | PlaceCan | \n| ------- | ------- | ------- | ------- | ------- |\n| 1 | 100 | 100| 50 | 50 |\n| 2 | 90 | 100| 30 | 50 |\n| 3 | 100 | 90| 50 | 60 |\n| Average| 96.6 | 96.6 | 43.3 | 53.3 |\n\nOur experiments indicate that the training of our approach is stable, and the results are consistent across different seeds.\n\n**[R4C3]**\n\n> Missing MetaWorld Results. \n\nWe thank Reviewer 4 for the careful review and for pointing out the missing experimental results. We sincerely apologize for this oversight. Here is a brief summary of the experimental results. We present the results based on three task levels, following the settings in Masked World Model (CoRL'23). We report the average success rate across three task levels: easy, medium, and hard. In total, there are 45 tasks. All experiments were trained with 20 demonstrations, evaluated with 3 seeds, and for each seed, the success rate was averaged over five different iterations. We used all available modalities for our method. The experimental results are shown below:\n\n| Method | Easy (28) | Medium (11) | Hard (6) | \n| ------- | ------- | ------- | ------- |\n| Diffusion Policy (RSS'23) | 80.4 | 32.6 | 9.4 | \n| IBC (CoRL'21) | 68.3 | 15.2 | 10.6\n| EmbodiedGPT (NeurIPS'23) | 60.1 | 24.5 | 7.9 | \n| **Ours** | **89.6** | **65.3**| **29.6** |\n\nThe numbers in parentheses indicate the number of tasks for each specific task level. The experimental results support our conclusion that Any2Policy are able to achieve better performance than the baseline across different levels of task difficulty. \n\n**[R4C4]**\n\n> Comparison with Mutex. \n\nWe thank the reviewer for raising this point. Mutex is indeed a significant work that explores multimodal task specification. To strengthen our work, we conducted a comparative study with Mutex. Specifically, we selected four real-robot tasks and compared them with Mutex using the same training data. Both methods were evaluated 10 times, and we report the average success rate of each method.\n\n| Method | PlaceBread | CloseLaptop | InsertPlug | PlaceCan | \n| ------- | ------- | ------- | ------- | ------- |\n| Mutex |   80 | 90 |  20 | 10 | \n| **Ours** | **100** | **100**| **50** | **50** |\n\nOur experimental results indicate that Any2Policy performs better than Mutex. We believe that the performance gain comes from the additional modalities introduced by our method.\n\n\n**[R4C5]**\n> I'm curious to know whether the video instructions were recorded using the same embodiment or if they include cross-embodiment examples, such as demonstrations by humans?\n\nYes, the video instructions were recorded through human demonstration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a multi-modal robotic manipulation dataset encompassing five input modalities: image, text, audio, point cloud, video. This paper demonstrates that a simple approach integrating all these modalities lead to improved performance in robotic manipulation tasks. 

Despite its limited technical novelty, the reviewers have reached a consensus that the curated dataset and comprehensive empirical evaluations offer valuable insights to the robotic learning community. As such, this paper is recommended for acceptance to NeurIPS.

However, most of the reviewers have concerns on this paper's presentation and organization. The author responses during the rebuttal period have resolved some of these issues, but it does not seem sufficient. The authors should carefully address the reviewers' comments and improve the quality of the paper together with more comprehensive experimental results beyond the ones included in the rebuttal.
Decision: Accept (poster)