Any2Policy: Learning Visuomotor Policy with Any-Modality

Authors: yichen  zhu
zhicai  ou
feifei  feng
jian  tang
midea  group
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 12:26:02

Abstract: Humans can communicate and observe media with different modalities, such as texts, sounds, and images. For robots to be more generalizable embodied agents, they should be capable of following instructions and perceiving the world with adaptation to diverse modalities. Current robotic learning methodologies often focus on single-modal task specification and observation, thereby limiting their ability to process rich multi-modal information. Addressing this limitation, we present an end-to-end general-purpose multi-modal system named Any-to-Policy Embodied Agents. This system empowers robots to handle tasks using various modalities, whether in combinations like text-image, audio-image, text-point cloud, or in isolation. Our innovative approach involves training a versatile modality network that adapts to various inputs and connects with policy networks for effective control. Because of the lack of existing multi-modal robotics datasets for evaluation, we assembled a comprehensive real-world dataset encompassing 30 robotic tasks. Each task in this dataset is richly annotated across multiple modalities, providing a robust foundation for assessment. We conducted extensive validation of our proposed unified modality embodied agent using several simulation benchmarks, including Franka Kitchen, Meta-World, and Maniskill2, as well as in our real-world settings. Our experiments showcase the promising capability of building embodied agents that can adapt to diverse multi-modal in a unified framework.

Main paper: 
 introduction: 
 What is the ultimate form of robots? It should possess the ability to both listen and read, learn from demonstrations, and perceive the three-dimensional world with an understanding of time. While recent advancements in robot learning have explored various modalities for task specification and environmental perception -such as text, speech, images, videos, and point clouds -most prior research has approached these modalities as distinct challenges.An expanding field of research within artificial intelligence, spanning various disciplines, indicates that simultaneous learning across different modalities, such as vision-language [1, 2, 3, 4, 5] , video-text [6, 7, 8, 9, 10, 11] , and vision-touch [12, 13, 14, 15] , leads to the development of more comprehensive and effective representations. The enriched representations derived from this approach lead to a more profound comprehension of each modality on a standalone basis. This multi-faceted concept finds backing in a number of cognitive science and psychology studies [16, 17] , which propose that human learning is significantly enhanced when it integrates multiple cues, compared to relying on a single modality in isolation. By adopting this multi-modal approach, which leverages the collective strength of various senses or inputs, we emulate the natural human method of information acquisition and processing, indicating a path toward a more integrative methodology in robotics. The overall framework of Any2Policy. The Any2Policy framework is structured to handle multi-modal inputs, accommodating them either separately or in tandem at the levels of instruction and observation. We design embodied alignment modules, which are engineered to synchronize features between different modalities, as well as between instructions and observations, ensuring a seamless and effective integration of diverse input types.Building on the proven success of multi-modal learning in various disciplines, our goal is to develop a multi-modal embodied agent capable of processing a wide array of instruction and observation modalities. Our method, referred to as Any-to-Policy Robot Model (abbreviated as Any2Policy), is illustrated in Figure 1 . The Any2Policy architecture is structured in two primary components. Initially, we harness multi-modal encoders to extract relevant features from the input. These features are then transformed into standardized token representations, ensuring uniformity at the neural network level via projection layers. In the second component, we introduce embodied alignment, which processes multi-modal signals, enriched with specific instructions. These signals are routed to encoders following projection, ultimately leading to the generation of actions for manipulation.In support of this project, we are releasing a substantial real-world dataset consisting of 30 tasks, where each task includes 30 trajectories, all annotated with multi-modal instructions and observations, mirroring the setup used in our experiments. The purpose of this dataset is to foster and encourage future research in the area of multi-modal embodied agents.We have demonstrated the efficacy of our Any2Policy approach through rigorous testing against this benchmark. The results clearly indicate that Any2Policy is highly adept at interpreting and responding to a variety of modalities. We further conduct experiments on two simulated robotics benchmarks to reinforce the strong generalizability of our approach compared to existing methods. Our project is at any2policy.github.io/.In summary, our contributions are the follows:• We introduce any-to-policy models that enable a unified embodied agent to process various combinations of modalities, effectively facilitating instruction and perception of the world. • We present novel embodied alignment learning techniques designed to seamlessly align instructions and observations, enhancing both the effectiveness and efficiency of policy learning. • We offer a multi-modal dataset tailored for robotics, encompassing 30 distinct tasks. This dataset covers a wide spectrum of modalities in both instruction and observation. 
related work: 
 Multi-Modal Learning. Recent large vision language models (VLMs) use pretrained image encoders as part of the larger model; some pretrain it with supervised classification, some use the pretrained CLIP encoders [18] , and some with custom multi-modal pretraining. Inspired by GPT-4, which has been released and demonstrates many advanced multi-modal abilities, recent models present more lightweight versions of VLM that align the image inputs with a large language model using proper instructional tuning [19, 20, 21, 22] . These vision-language models also showcase many advanced multi-modal capabilities after the alignment. Another line of research [23, 24, 25, 26, 27, 28] seeks to demonstrate the transformer's capability to deal with multiple modalities in a single neural network and then perform image generation, video-question answering, and multi-turn dialogue, graph learning [29, 30, 31] according to the input modalities. Different from these prior works, we focus on robotics. Our design multi-modal input includes both instruction and observation, which potentially makes the robot more generalizable to diverse scenarios.Embodied agent with Diverse Modalities. The study of embodied AI has been extensively explored through instruction and observation methods. A series of studies concentrate on learning using varied task specifications. This includes language-conditioned policy learning [32, 33, 34, 35, 36, 37, 32, 38, 39, 40] , instruction following agents [41, 42] with 3D observation [43, 44, 45, 46, 47] , visual goalreaching [48, 49, 50, 51] , video demonstrations [52, 53, 54, 55, 56, 34, 57, 58] , or combination with multiple modalities [59, 60, 61] . Notably, most systems are specialized for certain task specifications, with few like VIMA and MUTEX engaging with multiple modalities, albeit focusing on a single observation type.Representation learning is critical in high-dimensional control settings, particularly when managing visual observation spaces. Recent advancements in this field have explored different modalities in feature learning for robotics, especially for images with the unsupervised pretrained visual representation [62, 63, 64, 65, 66, 67, 68, 69] , or using visual encoders from pretrained vision-language models [70, 71, 72, 73, 74, 75, 76, 77] for robotics [78, 79, 80, 33, 81, 82, 59, 83, 84, 61] . These approaches often neglect instruction or rely solely on text-based instruction, lacking exploration of diverse instructional modalities.Our proposed Any2Policy framework differs from previous work by focusing on both instruction and observation levels and utilizing various modalities. This approach aims to enhance learning and inference for embodied agents, offering a more holistic and integrated perspective compared to earlier, more modality-specific research.Robot Manipulation and Datasets. A diverse array of robot manipulation tasks necessitates various skills and task specification formats, including instruction following [32] , one-shot imitation [85] , rearrangement [86] , reasoning [87, 88, 89] , and planning [90, 91] . Most existing datasets predominantly feature 2D images for observations and, in some cases, include language instructions to prompt specific robotic actions, as seen in datasets like Franka-Kitchen [92] and CALVIN [93] . VIMA-Bench [59] introduces a dataset that combines language instructions with images in multimodal prompts. Meanwhile, Maniskill [94] provides 3D visual information, enabling models to interpret point clouds as observations. Our dataset, RoboAny, stands out as the first to support a comprehensive range of modalities in robotics. It encompasses both instructions and observations across images, videos, audio, language, and point clouds, offering unparalleled diversity in robotic task specification and learning. 
methodology: 
 Our objective is to develop a comprehensive policy capable of executing a variety of tasks, utilizing a dataset of demonstrations that are annotated with multi-modal instructions and observational data from various sources such as images, videos, point clouds, text, and speech. We acknowledge that the modality of task specifications and observations may differ depending on the specific scenario, and our approach is tailored to adapt flexibly to these varying modalities.We start with a formal description of our settings and provide mathematical annotations. For each task T ∈ {T 1 , T 2 , • • • , T n }, we assume that the embodied agents learn from demonstrations obtained through teleoperation,D i = {d i , • • • , d m }, where m is the number of demonstrations for task T i . Each demonstration d j i presents the form of a sequence of observations and expert actions,d j i = [(s, o, a) j i ],where a ∈ A denotes the actions, s ∈ S represent instruction, and o ∈ O is the observation.The instruction of task T i is specified with four modalities: text instructions s ∈ {L1 i , L 2 i , • • • , L k i }, audio instruction s ∈ {A 1 i , A 2 i , • • • , A k i }, image goal instruction s ∈ {M 1 i , M 2 i , • • • , M k i } and video demonstration s ∈ {V 1 i , V 2 i , • • • , V k i }.Similarly, the observation of task T i , is detailed by o ∈ {m 1 i , m 2 i , • • • , m k i }, point cloud o ∈ {p 1 i , p 2 i , • • • , p k i }, and video o ∈ {v 1 i , v 2 i , • • • , v k i }.Capital letters are used to denote instructions, while lowercase letters represent observations. Our objective is to develop a multi-task policy, denoted as π(a|s, o), which generates continuous actions a in response to the current observation o and given instruction s. The policy aims to proficiently execute n tasks included in the training dataset D under new initial conditions, such as varying positions of objects. Moreover, a critical aspect of this policy is its capacity to generalize effectively to environments, entities, and tasks that have not been previously encountered or represented in the training data.A pivotal characteristic of an Any2Policy agent is its capacity to leverage multi-modal information throughout its training phase, exhibiting the ability to operate competently when restricted to a single modality. It adapts its comprehension and actions to suit the available sensory input. This versatility brings complexities in crafting multi-modal models, requiring careful consideration in their design to ensure seamless adaptability and robust performance. To address these, we introduce two modules: 1) Multimodal encoders to promote cross-modal interactions of all modalities into a shared latent space for instruction and observation, individually, and 2) Embodied alignment to take in the instruction token and merged with observation tokens via cross-attention. We provide an overview in Figure 2 .Multimodal Encoder. There are many choices for the multi-modal encoders, for instance, CLIP [18] for language and vision transformer for images. Specifically, we denote the encoder as a mapping function P (•), and for each modality, we need a particular model to extract useful modality-agnostic representation to prepare for policy learning. To make our framework more consistent, we would like to share the encoder if the instruction and observation have the same modality. Then, we would have five modality encoders for text, audio, image, video, and point cloud, respectively.Using different backbone models for each modality can be cumbersome and hard to maintain. Thus, we leverage existing well-established models, ImageBind [25] , a unified high-performance encoder across five modalities, to encode inputs of various modalities. With the help of ImageBind, we are spared from managing many numbers of heterogeneous modal encoders. Notably, the multi-modal encoders extract semantically meaningful representations from the input modality, which aims to provide essential knowledge to the policy learning networks for accurate manipulation.Embodied Alignment. We leverage a projection layer to map different modalities' feature representations from P (x) into unified representations. In particular, we want to avoid heavy computational costs during training, so that we freeze all the layers in ImageBind, and only keep the projection layers learnable. This operation is kept the same for both the instruction segment and the observation segment.As a result, the projection layers ensure that the tensor of different modalities are in the same form. Nevertheless, the process is fraught with challenges, 1) aligning distinct modalities within a segment, particularly when diverse types such as images and point clouds coexist, is difficult. Simultaneously, it's vital to control computational costs, which can escalate with the addition of modalities, and 2) the integration of instructions with observations is complex. This complexity arises partly because instructions and observations don't always correspond one-to-one; often, a single instruction relates to multiple observation frames, and for most time steps, observations exist without accompanying instructions. Additionally, the modality gap between instruction and observation can be an obstacle in creating effective feature representations for manipulation tasks.To address the first challenge, we employ a transformer architecture. Each modality undergoes tokenization, with the token count varying between modalities. For example, images and videos typically require more tokens to encapsulate visual information. We use 81 tokens for images, 81 × t for videos (where t represents the number of frames), and 256 tokens for point clouds. For text and audio, a single token is generally sufficient due to their shorter length. It's important to note that the computational demands of self-attention layers increase quadratically with the number of tokens, leading to potentially high inference costs for visual modalities. Therefore, it is crucial to use model compression methods [95, 96, 97, 98, 99, 100, 101, 102, 103, 104] to reduce the computational cost.We follow RT-1 [35] to utilize TokenLearner [105] for pruning redundant visual tokens. We apply TokenLearner individually to each visual modality, such as images, videos, and point clouds. After token pruning, the token count for images and videos is reduced to 8, while for point clouds, it's brought down to 16. These tokens are then amalgamated, using a modality token to distinguish between different modalities. We insert a modality token between every two modalities. We also use absolute position embedding to maintain the token order. Consequently, the maximum token count, when integrating all three visual modalities, is significantly reduced to as few as 34 (8+8+16+2), substantially enhancing computational efficiency. For the transformer model, we use one Transformer block with self-attention and feed-forward networks.The second major challenge is how to align instruction with observation. We seek the help of crossattention. Specifically, when we are presented with an observation sequence, P o , and an instruction sequence, P s , we project P o into the key (K) and P s into the value (V). We create a set number of learnable query (Q) embedding as input for the cross-attention. This learnable query embedding aims to adapt different modalities for efficient fusion between instruction and observation. This setup enables us to align the two tensors through a cross-attention mechanism, mathematically represented as:EQUATIONIn this equation, d h denotes the dimensionality, and the construction of the query, key, and value components aligns with the standard framework utilized in BERT [106] . The elegance of the method lies in its ability to effectively incorporate relevant representations from the instruction into the observation, thus ensuring a more coherent and informative interaction between the two. In our experiments, we use 16 queries, where the dimension of queries corresponds to the cross-attention layers.Policy Networks. Designing a multi-task policy comes with its share of challenges, and one critical aspect involves choosing an appropriate conditioning mechanism. In our framework, the policy networks are conditioned based on the observation sequence and occasionally an instruction sequence. As aforementioned, this conditioning is achieved through a series of cross-attention layers, which establish connections between the policy network and the input modalities. Each Transformer block comprises self-attention, cross-attention, and a feed-forward network. Additionally, we incorporate residual connections that link the higher layers with the input rollout trajectory sequence. We append one history action token to the model, similar to VIMA approach [59] . It is important to note that the instruction aligns with the observation at every time step. To optimize efficiency, we store the instruction's representation to avoid repetitive computations. The decoder is constructed using L Transformer blocks. Finally, we append a Multi-layer Perceptron (MLP) layer to generate continuous actions. We use behavior cloning mean squared loss as the optimization objective.In our pursuit to develop a robot model that allows input with any modality, we have established a new dataset featuring a range of tasks, each accompanied by multiple instructions and observations tailored to real-world scenarios. We developed two workstations, one of which is depicted in Figure 3 . This dataset comprises n = 30 distinct tasks. These tasks vary from straightforward pick-and-place actions, such as "pick up the yellow cube," and "place the toy bear in the blue box to the right," to more complex contact-rich tasks like "open the drawer" as well as tasks demanding to reason, for instance, "sort the cube with the same color." Although all tasks are executed within the same environment, they involve a variety of objects selected from a set of 70. Each task is exemplified through m = 30 human-collected trajectories. Further, every task is annotated with k = 5 distinct instructions. To generate these diverse instructions, we utilized GPT-4, creating prompts to formulate alternative descriptions. We have carefully filtered these descriptions to ensure they accurately represent the specific task-relevant objects, avoiding any potential confusion due to synonyms. Additionally, we have incorporated speech signals from various characters of the Amazon Polly service to enrich our dataset with diverse speech descriptions. In terms of observation, we recorded the video during our data collection process. The depth information captured in these recordings was then converted into point cloud data, enabling detailed analysis and application of the collected data. All data are collected via teleoperation. For long-horizon tasks, we operated the robot to move back to the initial state and then start the next move. In the appendix, we provide detailed descriptions of our hardware for two workstations and more qualitative examples of our collected data.A key advantage of the Any2Policy framework is its ability to outperform modality-specific models by training on multiple modalities for instruction-observation pairs. It raises the question: can including more modalities at the inference stage further improve performance? Table 4 presents our experimental findings. We used the text-image pair as the baseline, a common setup in robotics. Adding additional instructional modalities, such as audio, image-based end-goals, and video demonstrations, consistently increased the success rate. Interestingly, although text and audio convey similar instructions, their combination still yielded a slight performance improvement of 1%. For observations, significant performance gains were observed when adding video and point cloud data, integrating temporal and 3D spatial information. Combining multiple instructional and observational modalities also improved performance; for example, incorporating video demonstrations in instructions and video in observations led to a 12% increase in the success rate.4. Comparison of Any2Policy with state-of-the-art robot models. To further assess the effectiveness of Any2Policy representations, we compare them with other robot models, as shown in Table 3 . Any2Policy consistently outperforms both T5 [110] and R3M [109] across various modalities, highlighting the advantage of incorporating multiple modalities during training. Furthermore, Any2Policy significantly surpasses VIMA [59] , a recent method that uses text goals and object images for task specification. This demonstrates that Any2Policy not only outperforms single-modal counterparts, but is also highly effective in employing multiple modalities during inference.5. Other results. We observe that the performance of image-point cloud and video-point cloud pairings achieves relatively lower scores compared to other modality pairs. However, when we use text or audio as instruction, the performance of point cloud is typically better than image or video due to its accurate position in three-dimensional space. This indicates that there is still room to improve model performance with point cloud inputs. These results highlight the inherent complexity of integrating visual modalities like images and videos with point cloud data, which often requires sophisticated fusion techniques to effectively combine spatial information with traditional 2D features. 
experiments: 
 This section aims to demonstrate the generalizability of our proposed Any2Policy agent. We evaluate our method in two setups. Initially, we conduct experiments in a real-world setting using our own collected dataset, which comprises various modalities. This is aimed at demonstrating the efficacy of our approach across a wide range of instruction and observation modalities. Subsequently, we assess the performance of our method on three simulated benchmarks. Each benchmark is limited to one or a small number of modalities. This phase aims to establish that, by leveraging training on cross-modal information, our method can still deliver competitive or superior results compared to standard benchmarks, even in the presence of a single modality.Evaluation Setup. We conduct our evaluations using a newly constructed dataset of multimodal instruction and observation. The dataset is divided into training, validation, and testing subsets, with a split of 7/1/2, respectively. We position the objects randomly, allowing tasks to be performed on unseen objects.Implementation details. We use an initial learning rate of 3e-5 with the AdamW [107] optimizer, a weight decay of 1e-6, and a linearly decaying learning rate scheduler with a warm-up covering the initial 2% of the total training time [108] . We apply a gradient clipping of 1.0. All experiments are evaluated over 10 trials to obtain the mean success rate. The action space is the absolute joint position.Experiment Results. We aim to answer the following questions with our experiments.1. Does Any2Policy outperform single-modality trained methods? We compare Any2Policy with modality-specific approaches in Table 1 . In particular, all implementations and network architecture remain constant across both settings, except that Any2Policy is trained on a variety of modalities while the modality-specific method is limited to particular ones. We observe that Any2Policy consistently achieves superior performance across all instruction-observation pairs compared to the modalityspecific approach. Given that the total volume of training data is identical due to fixed training steps, this enhanced performance is attributed to learning from multiple modalities, which aids in better model generalization.2. The significance of embodied alignment. The impact of embodied alignment is specifically explored in our experiments presented in Table 2 . We experiment by removing the embodied alignment module and simply concatenating the outputs of projection layers from different modalities. TokenLearner is still utilized to manage the memory burden of numerous tokens. The alignment between instruction tokens and observation tokens is then achieved using an MLP layer, followed by the policy network. Notably, there is a significant performance drop when embodied alignment is omitted.Dataset. Our experiments are based on the following three datasets in our simulation. These three datasets represent three different instruction-observations modalities. Specifically, Franka Kitchen [92] uses text-image and ManiSkill2 [94] uses text-image and text-{image, point cloud}. Comparison with R3M, BLIP-2, and EmbodiedGPT. On all tasks except for Knobs-left with 25 demonstrations, we obtained superior performance over SOTA methods.The objective of our simulated environments is to evaluate the efficacy of our proposed method under conditions where modalities are limited.Implementation Details. All models are trained on A100 GPUs, implemented in PyTorch [111] .The Franka-Kitchen are trained for 40K steps. We use weight decay of 1e-6, cosine learning rate scheduler with warmup steps of 2% total steps. The gradient clip of 1.0 is also applied. We use Adam optimizer with initial learning of 1e-3 and 3e-4 for Franka Kitchen and Maniskill-2. Note that Maniskill-2 does not have text instructions. We augment the task description into instruction via GPT-4. It is then tokenized and prepend with image backbone via FiLM [112] .Evaluation: We evaluate our approach using 30 rollouts from the BC learned policy. We use the mean success rate of the final policy as our metric. When providing task suite metrics we average the mean success rate across camera configurations.Experimental results on Franka Kitchen. In our experiments, we benchmarked our model against two state-of-the-art methods: R3M [109] , and BLIP-2 [22] , a sota vision-language model, and Embodied-GPT [113] , a multi-modal model designed for robotics. Our policy network was trained using few-shot learning, utilizing either ten or twenty-five demonstrations. We assessed the success rate of these models in 100 randomized evaluations across five different tasks in each benchmark. These evaluations were conducted under two settings, each with five separate runs and from two different camera perspectives, using only visual observations. The results, illustrated in Figures 4 for the Franka Kitchen, clearly show that Any2Policy outperforms the baseline methods, on both 10 and 25 demonstrations, except for Knobs-left with 25 demonstrations. 
conclusion: 
 Humans interact with and perceive the world through information from multiple modalities. Existing research in robotics predominantly concentrates on policy learning using a single modality. In contrast, this paper introduces the Any2Policy framework, which adeptly demonstrates the integration of various sensory modalities into a cohesive model. This framework efficiently processes and responds to multi-modal data for robotic tasks. The overall framework, coupled with its multi-modal dataset, represents a significant advancement in the field of embodied AI. Our findings underscore the potential of Any2Policy to augment robotic adaptability and performance, thereby highlighting the importance of multi-modal methodologies in the development of more generalized and versatile robotic systems.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justfication: The data will be attached in the webpage.Guidelines:• The answer NA means that paper does not include experiments requiring code. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We specify all the training and test details.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Because the evaluation of robot manipulation is extremely time-consuming, we report the average success rate over 20 trails on our real-world experiments. For the simulation environment, we follow previous methods which report the average success rate over 30 trails. Details are specified in the paper.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We specify the potential positive/negative societal impacts in the appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [NA] Justification: [NA] Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: This paper presents a multi-modal robotic manipulation dataset encompassing five input modalities: image, text, audio, point cloud, video. This paper demonstrates that a simple approach integrating all these modalities lead to improved performance in robotic manipulation tasks. 

Despite its limited technical novelty, the reviewers have reached a consensus that the curated dataset and comprehensive empirical evaluations offer valuable insights to the robotic learning community. As such, this paper is recommended for acceptance to NeurIPS.

However, most of the reviewers have concerns on this paper's presentation and organization. The author responses during the rebuttal period have resolved some of these issues, but it does not seem sufficient. The authors should carefully address the reviewers' comments and improve the quality of the paper together with more comprehensive experimental results beyond the ones included in the rebuttal.

Reviews: review: paper_summary: paper_summary: The paper aims to enhance the generalizability of robotic agents by enabling them to handle tasks using diverse modalities such as text, audio, images, and point clouds. The authors introduce a multi-modal system named Any-to-Policy, which utilizes a versatile modality network to adapt various inputs and connects with policy networks for effective control. To support this approach, they assembled a comprehensive dataset covering 30 robotic tasks annotated across multiple modalities. Extensive experiments demonstrate the system's capability to adapt to diverse multi-modal inputs in both simulated and real-world settings, showcasing its potential to improve the generalizability and performance of embodied agents in various scenarios when comparing various baselines.
strength_weakness: strength_weakness: 1. While the assembly of a multi-modal dataset is a significant contribution, the paper could improve by providing more detailed information on the diversity and representativeness of the tasks and scenarios included in the dataset. For instance, it would be beneficial to know more about the variation in objects, environments, and task complexities. This information would help to assess whether the dataset sufficiently covers the wide range of scenarios the framework is intended to handle.

2. Although the paper compares the Any2Policy framework with several state-of-the-art models, it could benefit from a more extensive comparative analysis. For instance, including a wider variety of baseline models e.g. RT-1, RT-2, RT-X, Octo, OpenVLA etc. and providing a detailed discussion on the differences in performance could help to better highlight the strengths and weaknesses of the proposed approach. Additionally, a qualitative comparison, showing example outputs or behaviors from different models, could provide valuable insights.

3. The paper includes some ablation studies to assess the impact of different components of the framework. However, these studies could be expanded to provide a deeper understanding of the contribution of each component given that Any2Policy is a fairly complex system. For example, the authors could investigate the impact of different encoder architectures or the role of specific modalities in greater detail. This would help to identify the most critical elements of the framework and guide future improvements.
questions: questions: See the section above.
limitations: limitations: Yes
paper_summary: paper_summary: This submission develops a new model that can handle many different modalities as input for instruction and observations (video, text, image, pointcloud etc.). Different modalities are encoded via different frozen encoders (with projection layers kept trainable) to a shared representation to then be passed into a policy that then generates actions. The submission constructs a multi-modal dataset for several evaluation benchmarks (3 simulated, 1 real), and trains a policy on this multi-modal data and show it out-performs policies trained on a specific modality of data. While work in the past has built multi-modal models in language/vision, this is the first to do so in robotics. Cross attention is used where the observation sequence of tokens are keys and the instruction sequence of tokens are values.
strength_weakness: strength_weakness: - The performance of AnyPolicy looks very similar to EmbodiedGPT in the franka kitchen setting in Figure 4. The figure 4 caption appears to also be wrong as it seems EmbodiedGPT outperforms AnyPolicy on both Knobs Left and Light Left, not just Knobs left. Moreover, no error bars are shown and with results so similar over just 100 evaluations in simulation, it is hard to say if AnyPolicy out performs embodied GPT.
- FrankaKitchen, while a useable benchmark, has almost no initial randomization and is incredibly easy given objects in the environment are not randomized, making results reported on it not mean that much. It would be great to see comparisons on harder manipulation benchmarks although I realize it might not be possible to run experiments now.
- Very few details about the real world dataset are provided. What are the 30 tasks? What are some example instructions? What are some example audio?
questions: questions: Questions:
- Are there meta-world results? I can't find any figures/tables on it, only see results on ManiSkill2 and Franka Kitchen.
- The name of the dataset this paper curated, RoboAny, is mentioned once in the paper in passing and never again. Is there a reason for that?
- Can there be more clarification around how the ablation of no embodied alignment module works? My understanding is if you remove it you simply directly use the encoded tokens and put them all in one big sequence as input to the transformer. However the original encoders might output different shapes for the tokens so I am not sure how this is consolidated.
- Is AnyPolicy trained on data from all benchmarks or is a separate policy trained per benchmark. In other words, is the multi-task policy multi-task over all tasks in a benchmark, or all tasks in all benchmarks?
- How does the modality-specific model work, what is its architecture compared to AnyPolicy?
- How come StackCube has such a high success rate but PickCube has such a low success rate? My understanding is in ManiSkill2 StackCube is much harder than PickCube because it requires careful placement on a cube and releasing the cube, whereas in PickCube you simply grab the cube and move to a random goal.
- How many demonstrations are used for ManiSkill2? This detail could not be found anywhere.
- How is R3M used in the Video-Image result in table 3? How does it process video input and produce video tokens? 

Typo: 
- "We further conduct experiments on two simulated robotics benchmarks to reinforce the strong generalizability of our approach compared to existing methods" in section 1. I think three benchmarks were tested not two.
- Figure 4 shows 110% on the y-axis. It should be clamped to [0, 100].

Unfortunately given the significant lack of details on data, and how some ablations work (see questions below for more) I recommend a reject in the current state. Moreover the performance of the model after combining several new modalities does not appear to out perform other baselines from other papers that much to warrant the amount of extra work on a simple task like FrankaKitchen. I am happy to raise my score to the accept range if these weaknesses and questions are addressed.
limitations: limitations: Limitations are provided in the appendix and discuss limitations from the perspective of additional things that could be done (more modalities, more robots etc.). I think limitations around data efficiency could be important to bring up. Just how many demonstrations are needed? Real-world imitation learning suffers greatly due to the expensive cost of teleoperation (especially if high success rates are desired).
paper_summary: paper_summary: This paper proposes the simultaneous fusion of image, text, point cloud, video, and audio—four modalities—in robotic manipulation tasks, while also considering the integration of information from both instruction and observation. Through the transformer architecture and cross-attention mechanism, embodied alignment of multiple modalities is achieved. Furthermore, this paper constructs a real multimodal dataset and conducts a detailed experimental analysis of the impact of additional modality information on performance.
strength_weakness: strength_weakness: 1. The technical contribution of this paper is limited. Although the paper analyzes some difficulties in the process of fusing multimodal information and embodied alignment, the overall model architecture is still relatively straightforward and simple.

2. The scale of the dataset is relatively small compared to the widely used robotic operation datasets currently available. Considering that the model architecture and training methods for multimodal information fusion typically require a large amount of data, it may be necessary to have a larger dataset to verify stronger generalizability or to draw more robust conclusions on multimodal information fusion. Additionally, some existing datasets are already capable of including modalities such as text, image, video, and point cloud simultaneously. The main contribution of this paper seems to lie in the addition of rich paired audio data. If one considers expanding the scale of the dataset as well as costs, would it be a better choice to augment the existing datasets with information from modalities such as audio?

3. The experimental section does not yield particularly new conclusions. Generally speaking, it is not surprising that the addition of new modalities, especially for robotic manipulation tasks, typically leads to better performance.
questions: questions: Considering the excellent development of current multimodal pre-trained models, and that the models in this paper are trained from scratch, is it feasible to incorporate existing pre-trained models such as text-image, audio-image, image-point cloud, and text-point cloud models into the current training framework?
limitations: limitations: The authors have adequately addressed the limitations andsocietal impact of the work.
paper_summary: paper_summary: This paper aims to enable robots to understand tasks and their environments using multiple modalities such as text, audio, images, video, and point clouds. To accomplish this, the authors introduce Any2Policy, a versatile framework designed to process multi-modal inputs at the instruction and observation levels. Within this framework, they developed an embodied alignment module that effectively integrates multi-modal features. Additionally, the use of ImageBind and TokenLearner helps to mitigate tedious encoder design and enhance training efficiency, respectively, which appears to be a practical approach. The paper's thorough evaluations, conducted on 30 real-world tasks and three distinct simulation benchmarks, convincingly demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: However, there are still some concerns regarding the experiments.

1. While the authors have conducted extensive ablations on real-world experiments, replicating these results in practice may be challenging for follow-up researchers. To facilitate better benchmarking, it would be advantageous if the effectiveness of each proposed method could also be demonstrated in simulated experiments.

2. Due to the high cost associated with evaluating real-world tasks across different random seeds, reporting variance is difficult. Nonetheless, it would be beneficial if the authors could assess the impact of multiple random seeds in simulation tasks to provide more robust statistical insights.

3. The authors appear to have omitted the results from the meta-world evaluations. These results were neither found in the main text of the paper nor in the appendix, despite the authors claims that they were reported.

4. MUTEX [1] is another significant work that explores task specifications using multi-modal inputs. While Any2Policy incorporates a broader range of observation modalities (image, point cloud), including the comparison with MUTEX could enrich the analysis, especially given the limited range of baseline methods evaluated in this paper.


[1]  Mutex: Learning unified policies from multimodal task specifications
questions: questions: 1. I'm curious to know whether the video instructions were recorded using the same embodiment or if they include cross-embodiment examples, such as demonstrations by humans?
limitations: limitations: The authors have properly discussed the limiations and societal impacts in the Appendix.

Comments: title: Response
comment: Thanks for the authors' sincere response. Despite the objective shortcomings that I mentioned, I generally recognize the contribution and solidity of this paper. Therefore, I maintain a positive attitude towards this paper overall, and I will discuss the review results with other reviewers in the follow-up.
Thanks for the comprehensive response. 

R2C2: I'm aware you test on other benchmarks, but surprised you keep FrankaKitchen in the main text and then the harder ManiSkill 2 benchmark in the supplemental (many people often miss the supplemental). Is there a reason for this? At minimum since there are MetaWorld results you can include those in the main text (they are far more impressive and significant, larger differences in success rates on harder tasks with a bit more diversity than franka kitchen).

R2C10: What do you mean keyframe, how are keyframes selected? If these are hand picked or by some algorithm, could it be that the keyframe selection is poor leading to worse results?

All my concerns are essentially covered. My main concern left really is the paper does not seem to be polished given some key figures were missing to begin with (and are only present in openreview comments, not revised pdfs submitted here).

I have raised my score to 6 to reflect that I do wish to accept the paper. It is straightforward and fairly simple, easy to understand, and even if the results are highly expected, someone had to do the experiments and I really appreciate the efforts, I imagine a good amount of engineering is necessary to get this kind of project working. But I think its presentation and organization could be better. A big presentation issue for example is with regards to the choice to only include franka kitchen in the main text, without including MetaWorld or ManiSkill2.
Thanks for the careful comments. My major concerns are well-resolved. So, I'm happy to increase my confidence to reflect this (from 3 to 4).