Contrastive Imitation Learning for Language-guided Multi-Task Robotic Manipulation

Authors: teli  ma
jiaming  zhou
zifan  wang
ronghe  qiu
junwei  liang
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-06 08:50:49

Abstract: Developing robots capable of executing various manipulation tasks, guided by natural language instructions and visual observations of intricate real-world environments, remains a significant challenge in robotics. Such robot agents need to understand linguistic commands and  distinguish between the requirements of different tasks. In this work, we present $\mathtt{\Sigma\mbox{-}agent}$, an end-to-end imitation learning agent for multi-task robotic manipulation. $\mathtt{\Sigma\mbox{-}agent}$ incorporates contrastive Imitation Learning (contrastive IL) modules to strengthen vision-language and current-future representations. An effective and efficient multi-view querying Transformer (MVQ-Former) for aggregating representative semantic information is introduced. $\mathtt{\Sigma\mbox{-}agent}$ shows substantial improvement over state-of-the-art methods under diverse settings in 18 RLBench tasks, surpassing RVT by an average of 5.2% and 5.9% in 10 and 100 demonstration training, respectively. $\mathtt{\Sigma\mbox{-}agent}$ also achieves 62% success rate with a single policy in 5 real-world manipulation tasks. The code will be released upon acceptance.

Main paper: 
 introduction: 
 One of the ultimate goals of robotic manipulation learning is to enable robots to perform a variety of tasks based on instructions given by humans in natural language. This requires robots to understand and distinguish minor variations in linguistic commands and visual cues. However, training robots is difficult due to the limited rewards available in simulated environments and the lack of extensive real-world data. Imitation learning is an effective off-policy method that avoids complex reward design and low-efficient agent-environment interactions [2, 3, 4, 1] . In this paper, we focus on imitation learning for 3D object manipulation.Previous works have mainly concentrated on improving the perception ability of robotic agents, but ignoring the development of discriminating different instructions and related tasks. A portion of these studies has been directed towards enhancing the transferability from 2D pre-trained visual representation to the real-world [2, 7, 8, 9] . Nonetheless, to maintain geometric details for both simulated and real-world environments, 3D vision learning dominates in instruction-guided manipulation [4, 3, 10, 1, 11, 12, 13, 14] . For instance, C2FARM [4] leveraged 3D ConvNets to aggregate visual representations based on pre-constructed voxel space. PerAct [3] constructed voxelized observations and discrete action space on RGB-D images, utilizing Percevier [15] Transformer to encode features. Moreover, PolarNet [11] directly encoded the point cloud features reconstructed from RGB-D to predict actions. However, these works do not address how to train visual representations to align with linguistic features and differentiate between multiple tasks.Previous methods [2, 3, 16, 11, 1, 12, 13, 14] can be summarized as training the agent to learn a parameterized policy π θ that imitates the target policy π + based on data collected using the target Figure 1 : Left: t-SNE [5] visualization of multi-task representation learning without/with contrastive IL, and learning with contrastive IL shows a much more obvious separation of features belonging to different tasks. Right: Visualize the interested regions with Grad-CAM [6] , which shows accurate object-level understanding. policy via behavior cloning (BC) loss. These approaches typically involve mapping visual representations ϕ, linguistic representations ψ, and vision-language interactions δ to low-level endeffector actions. To tackle the aforementioned misalignment problem, we introduce an end-to-end contrastive Imitation Learning (contrastive IL) strategy to the original language-conditioned policy learning, inspired by contrastive Reinforcement Learning (contrastive RL) methods [17, 18, 19, 20] . The contrastive IL framework strengthens the training process by establishing: 1) a relationship between language instructions and corresponding visual inputs, which helps differentiate multi-task representations, and 2) a link between current and future states, which enhances the recognition of movement trajectories across different tasks. This contrastive loss provides additional supervision to the feature extraction ϕ and interaction δ components (as shown in Fig. 2 (b )), complementing the standard BC loss.Based on the contrastive IL, we present an end-to-end trained language-conditioned multi-task agent to complete 6-DoF manipulation, dubbed as contraStive Imitation learninG for Multi-tAsk manipulation agent (SIGMA-agent, abbreviated as Σ-agent ). Σ-agent follows the state-of-the-art baseline model, RVT [1] , and leverages the re-rendered virtual images from RGB-D reconstruction to explicitly represent visual information. We propose a Multi-View Querying Transformer (MVQ-Former) [21, 22, 23] to minimize the number of tokens for efficient contrastive IL. The Σ-agent framework offers guidance on how to incorporate contrastive learning into existing imitation learning methods, while the inference process stays the same.Experiments on RLBench [24] and real-world tasks demonstrate the effectiveness of the Σ-agent . The results on RLBench show Σ-agent significantly outperforms previous agents in both the 10 demonstrations (by +5.2% on average) and the 100 demonstrations (by +5.9% on average) training under the setting of one policy for 18 tasks with 249 variations. Also, we integrate the contrastive IL module into existing methods (PolarNet [11] , RVT [1] ) and experiment Σ-agent on another simulation environment (Ravens [25] ). The significant improvements show the general applicability of our approach across various models and environments. Σ-agent also achieves 62% multi-task success rate on average with a single policy over 5 tasks in real-world robot experiments. 
language-conditioned robotic manipulation: 
 Language-conditioned robotic manipulation has emerged as a pivotal research branch in the robotics domain due to its extensive applicability in human-robot interaction [26, 2, 3, 1, 11, 16, 12, 13, 14, 27, 28, 29, 30] . Many previous studies have delved into vision-based representations and strategies for vision-language interaction in policy learning. For example, RT-1 [31] encodes multi-modal tokens via a pre-trained FiLM EfficientNet model and feeds them into the Transformer for multi-modal information aggregation. The later version RT-2 [32] leverages the auto-regressive generative capacity of LLMs to project visual tokens into linguistic space, and uses LLMs to generate the actions directly. Diverse benchmarks have been curated to benchmark the language-conditioned manipulation [24, 25, 33, 34, 35, 36] . In this paper, we mainly focus on RLBench [24] , which provides hundreds of challenging tasks and diverse variants covering object poses, shapes, colors, sizes, and categories to evaluate agents based on RGB-D cameras.Numerous efforts have been made in this challenging benchmark. C2FARM [4] , PerAct [3] and GN-Factor [14] harness the 3D voxel representation for policy learning. C2FARM [4] detects actions at two levels of voxelization in a coarse-to-fine manner. PerAct utilizes the Perceiver network [15] to encode 3D voxel features to predict the next keyframe positions with lower voxel resolution than C2FARM [4] . To refine the 3D scene geometry understanding, GNFactor [14] incorporates a generalized neural feature fields module to distill 2D semantic features into NeRFs [37] based on Per-Act [3] . Besides the voxelized features, policy learning based on 3D point cloud representation has gained significant attention such as PolarNet [11] , Act3D [12] and ChainedDiffuser [13] . PolarNet trains agents on 3D point clouds constructed from RGB-D and adopts pre-trained PointNext [38] to extract point-wise features. Both Act3D [12] and ChainedDiffuser [13] employ a coarse-to-fine sampling strategy to select 3D points in space and feature them with relative spatial attention, while ChainedDiffuser [13] synthesize end-effector trajectories with a diffusion model. Our work follows RVT [1] , which re-renders virtual view images from reconstructed 3D point clouds and processes the images using a Transformer network.A large volume of prior work combines representation learning objective with RL objective [39, 40, 41, 42, 43, 44] . Contrastive learning has gained significant attention among these representation learning methods [40, 41, 45, 46, 43, 47] . Recently, the paradigm of unifying the representation learning and reinforcement learning objective has emerged as a research hotspot in the field of RL [19, 45, 18, 48, 49, 50, 51] . For instance, C-learning [19] regards goal-conditioned RL as estimating the probability density over future states, learning a classifier to distinguish the positive future state from the random states. Eysenbach et al. [17] demonstrate that contrastive representation learning can be used as value function estimation, connecting the learned representations with reward maximization. Zheng et al. [18] propose to stabilize contrastive RL in offline goal-reaching tasks, analyzing the intrinsic mechanism of contrastive RL deeply to explore ingredients for stabilizing offline policy learning. Note that the contrastive RL methods above mainly focus on reinforcement learning with reward updating. One of the most similar works to ours is GRIF [52] , which learns linguistic representations that are aligned with the collected transitions in the trajectory via contrastive learning. However, our contrastive IL is different from GRIF [52] in three aspects. First, contrastive IL is an end-to-end training paradigm, while the GRIF [52] decouples the contrastive representation pre-training and policy learning into two phases. Second, we target the 3D multi-task setting in this paper, whereas GRIF [52] utilizes RGB images and single policy training. Lastly, GRIF [52] performs contrastive learning on (state, goal) pairs with language instructions. For contrastive IL, we perform contrastive learning to refine both the feature extraction and vision-language feature interaction. More related work is present in Appendix D. 
method: 
 Fig. 2 provides an overview of the Contrastive IL design in Σ-agent (Fig. 2 (a) ) and how to apply to the previous language-guided imitation learning paradigm (Fig. 2 (b) ). In this section, we introduce the components of Σ-agent . Additional details about Σ-agent are provided in Appendix C.We assume a language-conditioned Markov decision process (MDP) defined by states s t ∈ S, actions a t ∈ A and language instructions l ∈ L. S, A are the state and action spaces, and LUse the stick to drag the cube onto the teal targetLanguage Encoder represents the set of language instructions. The goal is to learn a policy π : S × L → A to maximize the expected rewards of predicted actions. Following previous work [3] , we utilize behavior cloning to maximize the Q-function without specific rewards. Therefore, the objective of the policy learning can be formulated as:Visual Encoder … … Feature Fusion Proj … q v q v q v,l q v,lπ θ (a | s, l) π θ (a | s, l) π + π + ψ(l) ψ(l) ϕ(s) ϕ(s) δ(s, l) δ(s, l) a a BC Loss a + a + (b) Language Instruction (a)EQUATIONwhere θ means parameters of the policy network and D represents the transitions from demonstrations collected for behavior cloning. Note that D is sampled from the expert policy π + , and we train the θ to drive π θ to mimic the π + . In this paper, the state s t includes aligned RGB and depth images from the front, left shoulder, right shoulder, and wrist position. We follow RVT [1] We acquire re-rendered virtual images following RVT [1] from 5 cubic viewpoints: the front, left, right, behind and top. Each view image comprises RGB, depth, and (x, y, z) coordinates channels.The visual encoder comprises a patch embedding layer and a two-layer self-attention Transformer.We split the images into 20 × 20 patches and leverage an MLP layer to project the embeddings of patch tokens for self-attention. For the self-attention Transformer, each patch token only attends to other tokens within the same virtual view image, which aims to aggregate the information from the same view. The visual encoder is trained from scratch with normalized initialization.For the language encoder, we follow previous works [3, 16, 11, 1, 12] and use pre-trained language encoder from CLIP [53] for fair comparison. The language encoder remains frozen during training.The language token embeddings from the encoder are then projected by a trainable MLP for the cross-attention with visual tokens.With the extracted visual features from the visual encoder, we follow the query Transformer [23, 53, 21, 22] to pre-define a set of learnable queries. These queries are utilized by the contrastive IL module, minimizing the computational complexity by reducing the number of original visual tokens. The number of learnable queries is set to 5, one for each virtual view to aggregate the intra-view information. The MVQ-Former in Fig. 2 consists of two cross-attention layers, where the query tokens co-attend to the extracted visual features. We denote the queries produced by MVQ-Former as q v .The query tokens, visual tokens and language tokens are then concatenated together and fed into 4 self-attention layers for feature fusion. During this process, the queries interact sufficiently with both the visual and language features, resulting in queries at this stage named q v,l . The q v and q v,l are utilized for contrastive IL as described in the following section. Finally, the context features (denote as v) after the self-attention layers are input to the decoder for action prediction.For multi-task agents conditioned by language instructions, we propose the key question: What are effective task representations for accurate, fine-grained controls? Two factors matter. First, aligning task and language representations is essential for the agent to comprehend correlations and differentiate between tasks. Second, the agent needs discriminative features to accurately perceive the current state.Learning Objective. Inspired by contrastive RL [19, 17, 20, 18] , we propose to integrate the contrastive representation learning into current imitation learning framework to train the policy in an end-to-end manner. The contrastive IL comprises state-language (s ↔ l) and (state, language)future (s, l ↔ g) contrastive learning to refine both the feature extraction and interaction as shown in Fig. 2 .For the s ↔ l, we define the similarity function as f ϕ,ψ = exp (ϕ(s t ) ⊤ ψ(l)/τ ) (τ : temperature parameter), where ϕ and ψ are the state and language instruction representations, respectively. The contrastive IL objective between states and language instructions is:EQUATIONwhere N is the batch size in training, and s + t , l + denotes corresponding positive states and instructions. L -and S -stand for all negative language and state samples in the current batch. Essentially this enforces the network to learn more discriminative representations by aligning state and language pairs in the joint embedding space. For the (s, l ↔ g), we randomly sample future states g from D, and applying network φ to encode. δ encodes the joint representation of (s t , l). The negative samples are from the other tasks in the same batch. Similar to Eqn. 2, we formulate the contrastive training loss between the future and state, language pairs as:L (s,l)↔g (s t , l, g) = 1 N N log f δ,φ ((s + t , l + ), g + ) f δ,φ ((s + t , l + ), g + ) + g -∈G -f δ,φ ((s + t , l + ), g -) + 1 N N log f φ,δ (g + , (s + t , l + )) f φ,δ (g + , (s + t , l + )) + (s - t ,l -)∈Ω -f φ,δ (g + , (s - t , l -)).(EQUATIONThe L (s,l)↔g aims to refine the representation of vision-language interaction δ, which is crucial for the vision-language understanding of agents. G -and Ω -signify negative space of future states and (s t , l) in the current batch. Based on the L s↔l and L (s,l)↔g , we re-formulate the learning objective in Eqn. 1 as:max θ E (s t ,a t ,l,g)∼D, g + ∼P π + (•|s t ,l)[λ log π θ (a t |s t , l)EQUATIONwhere λ is the coefficient. The positive future state g + conforms to the transition probability P π + of the target policy π + . More discussions can be found in Appendix C.2.Module Details. For the contrastive training between language and state observations, we choose the last token (i.e., [EOS]) as the feature representation of the whole text following CLIP [53] , which is then linearly projected into the multi-modal embedding space. The queries q v are projected along the token dimension to aggregate the representative visual features as shown in Fig. 2 (a) . Then, the visual token and text token are trained to align in the joint embedding space based on Eqn. 2.We utilize the next state in the trajectory as the future state. The visual encoder of the future state shares the same backbone parameters with that of the current state as shown in Fig. 2 . Hence, the φ in Eqn. 3 equals to ϕ in Eqn. 2. The queries q v,l , which include both the visual features of the current state and the language features, are projected to perform contrastive training with averagepooled future-state features. This process conforms to Eqn. 3. Note that the contrastive IL module is designed to enhance representations during training but is disabled during the inference process. 
simulation experiments: 
 Simulation Setup. RLBench is a robot manipulation benchmark built on CoppelaSim [54] and PyRep [55] . We follow the protocols of PerAct [3] and RVT [1] to test the model on 18 tasks in RLBench [24] . These tasks, which include picking and placing, bulbs screwing, and drawer opening etc., are all performed by controlling a Franka Panda robot with a parallel gripper. Details of the 18 tasks and their variations are provided in Appendix A. Implementation Details. We follow RVT [1] to perform training on cube-viewed re-rendered images from 3D point clouds. We evaluate Σ-agent with 10 and 100 demonstrations per task for training. Following previous work [24, 56, 3, 1] , we perform behavior cloning on the replay buffer of extracted keyframes rather than all frames from episodes. Similar to PerAct [3] , we adopt translation and rotation data augmentations during training, perturbing the point clouds randomly in the range of ±0.125m for translation and rotating the point clouds along the z-axis within ±45 • . For the training scheme, we train Σ-agent for 25K steps with a batch size of 96 and an initial learning rate of 9.6 × 10 -4 . The LAMB [57] optimizer is applied, with cosine learning rate decay and 2K warming steps. The training is conducted on 8×NVIDIA A6000 GPUs for around 22 hours. Σ-agent is evaluated on all 18 tasks with variants. The initial observations are given to the agent, and the agent explores to reach the final state by the observation-action loop. The agent scores 100 for reaching the final state and 0 for failures without partial credits. Following PerAct [3] , we report the average success rates on 25 episodes for each task and the average success rates on all 18 tasks.Comparison with the State-of-the-art Methods. We evaluate the Σ-agent five times on the same 25 episodes for each task and report the mean results due to the randomness of the sampling-based motion planner. The evaluation comprises two settings, training with 10 (Table 8 ) or 100 (Table 1 ) demonstrations per task. We re-implement the results of PolarNet (100 demos) and RVT (10 demos) since they are missing from the original paper. Other results are taken from the related literature.From Table 1 and 8, Σ-agent outperforms previous methods on both the 10 and 100 demonstrations by a large margin, up to 5.2% and 5.9% in average success rate over 18 tasks, respectively. In specific tasks, our Σ-agent achieves state-of-the-art performance on 13 out of 18 tasks for both the 10 and 100 demonstration settings.Moreover, we compare Σ-agent with two state-of-the-art models, Act3D [12] and ChainedDiffuser [13] , which are trained on 256 × 256 input resolution and tested on 100 episodes for each task. As shown in Contrastive Imitation Learning for Baselines. To validate the effectiveness of contrastive IL, we integrate the contrastive IL module into other baseline models. We choose PolarNet [11] and RVT [1] as the baselines to demonstrate improvements in representations of point clouds and 3D re-rendered images. The main network and inference pipeline are kept unchanged, with only a contrastive IL module incorporated, adding a negligible increase in parameter count during the training process. As shown in Table 3 , the contrastive IL module improves the performance of both Po-larNet [11] and RVT [1] by +2.8% and +1.8% in average success rate over 18 tasks, respectively. Specifically, the performance of most tasks is improved (13 out of 18 and 11 out of 18), with a largest margin of 13.9% improvement. These improvements demonstrate two points: first, our proposed contrastive imitation learning can transfer across multiple models. Second, the learning method is effective for both 3D re-rendered images (RVT [1] ) and point cloud representations (PolarNet [11] ).Future State and Language Ablations. We ablate the influence of future state and language contrastive IL as shown in Fig. 3 (a). We summarize three key points from the results. (1) Both language and future-state contrastive learning with current observations enhance performance. (2) Contrastive learning between current observations and language instructions accelerates the convergence speed of agent training, achieving higher performance at the early stage of training. (3) When future-state contrastive learning is added to a model that already includes language-state contrastive learning, the performance achieves a more substantial gain. We believe that the limited improvement in only utilizing future-state contrastive learning can be attributed to the insufficient demonstrations as the trajectory is the unit for constructing contrastive learning categories. Consequently, without leveraging language features, it is challenging to learn discriminative features. Coefficient λ Ablations. In Eqn. 4, λ is the hyper-parameter that conditions the relation between contrastive IL and behavior cloning. We vary the λ ranging from [0, 1] to find the optimal value. From Fig. 3 (c), we train the agent with five different λ values and observe it makes slightly different results. We choose λ = 0.5 for agent training as it is the relatively optimal value. We conduct experiments on a real robot, a 6-DoF UR5 robotic arm. The Σ-agent is validated on 5 realworld tasks, including a total of 9 variants. For each task, we collect 10 human demonstrations and train the Σ-agent with a single policy from scratch using all task demonstrations. Details of the real-world setup and datacollecting are provided in Appendix B. Table 4 presents the real-world results. We test the Σ-agent in 10 episodes for each task, and it achieves an average success rate of 62% across all tasks. We show the failure cases in Fig. 8 .To analyze the reasons for failure: first, the limitation of a single front-view camera cannot provide precise visual information for tasks that require aiming, such as "Put tennis in barrel" and "Stack cups". Second, during the grasping process, imperfect grasp poses result in the translation or orientation of objects, worsening the collision problems. In the future, we plan to add an extra RGB-D camera at the wrist position to provide a first-person view. Additionally, integrating a pose estimation model for objects into Σ-agent would improve grasping poses and avoid collisions. 
conclusions and limitations: 
 In this work, we propose contrastive IL, a plug-and-play imitation learning strategy for languageguided multi-task 3D object manipulation. The contrastive IL optimizes the original imitation learning framework by integrating the contrastive IL module to refine both the feature extracting and interacting. Based on the contrastive IL, we design an end-to-end imitation learning agent Σ-agent utilizing the re-rendered virtual images from RGB-D input. Σ-agent is effective and efficient in both simulated environments and real-world experiments. However, we identify some limitations that exist. First, the scale and the number of camera views of our real-world experiments are limited. Second, the contrastive IL module in our method only demonstrates the effectiveness of closed-set scenarios like RLBench, but does not show the applicability in open-vocabulary scenarios. Lastly, like previous work, the generalization capacity of Σ-agent is limited, as shown in Table 7 . We leave these issues for future work. The RLBench setting in our paper is elaborated in Table 5 . We follow the PerAct [3] to use the 18 tasks with 249 variations in the RLBench. The examples of the 18 tasks and corresponding human instructions of specific variants in these tasks are shown in Fig. 4 .A.2 Effectiveness in Other Simulated Environments.Ravens [25] is a simulated benchmark environment built with PyBullet [59] for robotic rearrangement based on a Universal Robot UR5e. There are 3 simulated 640x480 RGB-D cameras covering the 0.5×1m tabletop workspace from the front, left shoulder and right shoulder. We train the model on six tasks in the Ravens environment [25] as shown in Table 6 , selecting from the seen split of CLIPort [27] . The models are trained on 100 demonstrations and evaluated on 100 evaluation instances for each task. The Ravens benchmark scores 0 for task failure and 100% for success. The Table 7 : Evaluation on the generalization capacity of models. We test the models trained on 18 seen tasks directly on the 14 unseen tasks selected from the 74 tasks of RLBench [24] . A.3 Visualization.To qualitatively illustrate the effectiveness of contrastive IL, we visualize the multi-modal similarity between visual observations and language instructions. As shown in Fig. 5 , we compute the cosine similarity between the embeddings of observations in a trajectory of close jar and different language instructions. For Σ-agent without contrastive IL module, we utilize the original text projection layer of CLIP [53] to project the textual embeddings for similarity computation. It is evident that contrastive IL amplifies the alignment between language instructions and local visual observations. For example, in the third keyframe, the agent without CL tends to confuse the close jar task with screw bulb and sort shape, as the three task instructions have close similarity with the visual observation. 
 
 Appendix: We conduct real-world experiments on a table-top setup with a 6-DoF UR5 robotic arm. The visual input is provided by a third-person perspective Orbbec Femto Bolt2 (RGB-D) camera mounted directly above and facing forward. The camera streaming the RGB-D images of 1280 × 960 (hardware D2C align) at 30 Hz. The images will be resized to 640 × 480 for feeding into the model. The extrinsics of the RGB-D camera and robotic arm are calibrated via hand-eye calibration. We mount the ARUCO3 AR marker in the base of UR5 arm. Additionally, we mount a DJI Osmo Action 4 4for recording inference demos. The hardware setup is shown in Fig. 6 .We collect the human demonstrations by manual teaching, dragging the robotic arm to pre-defined keypoints and collecting the visual observations and gripper poses. These poses are executed with a motion-planner using ROS5 and MoveIt6 . We define 5 tasks to experiment, including Stack cups, Put fruit in plate, Hang mug, Put item in barrel, and Put tennis in mug. We The Σ-agent is trained on 50 demonstrations for 5 tasks. The training samples are augmented with ±0.125m translation perturbations and ±45 • yaw rotation perturbations following PerAct [3] . The When the g + is the next state of s t in Eqn. 3, the f δ,φ maximizes the similarity between (s t , l) and s t+1 . In other words, training f δ,φ aims to maximize the transition probability from s t to s t+1 with the language instruction l by minimizing the distance between corresponding pairs [(s t , l), s t+1 ], while simultaneously maximizing the distance of negative pairs. Therefore, training of f δ,φ with L (st,l)↔g=st+1 is beneficial for maximizing the P π (s t+1 |s t , l), thereby maximizing the Q π (a t |s t , l). f δ,φ can be an extra critic function to facilitate the policy π mimicking target policy π + in the level of representation learning.Learning a single agent for multiple tasks is of vital significance to robotic learning. One of the main challenges in multi-task learning is the conflicting representations and gradients among different tasks. Previous works address multi-task learning via strategies like knowledge transfer [60, 61] , representation sharing [62, 63, 64] , and gradient surgery [65] . With the advent of large visionlanguage models [53, 21, 22, 66] and LLMs [67, 68, 69] , language instructions serve as complementary hints to differentiate task representations in policy learning [31, 32, 16, 11, 3, 1, 12, 13, 14] .In this paper, we leverage contrastive learning to amplify the distinction function of linguistic hints.https://www.orbbec.com/products/tof-camera/femto-bolt/https://github.com/pal-robotics/aruco_roshttps://www.dji.com/cn/osmo-action-4https://ros.org/https://docs.ros.org/en/kinetic/api/moveit_tutorials/html/ 


Decision: This paper proposes an end-to-end imitation learning method which introduces auxiliary contrastive learning objectives to align visual representations with language instructions as well as the language-visual representation with the goal. Additionally, a multi-view querying transformer is proposed as an important component for efficiently incorporating contrastive imitation learning into the RVT framework. 

Overall, reviewers appreciated the motivation of the proposed method, the diverse simulated benchmark and real-world experiments, and the high writing quality. However, reviewers also raised significant questions and concerns. Of particular note:

- Most reviewers note that the empirical gains are marginal. Furthermore, the ablations suggest that the performance improvements can not necessarily be attributed to the novel contributions of the work.
- Numerous reviewers request clarity on comparing the work against RVT and RVT2, both in the form of discussions as well as analysis.
- Some reviewers raise questions about the novelty of the work, since prior robot learning works have also incorporated auxiliary contrastive losses.
- Some reviewers note that it’s unclear whether the method can scale to more complex or unseen tasks.

The reviewers raise other questions and also provide helpful suggestions for improvement. It would be helpful for authors to address these during the rebuttal phase.

Update: The authors provided thorough and helpful responses to the numerous concerns raised by 7 reviewers. After discussions in the post-rebuttal phase, most reviewers are positive about the work and appreciate the statistically significant (if marginal) performance improvements over a variety of strong methods. *This work will be a valuable contribution to CoRL, so I am recommending an acceptance*. However, I recommend that the authors address the remaining reviewer feedback, especially those regarding performance and generalization. One reviewer provides the following helpful suggestions for the camera-ready:
- calling this goal-state contrastive learning is somewhat misleading. one would not normally associate the next state as a "goal". the semantics here need to be rephrased and we need to see an updated manuscript with a better interpretation of what the role of this "goal-state" contrastive learning is.
- we need to better understand why both language and goal contrastive learning are needed, and that why without language contrastive learning we do not see many benefits to goal contrastive learning.

Reviews: review: title: title: summary of Contrastive Imitation Learning
paper_summary: paper_summary: This study presents Σ-agent, an end-to-end imitation learning agent designed to address language-guided multi-task robotic manipulation.
main_review: main_review: **Summary**
This study presents Σ-agent, an end-to-end imitation learning agent designed to address the challenge of language-guided multi-task robotic manipulation. The core innovation lies in the application of Contrastive Imitation Learning (CIL) to enhance the understanding of vision-language and current-future representations. The Σ-agent introduces a Multi-View Querying Transformer (MVQ-Former) to aggregate semantic information and demonstrates significant performance improvements across 18 RLBench tasks, surpassing state-of-the-art methods.

In the methodology section, the article details the design of the Σ-agent, particularly how it utilizes contrastive learning to optimize feature extraction and vision-language interaction. Experimental results indicate that the Σ-agent achieves average success rates of 5.2% and 5.9% improvement in 10 and 100 demonstration training, respectively. Additionally, it achieves a 62% success rate in five real-world manipulation tasks, demonstrating its effectiveness in real-world scenarios.

**Strengthens**

1.This paper innovatively integrates contrastive imitation learning into the imitation learning framework, enabling robots to better understand and differentiate multi-task language instructions.

2.The proposed MVQ-Former effectively aggregates information from multiple views, reducing computational complexity while enhancing model performance.

3.The experiments in both RLBench and real-world tasks demonstrates the effectiveness of the proposed method, showcasing its potential applications in multi-task learning and real-world environments.

**Weakness**

1.The model may still face challenges when handling complex tasks, particularly those requiring precise visual information, as the current visual input may not suffice.

2.Dependence on Human Demonstrations: What are the implications of relying on human demonstrations for training? How might this affect scalability and applicability in diverse environments?

3.Generalization to Unseen Tasks: How does the Σ-agent perform when faced with tasks that were not part of the training set? Is there evidence to suggest that the model can generalize effectively to novel tasks?

4.Future Directions: What are the next steps for improving the Σ-agent? Are there plans to enhance its robustness or expand its capabilities beyond the current scope?
questions: questions: See weakness.
limitations: limitations: 2
review_summary: review_summary: Overall, this article proposes a novel learning approach in the field of robotic manipulation, effectively improving model performance through contrastive imitation learning, while still facing challenges in real-world applications.
title: title: Official Reivew - Submission 269
paper_summary: paper_summary: This paper proposes $\Sigma$-agent, a novel multi-task robotic manipulation learning agent. $\Sigma$-agent introduces new contrastive imitation learning modules that help to learn discriminative features to accurately perceive the current state and also to align task and language representations better. Then, a multi-view querying transformer module (MVQ-Former) is proposed to aggregate visual information into a set of 5 learnable queries (one per available input camera) that are fused later on with language features to produce the context features from which the final robot action is decoded. $\Sigma$-agent outperforms other baselines on 18 RLBench tasks. It shows how the proposed contrastive imitation learning modules can be plugged during training to previous methods (RVT and PolarNet) to improve its final performance. Finally, the paper demonstrates that the method can be deployed on a real-robot performing 5 different manipulation tasks.
main_review: main_review: ### Strengths

- The proposed contrastive imitation learning modules and MVQ-Former seem to be novel enough in the context of multi-task learning and $\Sigma$-agent outperforms previous state-of-the-art methods in RLBench and Ravens.
- The proposed contrastive IL module is easy to plug into previous methods such as PolarNet and RVT to improve its performance on RLBench. 
- The experiments show how Σ-agent can be deployed on a real robot with a 62% success rate across 5 different tasks. Also, the authors include nice videos of some real robot executions.
- The paper is well written and the visualization in Appendix Figure 5 is very nice to show that the learned visual representations are well aligned with the task instructions representations. 

### Weaknesses
- The contrastive IL module improvements when integrated with PolarNet and RVT seem marginal, as it only slightly improves its original performance by +2.8 and +1.8 respectively.
- The real-world experimental videos are missing the failure cases. It would be nice to see some of the failure modes to understand better the method limitations.
- The performance in the ablation graph Figure 3 (a) w/ language and goal and w/ language seems very close and especially in the last data point seems that adding a goal is very beneficial. What happens when the training is longer than 25K steps? Have the performances already plateaued by this point? 
- It is a bit strange that Table 2 comes before Table 1 in the text.
questions: questions: Please, take a look at the weaknesses in the review. Questions should be addressed to make the paper stronger.
limitations: limitations: 3
review_summary: review_summary: My recommendation is Weak Accept. I believe that $\Sigma$-agent and specifically the proposed contrastive imitation learning modules will benefit the robotics community. While contrastive learning already exists and has been well studied, the proposed modules are novel and can be easily integrated with other language-guided robotic manipulation agents. However, when integrated with RVT and PolarNet, the performance only improves slightly. Further discussion with the authors and other reviewers' points of view can help towards the final decision.                                                             ||||||    Update: I want to thank the authors for the effort and work put into the rebuttal to respond to the 7 reviews. I will keep my final recommendation as Weak Accept. The authors addressed most of my concerns. However, I still believe that the improvements of this work when plugged into other methods remain very limited.
title: title: Review of Submission269
paper_summary: paper_summary: This paper introduces a contrastive imitation learning approach for language-guided multi-task robotic manipulation. The paper builds upon the RVT [1] with the key additions being (i) the integration of contrastive learning into the imitation learning framework to refine vision feature extraction and vision-language feature interaction, and (ii) introducing a multi-view querying transformer for efficient contrastive RL. The authors evaluate their method on the RLBench benchmark demonstrating some improvements over state-of-the-art methods. The authors also test their method on a few real-world tasks.   [1] RVT: Robotic View Transformer for 3D Object Manipulation
main_review: main_review: ## Strengths

1. The paper presents an intuitive approach to incorporate contrastive learning into imitation learning for improving multi-task robotic manipulation.
2. The authors provide a comprehensive baselines in the RLBench experiments and also provide ablation studies analyzing the impact of different components of their method.
3. The paper is overall well-written, well-organized and easy to follow - great job on the writing!

## Weaknesses

1. While the addition of contrastive loss shows improvements, I am not convinced that they are significant. The paper does not report the variance of success rate across tasks in RLBench (which I think is important, given that RLBench uses a sampling-based planner AFAIK). Looking at the results in RVT [1] paper, it seems like the variance on some tasks can be as high as 7%. Considering this, the reported improvements (5.2% and 5.9% over RVT in 10 and 100 demo cases) may not be as significant.
2. I am not entirely sure what the purpose of some of the experiments and reported results in the paper is. For example, Table 2 compares, besides the success rate, the training time of proposed approach with Act3D and ChainedDiffuser showing that the proposed approach is significant faster to train, but fails to acknowledge here that RVT is probably equally faster. Also, I am not sure what the purpose of real-world experiments is if there are no baselines in it to compare with. RVT already demonstrates that you could deploy this pipeline on real system, and as authors acknowledge in the paper, during inference the proposed approach is no different from RVT. 
3. While authors demonstrate that using the contrastive IL module with other approaches (say RVT) could lead to better results (a 1.8% improvement for RVT), it seems like the module seems to make RVT performance worse in majority of the tasks (13/18 tasks are worse with the module). I am not sure if the slight average success rate improvement is enough to claim that the contrastive learning module could potentially be useful in other architectures.

## Minor Comments/Nits

- The paper would benefit from a more detailed explanation of the MVQ-Former and how it’s different from the multi-view transformer in RVT paper, as this seems to be an important component of the architecture.
- Figure 2 is dense and a little harder to follow.
- Some parts should be more clearly defined with in the paper, e.g., I shouldn’t have to go through RVT paper to understand what collision state is in the action space.

[1] RVT: Robotic View Transformer for 3D Object Manipulation
questions: questions: In addition to the questions in the previous sections, I have the following questions:

1. Do you have any qualitative examples of why the proposed approaches performs better than say RVT in some of the RLBench tasks?

2. In limitations, you state that the large discrepancy between RLBench and real-world environments leads to failure of sim-to-real transfer. Could you clarify this statement? Do you mean that “conclusions made in sim possibly may not hold in real-world” or “training in sim and deploying zero-shot in real-world wouldn’t work”? I usually associate sim-to-real with the latter, but that doesn’t make a lot of sense in this paper’s setting.
limitations: limitations: 2
review_summary: review_summary: The main contribution of the paper is the idea of using contrastive learning in the RVT framework, and I think the results from RLBench don't necessarily show yet that this led to significant improvement in performance.
title: title: CoRL 2024 Conference Submission269 Reviewer h5Zm
paper_summary: paper_summary: This paper introduces an end-to-end imitation learning framework for language-conditioned multiple 6-DoF manipulation tasks for agents. It presents a novel method called contrastive imitation learning to refine the visual representation and also the vision-language representation. It also proposes the Multi-View Querying Transformer to improve efficiency by minimizing the token numbers. Experiments are performed on several benchmarks and various manipulation tasks. The contrastive IL module is integrated with baseline models to demonstrate its effectiveness in different visual representations.
main_review: main_review: *Strengths*:
- Contrastive imitation learning is skillfully designed to enhance the representation of language-conditioned manipulation tasks. It effectively utilizes discriminative features to accurately perceive the current state and align representations with task correlations.
- The simulation experiments are thorough, encompassing multiple manipulation tasks to demonstrate significant improvements over previous baselines. The efficiency of the proposed contrastive imitation learning modules is clearly showcased.
- The paper effectively formulates key questions for language-conditioned multi-tasking. It provides a comprehensive introduction to the proposed objectives and their detailed explanations.

*Weakness*:
- While the paper claims that the proposed MVQ-Former efficiently reduces the number of visual tokens, it would benefit from more in-depth analysis and concrete comparisons to substantiate this claim.
- In real-world experiments, the results for several simple pick-and-place tasks show unsatisfactory success rates. The reasons provided for these appear to be relatively straightforward to address, suggesting room for improvement in real-world experiments.
- The results of the simulated experiments indicate a significant improvement in tasks such as "put in drawer" and "close jar." However, the paper would benefit from a more detailed analysis explaining why the proposed method excels in these specific tasks.
- The baselines are reimplemented using different numbers of demonstrations for training. However, more detailed comparisons and analyses regarding the impact of varying demonstration numbers and the reasons why using different numbers of demonstrations would strengthen the paper’s claims.
questions: questions: - Efficiency of MVQ-Former: Can you please provide a more detailed analysis and concrete comparisons to substantiate the claim that the proposed MVQ-Former efficiently reduces the number of visual tokens?
- Real-World Experiment Improvement: The success rates in real-world pick-and-place tasks are not very satisfying. Could these rates be improved by relatively simple means, such as using multi-view cameras to provide more precise visual information or adding an extra RGB-D camera at the wrist position as proposed? If these improvements are feasible, please provide several results.
- Task-Specific Improvements: In simulated experiments, the method shows a significant improvement on tasks like "put in drawer" and "close jar." Can you please provide a more detailed analysis explaining why the proposed method excels in these specific tasks?
- Baseline Reimplementation and Demonstration Variability: The baselines are reimplemented using different numbers of demonstrations for training. Can you include more detailed comparisons and analyses on the impact of varying demonstration numbers on the performance of the proposed method?
limitations: limitations: 3
review_summary: review_summary: Overall it is a good work with comprehensive results in the simulated experiments. The paper is easy to read and the proposed method is easy to understand. I will be more confident in recommending the paper for acceptance if the questions are well answered or addressed.
title: title: Review 1
paper_summary: paper_summary: The paper titled “Contrastive Imitation Learning for Language-guided Multi-Task Robotic Manipulation” presents a language-conditioned imitation learning framework for robotic manipulation. The contribution consists of augmenting the standard imitation learning objective with two auxiliary contrastive learning objectives to promote alignment between 1) the language instruction and the visual representation and 2) the language instruction + visual representation and the goal. This objective is used to train a multi-task, multi-view policy architecture, which is evaluated against a comprehensive set of baselines. The simulated results indicate that 1) the overall proposed system (new objective + policy architecture) outperforms existing methods and 2) contrastive objectives marginally improve the performance of baselines. The proposed method is validated on a real robot.
main_review: main_review: Quality: The paper quality is quite high: the content is well organized under appropriate subsections, and the experiments and results are presented elegantly. There are few, if any, spelling or grammatical errors.

Clarity: Some sections of the paper are clear (e.g., the experiments section onwards is clear), while others have room for improvement in terms of clarity and rigor. For example, the introduction does not clearly articulate the core research problem (and its importance) addressed by the paper (i.e., lines 34-35). The essence of the method is actually quite simple, yet Figure 2 overcomplicates it – and the caption of Figure 2 contains symbols that are not represented in the figure, which is a point of confusion. There are terms used to describe the methodology that are not defined beforehand, e.g., what is a “discriminative feature” and what does “interacting feature” mean? I recommend defining these terms before use.

Originality: The idea of using auxiliary contrastive learning objectives to promote or bolster policy learning in an imitation learning setup is not original. However, the specific objectives used here appear to be original.

Significance: Objectives to regularize various aspects of robotic policy learning are relevant to the robot learning community. However, the contributions proposed in this paper yield marginal improvements over existing baselines.

Summary of strengths:
- The proposed method is easy to integrate with existing imitation learning pipelines and is perhaps a cheap solution to slightly boost policy performance.
- The evaluation of the method and baselines is quite comprehensive and well presented. The reviewer appreciates the additional ablations conducted both on the proposed method (Figure 3) and the baselines (Table 3). The analysis is also quite clear.
- The proposed contrastive learning objectives are shown to reduce training times.

Summary of weaknesses:
- The results are a bit underwhelming. While the proposed contribution performs best in aggregate by ~6%, it is unclear why the contrastive objectives harm performance on some tasks. Similarly, the contrastive learning objectives do not yield significant increase in success rates when implemented on the baselines (<3% in aggregate). 
- The ablation in Figure 3 indicates that one of the two proposed objectives (goal contrastive loss) contributes minimally to policy performance.
- The limitations stated in Section 5 are more so general limitations of language-conditioned imitation learning rather than concrete limitations associated with the proposed method.
questions: questions: - A more compelling and intuitive motivation should be established for each of the contrastive learning terms. At the present moment, the motivation for these terms in the form of a need for feature “alignment” is vague to me. The language-state term is more intuitive than the goal term. 
- Can the authors please explain the discrepancy of results across tasks? Why does contrastive learning benefit some but harm others? Can this be improved?
- Quality / Clarity: I encourage the authors to simplify Figure 2 and address the above comments pertaining to quality and clarity.
limitations: limitations: 2
review_summary: review_summary: The paper is in relatively good condition in that it is well organized, quite clear, and experiments are extensive. The contribution contains some originality, and while it does improve policy performance (in aggregate) and learning efficiency, the significance of the results (and hence the contribution) is questionable. On these bases, my recommendation is weak reject.
title: title: Review of Contrastive Imitation Learning for Language-guided Multi-Task Robotic Manipulation
paper_summary: paper_summary: This paper introduces an imitation learning approach that leverages contrastive learning to align the image + language embeddings and the <image, language> + goal image embeddings. The vision encoder is additionally trained on re-rendered virtual images (following RVT). The evaluations are conducted on RLBench and a set of real-world manipulation tasks, and demonstrates improvements over RVT in the multi-task setting.
main_review: main_review: Strengths:
- This work presents a contrastive learning-based approach to learning from both vision and language modalities for robotics. To the best of my knowledge, this is novel to the robotics community and is also an important problem.
- The experiments show that the contrastive representations enable some performance improvements compared to the non-contrastive representations.
- This method can be applied to real-world problems with relatively low amounts of demonstration data. The additional cost to training is also low.
- Writing is clear and easy to follow.

Weaknesses:
- The improvements are minor (2-3% improvement in success rate), especially compared to differences between different base models (e.g., between PolarNet and RVT), which suggests that the impact of the proposed contrastive representations is less significant compared to other model choices. Perhaps, for this particular setting with relatively few tasks, aligning vision and language modalities is not as important.
- Given that there is prior work on aligning text and vision modalities with contrastive losses [1, 2], the main difference in this work is additionally contrasting the <image, language> to goal image representations. This additional loss, however, has less impact on performance according to the ablations in Figure 3(a).
- The related work section could be improved by including references to works on (contrastive) alignment of vision-language modalities outside of robotics.

[1] Jia et al. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. ICML 2021.
[2] Li et al. Align before Fuse: Vision and Language Representation Learning with Momentum Distillation. NeurIPS 2021.
questions: questions: Clarification questions:
- In Table 2, is $\Sigma$-agent using RVT as the base model? If so, what is the performance of RVT in the 100-episode per task setting?
- How much training time does contrastive learning add to RVT and to PolarNet? In Table 1, is it correct that $\Sigma$-agent applies the contrastive learning module to RVT and is faster to train compared to RVT?
- For real robot experiments, how does the performance compare to the base model without the contrastive representation?

Other questions:
- What would happen if this is to scaled to a much larger number of tasks? How does dataset size and number of training tasks affect performance?
- Does this alignment of vision and language modalities enable better generalization to new tasks and task variations?
- Currently, the variations in tasks are reference variations in the color and object instance. Have the authors tried testing other variations such as position/direction (e.g., "stack the cup located in the top-right corner on top of the cup in the bottom-left corner"), size (e.g. "stack the smaller cup on top of the bigger cup"), and shape? It would also be interesting to organize the results in terms of different types of variations to see which variations are harder for the model to handle.
- Have the authors tried varying the camera pose? Some of the aforementioned variations (position/direction) depend heavily on the camera perspective. It'd be interesting to see whether aligning the vision-language modalities allow generalization to new camera poses and references to position/direction.
limitations: limitations: 3
review_summary: review_summary: This paper studies an important problem that is relevant to the robot learning community, and presents an approach that is simple to integrate into existing approaches. However, from the present experiments, it's not entirely clear to me that there are strong advantages, and stronger analysis may highlight where this approach would be most beneficial.
title: title: Well written paper. Some concerns about novelty and implementation
paper_summary: paper_summary: The authors introduce an imitation learning method that incorporates contrastive learning in the training objective to better align the underlying vision and language representations and improve downstream policy performance. The contrastive learning objective is two fold: (1) aligning states and language representations for the same task, and (2) aligning joint state-language and goal representations for the same task. The authors also propose a Multi-View Querying Transformer to improve the training efficiency of the model. On the simulated RLBench suite, the authors show that the model generally outperforms existing approaches, including RVT.
main_review: main_review: The paper is generally well written and easy to follow. The introduction lays out the problem motivation well, the related works section is thorough, and the method section is well laid out. The experiments and comparisons are also thorough, with the ablations being informative in breaking down the effect of the language vs. goal based contrastive learning. However I have a few specific concerns:
* The work, for the most part, has somewhat low technical novelty. Adding auxiliary contrastive learning objectives is a useful tool that has been established in numerous works, as the authors point in the related works section. And while there is an improvement in performance compared to prior work like RVT, the performance gain of ~5% is somewhat small.
* line 187 states: “for the goal and state-language contrastive training, we utilize the next state in the trajectory as the goal state.” Why is only the immediate next step chosen as the goal state? It seems like a rather constraining choice. I would be curious to see what happens if set the goal state as the state 10 steps ahead, or the final state of the trajectory. Additionally, the ablation in Fig 3a does not really show any gains from incorporating the goal-based contrastive objective. This makes sense if the goal state we are choosing is only a single tilmestep away from the current state.
* I’d like to see a comparison to RVT2, which claims to provide state-of-the-art results with significant gains over RVT. The contrastive learning techniques used in this paper can be applied to RVT2, and it would be interesting to see whether we there is an improvement in that setting as well.
* The limitations mentioned in the final section are rather generic and not specific to the ideas presented in this paper. I would have liked to see the authors point out specific observations where the method is not performing as well as it should, and propose some ideas to address these limitations.
questions: questions: Please see the questions I raised in my review above.
limitations: limitations: 2
review_summary: review_summary: I think at least one of the following is needed: (1) more technical novelty; (2) larger performance gains over the current state-of-the-art methods.