VIP: Vision Instructed Pre-training for Robotic Manipulation
Authors: Zhuoling Li
LiangLiang Ren
Jinrong Yang
Yong Zhao
Xiaoyang Wu
Zhenhua Xu
Xiang Bai
Hengshuang Zhao
Venue: ICML 2025 poster (2025)
Keywords: vision instruction
robotic manipulation pre-training
imitation learning
Primary area: applications->robotics
TL;DR: We propose how to utilize vision instruction to improve robotic manipulation pre-training.

Abstract: The effectiveness of scaling up training data in robotic manipulation is still limited. A primary challenge in manipulation is the tasks are diverse, and the trained policy would be confused if the task targets are not specified clearly. Existing works primarily rely on text instruction to describe targets. However, we reveal that current robotic data cannot train policies to understand text instruction effectively, and vision is much more comprehensible. Therefore, we introduce utilizing vision instruction to specify targets. A straightforward implementation is training a policy to predict the intermediate actions linking the current observation and a future image. Nevertheless, a single future image does not describe the task target in insufficient detail. To handle this problem, we propose to use sparse point flows to provide more detailed information. Extensive tasks are designed based on real and simulated environments to evaluate the effectiveness of our vision instructed pre-training (VIP) method. The results indicate VIP improves the performance on diverse tasks significantly, and the derived policy can complete competitive tasks like ``opening the lid of a tightly sealed bottle''.

VIP: Vision Instructed Pre-training for Robotic Manipulation
Zhuoling Li'
Liangliang Ren?
Jinrong Yangi. Yong Zhao? Xiaoyang Wu
Zhenhua Xu"
Xiang Bai Hengshuang Zhao
https://lizhuol wwwphhhwwwiii..................e SeVpRRT_ebbage/////
Abstract
up green block Text Instruction
Text
The effectiveness of scaling up training data in robotic manipulation is still limited. A primary challenge in manipulation is the tasks diverse, and the trained policy would be confused if the task targets are not specified clearly. Existing works primarily rely on text instruction to de- scribe targets. However, we reveal that current robotic data cannot train policies to understand text instruction effectively, and vision is much more comprehensible. Therefore, introduce utilizing vision instruction to specify targets. A straightforward implementation is training pol- icy to predict the intermediate actions linking the current observation and a future image. Neverthe- less, a single future image does not describe the task target in insufficient detail. To handle this problem, propose use sparse point flows to provide more detailed information. Extensive tasks are designed based on real and simulated environments to evaluate the effectiveness of our vision instructed pre-training (VIP) method. The results indicate VIP improves the performance on diverse tasks significantly, and the derived policy can complete competitive tasks like "opening the lid of tightly sealed bottle".
focus the green
Instructed Policy
Policy Attention Map
Vision
Vision Observation
the green block correctly.
Instructed
Policy
Vision Instruction
Policy Attention Map
Figure I. Visualization comparison between the action attention maps of the text instructed policy and vision instructed policy. We can observe that the text instructed policy is confused about which region to concentrate on, while the vision instructed policy focuses on the target correctly. This phenomenon suggests that vision instruction is comprehensible by policy networks.
& Kragic, 2019). This is partly because manipulation data is ambiguous due to its multi-modas nature (Chi et al., 2023). Given the same observation, there could exist multiple ob- jects manipulate and the potential actions diverse, such as picking, sorting, and rotating.
To handle this ambiguity, it is important to describe task targets clearly to the policy. A common practice in existing manipulation pre-training paradigms is describing task tar- gets using text instructions like "pick up the green block", These paradigms expect that the trained policy understands what the green block is in the input image and predicts the action sequence of picking it up. Therefore, the policy needs to align the information between text and vision, such the words "green block" and the appearance of this green block. However, we find that existing manipulation data not di- verse sufficiently to train a policy to own this capability, which demands millions image-text pairs as suggested in previous literature (Liu et al., 2024). As shown in Fig. 1, the text instructed policy fails to concentrate the green block specified in the text instruction, and the robot hand often grasps a block in false color.
1. Introduction
Following the data scaling law, the natural language pro- cessing and computer vision communities have achieved remarkable breakthroughs (Achiam et al., 2023; Liu et al., 2024). the robotics community, there are also many at- tempts of incorporating more robotic data (O' (e'Neil et al., 2024; Kim et al., 2024). However, the effectiveness of large- scale training in robotic manipulation is still limited (Billard
'HKU #CVTE SSYSU *THU HHUST Project leader. Corre- spondence to: Hengshuang Zhao
In this work, we reveal that vision instruction, such as a future image predicting the grasp moment, much more comprehensible by policies in the current robotic data di-
Proceedings of the = International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

VIP: Vision Instructed Pre-training for Robotic Manipulation
versity scale. As illustrated in Fig. the vision instructed policy focuses on the correct block and grasps it success- fully. This is because both the observation and instruction lie in the vision domain, avoiding the need for numerous data to bridge the feature gap between different domains. We can also understand this phenomenon from the view of analyzing human infant behaviors, as existing robotic policies are built upon infant-level intelligent networks like ResNet (He et al., 2016) due to real-time deployment re- quirements. When we tell an infant to pick up green block, the infant cannot understand the instruction even after many iterations of text explanations. However, if we show the infant what is the green block and how to grasp it through vision, the infant could gradually make correct reactions.
given the online specified order. In simulated experiments. we build real-time human hand pose acquisition system to teleoperate robotic hands in Isaac Gym (Makoviychuk et al., 2021) and design several tasks of transporting and stacking color blocks following task instructions. The experimental results demonstrate the superior performance of VIP.
2. Related Work
Demonstration Learning based Robotic Manipulation. Robotic manipulation has advanced markedly due to the integration of machine learning techniques (Fang et al., 2019). Among existing methodologies, demonstration learn- ing has garnered significant attention for its training effi- ciency (Zhao et al., 2023). The fundamental premise of demonstration learning is that human teacher performs task while the robot records the relevant data, such as sensory inputs, actions, and corresponding outcomes. This recorded data is subsequently used to train models that allow the robot to replicate the demonstrated behavior in similar situations (Florence et al., 2022).
Therefore, vision instruction is more suitable for prompt- ing policies in the current diversity scale of robotic data. A natural idea of using vision instruction is feeding the policy with future images besides the current observation, and the policy is optimized to predict correct actions that make the robot reach the status described in future images. However, a single future image is insufficient to describe ma- nipulation dynamics and more frames lead to dramatically increasing computation cost. To specify the manipulation procedures clearly while maintaining an acceptable compu- tational burden, we propose to represent the intermediate action information with sparse point flows. Specifically, we sample sparse points in the current vision observation and track the moving dynamics of these points. Therefore, the input to the pre-trained policy in VIP includes the current image, future image describing the target, and the sparse point flows between the current and future images.
Existing demonstration learning policies can be broadly cat- egorized into two groups, i.e., explicit policies (Fu et al., 2024b) and implicit policies (Chi et al., 2023). Among them, explicit policies directly map environment observa- tions to actions, and the policy output is supervised with human demonstration trajectories by computing regression losses (Fu et al., 2024a). By contrast, implicit policies de- fine the distributions of actions with energy- -ased models, where predicting the next action is framed as identifying the manipulation trajectory with minimal energy (Chi et al., 2024). This modeling approach allows for the natural rep- resentation of multi-modal distributions in manipulation trajectories. Consequently, some studies suggest implicit policies are more advantageous for robotic manipulation learning (Florence et al., 2022). Nevertheless, we contend that explicit policies offer faster response speeds due to their simplicity, which is crucial for robotic manipulation. In addition, the iterative decoding mechanism inherent in Transformer models is similar to the denoising process in implicit policies, and thus can also handle the multi-modal ambiguity in robotic manipulation to some extent. Hence, in this work, we develop fully Transformer-basse policy adopting the explicit prediction paradigm.
However, sparse point flows and future images are unavail- able during inference. To handle the lack of sparse point flows, we progressively remove them during pre-training by random masking. This design gradually boosts the action prediction challenge and helps the policy learn more mean- ingful representation (Oquab et al., 2024). For the future image in inference, we replace it as the cropped region of the object to manipulate from the current observation. We find this design achieves better performance, as it excludes the disturbance from image background while also clearly spec- ifying the target object. Additionally, this design enables us to specify the object manipulation order dynamically.
Combining the above designs, Vision Instructed Pre- training (VIP) is derived. Based on VIP, we pre-train our de- signed fully brassedrmer-based policy using 1.7B of manip- ulation data (Khazatsky et al., 2024). Extensive real-robot and simulated tasks are designed to verify the performance. For roal- experiments, we devise tasks to validate dif- ferent capabilities of policies from various perspectives, and the tasks include pouring blueberries into a juicer cup, open- ing the lid of a tightly sealed bottle, and cleaning plates
Robotic Pre-training. Recent advancements in natural language processing and computer vision demonstrate the efficacy of first pre-training models on large-scale data and then fine-tuning them for specific downstream applications (Achiam et al., 2023; Wang et al., 2023b). Drawing inspira- tions from these successes, the robotic learning community begins to explore pre-training paradigms to enhance robotic manipulation capabilities (Radosavovic et al., 2023; Nair et al., 2023; Majumdar et al., 2023). The principal idea be-

---

VIP: Vision Instructed Pre-training for Robotic Manipulation
Policy Vision Encoder
Current Observation l
Action
Policy Action Decoder
Future O bservation by
Action T
Progressive Mask
Numerous Manipulation Data
Sparse Point Flows
Figure 2. Overall pipeline of VIP. The input to the pre- tre-trained policy includes two image frames (the observation frame and future frame) and sparse point flows, which describe the changing dynamics of the scene. The sparse point flows gradually removed by the progressive mask module during pre-training.
hind robotic pre-training first expose the robotic policy to wide range of tasks and environments, allowing it to learn generalizable representations for diverse robotic tasks (Brohan et al., 2022). Subsequently, a fine-tuning phase on specific manipulation tasks is conducted, utilizing the previously gained prior knowledge to enhance performance and efficiency (Zitkovich et al., 2023).
involves multi-object manipulation or there are many poten- tial operation steps (Shi et al., 2023). To alleviate this prob- lem, instructions are demanded to guide policies with task- specific information. In previous works, the instructions are mostly represented as natural language (Brohan et al., 2022; Zitkovich et al., 2023). Alternatively, some researchers have also explored voice instructions (Shi et al., 2024), but voice information similarly poses learning difficulties. By con- trast, images are more readily comprehensible for policy networks, as the commonly adopted backhones of these net- works are already pre-trained on extensive image datasets (He et al., 2016). This pre-existing visual understanding in robotic policies is akin to the innate vision comprehension in human infants. Despite the potential of visual instruc- tions, their applications remain unexplored in the context of robotic manipulation. Existing studies on visual instructions have primarily focused on goal images within game-based reinforcement learning (Yuan et al., 2024) and navigation (Majumdar et al., 2022). This work aims to bridge the gap in exploring visual instructions for robotic manipulation.
Pre-training policy requires a substantial amount of data (Fang et al., 2020). However, robotic manipulation data is expensive to collect. To mitigate this issue, some methods generate data through simulated environments (Wang et al., 2023a). However, significant discrepancies in appearance and motion dynamics between simulated and real-robot data limit thier effectiveness. Several methods employ large language models generate grasp positions for objects in 2D images (Vuong et al., 2024), but this approach is constrained by its woo-dimensional output, whereas robotic manipulation occurs in a collaborative efforts among various institutions have led to the creation large-scale datasets by merging existing data sources (O' Neill et al., 2024) or collecting new data across diverse scenarios (Khazatsky et al., 2024). Thanks these datasets, a handful of promising pre-trained policies are de- rived (Kim et al., 2024). Nevertheless, robotic manipulation trajectories remain ambiguous if without appropriate task instructions as prompt. Existing pre-training algorithms pre- dominantly use text instructions to inform the policy, which restricts the pre-training effectiveness, as previously noted.
space. Recently,
3. Method
3.1. Vision Intructed Pre-training
A data sample for robotic manipulation pre-training consists of two parts, video sequence V {1,, 12, -.. Ir) and the corresponding robot actions A = =āā, u2 ãr J. In a real robotic manipulation application, future information is unavailable, and a policy x needs to predict future actions A using only the current observation 7, However, manipu- lation trajectories are highly diverse. Directly regressing A based on Jı is too ambiguous for æ. To address this problem, instruction is demanded in pre-training to disclose future information to π to guide the generation of future actions.
Robotic Instruction. Robotic manipulation learning is in- trinsically a long- sequence regressgee problem, often involving thousands of action steps within a short dura- tion (Chen et al., 2024). Therefore, a basic challenge in manipulation lies in the ability of a policy to determine the appropriate actions based on the current observation. This problem especially serious if the task to perform
The previous robotic pre-training paradigms often disclose

---

VIP: Vision Instructed Pre-training for Robotic Manipulation
future manipulation trajectory information with text-based task description. These methods take vision observation and text description as input to predict actions, the process of which demands aligning the information among three domains (vision, text, and action). Nevertheless, we reveal that existing robotic data is not diverse enough to align text and vision representation. In addition, it is challeng- ing to elaborate manipulation procedures clearly based on text description. Different from previous paradigms, this work proposes VIP, which pre-trains a policy to predict actions based on solely vision information. As illustrated in Fig. 2, the vision information includes current observation J1, future observation Iy, and sparse point flows. Notably, each observation could contain multiple images captured by different cameras at the same timestamp.
Figure 3. The conceptual diagram of sparse point flow. Consecu- tive frames in video comprise numerous pixels and contain much redundant information for describing the movement of robot hand. By contrast, small group of points tracking moving pixels, namely sparse point flows, are much more efficient.
sparse point flows in moving image regions is much more efficient. Compared with using an image sequence where every frame comprises tens of thounsands of pixels, the sparse point flows only have tens of points and can describe the robot hand manipulation process clearly.
In VIP, we first transform 1 and I visual features F1 and Fe by shared encoder like ResNet (He et al., 2016) in the pre-trained policy. Besides, we utilize sparse point flows in moving image regions to describe the missing intermediate robot manipulation information. We gradually remove the point flows through the progressive mask module shown in Fig. 2, and the remaining flows are transformed into the feature F, by simple MLP layer. Refer to Section 3.2 for details about how the point flows are generated and masked. According to Fig. 2, F1 is the environment observation. Fr and Fp are the vision instruction, which describes the future moving dynamics. Fi, Fr, and F,, are input to the action de- coder (e.g., Transformer decoders or diffusion heads) of the pre-trained policy to produce T action predictions {a, yr and corresponding Laplacian uncertainty values {a.){-1- We optimize the pre-trained policy by minimizing loss L constructed based on (ue)T. {or))-
In this work, generate sparse point flows based Co- Tracker (Karaev et al., 2024), state-of-the th-the-art point tracker. A simple implementation of using CoTracker is first ran- domly sample some points in the first frame of a video sequence and then provide these points to CoTracker. Co- Tracker can automatically track the point flows in subse- quent frames. However, this implementation suffers from two drawbacks: (i) We only concern the changing regions in the frames of a video, while this implementation wastes many points in the static background. (ii) In many frames, the robot hand is invisible or not moving. These frames should not be incorporated into the pre-training data.
To handle the aforementioned two drawbacks, we adopt a two-stage strategy to generate the sparse point flows. Specif- ically, we first uniformly sample points in the first frame and produce point flows using CoTracker. Then, we remove the point flows that keep static among frames. If the remaining moving flows are too few (e.g., fewer than three), we do not add this video sequence into pre-training data. Conversely, if there are enough moving flows, we randomly sample more points around these remaining moving flows to pro- duce more sufficient point flows. After these two phases of point sampling, the derived sparse point flows describe the changing dynamics of the video sequence in both efficient and detailed manner.
and {āe)T=1 as:
2 a: ael + +logor).
(1)
Notably, fo has no ground truth and is learned in an unsupervied manner. Refer to previous literature (Li et al., 2022; Kendall & Gal, 2017) for why uncertainty can be captured in this way. According to Eq. (1), we can observe that the learned uncertainties {ot) exhibit higher values for more ambiguous action segments. Consequently, larger {or)²"z result in a smaller penalization for the discrepan- cies between predicted actions {ar (at)7= and demonstrated actions (ā. )[_ This property enables π to concentrate on more deterministic action segments.
Nevertheless, we find that too abundant point flows prevent the policy from learning very meaningful vision represen- tation. There are mainly two problems. On the one hand, infering robot hand actions from point flows easier than using vision observations. Therefore, very sufficient point flow information makes the policy become lazy to explore the feature in vision observations. On the other hand, the point flows are unavailable in inference, so adopting point flows during gre-training causes an input information gap be- tween pre-training and inference. To address these two prob
3.2. Sparse Point Flow
Consecutive frames in video contain numerous redundant information for recording the changing dynamics of a robot. Therefore, employing a video sequence to describe manip- ulation procedures to a pre-trained policy leads to huge computation cost. Differently, as depicted in Fig. 3, using

---

VIP: Vision Instructed Pre-training for Robotic Manipulation
lems, we gradually remove point flows during pre-training by masking them with increasing probability. Specifi- cally, assuming there are totally N pre-training iterations, we randomly mask On of point flows at the ah iteration, x. where 0s, = min(1.22 x N. 1). In this way, the point flow is completely removed at the end of pre-training, and thus the above two problems are tackled.
World Model (Sim)
Framer Obs.
Frame t + GT
Frame t+1 Pred.
World Model (Real)
3.3. Vision Instruction after Pre-train
As the robots used in applications are often different from the robots for gathering pre-train data, a small handful of demonstration data is needed to fine-tune the pre-trained pol-
icy. To maintain usage consistency, The implementation of the policy in fine-tuning is the same the one in inference. Therefore, we elaborate them together in this part.
Frame ( O
Frame + 1 GT
Frame [+1 Pred.
Cropped Image
The main difference between pre-training and the subse- quent fine-tuning and inference phases is the vision instruc- tion. As shown in Fig. 2, the vision instruction in pre- training consists of two parts, the future frame and sparse point flows. As introduced in Section 3.2, the sparse point flows are completely removed at the end of pre-training, so the problem is how to get the future frame. To address this problem, we first try training diffusion based world model (Parmar et al., 2024; Ho et al., 2020) to predict the future image. It can be observed from Fig. 4 that the world model predicted image more accurate in simple simulated sce- narios but not satisfactory in real scenarios. For example, shown in the second row of Fig. 4, the world model output is easy to collapse to being the same as the input image.
Frame Obs.
Frame Obs.
Cropped Image
Figure 4. Visualization of different vision instructions. The threc columns of images in the first and second rows show the world model input, future ground truth, and future image prediction simulated and real scenarios. The third row illustrates the cropped image of the object to manipulate.
order dynamically by providing different cropped image regions during inference. The cropped image regions can be specified in various ways, e.g., user eye gaze recogniza- tion, user instruction understanding based on LLM, an external detection or segmentation model, etc.
There are mainly two reasons resulting in this phenomenon. First of all, the future observation 0141 is affected by both the current state 8, and action at. However, the input to the world model only includes st, because de unavailable. Why predict 0 is to use 0++1 for guiding the genera- tion of a, In other words, predicting 0++1 demands a, and outputting a, depends on 0++1, which is an endless loop. Hence, is not input to the world model. This restricts the future image prediction accuracy. Secondly, compared with simulated environments, the object interaction dynamics like collision in real scenarios is more complex.
3.4. Vision Instructed Robotic Transformer
To fully exploit the potential of VIP, we develop fully Transformer- based policy named Vision Instructed Robotic Transformer (VIRT). In VIRT, we implement the policy vision encoder in Fig. with twelve Transformer encoders. These encoders are initialized from the weight of DINOv2 (Oquab et al., 2024). Before inputting I₁ and Ie to the encoders, we randomly mask ratio of their pixels. This operation forces VIRT to be more sensitive to image details, as it can only depend on random remained image pixels to predict actions. The policy action decoder in Fig. is modeled as three Transformer decoders. T learnable queries are randomly initialized to interact with Fi, F., and F, in decoders produce {ar and {or)
According the above analysis, we cannot employ a fu- ture frame as the vision instruction. To bridge this gap, we propose to replace the future frame in pre-training as the cropped image region of the object to manipulate during fine-tuning and inference. Although different from future frames, the cropped image also specifies where the robot hand should move towards for manipulation, as illustrated in the third row of Fig. 4. Our experimental results indicate that the pre-trained policy can adapt to using this new vision instruction based on only a little fine-tuning data. In addi- tion, as the cropped image delivers which is the next object to manipulate, users can change the object manipulation
4. Experiments
Experimental platforms. We evaluate the effectiveness of our method in both real and simulated environments. For the real environment, we conduct experiments using the Cobot Magic robot (Agilex, 2024). As shown in Fig. 5, the robot

---

VIP: Vision Instructed Pre-training for Robotic Manipulation
Right Camera
MasterAm Move
Pour Bluebanness
Master Amms
Left Camera
Puppet A Move
Frunt Camera
Ams
Battery Computer
Open the Lid
(a) The Cobot Magic Robot
(b) Robot Teleoperation
Figure 5. Illustrations of the Cobot Magic robot and how it is tele- operated. The robot has two master arms and two puppet arms,
Cleen the Tablc
Figure 7. Illustrations of the three designed real-robot tasks, which include Pour Blueberries, Open the Lid, and Clean the Table.
(a) Hand Pose based Arm Control (b) Hand Pose based Gripper Control
Figure Illustrations of how we teleoperate the robot Isaac Gym. We build a real-time hand pose acquisition system to map the human hand pose to robot joint rotations. The orientation and translation of the palm used to control the end of the robot arm. The distance between the thumb and index finger is employed to determine the opening closing of the gripper.
Movea Single Box
Transpor Spocified Box Stack Spocified Boxcs
Figure 8. Illustrations of the three designed simulated tasks, which include Move a Single Box, Transport the Specified Box, and Stack the Specified Boxes.
is integrated with four robot arms, i.e., two master arms and two puppet arms. When collecting demonstration data, we manually control the master arms, and the puppet arms imitate the actions of the master arms in real time. After fine- tuning the pre-trained policy with the demonstration data, the policy directly controls the puppet arms in inference. Three cameras are deployed on the robot, which are the right camera, front camera, and left camera, respectively.
of the robot gripper is controlled by the distance between the thumb and index finger of the human hand.
Evaluation tasks. To fully evaluate the effectiveness of our proposed techniques, we design three real-robot tasks and three simulated tasks. As depicted in Fig. 7, the three real-robot tasks include Pour Blueberries, Open the Lid, and Clean the Table. We collect 100 demonstrations of data for every task. In the Pour Blueberries task, the robot needs to first remove the juicer cup from the juicer and place it on the table. Then, the robot picks up the plate containing blueberries and pours all blueberries into the juicer cup. For the Open the Lid task, the robot uses a robotic hand to hold bottle with tightly screwed lid. The another hand first needs to grasp the lid. After series of twists, the robot gradually unscrews and removes the lid from the bottle. In the Clean the Table task, three plates of different colors and a small cabinet are positioned on a table. The robot is required to move the plates onto the cabinet in a color order that randomly specified during test. The three tasks test the multi-step operation, precise manipulation, and instruction following capabilities of policies, respectively.
The simulated environments are built based on Isaac Gym (Makoviychuk et al., 2021), which supports GPU-based ef- ficient physics simulation. A Franka Panda robotic arm is deployed in each simulation environment to manipulate ob- jects, with four cameras strategically positioned to observe the scene from various angles, including three peripheral views and one hand view. Unlike previous approaches that rely on manually crafted script rules for generating ma- nipulation demonstrations (Zhao et al., 2023), we build a real-time hand pose acquisition system to teleoperate the simulated robotic arm, which better mimics the real demonstration data distribution. Specifically, Leap Motion Controller (Ultraleap, 2013), which a binocular infrared camera, is adopted estimate the hand translation and orientation. Then, as shown in Fig. 6, we map the trans- lation and orientation of the hand palm to the robot arm end-effector position using pre-defined rules, and the joint rotation angles of the robot arm are derived based on inverse kinematics (Kucuk& & Bingul, 2006). The opening or closing
The three simulated tasks are Move Single Box, Transport the Specified Box, and Stack the Specified Boxes, as visu- alized in Fig. 8. We collect 50 demonstrations for the first task and 100 demonstrations for each of other two tasks. In Move a Single Box, the robot needs to transport the box on

---

VIP: Vision Instructed Pre-training for Robotic Manipulation
Table I. Effectiveness study of VIP both real and simulated environments.
Policy
Pre-train Pour Blueberries
Open Lid f Clean Table t Move Box f Transport Box
Stack Boxes
Speed t
ConvMLP
0.00
0.00
0.00
0.24
0.12
0.03
17.54
ConvMLP
0,00
0.00
0.00
0.38
0.17
0.11
17.54
Diffusion
X
0.19
0.46
0.24
0.72
0.51
0.43
27.32
Diffusion
0.28
0,54
0.28
0,85
0.60
0.56
27.32
ACT
0.28
0.51
0.25
0.84
0.61
0.47
43,48
ACT
0.34
0.59
0.30
0.90
0.65
0,58
43.48
VIRT (Ours)
0.30
0.65
0.29
0.87
0.65
0.54
39.22
VIRT (Ours)
0.42
0.71
0.37
0.92
0.74
0.68
39.22
table into a container. For Transport the Specified Box, five different colors of boxes are randomly located on table, and the robot should move the box described by random instruction to the container. Differently, in Stack the Specified Boxes, two boxes are specified. The robot is expected to move the first box in the container and then stack the second box on the first box.
Table Comparison among various instructions.
Pre-train Inference Move Box Transport Box
Stack Boxes
F
Cropped
0.87
0.64
0.50
S
Cropped
0.78
0.51
0.36
F+S
Text
0.85
0.19
0.06
F+S
Future
0.88
0.67
0.54
F+S
Cropped
0.92
0.74
0.68
VIP, the pre-trained model parameters are updated using AdamW (Loshchilov, 2017) and the learning rate is le 5. The action prediction horizon T and image masking ratio are set to 20 and 0.5. Without special statement, the cropped image is obtained from YOLOv 10-small (Wang et al., 2024). We pre-train policies using Droid (Khazatsky et al., 2024) due to its large scale data volume and scene diversity. Besides, the manipulation trajectories Droid present high ambiguity and are helpful for verifying the superiority of VIP. The pre-training consists of 120K iterations and fine-tuning comprises 8K iterations.
Implementation details.
single RTX4090 GPU) in Table 1.
According the results, can observe that our designed pre-training paradigm is effective for all the policies in vari- ous model architectures. The success rates of these policies are boosted both simulated and real robotic manipulation environments, indicating the value of incorporating more diverse training data.
Comparing the various policies, it is found that VIRT achieves the best performance, and its inference speed is also promising. For ConvMLP, its primary problem is its output head is a naive MLP, which is fast but fails to estimate actions precisely. In Diffusion Policy, since adds random noise to action labels during training, the policy network required to learn to recover actions from any noise. This requirement makes Diffusion Policy generally demand more data converge well. VIRT is superior to ACT thanks to its broader perceptive field in encoders and the well-learmed encoder representation inheriting from DINOv2.
4.1. VIP Effectiveness
We study the effectiveness of VIP based the aforemen- tioned six tasks comprising Pour Blueberries, Open the Lid, Clean the Table, Move Single Box, Transport the Specified Box, and Stack the Specified Boxes. Besides our developed VIRT, we further evaluate the effectiveness of pre-training on representative policies such as ConvMLP (Zhang al., 2018), Diffusion Policy (Chi et al., 2023), and ACT (Zhao al., 2023). Among them, ConvMLP is the most commonly adopted baseline, which first extracts im- age feature using convolutional neural network (CNN) and then regresses actions based on the extracted feature. Dif- ferent from Conv Diffusion policy decodes the action chunk through iterative denoising. ACT consists of a CNN backbone, encoders, and decoders. Its basic architecture is similar to VIRT. For fair comparison, the cropped image region is input all these policies as vision instruction. We pre-train these policies with the proposed pre-training paradigm described in Section 3 separately and then fine- tune them with the demonstration data of every task. These policies are tested for 100 times on each task, and we report their success rates as well as inference speeds (test on
4.2. Instruction Comparison
In this part, we analyze the effectiveness of different instruc- tions used in pre-training and inference. The experiments are conducted based on the VIRT policy with the three simulated tasks. The experimental results are presented in Table 2. The studied pre-training instructions include only future image (F), only sparse point flows (S), and us- ing both them (F+S). The analyzed inference instructions are text instruction, world model predicted future image, and cropped images of target objects. According to the re- sults, we can find that solely using future image or sparse point flows does not lead to effective pre-traie pre-training due to the lack of sufficient manipulation process description. By

---

VIP: Vision Instructed Pre-training for Robotic Manipulation
Table 3. Ablation study on designs of VIRT.
DINO Uncern Mask
Move Box Transport Box Stack Boxes
0.80
0.58
0.49
0.86
0.66
0.57
0,90
0.69
0.66
Brightness
Noise
Blur
V
0.92
0.74
0.68
Figure 9. Robustness analysis of VIRT to different disturbances, e.g. brightness change, vision noise, and image blur.
Table 4. Study on the data scaling law.
Pre-train Finc-tune Move Box
Transport Box Stack Boxes
Table 5. Analysis on the policy robustness.
0%
100%
0.87
0.65
0.54
Brightness Blur Noise Move Box Transport Box Stack Boxes
10%
100%
0.81
0.55
0.49
0.92
0.74
0.68
50%
100%
0.89
0.68
0.60
0.86
0.67
0.60)
100%
10%
0.40
0,25
0.06
0.88
0.71
0.65
100%
50%
0.77
0.58
0.43
0.75
0.64
0.52
100%
100%
0.92
0.74
0.68
trajectory parts that are less ambiguous. Ramdomly mask- ing pixels of input images forces the Transformer- policy to maintain its sensitivity to local features.
contrast, combining them results in significant manipulation performance boosts, confirming the effectiveness of VIP.
Additionally, we can observe that although the policy based on text instruction obtains comparable performance with the ones using vision instructions on the Move Single Box task, its performance on the other two tasks are much infe- rior. This is because there is only single box on the table in Move a Single Box task, while the policy needs to grasp the correct box out of multiple boxes according to the provided instruction in the other two tasks. As visualized in Fig. 1, the text instruction based policy does not really understand the instruction content and often grasps an incorrect target. Moreover, it is found that the vision instruction based on cropped image is superior than employing future image. This is because of two reasons. On the one hand, the future image quality predicted by a world model is limited. On the other hand, the cropped image removes the disturbance from image background and guides the generated manipulation trajectories more clearly.
Scaling law. In this part, we study whether the data scaling law appears in the pre- pre-training and fine-tuning procedures of robotic manipulation. Specifically, we conduct experi- ments using different amounts of the total pre- training and fine-tuning data, and the experimental results are presented in Table 4. According the results, we can first observe that increasing both the pre-training and fine-tuning data is beneficial to improving the manipulation success rates. Interestingly, comparing the 1 row (no pre-training is con- ducted) and 2nd row of the results, it can be found that pre-training with only a little data harms the policy perfor- mance. This is because the policy is prone to over-fitting to the pre-training data domain if the pre-training data insufficient, and there is significant gap between the pre- training and fine-ture fine-tuning data domains. In addition, we can find that increasing the fine-tuning data volume boosts exe- cution success rates more significantly, which is because the fine-tuning data aligns better with inference observations.
4.3. Method Analysis
Policy robustness. This part analyzes the robustness of VIRT different unseen enviromment disturbances. As illustrated in Fig. 9, the studied disturbances include en- vironment brightness, vision noise, and image blur. For environment brightness, we reduce the lightness intensity by 40%. To analyze the influence of random vision noise, we add random Gaussian noise to the perceived images. For image blur, we smooth the perceived images with a blur kernel with the shape of 3 3. We replace the in- put the trained policy as the images with the studied noise and then test the success rates on various evaluation tasks. The experimental results are presented in Table 5. As shown, VIRT presents promising robustness to differentt environment changes thanks to the robust representation obtained from the large-scale pre-train stage. In addition, the Gaussian noise unseen in training data leads to the most
Ablation study. This part conducts an ablation study on the designs in VIRT that are not clearly analyzed before. We mainly study the influences of three designs, i.e., initializing encoder weight with DINOv2, uncertainty in supervision loss, and randomly masking the input images. Notably, to show the value DINOv2 representation more clearly. when DINOv2 is not adopted, we replace the encoders in VIRT as ResNet18, the backbone that widely adopted in previous robot policies. The experimental results are reported in Table 3. As shown, all these designs improve the success rates of VIRT on the three evaluated tasks sig- nificantly. Specifically, DINOv2 initialization provides the policy with the initial ability of extracting discriminative rep- resentation. Incorporating uncertainty into the supervision loss in Eq. (1) helps the policy concentrate on predicting the

---

VIP: Vision Instructed Pre-training for Robotic Manipulation
significant performance degradation.
Chi, C., Xu, Z.., Pan, C., Cousineau, E., Burchfiel, B., Feng. S., Tedrake, R., and Song, S. Universal manipulation interface: In-the-wild robot teaching without in-the-wild 2024.
5. Conclusion
robots. arXiv:2402.
In this work, we have revealed that vision instruction is more comprehensible than text instruction for current robotic poli- cies. Based on this insight, we have designed VIP, which utilizes vision observations and sparse point flows to predict actions. To bridge the information gap between pre-training and applications, we have proposed to remove sparse point flows progressively during pre-training and replaced future images as cropped images of objects to manipulate. By combining the strengths of VIP with our developed policy VIRT, VIRT learns to complete diverse challenging tasks. Extensive experiments have been conducted in both real and simulated environments confirm the effectiveness of our proposed techniques from various perspectives.
Fang, B., Jia, S., Guo, D.. Xu, M., Wen, S., and Sun, F. Survey of imitation learning for robotic manipulation. International Journal of Intelligent Robotics and Appli- cations, 2019.
Fang, H.-S., Wang, C., Gou, M., and Lu, C. Graspnet- Ibillion: A large-scale benchmark for general object grasping. In CVPR, 2020.
Florence, P., Lynch, C., Zeng, A., Ramirez, O. A., Wahid, A., Downs, L., Wong, A., Lec, J., Mordatch. I., and Tompson, J. Implicit behavioral cloning. In CoRL, 2022.
Fu, Z., Zhao, Q., Wu, Q., Wetzstein, G., and Finn, C. Hu- manplus: Humanoid shadowing and imitation from hu- mans. In CoRL., 2024a.
Impact Statement
This paper presents work whose goal is to advance the field of robotic manipulation. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Fu, Z., Zhao, T. Z., and Finn, C. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body telcoperation. arXiv:2401.02111 2024b.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, 2016.
Acknowledgement
This work is supported by the National Natural Science Foundation of China (No. 62441615, 62422606, 62201484).
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. NeurIPS, 2020.
References
Karaev, N., Rocco, I., Graham, B., Neverova, N., Vedaldi, A., and Rupprecht, C. Cotracker: better to track together. In ECCV, 2024.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L.,, Almeida, D., Altenschmidt, J., Alt- man, S., Anadkat, S., al. Gpt-4 technical report. arXiv:2303.08774, 2023.
Kendall, A. and Gal, Y. What uncertainties do need in bayesian deep learning for computer vision? NeurlPS, 2017.
Agilex, R. Cobot magic robot. https://githi: com/ agilexroboticss wwtpliccc/coottoieeeeeeeeeeoeeee cobot_ coobottmagicc, 2024.
Khazatsky, A., Pertsch, K., Nair, S., Balakrishna, A., Dasari, S., Karamcheti, S., Nasiriany, S., Srirama, M. K., Chen, L. Y., Ellis, K., et al. Droid: large-scale in-the-wild robot manipulation dataset. In RSS, 2024.
Billard, A. and Kragic, D. Trends and challenges in robot manipulation. Science, 2019.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., et al. Rt-1: Roboties transformer for real-world control at scale. arXiv:2212.068177 2022.
Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakr- ishna, A., Nair, S., Rafailov, R., Foster, P., Sanketi, P. R., Vuong, Q., et al. Openvla: An open-source vision- language- action model. In CoRL,
Chen, T., Cousineau, E., Kuppuswamy, N., and Agrawal, P. Vegetable peeling: A case study in constrained dexterous manipulation. arXiv:2407.07888 2024.
Kucuk, S. and Bingul, Z. Robot kinematics: Forwurd and inverse kinematics. INTECH Open Access Publisher London, UK, 2006.
Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., and Song, S. Diffusson policy: Visuomo- tor policy learning via action diffusion. IJRR 2023.
Li, Z., Qu, Z., Zhou, Y., Liu, J., Wang, H., and Jiang, L. Di- versity matters: Fully exploiting depth clues for reliable monocular 3d object detection. CVPR, 2022.

---

VIP: Vision Instructed Pre-training for Robotic Manipulation
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tuning. NeurlPS, 2024.
Vuong, A. D., Vu, M. N., Le, H., Huang, B., Binh, H. T. T., Vo, T., Kugi, A., and Nguyen, A. Grasp-anything: Large- scale grasp dataset from foundation models. In ICRA, 2024.
Loshchilov, I. Decoupled weight decay regularization. arXiv:1711.05101 2017.
Wang, A., Chen, H., Liu, L., Chen, K., Lin, Z., Han, and Ding, G. 1olov10: Real-time end-to-end object detection.
Majumdar, A., Aggarwal, G., Devnani, B., Hoffman, J., and Batra, D. Zson: Zero-shot object-goal navigation using multimodal goal embeddings. NeurIPS, 2022.
14Xi::205..14555 2024.
Wang. R., Zhang. J., Chen, J., Xu, Y., Li, P., Liu, T., and Wang. H. Dexgraspnet: A large-scale robotic dexterous grasp dataset for general objects based on simulation. In ICRA, 2023a.
Majumdar. A., Yadav, K., Arnaud, S., Ma, J., Chen, C., Silwal, S., Jain, A., Berges, V.-P., Wu, T., Vakil, J., al. Where are we in the search for artificial visual cortex for embodied intelligence? NeurIPS, 2023.
Wang, W., Dai, J., Chen, Z.,, Huang, Z., Li, Z.,, Zhu, X., Hu, X., Lu, T., Lu, L., Li, H., et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In CVPR, 2023b.
Makoviychuk, V., Wawrzyniak, L.,, Guo, Y., Lu, M., Storey, K., Macklin, M., Hoeller, D., Rudin, N., Allshire, A., Handa, A., et al. Isaac gym: High performance gpu based physics simulation for robot learning. NewrIPS, 2021.
Yuan, H., Mu, Z., Xie, F,, and Lu, Z. Pre-training goal-based models for sample-efficient reinforcement learning. In ICLR, 2024.
Nair, S., Rajeswaran, A., Kumar, V., Finn, C., and Gupta, A. R3m: A universal visual representation for robot manipulation. In CoRL, 2023.
Zhang, T., McCarthy, Z., Jow, O., Lee, D., Chen, X., Gold- berg, K., and Abbeel, P. Deep imitation learning for complex manipulation tasks from virtual reality teleoper- ation. In ICRA, 2018.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F,, El- Nouby, A., et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research Journal, 2024.
Zhao, T., Kumar, V., Levine, S., and Finn, C. Learning fine- grained bimanual manipulation with low-cost hardware. RSS, 2023.
O'Neill, A., Rehman, A., Maddukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., Jain, A., al. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collabora- tion O. In ICRA, 2024.
Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F. Wu, J., Wohlhart, P., Welker, S., Wahid, A., et al. Rt-2: Vision- litoo-language-action models transfer web knowledge to robotic control. In CoRL, 2023.
Parmar, G., Park, T., Narasimhan, S., and Zhu, J.-Y. One-step image translation with text-to-image models. arXiv:2403.12036. 2024.
Radosavovic, I,, Xiao, T., James, S., Abbeel, P., Malik, J., and Darrell, T. Real-world robot learning with masked visual pre-training In CoRL, 2023.
Shi, H., Xu, H., Clarke, S., Li, Y., and Wu, J. Robocook: Long- Long-horizon horizon busto-plastic object manipulation with di- verse tools. In CoRI, 2023.
Shi, L. X., Hu, Z., Zhao, T. Z., Sharma, A., Pertsch, K., Luo, J., Levine, S., and Finn, C. Yell at your robot: Improving on fhy-fly from language corrections. 1r111:2403.12910, 2024.
Ultraleap. Leap motion controller. https:
/ //leap2.uttrales cotds//oaac//////////////cc leap-motion txap--mottoon-oontrolllle//// 2013.


Reviews:
[{"review_id": "2kzZeDvpSc", "reviewer": "Reviewer_uDoz", "summary": "This paper proposes VIP, a framework that uses visual signals as guidance for visual pre-training in robotic tasks. Instead of using language as the instruction (which is ambiguous), this work proposes using vision-instructed policy as an alternative. The vision-instructed policy uses visual cues to instruct the policy on what to do and where to pay attention, which helps policy learning. During training, the visual signals (sparse points) are masked and annealed gradually, which helps policy training by making the sparse points unnecessary at inference time. Experiments show that the policy is able to boost performance in both simulated and real-world robots.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We have addressed the concerns of the Reviewer one by one in the following. The paper will be revised accordingly.\n\n## Q1: Ablation study in real-world experiments.\n\nIn the paper, we conduct ablation study mainly in simulation as the environment is easy to control, which benefits to ensuring fair comparison. Our experience shows that the experimental results between our built simulation environment and real robot are consistent. \n\nAs suggested by the Reviewer, we also conduct ablation study using the real robot and report the success rates on all the three our designed real-robot tasks. These three tasks evaluate the performance of robot manipulation policies from different perspectives. \n\n\nThe real-robot experiment results corresponding to Table 2 of the paper are as follows. We can see that the improvments of our proposed designs are more significant in the real-robot experiments.\n\nPre-train | Inference | Pour Blueberries | Open Lid | Clean Table |\n| :-: | :-: | :-: | :-: | :-: |\nF | Cropped | 0.37 | 0.63 | 0.30 |\nS | Cropped | 0.00 | 0.02 | 0.00 |\nF+S | Text | 0.37 | 0.63 | 0.14 |\nF+S | Future | 0.30 | 0.57 | 0.06 |\nF+S | Cropped | 0.42 | 0.71 | 0.37 |\n\nThe real-robot experiment results corresponding to Table 3 of the paper are as follows:\n\nDINO | Uncern | Mask | Pour Blueberries | Open Lid | Clean Table |\n| :-: | :-: | :-: | :-: | :-: | :-: |\nNo | No | No | 0.29 | 0.56 | 0.26 |\nYes | No | No | 0.33 | 0.63 | 0.31 |\nYes | Yes | No | 0.39 | 0.65 | 0.34 |\nYes | Yes | Yes | 0.42 | 0.71 | 0.37 |\n\nAccording to the reported results, all the designs in our proposed method are also effective on the real robot.\n\n## Q2: Experiment on public simulator benchmarks.\n\nIn the paper, we conduct experiments based on our own built simulation environment rather than previous public benchmarks because these public benchmarks usually generate demonstration data by rule-based motion control. Their generated data has a distribution gap with the real robot data collected based on teleoperation. Differently, our used training data in our own built simulation tasks is also obtained from teleoperation, so the experimental results are more convinicing.\n\nAs suggested by the Reviewer, we compare our method with previous SOTAs using the ManiSkill-v3 and RLBench benchmarks. We select them because they provide unified training data generation protocol, which ensures fair comparison. The training data and testing protocols are the same between VIRT and the compared methods. \n\nThe tasks in ManiSkill-v3 cover navigation, humanoid robot, manipulation, etc. We select the manipulation tasks in ManiSkill-v3 to conduct experiments. As mentioned before, the training data is generated based on classic motion control.  The results on ManiSkill-v3 are as follows:\n\nMethod | Push Cube | Pick Cube | Stack Cube | Draw Triangle |\n| :-: | :-: | :-: | :-: | :-: |\nConvMLP | 0.32 | 0.56 | 0.00 | 0.02\nDiffusion | 0.75 | 0.81 | 0.06 | 0.55\nACT | 0.84 | 0.89 | 0.20 | 0.68 |\nVIRT (Ours) | 0.94 | 0.96 | 0.48 | 0.80 |\n\nFor RLBench, although there are totally 100 tasks, many tasks are easy and previous methods have achieved very high success rates. Due to the reply character limit, we select six challenging and representative tasks that the success rate of the best method is still below 90%. We compare our method with the other methods that have reported their results in RLbench. We use the training data of PerACT and follow its testing protocol (the other compared methods also adopt this protocol). The results on RLBench are as follows:\n\nMethod | Insert Peg | Open Drawer | Place Cups | Put in Cupboard | Screw Bulb | Stack Cups |\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: |\nConvMLP | 0.00 | 0.04 | 0.00 | 0.00 | 0.00 | 0.00 |\nPolarNet | 0.04 | 0.84 | 0.00 | 0.12 | 0.44 | 0.08 |\nPerACT | 0.06 | 0.88 | 0.02 | 0.28 | 0.18 | 0.02 |\nACT3D | 0.27 | 0.93 | 0.03 | 0.51 | 0.47 | 0.09 |\nRVT | 0.11 | 0.71 | 0.04 | 0.50 | 0.48 | 0.26 |\nRVT-2 | 0.40 | 0.74 | 0.38 | 0.66 | 0.88 | 0.69 |\nVIRT (Ours) | 0.48 | 0.79 | 0.42 | 0.68 | 0.90 | 0.73 |\n\nThe results on the ManiSkill-v3 and RLBench benchmarks show VIRT achieves the best results, which further confirm the superiority of our method.", "claims_and_evidence": "This paper claims that using vision-instructed pre-training provides a performance boost to downstream tasks and proposes using target image + sparse key point as an effective pre-training strategy for simulated and robotic bi-manual manipulation tasks. The evidence in performance boosts backs up these claims.", "methods_and_evaluation_criteria": "Three real-world and simulated robotic tasks are used to evaluate the proposed method. These tasks are household tasks that can evaluate the performance of the visual policy. There is a missing opportunity in evaluating in standard robotic benchmarks such as ManiSkill or RobotSuite.", "theoretical_claims": "No theoretical claims made.", "experimental_designs_or_analyses": "The experiment design and analysis are adequate to back up the claims. I appreciate the extensive ablations to show each component of the proposed method. Based on the success rate, it is clear that the proposed pre-training method is effective in boosting the performance of the policy.", "relation_to_broader_scientific_literature": "This work tackles the important question of how to use large-scale pre-training data for training visual encoders for robotic tasks. Due to the embodiment difference, it is important to train visual feature extractors that can extract useful information from the existing datasets.", "essential_references_not_discussed": "Reference is adequately discussed.", "other_strengths_and_weaknesses": "### Strength\n- I find the idea of using visual signals and sparse key points as guidance an intuitive and useful idea for visual pretraining. Figure 1 shows the benefit of using vision-instructed policy well. The sparse point flow is an interesting idea to explore as a different modality to instruct policy learning.\n- Extensive ablations show that the combination of vision instruction and sparse keypoints achieves the best performance.\n- SOTA results achieved compared to baselines.\n- Robustness analysis shows that the policy learned with pre-training is robust to visual noises.\n\n### Weakness\n- Most of the ablations are conducted in simulation but not in the real world. Real-world experiments can lead to different results than simulation.", "other_comments_or_suggestions": "I suggest testing the proposed method on standard benchmarks to better compare it with SOTA methods.", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5EW6GRBRw1", "reviewer": "Reviewer_HYVi", "summary": "This paper introduces a novel pretrained method designed for general robotic manipulation tasks. The authors argue that text-instructed policies often fail to effectively focus on target objects, and therefore, they propose integrating more interpretable features for pretraining visual-based policies. These features include sparse point flow and future visual observations, which enhance the model's ability to comprehend and predict environmental dynamics. To validate their approach, the authors have developed a series of tasks conducted in both real-world and simulated environments. These tasks are designed to rigorously evaluate the performance and robustness of the proposed method under various conditions.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We have addressed the concerns of the Reviewer one by one in the following. The paper will be revised accordingly.\n\n## Q1: Visualization details of the teaser figure. \n\nIn the teaser figure, we visualize the attention map between the action query $q_T$ at the last timestamp $T$ and image feature tokens $F$ in the last decoder. We use $q_T$ as query and $F$ as key to compute the attention map $A$. Then, we take the mean of $A$ across all channels and resize the obtained feature to the same shape of the input image. Finally, we visualize the feature and get the teaser figure. For fair comparison, the text and vision instructed policies adopt the same network except the input instructions are different. The policy parameter number is 38.6M.\n\n## Q2: The success of previous text-instructed methods. \n\n* The mentioned pre-train methods like R3M are based on natural videos (like human hand operation videos). Although these models provide promising initialization weight for robot applications, their effectiveness is limited due to embodiment and task gaps. Differently, what we study is another setting: how to use robot manipulation data to conduct pre-train, where the data is more expensive to collect but provides more direct generalization.\n\n* There are text-instructed VLAs based on robot manipulation data, like OpenVLA. However, we find these methods overfit to specific text instructions. For example, OpenVLA behaves well given ''grasp the cup'' but fails given ``pick up the cup''. This is because manipulation data is expensive to collect and not diverse enough to cover common text instructions, although the works like OpenVLA have used up most public data resources.\n\n* We have added comparison experiments with previous pre-train methods. For fair comparison, we integrate their pre-trained backbone into VIRT in the same way as our VIP pre-trained backbone. If the pre-train method relies on text instruction, we provide the policy with text instruction during fine-tune and test. We report comparison results using all the simulated and real-robot tasks as follows:\n\nMethod | Pour Blueberries | Open Lid | Clean Table | Move Box | Transport Box | Stack Boxes |\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: |\nR3M | 0.26 | 0.62 | 0.25 | 0.83 | 0.68 | 0.51 |\nOpenVLA | 0.30 | 0.67 | 0.35 | 0.88 | 0.66 | 0.60 |\nE-RADIO | 0.34 | 0.68 | 0.32 | 0.84 | 0.69 | 0.62 |\nTheia-B | 0.38 | 0.65 | 0.36 | 0.85 | 0.65 | 0.64 |\nVIP (Ours) | 0.42 | 0.71 | 0.37 | 0.92 | 0.74 | 0.68 |\n\nAccording to the results, we can find that although the compared methods generally use more pre-train data than ours, our proposed method achieves better performance.\n\n## Q3: Comparison with other pre-train methods.\n\nAs suggested by the Reviewer, we have conducted more comparison between our pre-train method and other pre-train methods. Due to character limit, the results are reported in the above Table in the Reply to Q2. The results suggest that our pre-train method achieves the best performance on all tasks, which further reveals the superiority of our method. \n\n## Q4: Comparison on public benchmarks.\n\nAs suggested by the Reviewer, we compare our method with previous SOTAs using the ManiSkill-v3 and RLBench benchmarks. The training data and testing protocols are the same between VIRT and the compared methods. Refer to the reply to Q2 of Reviewer uDoz for more experiment details.\n\nThe results on ManiSkill-v3 are as follows:\n\nMethod | Push Cube | Pick Cube | Stack Cube | Draw Triangle |\n| :-: | :-: | :-: | :-: | :-: |\nConvMLP | 0.32 | 0.56 | 0.00 | 0.02\nDiffusion | 0.75 | 0.81 | 0.06 | 0.55\nACT | 0.84 | 0.89 | 0.20 | 0.68 |\nVIRT (Ours) | 0.94 | 0.96 | 0.48 | 0.80 |\n\nFor RLBench, we use the training data of PerACT and follow its testing protocol. The results on RLBench are as follows:\n\nMethod | Insert Peg | Open Drawer | Place Cups | Put in Cupboard | Screw Bulb | Stack Cups |\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: |\nConvMLP | 0.00 | 0.04 | 0.00 | 0.00 | 0.00 | 0.00 |\nPolarNet | 0.04 | 0.84 | 0.00 | 0.12 | 0.44 | 0.08 |\nPerACT | 0.06 | 0.88 | 0.02 | 0.28 | 0.18 | 0.02 |\nACT3D | 0.27 | 0.93 | 0.03 | 0.51 | 0.47 | 0.09 |\nRVT | 0.11 | 0.71 | 0.04 | 0.50 | 0.48 | 0.26 |\nRVT-2 | 0.40 | 0.74 | 0.38 | 0.66 | 0.88 | 0.69 |\nVIRT (Ours) | 0.48 | 0.79 | 0.42 | 0.68 | 0.90 | 0.73 |\n\nThe results on the ManiSkill and RLBench benchmarks show VIRT achieves the best results, which further confirm the superiority of our method.\n\n## Q5: How image and point flow are integrated.\n\nAs shown in the pipeline of the paper, the images and points are encoded as tokens separately. Then, we directly concatenate them as the feature input to Transformer decoders. We will release our code publicly to help other researchers reproduce our work.\n\n## Q6: Image random mask in pre-train.\n\nThe image random mask improves the robustness of the pre-train policy but cannot directly enable the policy to learn useful representation. Therefore, the future action prediction task is necessary in pre-train.", "claims_and_evidence": "The main claim regarding the drawback of text-instructed policies—that they may misinterpret the attention focus on the target object—is not sufficiently substantiated. While the authors provide a visualization in Figure 1 to compare text- and vision-instructed policies, two critical issues undermine the credibility of this comparison. First, the specifics of the text-instructed policy used for the visualization are not detailed, which raises questions about the validity and reproducibility of the results. Without transparency regarding the implementation and parameters of the text-instructed policy, it is difficult to assess the fairness of the comparison. Second, the claim appears to overlook the significant success and widespread adoption of text-instructed pretraining methods in the field. Text-instructed policies have demonstrated remarkable achievements across various pretraining tasks, making the assertion that they are inherently flawed seem overly subjective and potentially biased.", "methods_and_evaluation_criteria": "Yes, integrating point cloud flows and future images into the pretraining framework could lead to more robust and generalizable policies.", "theoretical_claims": "This paper contains no theoretical claims.", "experimental_designs_or_analyses": "The soundness of the proposed method is not fully established. While the paper introduces a novel pretraining paradigm and, in Section 4, conducts experiments to validate the importance of individual components and the overall improvements for downstream tasks, it lacks a crucial comparison with other existing pretraining methods. This omission leads to the confusion about how the proposed method performs relative to previous pretraining methods. Without such comparisons, it is difficult to determine whether the claimed advancements are truly superior to state-of-the-art alternatives.", "relation_to_broader_scientific_literature": "The authors propose a novel pretraining method for robotic manipulation tasks. Unlike previous text-driven pretraining methods such as R3M and CLIP, this work shifts the focus to a vision-based paradigm. Text-driven methods typically align given prompts with current visual observations to generate conditional features for downstream policies. The authors identify potential limitations in text-instructed policies, like \nmisinterpretations or illusions in aligning textual instructions with visual contexts. To address this, they design a novel vision-based pretraining framework. A key contribution of the proposed method lies in its explicit use of future images and point cloud flows, which clearly indicate the intended direction of target actions. By focusing on visual cues rather than relying on potentially ambiguous textual instructions, the proposed method could be able to overcome some of the inherent limitations of text-driven approaches.", "essential_references_not_discussed": "This paper introduces a novel pretraining paradigm; however, it overlooks one of the most widely recognized and influential pretraining methods in the field, like R3M, which was published at CoRL 2022.", "other_strengths_and_weaknesses": "Strength:\n\n1.The integration of additional visual features, such as future images and point flows, for pretraining a robust feature representation network is a notable strength. This approach enhances the performance of downstream action policies without relying on text embedding networks like CLIP, offering a more direct guidance for robotic manipulation tasks.\n\n2.The authors conduct experiments in both real-world and simulated environments, which helps verify the practical effectiveness and versatility of the proposed method.\n\nWeaknesses:\n\n1.The claim regarding the drawbacks of text-instructed policies is not well-supported. Text-driven pretraining paradigms, such as R3M, are widely adopted in the field and have demonstrated strong generalizability in robotics. The lack of a fair comparison or evidence to substantiate this claim undermines its validity.\n\n2.A significant limitation is the absence of experiments comparing the proposed method with other established pretraining approaches, such as CLIP-based or R3M-based methods. This omission weakens the confidence in the superiority or even the competitiveness of the proposed method against existing alternatives.\n\n3.The presentation of the algorithm's details is insufficient. While the paper mentions the integration of images and point flows during training, it fails to provide a clear explanation of how these features are processed or combined, leaving critical implementation aspects unclear.\n\n4.Although the authors propose a general pretraining method, they do not evaluate it on widely used benchmarks like RLBench or Robomimic. Such evaluations are essential to demonstrate the generalizability and robustness of the proposed method across diverse tasks and environments, which is a key expectation for a general pretraining framework.", "other_comments_or_suggestions": "1. Why not incorporate masks directly into the pretraining stage? Utilizing masks during pretraining could enhance consistency between the pretraining and inference phases. If the mask image is validated to be effective, it seems like the current pretraining process can also \nrely on it instead of future images.\n\n2. Is the pretraining model directly employed in the simulation environment? If so, the ability of the pretrained model should be avaluated in other imitation learning benchmark like RLBench or Robomimic.", "questions_for_authors": "Please refer to the comments_suggestions and the strenghs_weaknesses sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TvwRKTV9pu", "reviewer": "Reviewer_Ee7z", "summary": "In this work, based on the observation that current policies cannot capture features from text instruction effectively, due to data scale, the authors propose a model named VIP which utilizes vision instructions to specify manipulation targets as an alternative. Specifically, the input of VIP is the current observation, future observation, and a corresponding sparse point flows between the current and future images. Since sparse point flows and future images are not available during inference time, they are progressively randomly masked during pre-training and future images are replaced with the cropped region of the object to manipulation in the current observation. The effectiveness of VIP is demonstrated with three tasks in the real world and three tasks in simulation(IsaacGym).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We have addressed the concerns of the Reviewer one by one in the following. The paper will be revised accordingly.\n\n## Q1: Experiments with more distraction.\n\nAs suggested by the Reviewer, we have added experiments with stronger distraction in both simulated and real robot environments. For simulation, we select a set of challenging tasks in RLbench, and these tasks require challenging actions like insert, open, and screw. We compare our method with previous SOTAs using the same training data and testing protocol. The results are reported as follows:\n\nMethod | Insert Peg | Open Drawer | Place Cups | Put in Cupboard | Screw Bulb | Stack Cups |\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: |\nConvMLP | 0.00 | 0.04 | 0.00 | 0.00 | 0.00 | 0.00 |\nPolarNet | 0.04 | 0.84 | 0.00 | 0.12 | 0.44 | 0.08 |\nPerACT | 0.06 | 0.88 | 0.02 | 0.28 | 0.18 | 0.02 |\nACT3D | 0.27 | 0.93 | 0.03 | 0.51 | 0.47 | 0.09 |\nRVT | 0.11 | 0.71 | 0.04 | 0.50 | 0.48 | 0.26 |\nRVT-2 | 0.40 | 0.74 | 0.38 | 0.66 | 0.88 | 0.69 |\nVIRT (Ours) | 0.48 | 0.79 | 0.42 | 0.68 | 0.90 | 0.73 |\n\nWe can see that our method outperforms all previous methods. \n\nIn addition,  we devise two new real robot tasks to verify the performance of our method. In both tasks, there are multiple targets randomly placed on a table. In the spoon deliver task, the robot hand needs to pick and deliver randomly placed spoons to a tray. This task is more challenging because spoons are smaller and the robot must learns to grasp the necks of the spoons in diverse poses. In the wire deliver task, the manipulation target is non-rigid and presents non-regular shapes. The policy must delivers all seen objects to achieve one successful trial. The success rates of different methods on these two new tasks are as follows:\n\nMethod | CNNMLP | Diffusion | ACT | VIRT (Ours) |\n| :-: | :-: | :-: | :-: | :-: |\nSpoon Deliver | 0.00 | 0.06 | 0.14 |  0.42 |\nWire Deliver | 0.00 | 0.00 | 0.07 | 0.39 |\n\nWe can find that VIRT outperforms other methods more significantly on these more complex tasks. Moreover, we present anonymous video links of our method on these two tasks here: [Spoon Deliver](https://drive.google.com/file/d/1sEOaJ9PwOs8aV7beUxLOJdWNAVCEzFXL/view?usp=sharing) and [Wire Deliver](https://drive.google.com/file/d/111Jbot6Skb1uRL7u-a40oGxNglf4OYLo/view?usp=sharing).\n\n## Q2: Method for Moving Object Manipulation.\n\nYes, our method supports moving object manipulation due to its fast response speed and precise manipulation operation. We design a new task that the robot needs to pick the moving spoons on a moving belt and delivers them to a box. This task is challenging because (1) The spoons are small and in diverse poses. The robot hand must grasps the necks of spoons for successful picking up. (2) The spoons are moving, which means the policy must learns to predict the positions of spoons at the grasp moment rather than the current moment.\n\nThe success rate of VIRT arrives 0.82, while all the other test methods fail. This result attributes to two advantages of our VIRT: (1) VIRT shows very fast response speed. (2) The vision instruction helps VIRT focus on the moving spoons and better predict the future positions of these spoons.\n\nWe present an anonymous video link of our method on this new task here: [Move Manipulation](https://drive.google.com/file/d/111juTCA0EHkfemDUYPoUj0xdHg2Y6ldC/view?usp=sharing). To the best of our knowledge, this is the first end-to-end imitation policy that can perform moving object manipulation.\n\n## Q3: More discussion with relevant works.\n\nWe thank the Reviewer for this reminder. As suggested by the Reviewer, we will add all the references mentioned by the Reviewer and discuss the relevance of our work with them in the paper. The papers mentioned by the Reviewer are mostly about using video data cheaper than robot manipulation data (like human ego-centric videos and web videos) to conduct pre-train. The advantage of these works is the data is easy to obtain, but their effectiveness is limited due to embodiment gaps. Differently, what we study is another setting: how to use robot manipulation data to conduct pre-train, where the data is more expensive to collect but provides more direct generalization. Notably, these two settings of pre-train do not conflict with each other. We can first pre-train a policy using cheap web data and then fine-tune the policy based on the robot manipulation data based on our proposed method.\n\n## Q4: Parameter number.\n\nAs suggested by the Reviewer, we will report the parameter numbers of these methods in the paper. For the convenience of review, we also list them as follows:\n\nMethod | CNNMLP | Diffusion | ACT | VIRT (Ours) |\n| :-: | :-: | :-: | :-: | :-: |\nParameter Number | 76.7M  | 142.8M  | 115.0M  |  38.6M |\n\nAccording to the results, we can find that VIRT is the most lightweight model. This is mainly because VIRT adopts a very concise architecture consisting of only Transformer encoders and decoders.", "claims_and_evidence": "- This work claims that text-instructed policies cannot successfully capture the feature of manipulation targets because of the lack of sufficient manipulation data and image-text pairs. Figure 1, which displays the attention map of a text-instructed policy in a pick-and-place scenario, evidences this claim.\n\n- The effectiveness of vision instructions is supported by the policy attention map in Figure 1 and the strong performance in both real and simulated environments.", "methods_and_evaluation_criteria": "The proposed vision-instructed pre-training method makes sense for the manipulation tasks. The evaluations involving both tasks in the real world and the simulation are comprehensive, and the evaluation criteria are the success rates of each task.", "theoretical_claims": "This work is mainly an ML application in robotic manipulation tasks, so there is no need to provide theoretical proofs.", "experimental_designs_or_analyses": "It is promising that VIRT has achieved all state-of-the-art performance in the tasks given in the paper. However, those tasks involved are still very simple, pick-and-place tasks in a clean background, which cannot genuinely demonstrate the advantages of vision instructions over text instructions as proposed in the paper. It would be better if you could add more distractions such as more positional randomness of your manipulation targets, and more complex tasks, such as articulated object manipulation tasks, to your evaluation suite. The robustness to simple image transforms is not persuasive enough of its generalization capabilities and the effectiveness of the vision prompts.", "relation_to_broader_scientific_literature": "Current pre-training methods in robotics manipulation mainly apply text as the prompt, while using vision instructions is less explored.", "essential_references_not_discussed": "Some works which pre-train on human demonstration datasets and are fine-tuned for robotic manipulation are not discussed in this work. These works involve but are not limited to:\n\n[1] Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. In CoRL, 2022.\n\n[2] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3M: A universal visual representation for robot manipulation. In CoRL, 2022.\n\n[3] Jia Zeng, Qingwen Bu, Bangjun Wang, Wenke Xia, Li Chen, Hao Dong, Haoming Song, Dong Wang, Di Hu, Ping Luo, Heming Cui, Bin Zhao, Xuelong Li, Yu Qiao, Hongyang Li. Learning Manipulation by Predicting Interaction. In *RSS*, 2024.\n\n[4] Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Pieter Abbeel, Jitendra Malik, Dhruv Batra, Yixin Lin, Oleksandr Maksymets, Aravind Rajeswaran, Franziska Meier. Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence? In arXiv, 2023.\n\nBesides, there are also some works supporting utilizing both vision and text instructions, such as:\n\n[5] Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, Sergey Levine. Octo: An Open-Source Generalist Robot Policy. In RSS. 2024.\n\nare not discussed in this work.", "other_strengths_and_weaknesses": "The clarity of this work is great.", "other_comments_or_suggestions": "It would be better if Table 1 could provide the parameter numbers of each method for direct comparison.", "questions_for_authors": "Can VIRT also generalize to moving manipulation target scenarios?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2bfGu15ovu", "reviewer": "Reviewer_DB9D", "summary": "The authors propose a pretraining strategy for robotic manipulation in which all inputs are visual, with no language descriptions required.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We believe the Reviewer has significant misunderstandings of this work. In the following, we address the concerns one by one using more precise explanations and sufficient experiments. The paper will be revised accordingly to avoid these misunderstandings.\n\n## Q1: Insufficient support for vision instruction\n\nWe believe we do not mention RoboFlamingo in the paper. What we claim is in the current manipulation data status (very expensive to collect), vision instruction is more efficient for disclosing task information. \n\nWe agree models like OpenVLA and PI0 can pick up the shown green box given text instruction We do not claim that text instruction cannot specify target. What we highlight is the efficiency difference between text and vision instructions. Robot manipulation data is expensive, and models like OpenVLA have used up most public data resources (Open X-Embodiment dataset covers most popular manipulation datasets). However, we find that OpenVLA overfits to specific text representation. For example, when we use the instruction ''grasp the cup\", it works well. However, when we use ''pick up the cup'', the policy fails. This means even though we combine almost all public manipulation datasets, the data is still insufficient to cover common text instructions. By contrast, the shown generaliazation performance of our method is much better.\n\nTo further support our claim, we compare the performances of the models pre-trained by OpenVLA, PI0, and our VIP. As the zero-shot manipulation results of OpenVLA and PI0 are limited, we tune them with the same data as our policy for fair comparison. Specifically, We use the pre-trained feature extractors to extract feature, and employ three Transformer decoders to decode the extracted feature as actions to execute. Each pre-trained policy is test in two settings, freeze or not freeze the feature extractor during tuning. Text instructions are provided to VLA-based policies. The success rates are as follows:\n\nMethod | Freeze | Pour Blueberries | Open Lid | CLean Table | Move Box | Transport Box | Stack Boxes |\n| :-: | :-: | :-: | :-: | :-: |  :-: | :-: | :-: |\nPI0 | Yes | 0.07 | 0.48 | 0.14 | 0.82 | 0.44 | 0.29 |\nPI0 | No | 0.28 | 0.60 | 0.31 | 0.84 | 0.58 | 0.55 | \nOpenVLA | Yes | 0.14 | 0.45 | 0.19 | 0.81 | 0.45 | 0.36 |\nOpenVLA | No | 0.30 | 0.67 | 0.32 | 0.88 | 0.66 | 0.60 |\nVIP (Ours) | Yes | 0.32 | 0.63 | 0.34 | 0.88 | 0.69 | 0.61 |\nVIP (Ours) | No | 0.42 | 0.71 | 0.37 | 0.92 | 0.74 | 0.68 |\n\nAccording to the results, although PI0 and OpenVLA use much more pre-train data than VIP, VIP achieves better performance. This result supports our claim that vision instruction is more efficient than text instruction. The ablation study in Table 2 of the paper also supports this claim. \n\nIn summary, our method achieves superior or at least similar performance as OpenVLA and PI0 based on much less data, which favors addressing the expensive data collection cost in robot manipulation.\n\n## Q2: Vision instruction is more comprehensible\n\nIn this work, \"more comprehensible\" means the policy can learn to follow instructions based on less training data. As reminded by the Reviewer, we will replace \"comprehensible\" as \"easy to follow\".\n\nWe understand the concern of the Reviewer is VLA may can understand text instruction easily as it has been pre-trained using numerous image-text pairs. Actually, we have tried many VLAs during developing this work. The observation is large-scale image-text pre-train favors understanding text instruction in robot manipulation, but the required manipulation data volume required by text instruction is still significantly larger than vision instruction to learn to complete a task. This is due to the gaps between common image-text data and manipulation data. Image-text-action data is precious and far from being sufficient to train the policy to follow general text instruction following. The experiment results in the Reply to Q1 supports this observation.\n\n## Q3: How the cropped region is obtained\n\nIn test, we give a text instruction, and the text instruction is converted as vision instruction (cropped region) based on a 2D object detector.\n\nWe do not claim we should not use text instruction. It is more convenient for human to give instructions based on text than taking a photo. What we claim is vision instruction is more efficient to serve as the direct input to the policy. Therefore, we convert the text instruction from human as vision instruction and input the vision instruction to the policy. The experiment results in Table 2 of the paper indicates this design leads to much better performance than inputting text instruction to the policy.\n\n## Q4: Comparison with more pre-training methods\n\nAs suggested by the Reviewer, we conduct more comparison with the recommended pre-train methods. Besides the results in the Reply to Q1, find more comparison results in the anonymous link: [Exp Result](https://anonymous.4open.science/r/VIP_rebuttal-9331/README.md).", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Good", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The paper is well-written, which is easy to follow. The real-world experiments are interesting, which include dual-arm setting.\nHowever, my main concern lies in the motivation of this work, as described in the introduction, which highlights the drawbacks of language descriptions and emphasizes the importance of visual observations:\n\n1. The paper uses RoboFlamingo to argue that language-guided policy models struggle to follow instructions due to data scarcity. However, I do not find this example convincing, as RoboFlamingo is not pretrained on a large-scale robotic dataset paired with language descriptions. Other works, such as OpenVLA[1] and PI0[2], pretrain models on large-scale robotic demonstrations with language descriptions, enabling instruction following—at least in simple cases like the 'green block' example provided by the authors. While I partially agree with the claim made in the second paragraph of the introduction, I do not believe that RoboFlamingo and the 'green block' example are appropriate choices to support this argument.\n\n2. In the third paragraph of the introduction, the authors claim that visual prompts provide a more comprehensive representation for the model, leading them to use only visual information for pretraining, supported solely by a human intuition example. While I acknowledge that visual prompts have advantages over language prompts, their advantage should not be described as 'comprehensive to the model.' A visual model may be more comprehensive for a vision-motor policy, but this does not necessarily hold for a vision-language-action model.\nMoreover, the intuition-based example does not sufficiently support the authors' argument. A citation to prior research would be more convincing.\n\n3. If the claim that visual prompts provide a more comprehensive representation for the model is true, then how did the authors crop the image region of the object for manipulation during fine-tuning and inference without language guidance? If language is still used to guide the cropping process, then the claim that visual prompts alone are sufficient is inconsistent. The motivation should align with the expectation that the model fully leverages visual observations without relying on language descriptions.\n\n[1] Kim, Moo Jin, et al. \"Openvla: An open-source vision-language-action model.\" arXiv preprint arXiv:2406.09246 (2024).\n\n[2] Black, Kevin, et al. \"$\\pi_0 $: A Vision-Language-Action Flow Model for General Robot Control.\" arXiv preprint arXiv:2410.24164 (2024).", "other_comments_or_suggestions": "Additional baselines focusing on pretraining models for robotic tasks should be considered, such as R3M[3], VC-1[4] and others.\n\n[3] Nair, Suraj, et al. \"R3m: A universal visual representation for robot manipulation.\" arXiv preprint arXiv:2203.12601 (2022).\n\n[4] Majumdar, Arjun, et al. \"Where are we in the search for an artificial visual cortex for embodied intelligence?.\" Advances in Neural Information Processing Systems 36 (2023): 655-677.", "questions_for_authors": "Refer to weakness section", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes VIP, a vision-instructed pretraining framework for robotic manipulation that uses visual cues as instruction, instead of relying on potentially ambiguous language inputs. The key idea is to guide policy learning using purely visual signals, allowing robust pretraining while discarding the need for language-based annotation or alignment. The authors conduct extensive evaluations on both simulated and real-world tasks and demonstrate improvements over existing baselines.

Reviewer uDoz recommends accept, acknowleging the novelty of using visual signals as instruction and the strong empirical results, especially in real-robot settings. The reviewer’s concerns about real-world ablations were addressed by new experiments added in the rebuttal. Reviewer Ee7z also recommends accept after the rebuttal. Initially concerned with the simplicity of tasks, this reviewer was satisfied by the newly added results on more complex and dynamic environments (e.g., moving object manipulation and multi-target settings). Reviewer DB9D adjusts their rating to weak accept, for the additional clarifications around the motivation and claims about vision-vs-language instruction efficiency. The authors provided concrete comparison results against PI0 and OpenVLA, demonstrating better or comparable performance despite using less training data.

Reviewer HYVi remains at weak reject, maintaining that the drawbacks of text-instructed policies were not sufficiently justified and that certain implementation and comparison details were still lacking. Nevertheless, many of the key concerns were addressed in the rebuttal, including new experiments on RLBench and ManiSkill, network details, and expanded comparisons.

Given the overall positive consensus, the detailed and data-rich rebuttal, and the importance of the problem setting, I recommend acceptance.
Decision: Accept (poster)