PIVOT-R: Primitive-Driven Waypoint-Aware World Model for Robotic Manipulation
Authors: Kaidong Zhang
Pengzhen Ren
Bingqian Lin
Junfan Lin
Shikui Ma
Hang Xu
Xiaodan Liang
Venue: NeurIPS 2024 poster (2024)
Keywords: Robot manipulation; World model
Primary area: robotics

Abstract: Language-guided robotic manipulation is a challenging task that requires an embodied agent to follow abstract user instructions to accomplish various complex manipulation tasks. Previous work generally maps instructions and visual perceptions directly to low-level executable actions, neglecting the modeling of critical waypoints (e.g., key states of “close to/grab/move up” in action trajectories) in manipulation tasks.
To address this issue, we propose a PImitive-driVen waypOinT-aware world model for Robotic manipulation (PIVOT-R) that focuses solely on the prediction of task-relevant waypoints. Specifically, PIVOT-R consists of a Waypoint-aware World Model (WAWM) and a lightweight action prediction module. The former performs primitive action parsing and primitive-driven waypoint prediction, while the latter focuses on decoding low-level actions. Additionally, we also design an asynchronous hierarchical executor (AHE) for PIVOT-R, which can use different execution frequencies for different modules of the model, thereby helping the model reduce computational redundancy and improve model execution efficiency. Our PIVOT-R outperforms state-of-the-art (SoTA) open-source models on the SeaWave benchmark, achieving an average relative improvement of 19.45% across four levels of instruction tasks. Moreover, compared to the synchronously executed PIVOT-R, the execution efficiency of PIVOT-R with AHE is increased by 28-fold, with only a 2.9% drop in performance. These results provide compelling evidence that our PIVOT-R can significantly improve both the performance and efficiency of robotic manipulation.

PIVOT-R: Primitive-Driven Waypoint-Aware World Model for Robotic Manipulation
Kaidong Zhang!"
Pengzhen Ren Hang Xu*
Bingqian Lin' Xiaodan Liang Dataa Robotics
Junfan Lin²
Shikui Ma
Sun Yat-sen University
PPngg Cheng Laboratory
*Huawei Noah's Ark Lab
http://abliioggiihubbo///VVV
Abstract
sanguage-guiieee robotic manipulation is challenging task that requires bodied agent to follow abstract user instructions to accomplish various complex manipulation tasks. Previous work generally maps instructions and visual percep- tions directly to low-level executable actions, neglecting the modeling of critical waypoints (e.g., key states of "close to/grab/move up" in action trajectories) in manipulation tasks. Trivially fitting the data without revealing the relation between instruction and low-level executable actions, these models prone memorizing the surficial pattern of the data instead of acquiring the transferable knowledge, and thus are fragile dynamic environment changes. To address this issue, we propose a Prlmitive-driV waypOinT SaypOinTT-wwar world model for Robotic manipula- tion (PIVOT-R) that focuses solely on the prediction of task-relevant waypoints. Specifically, PIVOT-R consists of a Waypoint-aware World Model (WAWM) and a lightweight action prediction module. The former performs primitive action parsing and primitive-driven waypoint prediction, while the latter focuses on decod- ing low-level actions. Additionally, we also design an asynchronous hierarchical executor (AHE) for PIVOT-R, which can use different execution frequencies for different modules of the model, thereby helping the model reduce computational redundancy and improve model execution efficiency. Our PIVOT-R outperforms wate-of-the-ar (SoTA) open-source models on the Save benchmark, achiev- ing average relative improvement of 19. 19.45% across four levels of instruction tasks. Morcover, compared to the synchronously exccuted PIVOT-R, the exccution efficiency of PIVOT-R with AHE is increased by 28-fold, with only 2.9% drop in performance. These results provide compelling evidence that our PIVOT-R can significantly improve both the performance and efficiency of robotic manipulation.
Introduction
38 is key research problem of Embodied AI. This field aims to enable agents to follow abstract language instructions for performing various manipulation tasks. To complete the tasks, the agent needs to transform high-level language instruc- tions into low-level actions as well as capturing environmental dynamics for precise manipulation decision-making.
Language-guid robotic manipulation 22 33 61
50
12
Witnessed the immense success of vision-language foundation models (VLMs) works have explored the utilization of VLMs for facilitating language-guided robotic manipulation in
40
37
many
'Equal contribution Corresponding authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

the thet's placed uiddle
of the table.
User
Visual imago
User instructions
PIVOT-R WAWM
Current Ohservation
Pretrained VLM
Hi
World Model
close to the middle object
Primiiive
VLM
action
Scene prodiction mnodule
Ht.
Wagpoint
Action Head
Vision
Action prediction module
VgH
group
Decoder
Initial state
each
timestep
Atteecaaton 8,0 ARocticc [0.0.0].
Low-level
Intermediate state
Action
Images/features
prapper
action
Waypoint
(a) Sequentially executed robol munipulation model
(b) PIVOT-R
Figure I: Comparison of PIVOT-R and other models. (a) Sequentially executed robot manipulation model. They sequentially execute each module in the model at each timestep to perform manipulation reasoning (e.g., RT-2 (44. RT-X [9), RT-H [3], VILA [00 Octo [36. etc.) or world modeling (e.g., 60 etc.) This easily leads model redundancy and weak key manipulation node prediction capabilities. (b) PIVOT-R is primitive-driver driven waypoint-as aware world model with asynchronous hierarchical executors. It only focuses on the prediction of waypoints related to the manipulation task, and it is easier to predict key nodes in the manipulation task than other methods. In addition, PIVOT-R sets different execution frequencies for different modules to have higher execution efficiency and lower redundancy.
Surfer [22, Daydreamer [66] 3D- VLA
recent years 48
64 49 27 20 For example, RT-2 664, RT-X 49), RT-H [5] and RoboFlamingo
27 employ the VLM as the backbone and introduce large-scale vision language data for manipulation training, which significantly improve the generalization. VILA 200 resorts to GPT-4 (37 to generate sequential actionable steps for improving long-horizon planning. Daydreamer (56) have also tried to introduce world models into robot manipulation to help the models free themselves from a large amount of trial and error and improve learning efficiency. Despite extensive efforts made by researchers, two key challenges remain: (i) Weak key waypoint prediction and world modeling capabilities; (ii) High computational redundancy and inefficient execution.
addition, 3D-VI VLA
60) and
For the first challenge, such as "moving cup". humans intuitively apply their internal world models to seamlessly analyze and predict task-related key action flows: "getting close to the cup - grab the cup - move the cup + put down the cup". Similar to the approaches in navigation tasks, we define these key action frames as waypoints for manipulation tasks. Figure (b) right shows a robot manipulation task with three waypoints. How to enable robots to acquire this ability is very critical. To this end, RT-H (5) uses VLM to perform natural language parsing key action nodes and uses language to guide robot manipulation. However, it does not perform world modeling visual scene information. Therefore, some work (56) 34 60 42 have attempted to summarize general dynamic knowledge about the environment and predict future outcomes by introducing world models, to generate more executable long-term plans and accurate manipulation action decisions. However, they tend to model the world at each timestep of robot manipulation, leading to the neglect of waypoints which have more direct impact on manipulation success. To make matters worse, in the long-term lack of key waypoint guidance, the randomness of each action prediction may be continuously amplified due to the existence of low-level action similarities under local spatiotemporal conditions.
56 tend to treat different modules in the model equally and exccute all modules scquentially. which is not necessary and inevitably introduces redundancy of computation and causes great cost of resources. To this colliirresoltionn transformer that uses different exccution frequencies for different spatial and temporal resolutions to control coarse, precise, and dynamic tasks in real-time, thereby effectively reducing unnecessary computational redundancy and improving the real-time performance of robot manipulation. However, it lacks focus on world modeling capabilities and cannot predict critical nodes of manipulation tasks as accurately as humans.
For the second challenge, as shown in Figure (a), previous methods 64 5
49
end, MResT 43 proposes
Based on the above observations, shown in Figure (b). in this paper, we propose PIVOT-R, primitive-driven waypoint-aware world model with an asynchronous hierarchical executor for robot manipulation. PIVOT-R mainly consists of waypoint-awarc world model (WAWM) and an action

---

prediction module. Specifically, in WAWM, we first use the pre-trained VLM for primitive action parsing and use it as a primitive prompt for the scene prediction module to help the model perform modeling of the robot manipulation waypoint scene. Then, we use waypoints as cues for low-level action prediction. Thanks to WAWM's 's modeling of key waypoint information, PIVOT-R achieves an average relative performance improvement of 19.45 19.45% compared to the state-of-the-e thrtt--rrt (SoTA) open-source manipulation model on SeaWave's 's [42] 4-level instruction tasks. In addition, to reduce model redundancy, we also design an asynchronous hierarchical executor (AHE) for PIVOT-R, which sets a slow-to-fast execution frequency scheduler for the three modules of primitive action parsing, scene prediction, and action prediction in the model to help PIVOT-R improves execution efficiency. With the help of AHE, the execution efficiency of PIVOT-R integrated with VLM has not dropped significantly. Compared with synchronously executed PIVOT-R (all modules use the same execution frequency), the execution efficiency of PIVOT-R with AHE is increased by 28 times, while the performance only drops by 2.9%. Our contributions can be summarized as follows:
We show that modeling of waypoints prevents critical robot dynamics from being sub- merged in trivial robot manipulations, allowing models to benefit from enhanced dynamic environment modeling.
The proposed AHE significantly improves the execution efficiency of the model by setting different frequencies for different modules.
Extensive experimental results demonstrate that our PIVOT-R achieves significantly better performance than the SoTA baseline, such as Gato [4] and RT-1 77. in all settings.
2
Related Work
Language-Guin Guided Robotic Manipulation. Robotic Manipulation a long-standing research field in Embodied Artificial Intelligence. Benefiting from the flexibility and practicality of facilitating human- robot interaction, language-guided robotic manipulation has gained extensive research attention in recent years. Many benchmarks have been built to encourage the research of hinguage-guided robotic manipulation, such as RLBench [22], CALVIN (33], VLMBench [61] etc. Early methods improve the manipulation performance by introducing powerful representations (9] [59], elaborated network architectures [5] 13], or effective training mechanisms 32
44). With the rapid development of VLMs 40 37,, recent works have attempted to introduce VLMs improve the manipulation accuracy
and generalization to unseen scenarios/objects a trainable 48
64 49 27 or offline 21 35
manner. However, most previous approaches tend to learn direct mapping from multi-modal inputs to low-level actions, ignoring the explicit modeling of environmental dynamics. As a result, they may fail to make executable actions plans and not generalize well to complex environments. We have also noticed previous work on waypoints and primitive actions, but they often used a limited number of actions. For instance, CLIPort (45), Transporter [57]. GMRT [47], and VPG (58) are restricted to simple actions like pick/place/push limiting their use in complex tasks. Some language-
models 110 16. 30) define few primitive actions (S 5) and add prompts to aid decision-making PerAct 46). RVT [4] use robot states as waypoints to skip trivial action predictions. SUSIE [6] and [T predict sub-goals through video predictors, but there is an inconsistency between the predicted video and actions. In this work, we propose waypoint-aware world model to track key dynamics that happen during the manipulation. Our model fulfills asynchronous world modeling and action prediction, which significantly promotes both manipulation accuracy and efficiency. PIVOT-R supports 10 primitive actions and is extensible, making it effective in complex tasks.
World Models. World models aim to generate a predictive model of its surroundings, accounting for uncertainties and dynamic changes. They have been widely studied in video generation 4 53 g) navigation [5] 24, 39), and autonomous driving [52] 62 [4] areas. For example, Genie (8) introduces spatiotemporal video tokenizer and dynamies model autoregressively predict the next video frame. DriveDreamer [52) builds a world model deriving from real-world driving scenarios for enabling reasonable driving policy generation. With the great potential for acquiring insights into real-world motion and physics rules, some works have also introduced world models for robotic
manipulation tasks [56]
60) Daydreamer (56) applies the Dreamer [17] algorithm to train real-
world robots by online reinforcement learning. SWIM (34] collects human videos for training world model and fine-tuning it on a small amount of robot data. Nevertheless, they usually perform world modeling and makiin alternatively, bringing great difficulty for training and is also
3

---

Asynchronous Hlerarchical Executor
Waypoint-Aware World Model
Please the ite thats placed
Protesined
VLM
objoct.
grab'move
Salf-itiintin
Feod/ooward
Instraction
Fog-
alttenioo
+ Text Encoder
Scene prodiction
Robot
Image Encoder
medule
Lucone
(b) Scene prediction module
waggolitt
Put
Fo.
Fwt
Opt
Action prediction
Lues
Self-ar Self-afteution
FeedForwand
medule
Co bor MLP(S-Aa))
dttection
+
MLP
Robot status Se-tex
x LA
(a) Overall framework
(e) Action prediction module
Figure 2: PIVOT-R overview. It mainly consists of a waypoint-aware world model (WAWM) and an action prediction module, where two modules cooperate with each other through an asynchronous hierarchical executor (AHE). In WAWM, we first use pre-trained VLM to perform low-frequency primitive action parsing user instructions and provide waypoint indications for the scene prediction module. Then, the scene prediction module learns to model the world knowledge based on waypoints and manipulation trajectories. Finally, we use a lightweight action prediction module perform high-frequeee action prediction and execution.
inefficient. In contrast, our proposed WAWM only predicts task-relevant waypoints, empowering realistic and efficient world modeling for improving manipulation performance.
Vision-Language Foundation models. Vision-Language Foundation models (VLMs) have witnessed striking advancements in recent years. The ability to understand multimodal inputs and rich real-world knowledge storage of VLMs makes them highly adaptable for a wide range of downstream applications such image captioning [25 63) and visual question answering (29 25
Many works have also explored the utilization of VLMs in robotic manipulation tasks recently 48
49 27 21 20). MOO [88] leverages pre-trained vision-language model to improve zero-shot open- world object manipulation. RoboFlamingo 27 builds a Iisionnlanguage manipulation framework upon the open-source VLM OpenFlamingo
VILA 20 and CoPa [21] unleash the commonsense knowledge of GPT-4 for generating accurate and reasonable manipulation action decisions. this work, develop elegant combination of VLMs and world models for tackling the challenging inguage-guided robotic manipulation task, where we query the VLM, the world model, and the action execution model in an asynchronous way.
3
Architecture
Our goal is to build robot manipulation model that can respond accurately and timely to user instructions in various zero-shot complex and variable environments. To this end, as shown in Figure 2,
we introduce primitive-driven waypoint-aware world model for robot manipulation. Next, discuss the structural details of each module PIVOT-R in detail.
3.1 Problem Formulation
As shown in Figure 2 (a), we formulate the proposed PIVOT-R learning trainable robot manipula- tion model π, which maps the user's language instruction / and series of observation images Oc- and robot state St-A.a from the time step t h to the current time step t action A,. h represents the length of the historical frames, here it is set to 3. In addition, we also introduced a scene prediction module for WAWM to help the model model world knowledge. The overall formulation of PIVOT-R is as follows:
म (VLM(1, O,), Ot. St-nst) - MG A/e
(1)
where My and A are the waypoints and actions of the robot manipulation predicted by the model timestep t, respectively. In particular, the pre- pre-trained VLM to parse the primitive actions P

---

the current robot should take from the user instruction / based on the robot's observation image O Then, we use P as a waypoint indication for robot manipulation time step t, helping the robot to build prediction and modeling capabilities for future scene information and world knowledge. For each action trajectory Tra, it consists of a language instruction I and series of observation images O, robot status S, actions A, and waypoints M:
Tra- {4, [01, S1 Mi) [Or, ST, Ar, MT]}
(2)
where I' is the timestep length of the robot's manipulation trajectory. Note that because we use AHE, the primitive actions P input to the scene prediction module at different time steps & may be the same. The model can avoid redundancy caused by the alternating use of VLM and world models through low-frequency primitive action parsing, thereby improving training and inference efficiency. We adopt similar settings on the action prediction module to further improve the efficiency of the model.
3.2 Inputs and Outputs
We provide a detailed description of the inputs and outputs of PIVOT-R in Figure (a) as follows:
Language input. The user's language instruction is first combined with the prompt and used as the input of the pre-trained VLM to parse the primitive action represented by the short text. The details of the prompt are shown in Appendix [FI]] Specifically, in the example of the language instruction "Give me container of drinking water". the primitive action this time may be "approach/grab/pt down the container". Then, the parsed primitive action and original instruction I are encoded by text encoder as a text sequence P. Following 45 42
we employ pre-trained
CLIP [40] as the language encoder Emex-
Visual input. For visual observation of RGB image O, we use a pre-trained CLIP 40 visual
encoder Emmuge for encoding.
Robot state input. The robot state includes 6 dimensions of robot arm movement $ = (xr, y, 2, pitch, yaw). We use linear layers to encode them.
Outputs. The output of PIVOT-R is the feature FM; bx of the task-related waypoint image predicted by the scene prediction module and the robot action A₂ predicted by the action prediction module. Where 4, n = 49, and = 512 represent the batch size, number of tokens, and dimension of the feature FM,, respectively. The action A contains the delta state S of the robot's end-effector and the binary state G E {0, 1} of the gripper, i.e., = (S, (S.G)= 1x7
3.3 Network
Overall, PIVOT-R consists of a powerful waypoint-aware world model and a lightweight action prediction module, whose detailed information is described as follows:
Waypoint-Aware World Model (WAWM). By introducing waypoints as a data structural chunking mechanism, similar to tokenization in NI.,, we segment dense and irregular robot trajectories into meaningful sections, reducing the prediction burden. This hicrarchical approach decouples language-action interdependencies and leverages cross-trajecs trajectryy waypoint transition knowledge, improving action prediction accuracy. As shown in Figure 2 WAWM mainly includes a powerful VLM and scene prediction module sp Given user instruction 7, the VLM parses I to provide task-related waypoint prompts, which are used for guiding the scene prediction module Pap to conduct critical waypoint prediction.
Specifically, at each timestep f,, we combine the prompts with the user instructions 1 and the robot observation images O2 as the input of the pre-trained VLM to perform primitive action parsing related to the manipulation task. Then, the parsed primitive actions and the original user instructions / are combined as waypoint indicators P; for the scene prediction module. The above process can be expressed as:
Pt (1. VLM (Prompt()) O,))
For the scene prediction module & &ap. we use the waypoint waypoints P, related to the robot manipulation task prompt and the historical observation image O,-hx of the robot input to predict the waypoints feature FM; of the robot manipulation, that is, we have:
FM; 2 ap Erext (P)), Eimage (Ot-ha))
(4)
5

---

Table 1: Success rate and speed comparison of different methods in four levels of tasks (%).
Model
Level
Level
Level
Tevel
Mean
Time(ms)
Gato 41
34.74
30.53
23.16
20.00
27.11
139
BC-Z 23
41.05
32.63
23.16
25.26
30.53
12
Octo 36
69.79
48.48
34.69
33,58
46.64
18
SUSIE 6
78.89
48.48
32.50
47.26
434
RT-1 7
67.38
49.47
38,95
34.74
47.64
21
GR-1 55
77.08
55.56
37.31
34.33
51.07
35
Surfer 42
74.74
61.05
45.26
37.89
54.74
24
PIVOT-R
88.06 [13.32
77.55 (16.50 73.33 (28.07 1) 57.82 (19.93
74.19 (19.45
27
The model details of the scene prediction module are shown in Figure (b).
(b). It is stacked by LS 12 transformer layers. Each transformer layer consists self-attention layer, a cross- attention layer, and feed-forward layer.
Action Prediction Module. For the action prediction module t ap- we use the robot manipulation waypoint state features Far; predicted by the scene prediction module as prompts, and take the robot's s historical observation images Ot-hst and robot status St-h:t as input to predict the action A that the robot should take time Therefore, the prediction process of action A₁ can be expressed as:
A/ = Pap ( MM; Emnage (Ot-iet) (
The details of the action prediction module are shown in Figure 2 (c), which has the same structure as the scene prediction module consisting of a stack of LA transformer layers.
3.4 Asynchronous Hierarchical Executor
In addition, in order improve the execution efficiency of PIVOT-R, adopt asynchronous hier- archical execution mode to execute primitive action parsing, scene prediction, and action prediction respectively. Specifically,
(a), we different execution frequencies for these three parts according to needs. For primitive action parsing, it requires lot of computation using VLM so we use lower execution frequency V1. For the lightweight action prediction module, we adopt higher execution frequency
shown in Figure 2
These three execution frequencies conform to the following relationship: V1 0/ Uz. where U/ the execution frequency of the scene prediction module. Specifically, at timestep 1, if module has not finished processing the new request, it will return the previous result first.
3.5 Loss
The training loss of PIVOT-R mainly includes scene prediction loss Lyeene and action prediction loss Cast. Specifically, for scene prediction loss Lzceue. following I-JEPA [T], we calculate the average Le distance of features between the predicted waypoint state M' and the ground truth M, where M is encoded using a pre-trained CLIP image encoder Fimage. For action prediction loss L following RT-1 77, we use Cross Entropy Loss to calculate the loss between the predicted action A' and the ground truth action A. The total loss of PIVOT-R is L Lyceme Lact-
Experiments
We conduct experiments the challenging SeaWave address three key inquiries: 1) How effective is PIVOT-R in executing various complex language instructions? 2) How robust and generalizable is PIVOT-R to manipulation
42
benchmark. Our experiments aim to
out-of-distie diittritittiitt scenarios? 3) Which modules of PIVOT-R play an important role? 4) if there are cases where the robot retry and successfully perform action after an initial incorrect attempt?
4.1 Experiment Settings
AHE. We use multithreading to process each module separately. Each thread runs its own frequency, extracts the latest data from the corresponding buffer, and places the output results in the buffer. For example, the VLM gets data from the camera buffer and saves the output in the buffer after each update. Then, the scene and action prediction module updates at different

---

Instruction: "adjust the position of the green bottle that it is nearer to the blue one."
close to
grasp
close to
put down
Figure 3: Examples show the execution process of PIVOT-R. The text below the image describes the primitive actions to be performed next. Blue arrows indicate the direction of actions.
frequencies and reads the latest data from the cache of the previous module. Different modules will not be blocked by other parts. For the execution frequency of different modules in AHE, we 022 10, U3 30.
Primitive actions. We divide primitive actions according to the object-centered principle. including "close to", "grasp". "move up"" "move down", "release", "rotate + (direction)", "push (direction)", "pull + (direction)", "opee"""""""""""",,,,,,,""""""" and "close", a total of 10 types. For example, for the primitive action "close to", its description is defined as "move close the target object". More detailed information is presented in the Appendix A.1
Action prediction. For action prediction, PIVOT-R predicts delta XYZ positions and delta Euler angles for movement and binary state of the gripper. Similar to RT-1 we discretize each action dimension into 256 bins, ensuring that the value distribution of each bin is uniform.
More experiment settings are shown in Appendix
A.2
4.2 Benchmark and Baselines
Simulation Benchmark. We choose SeaWave [42], open-source benchmark to learn multi-level instruction tasks, as our experimental platform, and use the corresponding data as demonstration data for imitation learning. Its greatest advantage is that it provides progressive tasks, facilitating comprehensive comparison and analysis of the model's capabilities. It supports 8 skills, including daily operations such grasping and placing objects, opening and closing doors, and more than 3,000 different instructions. The Sea Wave dataset contains a total 13K data covering four different levels of language instructions. We train on this dataset and test on specially divided test set. A more detailed introduction in Appendix B
In addition, added primitive action and waypoint annotations to the dataset. For ground truth waypoints M, we define the action frame that meets one of the following two conditions in data collection the waypoint state of robot manipulation: 1) primitive action completion frame; 2) the speed of the robotic arm approaches zero or the state of the gripper changes. We annotate the frames that satisfy the conditions waypoints M along with the corresponding primitive actions.
Real-world Evaluation. We conducted real-world experiments, where we set up three tasks: (i) "Pick up": pick up the correct object from the table. (ii) "Put on": Pick up the object and place on the correct color block. (iii) "Push to": Push the object to the correct color block. We collected 400, 200, and 200 sets of demonstrations respectively. We tested each task 24 times to calculate the average success rate.
Baseline. In the experiment, we selected BC-Z (23, Gato [1]. RT-1 17). Octo [36, GR-1 (55), and Surfer 42 as the baseline models for the Sea Wave benchmark. BC-Z 23 includes pre-trained multilingual sentence encoder, a FiLM encoder, and two-layer MLP to decode robot actions. Gato 41. RT1 7 and Octo (36) all embed text and images, and then use Transformer to output actions end-to-end. They are currently relatively simple and effective methods. SUSIE (6] predicts sub-goals through video predictors and GR-1
(55) enhances model effectiveness with video generation pre- training. By predicting the future and explicit modeling of the action and scene prediction, Surfer [42 achieved the SoTA performance on Sea Wave with the same amount of data. We train these models on the full SeaWave dataset to allow for fair comparison.
7

---

Figure 4: We show demonstrations of real world evaluation. The first is "pick up the coke", the second row is "put the red bottle on the yellow block", and the third row is "push the object on the desk the pink block".
4.3 Results on Robotic Manipulation
Results on Weae We perform experiments on four levels of tasks in SeaWave, and the average success rate is in the last column. The results are shown in Table PIVOT-R substantially achieved a significant improvement on all tasks. Specifically, PIVOT-R achieved an average success rate of 79. 19.45% higher than the best baseline. Both the manipulation ability and the ability to understand instructions have been greatly improved. This confirms the effectiveness the primitive- driven approach.
We also show qualitative results, which are shown Figure demonstrates the example of bringing milk close to yogurt. The task process can be divided into five actions. Through the instruction of primitive actions and the prediction of waypoints, the model successfully completes the task.
is also important for robots to be able to operate in real-time. Since the hardware device and action space the same for all models, focus the inference speed of the models. As shown in the last column of Table we compare the inference time of the models. We calculated the average time for the model execute one step. It can be seen that BC-Z based on ResNet [9] the fastest. In addition, the inference speed of PIVOT-R and most other models are of the same order of magnitude, with only few milliseconds difference. Though simple, AHE's integration with WAWM is highly effective. PIVOT-R's VLM- primitive-driven WAWM for scene and action prediction, combined with AHE for asynchronous execution, improves efficiency by 28 times.
Results on Real World. The quantitative results shown in Table PIVOT-R improves the average success rate by 6%. The qualitative results are shown in Figure 4 Surfer and RT-1 usually fail due to position errors, while PIVOT- has higher accuracy. In the "push to" task, the performance of all models is suboptimal. This is because the downward force applied during the pushing process increases the resistance, making difficult for the models effectively predict and adapt to this change.
Table 2:
Performance of different methods on three real robot manipulation tasks (%). "Pick up": pick up the correct object from the table. "Put on": Pick up the object and place it on the correct color block. "Push to": Push the object to the correct color block.
Model
Pick up
Put
Push
Mean
Octo RT-1
34.72
27.78
4.17
22.22
40.28
22.22
19.44
27.31
GR-1
26.39
29.17
8.33
21.30
Surfer
41.67
29.17
31.94
34.26
PIVOT-R
54.17
41.67
25.00
40.28
4.4 Generalization Ability
We also perform experiments in different unseen scenarios on level 2, 3, and 4 tasks. New scenarios include unseen backgrounds (i.e., two unseen tables), changing light intensity, and more distractions (i.e., more objects). The results are shown in Table 3 PIVOT-R still maintains success rate far superior to other models, indicating that with the help of WAWM, the model captures key information and maintains good generalization in changing scenarios.

---

Table 3: Performance comparison on seen scenarios, different backgrounds, changing lights, and
more distractors (%).
Model
Seen
Unseen backgrounds
Changing lights
Distractors
Gato 4
24,56
20.83
23.33
16.67
BC-Z 23
27.02
19.17
18.33
21.67
Octo 36
38.92
40.83
37.50
35.83
RT-1
41.05
38.33
40.83
35.00
GR-1 55
42,40
40.83
35,00
37.50
Surfer 42
48.07
46.67
45.83
40.83
PIVOT-R
69.57 (21.0
59.17 (12.5
61.67 f)
55.83(15.0 ()
Table 4: Ablations studies of PIVOT-R in four levels of manipulation tasks (%).
Model
Level
Level
Level
Level
Mean
Time(ms)
PIVOT-R
88.06
77.55
73.33
57.82
74.19
27
PIVOT w/ PAC
82.22
73.33
69.17
51.67
69.10 (-5.09)
PIVOT w/ RSC
72.92
40.83
35.83
25.00
43.65 (-30.54)
PIVOT-R w/ next frame
72.92
45,92
34.33
24.63
44.45(-29.7)
PIVOT-R w/ interval frame
78.13
51.02
40.30
24.63
48.52( (2.52(225.7)
PIVOT-R w/ final frame
89.58
66.33
49.25
40.30
61.36(-10 12.8)
PIVOT-R w/ Qwen-VL
88.54
76.53
72.26
55.78
73.28( -
PIVOT-R w/ GPT-4
87.50
78.57
74.45
59.18
74.92(+0.7)
PIVOT-R w/o AHE
90.63
80.61
76.64
60.54
77.11(+2.9)
756
PIVOT-R w/ video decoder
85.42
70.83
62.77
46.26
66.32( -7.32)-7.8)
PIVOT-R w/ large action module
87.50
75.51
68.61
53.28
71.23(-24)
4.5 Ablation Study
In this section, we explore what is important in the design of the model. Specifically, We discuss the impact of waypoint selection, VLM, AHE, scene prediction supervision. and action prediction module design PIVOT-R's performance. We designed a series of ablation experiments. We made some assumptions and experiments: (i) Waypoint selection. We conduct experiments by selecting the primitive action completion (PAC) frame, robot state changes (RSC) frame, next frame, five frames apart, and the final frame of the trajectory as waypoints respectively. (ii) VLM's image and language understanding capabilities. We chose Qwen Qwen-VLB3 of the same size compare with the most powerful GPT-4[37] currently. (iii) Design of asynchronous architecture. We canceled the asynchronous architecture so that each module will be updated at every step. (iv) Design of scene prediction module. We refer to the design of MAE 18 and use predicted pixel-level images instead (v) Design of action prediction module. We use larger Transformer. Table
of feature prediction. shows the results of each ablation and the delta performance compared.
Waypoint selection. As shown in the results, the performance of PIVOT-R with only primitive action completion frames dropped by 5. 5.1%, the performance of PIVOT-R with robot state change frames dropped by 30.54%. Therefore, action completion frames the main contributing factor. Selecting the next frame or selecting the interval frame both caused significant drop in performance, indicating that the waypoint information for these two choices was too little confusing. The performance of selecting the final frame as a waypoint also dropped lot, indicating that it is an effective method to guide the model according primitive actions.
VLM selection. Different VLM models, whether they are models of the same level or the current largest and best models, do not bring significant performance changes. This shows that method does not strongly depend on VLM. PIVOT-R gives full play to the understanding and reasoning capabilities of VLM and makes up for the shortcomings of VLM in the dynamic world in the scene prediction module.
Other designs. Changing the model to synchronous serial structure has some improvements (2.9 1). it's 30 times slower. Considering the requirements real-time operation, we use an asynchronous parallel architecture, taking into account both success rate and speed. We also discussed the design of the scene prediction module. Compared with the original prediction at the high-level feature level, pixel-level prediction caused decrease in performance. We suspect that this is because pixel-level

---

Fail
Retry and Success
Unrecoverable
Figure 5: Left: Example retry and successful execution after a manipulation error occurred. Right: Example of retry still failing. After object is knocked down and rolled certain distance, it is difficult to successfully grab it again.
prediction focuses much on detailed information, causing key information to be ignored. Finally, we also test whether the design of the current action prediction module is reasonable and whether the action prediction module needs larger model to make better predictions. Experiments have shown that the action prediction module only needs small model to complete the task well, but larger model may cause over-fitting
4.6 Failure and Retry
This section discusses cases where the robot fails and whether it can be retried and successfully executed. As shown in Figure retries may be successful some cases. When the position of the gripper deviated and the object failed to be grasped, the second attempt to grasp was successful. However, in the case of Figure (right), the object knocked down and rolls certain distance, will be difficult to successfully grasp it again.
5 Discussions
Conclusion. In this paper, we propose PIVOT-R, a primitive-driven waypoint-aware world model. PIVOT-R focuses the execution of primitive actions. Predicting key waypoints in the future greatly improves performance. has achieved state-of-the-ard results on the Sea Save benchmark, and experiments have proven that it has good robustness. We also use asynchronous hierarchical exccutors to ensure fast enough exccution of the model. addition, we demonstrate that PIVOT- R has the potential to complete unseen instructions and tasks under the guidance of high-level VLM. Finally, we also demonstrate PIVOT-R's ability to improve performance through human demonstration. These results illustrate the potential of PIVOT-R
Limitations. We demonstrate the ability of PIVOT-R to complete tasks, even unseen tasks, through combination of primitive actions guided by instructions. However, action execution and instructions are sometimes inconsistent. For example, "push left" is required, the robot may execute "push front". Therefore, we also need to strengthen the consistency between high-level instructions and underlying actions, so that the robot can truly perform tasks according to our instructions, and even adjust according to requirements, just like real intelligent agent.
6
Acknowledgement
This work was supported in part by National Science Foundation of China Grant No. 0o.62476693, Guangdong Outstanding Youth Fund (Grant No. 2021B1515000061) Shenzhen Science and Tech- nology Program (Grant No. GJHZ202209 13 (3te)922221111140000 126600001). Nansha Key RD Program under Grant No. No.2022ZD014, The Major Key Project of PCL (No. PCL2024A04, No. PCL2023AS203).
10

---

References
[1] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint- embedding predictive architecture, 2023.
[2] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.013900 2023.
[3] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: versatile vision-language model for understanding. localization, text reading, and beyond, 2023.
[4] Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual repre- sentations from video. arXiv preprint arXiv:2404. 08471,, 2024.
[5] Suneel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quon Vuong, Jonathan Tompson, Yevgen Chebotar, Debidatta Dwibedi, and Dorsa Sadigh. Rt-h: Action hierarchies using language. arXiv preprint arXiv:2403.01823, 2024.
[6] Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pretrained image-editing diffusion models, 2023.
[7] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:22 22..2222.0600 12.06177 2022.
[8] Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, al. Genie: Generative interactive environments. arXiv preprint W4Xiv:2402.1331, 2024.
[9] Shizhe Chen, Ricardo Garcia, Cordelia Schmid, and Ivan Laptev. Polaret: 3d point clouds for language-guidee robotic manipulation. arXiv preprint arXiv:2309.15540 2023.
[10] Jeremy A Collins, Cody Houff, You Liang Tan, and Charles C Kemp. Forcesight: Text- Text-guided mobile manipulation with visual-force goals. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 10874-10880 IEEE, 2024.
[11] Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofrr Nachum, Joshua B. Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation, 2023.
[12] Xiaofeng Gao, Qiaozi Gao, Ran Gong, Kaixiang Lin, Govind Thattai, and Gaurav Sukhatme. Dialfred: Dialogue-enabled agents for embodied instruction following. IEEE Robotics and Automation Letters, (1449-0009910055 2022.
[13) Theophile Gervet, Zhou Xian, Nikolaos Gkanatsios, and Katerina Fragkiadaki. Act3d: 3d feature field transformers for multi-task robotic manipulation. 7th Annual Conference on Robot Learning, 2023.
[14] Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, and Dieter Fox. Rvt: Robotic view transformer for 3d object manipulation, 2023.
[15] Pierre-Louis Guhur, Shizhe Chen, Ricardo Garcia Pinel, Makarand Tapaswi, Ivan Laptev, and Cordelia Schmid. Instruction-driven history-aware policies for robotic manipulations. In Conference Robot Learning, pages 175-187. PMLR, 2023.
[16] Huy Ha, Pete Florence, and Shuran Song. Scaling up and distilling down: Language-guides robot skill acquisition. In Conference on Robot Learning, pages 3766-3777. PMLR, 2023.

---

[17] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint 1XXiv:9122000000 2019.
[18] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
[20] Yingdong Hu, Fanqi Lin, Tong Zhang, Li Yi, and Yang Gao. Look before you leap: Unveiling the power of gpt-4v in robotic 2023.
planning. arXiv preprini arXiv:2311. 17842,
[21] Haoxu Huang, Fanqi Lin, Yingdong Hu, Shengjie Wang, and Yang Gao. Copa: General robotic manipulation through spatial constraints of parts with foundation models. arXiv preprint arXiv:2403.0822 088448 2024.
[22] Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):301930026, 2020.
[23] Eric Jang, Alex Irpan. Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 991-1002. PMLR, 08-11 Nov 2022.
[24] Jing Yu Koh, Honglak Lee, Yinfei Yang. Jason Baldridge, and Peter Anderson. Pathdreamer: world model for indoor navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14738-14748, 2021.
[25] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[26] Xiaoqi Li, Mingxu Zhang, Yiran Geng, Haoran Geng, Yuxing Long, Yan Shen, Renrui Zhang, Jiaming Liu, and Hao Dong. Manipllm: Embodied multimodal large language model for object-centric robotic manipulation. arXiv preprint arXiv:2312.16217, 2023.
[27] Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al. Vision-language foundation models as effective robot imitators. arXiv preprint arXiv:231 1.01378, 2023.
[28] Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sich, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, and Ted Xiao. Evaluating real-world robot manipulation policies in simulation, 2024.
[29] Yunxin Li, Longyue Wang, Baotian Hu, Xinyu Chen, Wanqi Zhong, Chenyang Lyu, and Min Zhang. A comprehensive evaluation of gpt-4v on knowledge-intensive visual question answering. arXiv preprint arXiv:2311.07536 2023.
[30] Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Text2motion: From natural language instructions to feasible plans. Autonomous Robots, 4765):1345-1355, 2023.
[31] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
[32] Yecheng Jason Ma, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. Liv: Language-image representations and rewards for robotic control. In International Conference on Machine Learning, pages
PMLR, 2023.
[33] Oier Mees, Lukas Hermann, Erick Rosete-Beat and Wolfram Burgard. Calvin: A benchmark for language-conditionne policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 7(3):7327-7334, 2022.
12

---

[34] Russell Mendonca, Shikhar Bahl, and Deepak Pathak. Structured world models from human videos. arXiv preprint arXiv:2308. 10901, 2023.
[35] Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint arXiv:2402.07872 2024.
[36] Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. https //octo-models github.th...... 10 2023.
[37] OpenAl. Gpt-4 technical report, 2023.
[38] Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan- Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilck Hakkani-Tur Teach: Task-driven embodied agents that chat, 2021.
[39] Rudra PK Poudel, Harit Pandya, Chao Zhang, and Roberto Cipolla. Langwm: Language grounded world model. arXiv preprint arXiv:2311.17599 2023.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[41] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. generalist agent. arXiv preprint arXiv:2205.066 0615,, 2022.
[42] Pengzhen Ren, Kaiwen Zhang, Hetao Zheng, Zixuan Li, Yuhang Wen, Fengda Zhu, Mas Ma, and Xiaodan Liang. Surfer: Progressive reasoning with world models for robotic manipulation. 2023.
[43] Saumya Saxena, Mohit Sharma, and Oliver Kroemer. Mrest: Multi-resolution sensing for real-time control with vision-language models. arXiv preprint arXiv:2401.14502 2024.
[44] Rutav Shah, Roberto Martín-Martin, and Yuke Zhu. Mutex: Learning unified policies from multimodal task specifications. arXiv preprint arXiv:2309. 14320, 2023.
[45] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Proceedings of the 5th Conference on Robot Learning (CoRL), 2021.
[46] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Proceedings of the 6th Conference Robot Learning (CoRL), 2022.
[47] Elias Stengel-Eskin, Andrew Hundt, Zhuohong He, Aditya Murali, Nakul Gopalan, Matthew Gombolay, and Gregory Hager. Guiding multi-step rearrangement tasks with natural language instructions. In Conference on Robot Learning, pages 1501. PMLR, 2022.
[48] Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrishnan, Kuang- Huei Lee, Quan Vuong, Paul Wohlhart, Sean Kirmani, Brianna Zitkovich, Fei Xia, et al. -pen object manipulation using pre-trained vision- language models. arXiv preprint arXiv:2303.00900 2023.
[49] Quan Vuong, Sergey Levine. Homer Rich Walke, Karl Pertsch, Anikait Singh. Ria Doshi, Charles Xu, Jianlan Luo, Liam Tan, Dhruv Shah, al. Open x-embodiment: Robotic learning datasets and rt-x models. In Towards Generalist Robotx: Learning Paradigms for Scalable Skill Acquisition @ CoRI-2023, 2023.
[50] Yanming Wan, Jiayuan Mao, and Josh Tenenbaum. Handmethat: Human-robot communication physical and social environments. Advances in Neural Information Processing Systems, volume 35, pages 12014-1200 12026. Curran Associates, Inc., 2022.
13

---

[51] Hanqing Wang, Wei Liang, Luc Van Gool, and Wenguan Wang. Dreamwalker: Mental planning for continuous vision-language navigation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10873-10883, 2023.
[52] Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, and Jiwen Lu. Drivedreamer: Towards weal-world-driven world models for autonomous driving. arXiv preprint arXiv:2309.09777 2023.
[53] Xiaofeng Wang, Zheng Zhu, Guan Huang, Boyuan Wang. Xinze Chen, and Jiwen Lu. World- dreamer: Towards general world models for video generation via predicting masked tokens. arXiv preprini arXiv:2401.0998. 2024.
[54] Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. arXiv preprint 0rXiv:2311.17913 2023.
[55] Hongtao WU, Ya Jing, Chilam Cheang. Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. arXiv preprint 13X11:23/2.13139, 2023.
[56] Philipp Wu, Alejandro Escontrela. Danijar Hafner, Picter Abbeel, and Ken Goldberg. Day- dreamer: World models for physical robot learning. In Conference Robot Learning, pages 2226-2244 PMLR, 2023.
[57] Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, and Johnny Lec. Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robot Learning (CoRL), 2020.
[58] Andy Zeng, Shuran Song, Stefan Welker, Johnny Lee, Alberto Rodriguez, and Thomas Funkhouser. Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. In 2018 IEEE/RS International Conference on Intelligent Robots and Systems (IROS), pages 4238-4244 IEEE, 2018
[59] Tong Zhang, Yingdong Hu, Hanchen Cui, Hang Zhao, and Yang Gao. A universal semantic- geometric representation for robotic manipulation. arXiv preprint arXiv:2306. 10474, 2023.
[60] Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du Yining Hong, and Chuang Gan. 3d-via: A 3d vision-language-action generative world model. arXiv preprini arXiv:2403.09631 2024.
[61] Kaizhi Zheng, Xiaotong Chen, Odest Chadwicke Jenkins, and Xin Wang. VImbench: A com- positional benchmark for rision-and-lan and-language manipulation. Advances in Neural Information Processing Systems, 35:665-678. 2022.
[62] Wenzhao Zheng, Weiliang Chen, Yuanhui Huang, Borui Zhang, Yueqi Duan, and Jiwen Lu. Oceworld: Learning a 3d occupancy world model for autonomous driving. arXiv preprint arXiv:2311.16038, 2023.
[63] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing laiiuagl understanding with advanced large language models. arXiv preprins arXiv:2304. 10592, 2023.
[64] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-rr models transfer web knowledge robotic control. In 7th Annual Conference on Robot Learning. 2023.
14

---

SUMMARY OF THE APPENDIX
This appendix contains additional details for this paper. The appendix is organized as follows:
provides Experiment Details.
shows more Sea Wave Benchmark Details.
shows some Emergent Capabilities of PIVOT-R.
shows More Experiments.
shows More Results.
lists Prompt Details used in experiments.
A Experiment Details
A.1 Primitive Actions
Table The defined primitive actions and their textual descriptions.
Primitive Action
Description
Close to
Move close to the target object
Grasp
Hold or pick up the target object
Move Up
Lift the target object upwards
Move Down
Lower the target object downwards
Release
Let go of or put down the target object
Rotate + (direction)
Turn the target object
Push + (direction)
Push the target object
Pull (direction)
Pull the target object
Open
Open object, such as a door or container
Close
Close an object, such as door or container
A.2 Training Details
All experiments involved in this paper are conducted on a single GPU server with NVIDIA RTX- 4090 GPUs. PIVOT-R selects LLAVA the high-level VLM and selects Transformers of 12 layers and layers as scene prediction module and action prediction module respectively. We froze VLM and encoder, and PIVOT-R has trainable parameters of 30 M in total. The hyperparameter settings for PIVOT-R are shown Table
Table 6: The hyperparameter setting of Imitation Learning.
Hyperparameters
Value
LS
12
LA
Image encoder
VT-B-VVT-B/32
Text encoder
CLIP- VIT-B322
Transformers heads
Embedded dims
512
Learning rate
3e-5
dropout
0.1
B SeaWav Benchmark Details
In order to meet the needs of common robot operations, SeaWave designed skills, the detailed definitions are shown in Table And SeaWave proposes progressive tasks. Natural language is one of the most direct and effective ways of human-computer interaction. However, due the complexity and variability of external visual scenes and human natural language instructions, understanding
15

---

Figure 6: Scenes in Sea Wave. The left column represents the first pesspperspeetvve image, and the right column represents the third-peree surdd-perspective image. The scenes in the same row are the same.
and executing these instructions has become one of the key challenges in embodied AI research. To systematically analyze and study these challenges, SeaWave classified tasks into four levels according to the complexity of instructions and the ease of operation. Task definition and examples are described in Table The specific content is as follows:
Level 1:
The scene contains only object, and the robot receives explicit machine language commands consisting of verbs nouns. is used to evaluate the basic manipulation capabilities of the model.
Level 2: This task scenario contains multiple objects and the natural language instructions explicitly include the name of the target object. It used evaluate the model's ability to understand conventional natural language instructions.
Level 3: This task scene contains multiple objects, but the natural language instructions do not contain the name of the target object, but only provide expressions related to the functionality of the target object. It is used evaluate the model's ability infer the intent of human instructions.
Level 4: This task
contains multiple objects. The natural language instructions do not include the name of the target object but only provide expressions related to the function, appearance, or location of the target object. This instruction requires the model to have strong visual and language information processing capabilities at the same time. It is used evaluate the model's visual perception and decision-makingg capabilities.
16

---

Table 7: The list of skills on SeaWave and their description and success condition.
Skill
Description
Success Condition
Pick Target
Grasp target object and lift it place the target object on the table Grasp A and move it closer to B Open the door Close the door
The target object 10em away from table
Place Target
The target object stands upright on table
Move A Near B
is moved and cnds up being less than 10cm away from B
Open Door
The door is opened more than 80 degrees
Close Door
The door closed to than 10 degrees
Push Target Front
Push the target object forward Push the target object to the left or right Knock the target object over
The target object pushed forward 10cm The turget object moves 10cm to the left or right The target object falls on the table
Push Target Aside
Knock Target Over
0.30
Di
La distance between the festures of
current framnt.ffaame frame and milestone status
0.25
0.20
0.15
0.10
0.05
Miestone1
Milestone2
Milestone3
0.00
20
Step
Figure 7: Feature analysis. The blue points D1 represent the distance between Fo, and FM, in the spatial dimension, and the red points D2 represent the distance between FM; and M, in the spatial dimension. We fit curves to these points and draw confidence intervals for better observation.
We also show some scenes in Save in Figure 6 SeaWave is a highly simulated scene built based on UES and its scenes are realistic.
Table 8: The setting of progressive reasoning tasks.
Singlc/Multiple
Level
Objects
Capubility Assessment
Example
Single
Basic manipulation
pick the milk
Multiple
Natural language understanding
can you please take the gluestick off the table could you please go grab refreshing beverage for me retrieve object located behind the one right
Multiple
Intention inference
Multiple
Visual perception and decision decision-making
C Extra Studies
In this section, we analyze why PIVOT-R succeeds, exploring its generalization to new tasks and potential for further improvement by incorporating other datasets.
C.1 Feature Analysis
In this study, we performed an in in-depth comparative analysis. We define that Fo.. FM:. and FM, represent the features of O,, Mí, and M, respectively and try to explore the spatial distance relationships between the feature F'rt (red points) predicted by PIVOT-R and the real-time observed feature Fo. (blue points) relative to the feature Fas. As shown in Fig the Ly distance between Fo, and FM, gradually decreases as the task progresses, phenomenon that is critical the functionality of the action execution module. The main task of the action execution module is to adjust the action so that O, moves closer to M,, thereby reaching the target state. Fas (red dots) provides significant guidance for action prediction. FM; not only predict the possible locations of FM, in space but also show smaller variances in long-term series analysis, indicating that their predictions are more stable, thus greatly improving the accuracy and reliability of model manipulation.
17

---

Hcllo, I want use the coffee machine, but something is blocking it.
Let me assist you with that. The bottle in front of the coffee machine need to be removed.
Instruction is picking the bottle.
instr: pick the bottle action: close to
instr: pick the bottle action: grasp
instr: pick the bottle action: move
Figure 8: An example shows that when an out-of-distribution instruction encountered, the VLM's reasoning ability is used to parse it into learned task instruction, allowing the model to successfully complete the task.
instruction
Can you move the middle object to the left?
action
pick the middle object
push left
put down
close
clamp
move up
push left
move down
unclamp
Figure 9: An example shows that for an unseen task, based on the skills and actions that have been leamndd, PIVOT-R can break down the task and combine the actions complete the task.
C.2
Emergent Capabilities
C.2.1 Generalization to Out-of-distributiio Instructions
In this section, explore whether PIVOT-R can use the reasoning capabilities of VLM to understand out-of-distributton instructions. Although the model has only received instructions from the SeaWave training set, we can use VLM to parse the instructions into learned instructions, so that PIVOT-R can understand and execute out-of-distribution instructions. To this end, we designed prompt to let VLM analyze and process the new instructions. The details of the prompt are shown in the Section
We show an example. as shown in Figure g We qualitatively observe that for the instruction "Hello, I want use the coffee machine, but something blocking it.". VLM infers that the task that needs to be performed is "remove the bottle in front of the coffee machine.", and replaces the original instruction to the learned form "pick the bottle". At this point, PIVOT-R can complete the task based on the skills it has learned.
C.2.2 Generalization to New Tasks
Zero-shot unseen tasks generalization is very difficult. Nevertheless, we hope to prove that PIVOT-R can complete new tasks through the combination of existing primitive actions, because the primitive actions are shared between different tasks. This requires some appropriate adjustments. To this end, we provide new prompt let VL.M assist in completing this work. The details of the prompt are shown in the Section P3
18

---

Table 9: Experimental results for training with additional human data.
Model
Secn
Unseen backgrounds
Changing lights
Distractors
Origin
69.57
59.17
61.67
55.83
Co- Trainain 65.75
60.83
56.67
52.5
Pre-Train
67.78
63.
58.83
58.83
As shown in Figure 9 with the help of VLM, the new task is decomposed into existing primitive actions. Specifically, for the instruction "Can you move the middle object to the left?", VLM first decomposed it from the instruction level into "pick", "push left". and "put down", and then further decomposed it into the learned primitive actions. In the end, PIVOT-R completed the task according to the guidance of VLM.
Zero-shot unseen instruction and task generalization are very difficult. Nevertheless, we hope to prove that PIVOT-R can complete new tasks through VLM guidance and the combination of learned actions. Although the tasks are different, their primitive actions are shared. For example, for the unseen task "move the middle object to the left", VLM first decomposed it into the learned primitive actions "close to", "clamp". "move up". "push left". "put down" and "unclamp". Finally, PIVOT-R completed the task according to the guidance of VLM. More details are shown in Appendix
C.2.3 Train with Human Data
We also explored PIVOT-R's ability utilize other data. Embodied AI has been limited by lack of robot data. We see if we can use other data to enhance the model. Although some data do not contain robot actions, they are still helpful for training our scene prediction module. To this, We use the Ego4D dataset, which is large-scale first-person perspective video dataset. It contains more than 3,500 hours of data, and each video clip contains detailed annotation information to describe human behavior. We train based Anticipation Challenge", which aims to predict the next human-object interaction happening after given timestamp. Each piece of data contains 0.25s-1s Is video and the corresponding operation objects and operation action, which exactly meets the input and output label requirements of the scene prediction module.
the benchmark data of the "Short Term Object Interaction
Specifically, for Ego4D data, the scene prediction module accepts the input of the initial frame and the current action instruction, outputs the features of the predicted frame, and calculates the loss with the features of the annotated end frame. We use two training methods, one to mix Ego4D and SeaWave data for co-training, and the other is to use Ego4I) for pre-training first, and then use Sea Wave data for fine-tuning.
As shown in Table D we compared co-training and pre-training results. It can be seen that co-training does not bring better results. We guess it is because the data distribution is very different, making it difficult to train the model. Although the success rate of Pre-training has dropped slightly in seen scenarios, has improved significantly unseen backgrounds and more distractors scenarios. increasing by 4.16% and 3.00% respectively, indicating that PIVOT-R can make use of other data to improve the generalization ability.
D More Experiments
benchmark, scalable, repeatable and reliable proxy for real-world evaluation. We use this to verify the scalability PIVOT-R in the real world. As shown in Table I0 PIVOT-R outperformed the best bascline by nearly 10%.
We evaluated PIVOT-R on the latest SIMPLER 28
19

---

Table 10: Performance comparison of different methods SIMPLER benchmark (%). SIMPLER a simulation benchmark which evaluation can be scalable, reprodueible, and reliable proxy for real-world evaluation. It selects four tasks from Bridgedata.
Put Spoon on Towel
Put Carrot on Plate
Stack Green Blockon
Put Eggplant in Yellow Basket
Model
Mean
Yellow Block
RT-1-X
0.000
0.042
0,000
0.000
0.011
Octo-Base
0.125
0.083
0.000
0.431
0.160
Octo-Small
0.472
0.097
0.042
0.569
0.295
PIVOT-R
0.417
0.278
0.000
0.875
0.393
E More Results
More examples are shown in Figure 10
covering tasks various levels.
F
Prompt Details
F1
Prompt Details for Primitive Action Parsing
To ensure that the Vision Language Model (VLM) produces text that adheres to our criteria, we have meticulously crafted multi-stage dialogue process, complemented by comprehensive prompts. The procedure unfolds as follows: initially, we prompt the VLM to depict the scenario; subsequently. the VLM specifies the actions that need to be undertaken; and ultimately, the VLM determines the action to be executed at the current juncture.
We have total of three rounds of dialogue. The following are the prompts for each round of dialogue.
1. Describe the scene.
Given a task, which is for a mobile Franka panda robotic arm to learn a manipulation skill in the simulator.
Your task is to help me break down the process robot performing the task into several actions to help the robot better understand and execute.
Capabilities: The task can only be completed with a robotic arm, which can move, rotate and clamp.
You should output the response using the same format the following:
"scene": "You should description the scene"
Here is one example:
# 1
Input: Close the red jar.
Output: On the table, there is red jar, blue jar, and bottle cap
Can you do it for the following input:
Task: [task)
2. Imagine the actions that need to be done.
20

---

Level 1: "pick the coffee"
Level 2: " "Can you get me a bottle of yogurt?"
Level 3: "I need a container. Can you move it to the left?"
Level 4: "Can you move the green object closer to the one its left?"
Figure 10: Rollouts on multi-level tasks of the SeaWave benchmark.
Given task, which is for mobile Franka panda robotic arm to learn manipulation skill in the simulator. Your task is to help me break down the process of the robot performing the task into several actions to help the robot better understand and execute. Capabilities: The task can only be completed with a robotic arm, which can move, rotate and clamp.
You should output response using the same format as the following:
"act :: [
21

---

"action": "The action name", "target": "The target object"
},
#actions which are needed to complete the task a m
The actions you can choose include the following:
n n
move to move the gripper closer to object,
clamp use gripper to clamp the object, unclamp open gripper to unclamp the object,
screw rotate the gripper for opening closing lid, lift lift the object,
push push the object + (direction),
""" pull pull the object (direction),
Here is one example:
Input:
Task: close the red jar.
Scene: On the table, there is red jar, a blue jar, and a bottle cap.
Output:
"actions":
"action": "move to",
"target": "the bottle cap"
"action": "clamp" "target": "the bottle cap"
"action": "move to", "target": "the red jar"
"action": "rotate",
"target": "the bottle cap"
]
Can you do it for the following task:
Task: [task]
22

---

Scene: (scene)
3. Decide what action to take now.
Given task, which is for mobile Franka panda robotic arm to learn manipulation skill in the simulator. Your task is to help me break down the process of the robot performing the task into several actions to help the robot better understand and execute. Capabilities: The task can only be completed with a robotic arm, which can move, rotate and clamp.
You should output action that should be done the current moment, and only can output one. You should output response using the same format as the following json file, and don't need to output escape characters
"do_ action" {
"action": "The action name", "target": "The target object"
}
The actions you can choose include the following:
move to : move the gripper closer to an object,
clamp
gripper to clamp the object, open gripper to unclamp the object,
unclamp :
screw rotate the grüpper for opening or closing lid, lift lift the object,
push push the object (direction), pull pull the object (direction),
Here is one example:
Input:
Task: close the red jar.
Scene: the table, there is jar, a blue jar, and a bottle cap.
Actions:
"action": "move to", "target": "the bottle cap"
"action": "clamp" "target": "the bottle cap"
"action": "move to",
"target": "the red jar"
},
"action": "rotate",
23

---

"target": "the bottle cap"
Output:
{
"do_ action":
"action": "move to", "target"; "the red jar"
Can you do for the following task:
Task: (task}
Scene: {scene)
Actions: (actions)
Prompt Details for New Instructions
In order to be able to process out-of-distribution instructions, let VLM process the commands first and parse them into learned tasks. this, we set a prompt as shown below.
Given task, which is for mobile Franka panda robotic arm to learn manipulation skill in the simulator. Your task is to help me break down the process of the robot performing the task into several actions to help the robot better understand and execute. Capabilities: The task can only be completed with robotic arm, which can move, rotate and clamp.
You need to give an instruction base the skills you have learned according to the given tasks. You should output the response using the same format as the following json file:
" "You should description the instruction here",
mmn
The skills you have learned:
Pick Target: Grasp the target object and lift it, Place Target place the target object on the table, Hove Near : Grasp and move it closer to B, Open Door Open the door,
Close Door Close the door, Push Target push the object + (dérection),
Knock Target Over Knock the target object over,
Here is one example:
Input:
Task: The bottle is on the edge of the table, it's too dangerous.
24

---

Output:
"
"Push the bottle front",
you you do it for the following task:
Task: (task}
F.3 Prompt Details for New Tasks
In order to be able to solve new tasks, we let VLM process the commands first and parse them into learned tasks and actions. To do this, we set prompt as shown below.
Given a task, which for mobile Franka panda robotic arm to learn manipulation skill in the simulator. Your task is to help me break down the process of the robot performing the task into several actions to help the robot better understand and execute. Capabilities: The task can only be completed with robotic arm, which can move, rotate and clamp.
You need to complete given task, based on the skills and actions you have learned. You should output the response using the same format the following json file:
"ins innstruction: "You should description the instruction here", "actions": [
"The action name". "target": "The target object"
"action":
# actions which are needed to complete the task
]
"do_ action" {
"action": "The action name", "target": "The target object"
mmn
The skills you have learned:
Pick Target: Grasp the target object and lift it, Place Target place the target object on the table, Hove Near : Grasp and move it closer to Open Door Open the door,
Close Door Close the door, Push Target push the object + (direction),
Knock Target Over Knock the target object over,
The actions you can choose include the following:
move to move the gripper closer to an object,
25

---

clamp use gripper to clamp the object, uncl amp open gripper to unclamp the object, screw rotate the gripper for opening or closing lid, Lift lift the object, move move the object (direction),
Here is one example:
Input:
Task: Hello, I' on your right, can you bring me the object on the table.
Output:
"instruction": "Pick up the object and right", "actions": [
"action": "close to", "target": "the object"
}
"action": "clamp", "target": "the object"
},
"action": "move up", "target": "the object"
},
{ "action": "move right",
"target": "the object"
}
]
"do_ action":
"action": "close to", "target": "the object"
}
mmn
Can you do it for the following task:
Task: [task)
26

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have clarified the contributions and important assumption in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitation discussions are presented in Sec 5
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
27

---

Answer: [NA]
Justification:
The theoretical assumptions and proofs not the focus of this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The dataset construction recipe and experimental details presented.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
28

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Our work will be open source after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ "cippscc/puullc//guu /ttee//gguddeeeeedouuuuumbbbbai ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provided the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
is computationally expensive to report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information on the computation resources is provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https::/neurr// http::/ss.pppp.......tacaasaaaaa
Answer: [Yes]
Justification: The research of this paper follows the NeurIPS Code of Ethics in both data and model perspectives.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly credited the asset owner and follow terms of use.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
31

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "38uKAHlghU", "reviewer": "Reviewer_XYkg", "summary": "The paper proposes PIVOT-R, a waypoint-based world model for robot manipulation. Concretely, given a language instruction, PIVOT-R converts it to a textual intermediate goal using a VLM and then feeds that as input into a scene prediction model that generates the waypoint. This waypoint is then fed into the action prediction model that predicts the robot action. The proposed method is compared with several baselines on the SeaWave benchmark and shows significant improvement in performance compared to numerous baseline methods.\n\n\nEdit: The authors have addressed most of my comments.", "questions": "- Could the authors describe the ablation \"PIVOT-R w/ video decoder\" and provide an intuition on why might it hurt performance.", "limitations": "The authors should address the limitations in more detail. Specifically, discussing the additional labeling requirement of waypoints and the generalizability and efficacy of the heuristics to come up with the waypoints would help.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper seems to outperform the baselines by a margin on manipulation tasks of increasing complexity due to long range tasks\n- The paper is well written and easy to follow\n- The paper makes important technical contribution on effectively combining large VLMs with scene prediction models.\n- Modulo some additional experiments (see below), the paper makes a strong empirical case for the proposed method (PIVOT-R) compared to baseline methods.", "weaknesses": "- The method is complex, and uses heuristics for defining intermediate waypoints, which does not look scalable at the outset.\n- For baselines, it looks like there is video generation pretraining (GR-1) and Scene decoding (Surfer), however an important baseline is missing: SUSIE [A]\n- Labeling of waypoints which is done in a heuristic way (zero hand speed, gripper state change, final frame, etc). It would be good to understand which is the most critical of these to get the most performance from the model. \n- It would help to quantify the the additional labeling cost, and how scalable it is for various tasks. \n- If I understand correctly, the baseline method Surfer which is the closest to the proposed method, seems to have 2 differences – a) splitting the task into waypoints, b) feeding the predicted scene as input. Ablation of these two changes to see which one contributes how much would help in understanding the paper. \n\n[A] Black, Kevin, et al. \"Zero-shot robotic manipulation with pretrained image-editing diffusion models.\" arXiv preprint arXiv:2310.10639 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your comprehensive and encouraging review! We respond to all the issues you pointed out in detail below. We hope our response and rebuttal revision will address your concerns.\n\n**Q1. The complexity and scalability of method.**\n\n**(i) Model design & implementation.** \n\nIn fact, there is no complex and cumbersome design in PIVOT-R. In terms of design, the core module of PIVOT-R is a waypoint-aware world model, which is mainly composed of an open source VLM and a scene prediction model for primitive parsing and scene modeling. In terms of implementation, AHE sets different frequencies for different modules and adopts multi-threaded asynchronous implementation, which greatly improves execution efficiency. Therefore, the introduction of WAWM will not significantly increase the complexity of the entire model, and PIVOT-R is simple in design and implementation. We will also release our code upon acceptance of the paper to facilitate future research.\n\n**(ii) Heuristic methods for defining waypoints.** \n\nThe heuristic method is a general waypoint definition method that has been adopted in many previous robot manipulation tasks [1-4]. PIVOT-R can achieve automated annotation of waypoints in various robot manipulation benchmarks using open-source VLM and scripts. Therefore, our waypoint annotation method has good scalability.\n\n[1] HSIAO K, LOZANO-PEREZ T. Imitation Learning of Whole-Body Grasps[C/OL]//2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, Beijing. 2006.\n\n[2] AKGUN B, CAKMAK M, JIANG K, et al. Keyframe-based Learning from Demonstration[J/OL]. International Journal of Social Robotics, 2012: 343-355.\n\n[3] SHRIDHAR M, MANUELLI L, FOX D. Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation[J]. 2022.\n\n[4] JAMES S, DAVISON AndrewJ. Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulation[J]. arXiv: Robotics,arXiv: Robotics, 2021.\n\n**Q2. PIVOT-R *vs.* SUSIE.**\n\nThank you for your suggestions. We compared PIVOT-R and SUSIE on the SeaWave benchmark, and the results are shown in Table 2 of the rebuttal PDF. As shown in the results, compared with SUSIE, PIVOT-R achieved an **average performance improvement of 26.93%** (74.19% *vs.* 47.26%) on four levels of SeaWave tasks. We found that this may be caused by the poor quality of the target image generated by SUSIE severely limiting the model's action execution when given unseen instruction input. In addition, compared with SUSIE, PIVOT-R still achieves a 9.17% (88.06% vs. 78.89%) performance improvement on Level 1 tasks without complex instructions. We think this is because SUSIE's diffusion model and low-level policy are trained separately, causing prediction deviations in the images that affect the low-level policy. In contrast, PIVOT-R's WAWM and action prediction modules are trained together, thus effectively avoiding this problem. We will add these results to the revised version.\n\n**Q3. Key factors in waypoint definition.** \n\nThank you for your suggestion. We mainly consider three markers: zero hand speed, gripper state change, and primitive action completion frame to define waypoints. Among them, both zero hand speed and gripper state changes belong to robot state changes, and the robot arm speed is zero when the gripper state changes, indicating a strong correlation between the two. And they can judge directly through the robot motion status port. Therefore, as defined by Line 240-241 in the article, the above three waypoint judgment conditions can be divided into two types: primitive action completion frame and robot state change frame (including gripper and arm). To illustrate the contribution of the above two waypoint selections methods, we added a set of ablation experiments shown in Table 4 of the rebuttal PDF. As shown in the results, the performance of PIVOT-R with only primitive action completion frames dropped by 5.1%, the performance of PIVOT-R with robot state change frames dropped by 30.54%. Therefore, action completion frames are the main contributing factor.\n\n**Q4. Annotation cost and scalability.**\n\n**(1) Cost.** For dataset annotation, we use the open source LLaVA 1.5 as the VLM for waypoint (*i.e.*, primitive action completion frame) judgment, and use scripts to automatically annotate waypoints in the robot's manipulation trajectory. As shown in Table 3 in our paper, PIVOT-R is not sensitive to the choice of VLM. We used the open source LLaVA 1.5 as VLM on 8 RTX4090s to complete the annotation of 13K trajectory data in 6 hours, which is affordable for practical implementation.\n\n**(2) Scalability.** The annotation conditions of waypoints depend on the state changes of the robot and the visual changes of the manipulation trajectory in the simulator, which are common and easily accessible in different benchmarks. Therefore, this waypoint annotation method has good scalability.\n\n**Q5. Ablation experiment for Surfer.** \n\nThe biggest difference between PIVOT-R and Surfer is that PIVOT-R adopts a waypoint-aware world model strategy. The results of using the next frame in Surfer as a waypoint in PIVOT-R have been shown in row 3 of Table 3 in our paper, which resulted in a very significant performance degradation (*i.e.*, performance loss of 29.7%, 74.19% *vs.* 44.45%). Taking the predicted scene as input is not the key difference between PIVOT-R and Surfer. Surfer and PIVOT-R essentially regard the robot action as the key factor in observing the image state transition, but they use different prediction orders. Thank you for this valuable comment, we will explain this in our revision.\n\n**Q6. Explanation of ablation for PIVOT-R w/video decoder.** \n\nThe pixel-level prediction settings of \"PIVOT-R w/ video decoder\" are inconsistent at the semantic level with the high-level primitive actions such as \"close to\" and \"grasp\" that need attention. This results in degraded model performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2J7C7sKkDx", "reviewer": "Reviewer_eMP4", "summary": "This paper introduces PIVOT-R, an approach for language-guided robotic manipulation. PIVOT-R consists of a Waypoint-aware World Model (WAWM) and a lightweight action prediction module, along with an Asynchronous Hierarchical Executor (AHE) to improve efficiency. The model achieves state-of-the-art performance on the SeaWave benchmark, demonstrating improvements in performance and efficiency compared to baselines.", "questions": "**Evaluation on Other Benchmarks:** Have you considered evaluating PIVOT-R on other well-known benchmarks to better contextualize the performance improvements? I may consider changing my rating if results on a more well-known benchmark are presented.\n\n**AHE Mechanism:** Could you elaborate more on the algorithm behind the AHE module?\n\n**Novelty:** Can you make a brief statement on the novelty of your work, particularly with respect to the prior work mentioned in the weaknesses section?", "limitations": "The paper discusses the limitations of the work, including the potential inconsistency between high-level instructions and underlying actions.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**Performance:** The proposed PIVOT-R model demonstrates strong performance improvements over chosen baselines, achieving state-of-the-art results on the SeaWave benchmark.\n\n**Thorough Evaluation on Chosen Benchmark:** The authors provide comprehensive results on the SeaWave benchmark against strong baselines, and perform thorough ablations for generalization and interpretability.", "weaknesses": "**Lack of Clarity:** The abstract and introduction do not clearly articulate the specific problem being addressed. While they mention the need for language-guided robotic manipulation and the limitations of previous approaches, the exact nature of the problem, such as motivating the need for waypoint prediction in the first place, is not succinctly defined. Similarly, I found some  explanations of the experiments hard to follow in Sections 4.5 and 4.6. This could be improved by decreasing the use of lingo specific to the project and spending more time motivating each experiment before diving into details.\n\n**Novelty:** The idea of breaking down tasks into action primitives and using waypoints is not entirely novel. Similar concepts have been explored in prior work, such as:\n\nCLIPort: https://arxiv.org/abs/2109.12098\n\nPerAct: https://arxiv.org/abs/2209.05451\n\nRAPS: https://arxiv.org/abs/2110.15360\n\nLearning Synergies between Pushing and Grasping with Self-supervised Deep \n\nReinforcement Learning: https://arxiv.org/abs/1803.09956\n\nAWE: https://arxiv.org/abs/2307.14326\n\nForceSight: https://arxiv.org/abs/2309.12312\n\nScaling up and Distilling Down: https://arxiv.org/abs/2307.14535\n\nText2Motion: https://arxiv.org/abs/2303.12153\n\nThe authors do not mention most of these works, and they do not go into sufficient detail in distinguishing themselves from prior work. The paper also seems to lack reference to neurosymbolic/TAMP/PDDL approaches, which are concepts relevant to the idea of action primitives.\n\n**Asynchronous Hierarchical Executor:** The AHE, while improving efficiency, appears to be a simple scheduler that can be implemented with very simple logic. This does not constitute a significant technical contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your detailed and constructive comments. We are delighted to see your recognition of PIVOT-R's state-of-the-art performance on the SeaWave benchmark. We respond to all the issues you pointed out in detail below. We hope our response and rebuttal revisions will address your concerns and lead to reconsideration of our work.\n\n**Lack of Clarity:** \n\n**(a) Waypoint prediction motivation.** \n\nIn fact, we have a detailed and concise statement about the motivation of waypoint prediction in lines 3-9 of the abstract and lines 41-42 of the introduction. The motivation for waypoint prediction is mainly to prevent the model from being drowned in trivial scene and action predictions, which is crucial for world modeling and manipulation skill learning.  \n\n**(b) Experimental motivation in Sections 4.5 and 4.6.** \n\nSection 4.5 discusses the impact of waypoint selection, VLM, AHE, scene prediction supervision, and action prediction module design on PIVOT-R's performance. Section 4.6 analyzes why PIVOT-R succeeds, exploring its generalization to new tasks and potential for further improvement by incorporating other datasets. These experiments are crucial for understanding each module's design effect and PIVOT-R's advantages.\nThanks for your valuable suggestions, we will revise the relevant expressions and reduce specific terminology for clarity.\n\n**Novelty. Q1. Primitive actions and waypoints.** \n\nThank you very much for providing so many valuable related works. We have noticed previous work on waypoints and primitive actions, noting that they often used a limited number of actions or lacked world models for support. For instance, CLIPort [1], Transporter [2], GMRT [3], and VPG [4] are restricted to simple actions like pick/place/push, limiting their use in complex tasks. Some language-guided models [5,6,7] define a few primitive actions (≤5) and add prompts to aid decision-making. In contrast, PIVOT-R supports 10 primitive actions, including unique actions like \"rotate/open/close,\" making it effective in complex tasks. Crucially, PIVOT-R is the first primitive-driven, waypoint-aware world model, using primitives to break down tasks and combining scene and action prediction modules to model physical world knowledge. This enables PIVOT-R to handle complex tasks based on user instructions. We will also add the reference to the neuro-symbolic and PPDL approaches in our revision. Thank you again for your suggestion.\n\n[1] SHRIDHAR M, MANUELLI L, FOX D. CLIPort: What and Where Pathways for Robotic Manipulation[J].\n\n[2] ZENG A, PETE F, TOMPSON J, et al. Transporter Networks: Rearranging the Visual World for Robotic Manipulation[J]. arXiv: Robotics,arXiv: Robotics, 2020.\n\n[3] STENGEL-ESKIN E, HUNDT A, HE Z, et al. Guiding Multi-Step Rearrangement Tasks with Natural Language Instructions[J].\n\n[4] ZENG A, SONG S, WELKER S, et al. Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning[C/OL]//2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid. 2018. \n\n[5] COLLINS JeremyA, HOUFF C, TAN Y, et al. ForceSight: Text-Guided Mobile Manipulation with Visual-Force Goals[J]. 2023.\n\n[6] HA H, FLORENCE P, SONG S. Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition[J]. 2023.\n\n[7] Lin K, Agia C, Migimatsu T, et al. Text2motion: From natural language instructions to feasible plans[J]. Autonomous Robots, 2023, 47(8): 1345-1365.\n\n**Novelty. Q2. AHE's technical contribution.** \n\nIt should be emphasized that the technical contribution of AHE is not isolated. It should be considered together with the waypoint-aware world model (WAWM). RT-H uses VLM for robot actions and language association, requiring VLM at every manipulation step, which reduces efficiency. In contrast, PIVOT-R's VLM-based primitive-driven WAWM for scene and action prediction, combined with AHE for asynchronous execution, improves efficiency by 28 times (Table 3 in our paper). Though simple, AHE's integration with WAWM is highly effective. We will emphasize this in the revised version. Thank you for the suggestion.\n\n**Questions. Q1. Performance of PIVOT-R on other benchmarks.** \n\nWe evaluated PIVOT-R on the latest SIMPLER [1] benchmark, a scalable, repeatable and reliable proxy for real-world evaluation. We use this to verify the scalability of PIVOT-R in the real world. As shown in Table 1 of the rebuttal PDF, PIVOT-R outperformed the best baseline by nearly 10%. Additionally, we conducted real-world experiments, where we set up three tasks: (i) \"Pick up\": pick up the correct object from the table. (ii) \"Put on\": Pick up the object and place it on the correct color block. (iii) \"Push to\": Push the object to the correct color block.  We collected 400, 200, and 200 sets of demonstrations respectively. We tested each task 24 times to calculate the average success rate. As shown in Table 3 of the rebuttal PDF, PIVOT-R achieved a 6% improvement over the best baseline, Surfer. The waypoint-aware world model enabled PIVOT-R to demonstrate excellent real-world capabilities. Real-world evaluation demonstrations are included in Figure 2 of the rebuttal PDF.\n\n[1] Li, Xuanlin, et al. \"Evaluating Real-World Robot Manipulation Policies in Simulation.\" *arXiv preprint arXiv:2405.05941* (2024).\n\n**Questions. Q2. AHE mechanism.** \n\nSpecifically, we use multithreading to process each module separately. Each thread runs at its own frequency, extracts the latest data from the corresponding buffer, and places the output results in the buffer. For example, the VLM gets data from the camera buffer and saves the output in the buffer after each update. Then, the scene and action prediction module updates at different frequencies and reads the latest data from the cache of the previous module.  Different modules will not be blocked by other parts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UwzLXaRPaW", "reviewer": "Reviewer_F1Qx", "summary": "In this paper, the authors present PIVOT-R, a primitive-driven waypoint-aware world model for robotic manipulation. PIVOT-R comprises two key components: a waypoint-aware world model (WAWM) that parses primitive actions and predicts primitive-driven waypoints, and an Action Prediction Module that decodes low-level actions. Additionally, they introduce an Asynchronous Hierarchical Executor (AHE) for PIVOT-R, which enhances execution efficiency by applying different execution frequencies to different modules. Experimental results demonstrate that PIVOT-R achieves state-of-the-art performance on the SeaWave benchmark.", "questions": "* I understand that different modules are assigned different execution frequencies, but what is the definition of the execution frequency? For example, when the authors set v1 = 3, v2 = 10, v3 = 30, does it mean that the VLM model is used 3 times and the action prediction module is used 30 times for each data sample? In addition, I am interested in knowing the average execution time for each module.\n* The authors claim that world modeling of waypoints can prevent critical robot dynamics from being submerged in trivial robot manipulations. It would be beneficial if the authors could provide an ablation study or relevant examples illustrating this phenomenon.\n* Given that the method operates in a closed-loop system, I wonder if there are cases where the robot can retry and successfully perform an action after an initial incorrect attempt? Or does an incorrect action always lead to task failure?\n* Typo: In line 181, there is a repeated 'waypoint'.", "limitations": "The authors have discussed the limitations, but they are encouraged to provide and analyze specific failure cases of the proposed method.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The design of setting different execution frequencies for different modules (AHE) makes sense and enhances execution efficiency.\n* The authors conduct extensive experiments, including comparisons with several baselines and ablated versions, and provide detailed discussion and analysis.\n* The paper is well-organized and easy to understand.", "weaknesses": "- The proposed framework has only been evaluated in simulated environments. The authors are encouraged to conduct real-world experiments to demonstrate the method's sim-to-real transfer capability.\n- The method relies solely on 2D RGB images without any depth information, which may hinder the accurate prediction of 3D robot actions, especially in unseen perspectives and scenarios. Additionally, transferring simulated RGB images to real-world RGB images without depth information is a non-trivial challenge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your comprehensive and encouraging review! We respond to all the issues you pointed out in detail below. We hope our response and rebuttal revision will address your concerns.\n\n**Q1. Real-world experiment.**\n\nWe have added additional real-world experiments in Table 3 of the rebuttal PDF, where we set up three tasks: (i) \"Pick up\": pick up the correct object from the table. (ii) \"Put on\": Pick up the object and place it on the correct color block. (iii) \"Push to\": Push the object to the correct color block. For these three tasks, we collected 400, 200, and 200 sets of data respectively, for a total of 800 demonstrations. Finally, we perform 24 sets of tests for each task to calculate the average success rate. As shown in the results, compared with the best baseline Surfer, PIVOT-R achieved a **6% performance improvement**. With the help of the waypoint-aware world model, PIVOT-R has demonstrated excellent real-machine capabilities. We also added demonstrations of real-world evaluation in Figure 2 of the rebuttal PDF.\n\n**Q2. Action prediction of 3D robots.**\n\nThe most advanced methods such as GR-1, RT-1, and Surfer are based on RGB, and for fairness and ease of comparison, we have followed their settings. Due to time constraints, we plan to introduce 3D information in our upcoming work.\n\n**Q3. Definition of execution frequency and execution time.**\n\nPIVOT-R's AHE executes the action prediction module once in each time step, and other modules are updated proportionally. That is, when the action prediction module is executed 30 times, the VLM and scene prediction modules are executed 3 and 10 times respectively. In addition, each inference time of the three modules of primitive action parsing, scene prediction and action prediction is about 177ms, 29ms and 18ms. The execution speed mainly depends on the action prediction module.\n\n**Q4. Too frequent waypoints cause the robot dynamics to be overwhelmed.** \n\nThe corresponding experimental results are shown in row 3 of Table 3 (*i.e.*, PIVOT-R w/ next frame) in out paper. As shown in Table 3, compared with PIVOT-R (using key frames as waypoints), the average performance of PIVOT-R w/ next frame (using the next frame of the robot's manipulation trajectory as waypoints) dropped by 29.7% (74.19% vs. 44.45%). This shows that the waypoint-aware world model can effectively improve PIVOT-R’s modeling capabilities of key robot dynamics. \n\n**Q5. The result of a robot encountering an error during initial execution.**\n\nIn some cases, retries may be successful. As shown in Figure 1 (left) in the rebuttal PDF, when the position of the gripper deviated and the object failed to be grasped, the second attempt to grasp was successful. However, in the case of Figure 1 (right), if the object is knocked down and rolls a certain distance, it will be difficult to successfully grasp it again.\n\n**Q6. Expression error.** \n\nThanks for your reminder, we will modify this typo in our revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kvbIzQuarF", "reviewer": "Reviewer_9FHi", "summary": "The paper introduces a Primitive-Driven Waypoint-Aware model for robotic manipulation tasks. Initially, the entire language instruction is broken down into specific sub-steps using VLM. Subsequently, each sub-step's corresponding feature (\"waypoint\") is predicted through image prediction to segment the entire task execution process. The article also employs different execution frequencies for different prediction steps to enhance the inference speed. Comparative experiments were conducted on the SeaWave benchmark, and the model's generalization ability in unseen scenarios and its learning capability from human videos were analyzed.", "questions": "My main concerns lie in the need for a clearer articulation of the motivation, a more explicit delineation of the contributions of this paper, and improvements to the experimental section.", "limitations": "The authors have adequately addressed the limitations and societal impacts.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The figures and tables are clear and easy to understand. The experimental section reproduces several renowned studies and tests them on a new benchmark. The ablation study is elaborate.", "weaknesses": "1. The Introduction section contains misleading information and overclaims regarding the motivation. The model proposed in this paper has limited relation to a “world model”. I think that only the first step of parsing the language instruction has utilized the VLM, which makes it hard to convince that the entire model can be called a \"world model\". However, almost the entire article emphasizes the content related to the world model. Additionally, the paper points out that a significant drawback of previous works is sequential execution, and proposes the idea of accelerating inference speed for this reason. But the method of this paper is still sequential execution, and the acceleration is merely achieved by artificially setting different execution frequencies for different reasoning steps, which is also hard to consider as a strong contribution of this paper.\n\n2. The method proposed in the article seriously lacks novelty, and the discussion of its relationship with related paper is also insufficient. I summarize that the article mainly proposes two methods: First, it decomposes language into multiple sub-goals, which the author refers to as waypoints. Similar methods were first proposed in SayCan, and the language parsing process in this paper is not learnable. Second, the article proposes predicting future waypoints as intermediate results. Similar future prediction methods have already been numerous, and related articles have also studied predicting future images (the approach of this paper) or the trajectory of future key points as different forms of sub-goals. Therefore, the technical contribution and innovation of this paper are quite lacking.\n\n3. The experimental results of this paper are not very convincing. \n\n-  a) The method of this paper has only been verified on a relatively limited SeaWave Benchmark. Moreover, for the method proposed in this paper, it is necessary to artificially process the data, such as limiting it to 10 primitive actions. And it is also necessary to add additional primitive action and waypoint annotations. This almost introduces additional annotation information that is not utilized in the baseline for comparison. \n\n-  b) It is strange that this paper chooses Octo, RT-1, and GR-1 as baselines, as the training cost of these papers is very high, and it seems that this paper needs to re-train these models on new data, which requires a very long time for both training and testing. From the perspective of the method of this paper, it should be compared with some sub-goal prediction methods, such as UniPi and AVDC. \n\n-  c) The results in Table 2 show that PIVOT-R's performance drops more in unseen scenarios compared to seen scenarios, how to explain this? \n\n-  d) The experimental results in Table 4 have almost nothing to do with the motivation of the paper. The fact that pre-training with human data can improve the performance and generalization of downstream tasks has already been proven in a large number of previous works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to see your recognition of the PIVOT-R experiment. But there are some serious factual errors in the review: \n\n(1) VLM is **not equal to** the world model; \n\n(2) AHE is **multi-threaded** rather than sequential execution; \n\n(3) The task parsing of SayCan and PIVOT-R is NOT at the same level, and the learnability of language parsing is unnecessary;\n\n(4) Scene prediction is strongly related to the waypoint-aware world model and should not be viewed in isolation; \n\n(5) The experimental design and the motivation of PIVOT-R are strongly related and complete. We provide detailed responses to all the issues you have pointed out. \n\nWe hope our response and rebuttal can address your concerns, and sincerely hope you can reconsider our work. \n\n**Q1.1.  About the world model.** \n\nVLM is not equal to the world model. The pioneering work of the world model [1] defines it as a model that can perceive the environment and predict changes in environmental states. In our PIVOT-R's waypoint-aware world model (WAWM), we use VLM to **perceive the scene and parse user instructions** to obtain primitive action descriptions. And we use the scene prediction module to **predict waypoint scenes** for capturing environmental dynamics. This fits the definition of a world model.\n\n[1] Ha, David, and Jürgen Schmidhuber. \"World models.\" *arXiv preprint arXiv:1803.10122* (2018).\n\n**Q1.2. AHE's technical contribution.** \n\nPIVOT-R's AHE is implemented in a multi-threaded manner, with each module's output stored in a buffer to prevent blocks between modules. As shown in Table 3 in our paper, AHE improves PIVOT-R's execution efficiency by 28 times (27ms vs. 756ms). This approach is valuable for models with multiple modules of varying efficiencies, providing significant enhancement and useful reference for other work.\n\n**Q2. Waypoints and technical contributions.** \n\n**(a) Waypoints.** \n\nSayCan uses LLM to decompose complex human instructions into a higher-level sub-task, such as \"find apples/go to table\". This is significantly different from PIVOT-R, which is a decomposition of low-level primitive actions proposed for robot manipulation tasks. \n\n**(b)Technical contribution.** \n\nWaypoints' technical contributions should not be reviewed individually. PIVOT-R is the first attempt to introduce waypoint predictions for world model learning. It parses complex user instructions into primitive actions through VLM, and combines scene and action prediction modules driven by primitives to realize the modeling of physical world knowledge. This enables PIVOT-R to focus on task-relevant key points instead of being drowned in trivial scene predictions, leading to significant improvement in both manipulation performance and execution efficiency. \n\n**(c) Language parsing is not learnable.** \n\nAs shown in the results of Lines 4 and 5 in Table 3 in our paper, the current open-source VLM is fully capable of handling the related primitive parsing tasks. Therefore, the introduction of VLM in the offline manner does not have a big impact on the whole performance.\n\n**Q3. Experiment.**\n\n**(a) Data annotation.** \n\nFirst, 10 primitive actions are sufficient for manipulation tasks. For example, [1,3] uses only pick and place actions, and ForceSight[2] defines 5 actions. Secondly, adding more annotation information to existing benchmarks is a common practice to improve model performance. Similarly, MOKA [4] and LEO [5] use LLM to annotate data for better performance. This annotation method itself is also one of the contributions of the work.\n\n[1] SHRIDHAR M, MANUELLI L, FOX D. CLIPort: What and Where Pathways for Robotic Manipulation[J].\n\n[2] COLLINS JeremyA, HOUFF C, TAN Y, et al. ForceSight: Text-Guided Mobile Manipulation with Visual-Force Goals[J]. 2023.\n\n[3] ZENG A, PETE F, TOMPSON J, et al. Transporter Networks: Rearranging the Visual World for Robotic Manipulation[J]. arXiv: Robotics,arXiv: Robotics, 2020.\n\n[4] Liu, Fangchen, et al. \"Moka: Open-vocabulary robotic manipulation through mark-based visual prompting.\" *arXiv preprint arXiv:2403.03174* (2024).\n[5] Huang, Jiangyong, et al. \"An embodied generalist agent in 3d world.\" *arXiv preprint arXiv:2311.12871* (2023).\n\n**(b) Reasons for choosing methods such as Octo and RT-1.** \n\nThis is mainly because they are state-of-the-art robot manipulation models.\n\n**(c) Comparison with other sub-goal prediction methods.** \n\nWe compared SUSIE [1], which uses InstructPix2Pix [2] for sub-object prediction, and low-level strategies for action prediction. As shown in Table 2 of the rebuttal Appendix PDF, PIVOT-R outperformed SUSIE by 26.93% (74.19% vs. 47.26%) on SeaWave tasks, demonstrating the effectiveness of WAWM in PIVOT-R for sub-goal modeling.\n\n[1] Black, Kevin, et al. \"Zero-shot robotic manipulation with pretrained image-editing diffusion models.\" arXiv preprint arXiv:2310.10639 (2023).\n\n[2] Brooks, Tim, Aleksander Holynski, and Alexei A. Efros. \"Instructpix2pix: Learning to follow image editing instructions.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n**(d) Why does PIVOT-R show obvious performance degradation on unseen scenes?** \n\nThis is normal for robot manipulation models, as they often struggle with generalization to unseen scenes not present in the training data, leading to perception or recognition errors. As shown in Table 2 in our paper, nearly all methods experience performance degradation in new scenarios. However, our model still achieves state-of-the-art performance.\n\n**(e) Train with Human Data.** \n\nThis experiment is mainly to prove that PIVOT-R can also benefit from human data. We hope to conduct large-scale data training to improve model performance in the future. Based on your suggestion, we will move this experiment to the appendix in our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their time, insightful suggestions, and valuable comments. We are happy that they appreciated our paper

- makes **important technical contribution** on combining large VLMs with scene prediction models (Reviewer XYkg);
- is **well-written, well-organized, and easy to follow** (Reviewer XYkg, F1Qx, and 9FHi);
- the design of the asynchronous hierarchical executor (AHE)  **makes sense and enhances execution efficiency** (Reviewer F1Qx);
- demonstrates **strong performance improvements** on the SeaWave benchmark and provides **thorough ablations** for generalization and interpretability (Reviewer eMP4, XYkg, and F1Qx).

We have also conducted additional experiments and provided necessary clarifications in our rebuttal, which are summarized below:

- In response to Reviewer eMP4 and F1Qx, we have added additional real robot experiments  (Table 3) and experiments on other benchmarks  (Table 1) in the rebuttal PDF. The results show that our PIVOT-R **significantly outperforms comparison methods** in both **real robot experiments and other benchmarks.**
- In response to Reviewer eMP4, we have highlighted our motivation and technical contribution of introducing waypoint prediction and asynchronous hierarchical executor, which are **crucial for enabling the world model to capture key dynamics and enhance execution efficiency**. We also provided a detailed analysis of the difference between our PIVOT-R and existing methods regarding action primitives and waypoint prediction. Note that both Reviewer XYkg and F1Qx **acknowledged the technical contribution of our paper.**
- In response to Reviewer 9FHi, we have clarified that **both the VLM and the scene prediction model** form the world model, which fits its definition of perceiving the environment and predicting changes in environmental states. We also highlighted that our core novelty is that we **introduce action primitives and waypoint prediction for improving world modeling**, which we believe is a valuable and inspiring idea for the robotic community. It is arbitrary to judge our novelty in simply action primitive decomposing and waypoint prediction.

We provide detailed answers to the reviewers' questions individually. We hope our response and rebuttal revision will address the reviewers’ concerns.

Meta-review: This paper introduces PIVOT-R, an approach for language-guided robotic manipulation. PIVOT-R consists of a Waypoint-aware World Model (WAWM) and a lightweight action prediction module, along with an Asynchronous Hierarchical Executor (AHE) to improve efficiency. The model achieves state-of-the-art performance on the SeaWave benchmark, demonstrating improvements in performance and efficiency compared to baselines. The paper has been well written and presented. Overall, a good paper.
Authors have addressed most of the concerns raised by the reviewers during rebuttal. Please include some of the minor revisions in the future edit of the paper.
Decision: Accept (poster)