Closed-Loop Visuomotor Control with Generative Expectation for Robotic Manipulation
Authors: Qingwen Bu
Jia Zeng
Li Chen
Yanchao Yang
Guyue Zhou
Junchi Yan
Ping Luo
Heming Cui
Yi Ma
Hongyang Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Robotic Manipulation
Visuomotor Control
Primary area: robotics
TL;DR: A closed-loop visuomotor control framework that incorporates feedback mechanisms to improve adaptive robotic control

Abstract: Despite significant progress in robotics and embodied AI in recent years, deploying robots for long-horizon tasks remains a great challenge. Majority of prior arts adhere to an open-loop philosophy and lack real-time feedback, leading to error accumulation and undesirable robustness. A handful of approaches have endeavored to establish feedback mechanisms leveraging pixel-level differences or pre-trained visual representations, yet their efficacy and adaptability have been found to be constrained. Inspired by classic closed-loop control systems, we propose CLOVER, a closed-loop visuomotor control framework that incorporates feedback mechanisms to improve adaptive robotic control. CLOVER consists of a text-conditioned video diffusion model for generating visual plans as reference inputs, a measurable embedding space for accurate error quantification, and a feedback-driven controller that refines actions from feedback and initiates replans as needed. Our framework exhibits notable advancement in real-world robotic tasks and achieves state-of-the-art on CALVIN benchmark, improving by 8% over previous open-loop counterparts. Code and checkpoints are maintained at https://github.com/OpenDriveLab/CLOVER.

Closed-Loop Visuomotor Control with Generative Expectation for Robotic Manipulation
Qingwen B2
Jia Zeng
Li Chen
Yanchao Yang
Guyue Zhou'
Junchi Yan?
Ping Luo
Heming Cui
Mi
Hongyang Li
Shanghai Al Lab
Shanghai Jiao Tong University
HKU 4 Tsinghua University
Equal contribution Corresponding authors
Abstract
Despite significant progress in robotics and embodied AI in recent years, deploy- ing robots for oong-horizon tasks remains great challenge. Majority of prior arts adhere to open-loop philosophy and lack real-time feedback, leading to error accumulation and undesirable robustness. A handful of approaches have endeavored to establish feedback mechanisms leveraging pixel-level differences or pre-trained visual representations, yet their efficacy and adaptability have been found to be constrained. Inspired by classic closed-loop control systems, pro- pose CLOVER, a closed-loop visuomotor control framework that incorporates feedback mechanisms improve adaptive robotic control. CLOVER consists a text-conditioned video diffusion model for generating visual plans as ref- erence inputs, measurable embedding space for accurate error quantification, and a feedback-driven controller that refines actions from feedback and initiates replans needed. Our framework exhibits notable advancement in real-world robotic tasks and achieves state-of-the-ar on CALVIN benchmark, improving by 8% over previous open-loop counterparts. Code and checkpoints maintained https:/ / gi thub. www httm//ppeDDiieea//Ct..
Introduction
Robotics and embodied generalists have gained enormous achievements in recent years, with the successful advancement representation learning and visual generation in computer vision [1, 2, 4], large Vvision-llnggua models [5, 6], policy learning [7, 8), etc. Remarkable behavior intelligence has been demonstrated in diverse and complex single-task settings from picking up a Lego block to solving Rubik's cube [9, 10]. However, deploying the robots for long-horizon manipulation tasks
remains a long-standing challenge [11, 12].
Some literature have attempted to tackle the problem with large language models, splitting prolonged job into detailed instructions for each minor movement [13, 14], or sub-goals. Though effective in high-level descriptions, texts could still be inadequate for detailed portrayals of the environment and robot state, leading to considerable issues under cross- goos-moophology or multi-ennvronmeets [15]. Therefore, recent efforts have started embracing vision as a universal medium to develop an embodied agent capable of planning diverse tasks through imagination and execution [16, 15, 17]. These approaches involve generative model for predicting future videos or goal images, followed by goal-conditioned policy for translating the visual plan into actual actions. Despite success, they adhere to an open-loop paradigm, i.e., proceeding with fixed sequence of actions without verifying whether the actual trajectory aligns with the planned one. For instance, when a robot is tasked to "grab Coke from the fridge", current works assume that the predicted sub-goal is the visual image of the door opening, and the robot should naturally achieve the state (sub-goal) prior to grabbing the bottle. However, the lack of error measurement and real-time feedback leads to accumulative deviation, undesirable robustness, and limited adaptability. particularly inadequate for long morgzoo tasks and dynamic environments [11, 18].
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Classic Closed-loop Control
17
Dested Value
Mecsurement
Current Volue
Bios
Actin
Controller
System
955
Human
Reforence
rrpuf
Feedback Loop
(b) Closed-loop Visuomotor Control
Adaptive
Transition Repian
Dested Suut good
Sub-goal
Guarent
Enor Mecsurement
BGX
Action
Visual
Feedback-Driven
Env.
Planner
Policy
Feedbock Loop
Figure 1: Motivation. The proposed CLOVER is inspired by the classic closed-loop control in automation systems (a). Our framework (b) employs visual planner to predetermine sequence of sub-goals (Section 3.1). Then these goals guide the policy to generate actions with an error measurement strategy (Section 3.2). Within the feedback loop, automatically replans when the sub-goal is infeasible, and adapts to to the next one upon achievement (Section 3.3).
We are inspired by the conventional closed-loop control system depicted in Figure I(a). It aims to regulate physical quantities such as actuator velocity by enhancing control precision via a feedback mechanism. Three major components states which could comprise multiple stages for prolonged task; the error measurement quantifies bias between the observed state and the planned sub-goal; and the controller adjusts output to reduce the deviation (error) [19]. In fact, several works have introduced analogous feedback in roboties by measuring errors with pixel appearance [20, 21] visual representations [22], e.g., CLIP features [23], and yet their performance and adaptability is limited, leaving accurate error quantification modeling unexplored.
worth mentioning, the reference input defines desired
To this end, we propose CLOVER, CLOsed-loop Visuomotor control framework with generative Expectation for Robotic manipulation. As shown in Figure 1(b), analogy to the classic closed-loop control system, the ingredients in our version are adapted accordingly. 1) Reference inputs. With video frames as the interface describing desired states, a text-conditioned video diffusion model generates future frames as reference inputs. To further facilitate the subsequent planning accuracy, endow the visual planner with the ability of depth map generation, and introduce optical flow regularization to prioritize motion consistency. 2) Error measurement. Given the limitations of pixel- wise metrics and the inadequacy of pre-trained visual representations, of a measurable embedding space to realize accurate and efficient error measurement between the observed and planned states. Our state embeddings are trained using an explicit error modeling approach, which yields a strong correlation with the process of converging towards or diverging from target states. 3) boedback-driver controller. We present simple yet effective control framework, comprising a controller and an error aware adaptive control strategy. The controller is optimized via an inverse dynamics objective [24] to achieve the predefined sub-goals. To address the issue of a goal-oriented policy failing to consistently achieve the desired state, our proposed framework, CLOVER, adopts an iterative refinement strategy. It continually adjusts its actions to minimize errors, re-evaluates and replans if sub-goals are unrealistic. To sum up, our contributions are three folds:
propose the establishment
We introduce CLOVER, a generalizable closed-loop visuomotor control framework that incorpo- rates feedback mechanism improve adaptive robotic control.
We investigate the mate-meaaurin attribute for latent embeddings and propose a policy by quanti- fying feedback errors explicitly. The error quantification settles the construction of execution pipeline to resolve the challenge of handling uncertaintics in vidco generation and task horizons.
Extensive experiments in simulation and real-world robots verify the effectiveness of CLOVER. It surpasses prior otate-of-the-arts by a notable margin (+8%) on CALVIN. The average length of completed tasks on real-world long- hong-horizon manipulation nearly doubles compared to RT-1.

---

2 Related Work
Closed-loop mechanisms for robotics. Hodel-ppedicctive control (MPC) is a classic and popular approach for leveraging learned dynamics for robotic control and has gained great success in learning robust closed-loop policies [25, 26, 27]. Nonetheless, many of these prior works require knowledge about the system state in the planned future which is often infeasible. Visual foresight [20, 28] integrates an auxiliary register network to calculate pixel distances between the current and goal images, consequently providing feedback. And some methods [21, 29] utilize an additional detection model with preset rules to estimate the current state. However, the adaptability of these methods is constrained to single pick-and-pul tasks and does not cover fong-horizon, multi-object manipulation tasks. HiP [30] investigates the feasibility of decomposed language sub-goals and the consistency of generated video plans through training with feedback. Like other visual planners [16, 15. 17], it lacks a quantitative assessment of trajectory achievement during test-time execution. Inner Monologue [18] leverages vision-language model to provide linguistic feedback for task success detection, but the substantial size of this model hinders the efficiency during test-test execution. In contrast, CLOVVRR constructs a measurable latent space from pixel observations and qualitatively measures deviations from planned goals for each action, thereby incorporating real-time feedback. Furthermore, the feedback mechanism incorporated into long-horizon manipulation tasks.
Diffusion model as a visual planner. Recently, it is trending to utilize diffusion models as visual planners to generate goal states. UniPi [15] seminally leverages internet data train text-conditioned video generator and uses an inverse dynamics model to estimate ultimate actions. UniSim [17] creates universal video diffusion model for simulating interactions and training policies through generative modeling. Ajay el al. [30] propose compositional foundation models for hierarchical planning, including task decomposition, visual planning, and action inference. They also utilize an additional
classifier to deal with the uncertainty of generation quality. SuSIE [16] utilizes an imagc-editing model high-level planner set achievable sub-goals for a low-level controller, while ADVC [31] infers actions from predicted video content with dense correspondences. Though prior techniques can synthesize visually reasonable future sub-goals, one challenge is that the lack of consistency constraints related to geometry and motion potentially diminishes the fidelity of generated videos for policy prediction and increases generation instability. In our work, an RGB-D video prediction model is introduced and constrained by the optical flow to enhance sub-goals' reliability. Moreover, the instability of visual plans from diffusion models is rarely discussed in the aforementioned literature. Contrarily, at the core of CLOVER, we adopt an policy state estimation to detect unreachable plans by measuring the distance between consecutive frames.
3 Methodology
We aim at building a generalizable framework that integrates the closed-loop philosophy into robotic visuomotor control. The overall system illustrated in Figure 1. Accordingly, in order to set the desired value before execution, we introduce a visual planner that generates consecutive sub- wub-goals (Section 3.1). In Section 3.2, we detail the structure of our feedback-driven policy to decode actions, and demonstrate how to measure the deviation from the current to the goal states. Finally, the overall test-time execution pipeline of CLOVER is leveraged in Section 3.3.
3.1 Visual Planner
The visual planner is to produce reliable sequence of future plans based on the initial observation On and task descriptions qr Inspired by previous successful attempts [15, 31, 17], we also employ the prevailing conditional diffusion model to realize text-conditioned video generation [32]. Derived from the image diffusion model of Imagen [33, 31], our model is designed to generate future videos (i.e., predicted sub-goals) spanning predetermined time frames, denoted as {O,, Óz.,ó,.... Ó } with K = 8. However, different from targets of high-resolution and meticulous structures for general generative models, visual planners for robotic manipulations highlight the need to understand spatial environments and robot movements. Therefore, designs of effectively integrating depth information and leveraging optical flow's s regularization are introduced in CLOVER to generate geometry-ave aawre and temporally coherent futures, which we describe below.
Text-conditioned RGB-D video generation. In the framework of amalgamating video prediction and goal-conditioned policy modules, generating visual plan that precisely corresponds to the
3

---

Visual Inputs
1. State Encoder
Error Measurement Embedoings
Fused
Multimodal
RGB-D Features
R
Encoder
(F [goss]
Rooat
Hour
Current
B
Observation
Images
RGB
Concall Comu
Taken
Encoder
RV
Aggregator
Action Decoder
Depth Maps
Dopth Encoddrr
MLP
Action E R'
Sub-goal 0g
Figure 2: Architecture of our feedback-driven policy. 1) The state encoder takes in both current observation along with the synthesized sub-goal -oal. A shared multimodal encoder generates fused RGB-D features. followed by two queries extracting informative features as the current and goal embeddings respectively. 2) The discrepancy of the two state embeddings explicitly modeled as errors. 3) The resultant residual in error measurement is ultimately decoded to the final action.
task description is prerequisite for accomplishing manipulation tasks. To encode language inputs, we employ the tokenizer and encoder from CLIP [23] as the basis, following [34]. In addition to the condition injection techniques outlined in Imagen [33], which integrate language embeddings into the latent space the diffusion model directly, our model further incorporates cross-attention based conditioning to enhance its language-following following ability. Moreover, utilizing classifier-frs free guidance [35], the visual planner demonstrates encouraging controllabiliit and generalization, being able to produce diverse and reasonable plans based task descriptions (See analysis in Section 4.2).
For the vision inputs, robots operating in the 3D space face great challenges in learning from 2D observations directly [36]. Therefore, considering the ease of acquisition of depth sensory nowadays robotics and its accurate spatial depiction of the environment, we incorporate geometric information from depth maps to assist in manipulation. To predict RGB-D videos, we adopt simple yet effective way. Specifically, the RGB image and depth map are concatenated on the channel dimension and embedded into unified latent space throughout all layers of the model. Compared to devising distinct branches for each modality, this yields satisfactory generation results with high consistency between modalities in practice. Moreover, the straightforward approach opens the potential for pre-training the diffusion model on large-scale RGB-only datasets to further enhance its capabilities [37,
Latent regularization with optical flow. Besides the easy acquisition of the depth modality, the robotic manipulation tasks also feature their interaction dynamics, i.e., the moving robot arm and interacted objects in the environment. Though existing works [16, 30] have utilized video diffusion models for visual plan generation, they fall short in considering the essential gaps between robot manipulation and general video data adequately, particularly the static camera position and robot-initiated movements [39]. Drawing inspiration from the importance of motion cues in robot manipulation, propose to incorporate optical flow foster the classic video diffusion models for manipulation tasks. Specifically, following the end-to-end optical flow estimation framework, RAFT [40], first build the pixel-wise correspondence map between the diffusion latent of two consecutive frames. This map is then utilized by subsequent modules to iteratively refine the optical flow estimation through lookup and update operations. Given the final estimation, our flow-based regularization term is formulated as:
explicit regularization term to further
K Σ k=
Ligg
|1x+1 |1++1-W - W(O F flo+1i>l
(I)
K
where Fi-Sk41 is the estimated optical flow, W represents the wrapping function and (Ox, Oc41 two consecutive frames in the ground-truth video. More details are provided in Appendix B.
3.2
Feedback-Driven Policy
As depicted in Figure 2, from current and desired visual inputs to the ultimate action output, our policy can be divided into the following components: 1) State Encoding: Deriving informative features from visual inputs and producing compact state embeddings that encode current and desired sub-goal states; 2) Error Measurement: Formulating the deviation from current to goal state; 3)
Action Decoding: Decoding the deviation signal into the action robot can actuate.

---

Sub-gooi
features while Sub-goal 2
Sub-goal goal
Sub-goal 4
Sub-goal 5
Sub-goal 6
Sub-goal 7
Sub-goal 8
Low variabiliity, low measurability
High variability, low measurabiiity
High variability, high measurability
Zoom View
Cap
Action Steps
Action Steps
Action Steps
(a) CUP Feature
(b) State Embedding w/o
[c] State Embeddiing (Ours)
Error Measuring
Figure 3: Comparison on the measurement ability of different embeddings. We visualize the cosine distance between embeddings of observations and generated sub-goals during roll-out process. (a) CLIP feature [23] and (b) state embeddings trained without error measuring do not hold clear interrelations among frames. While (c) state embeddings obtained from our policy distribute reasonably in the latent space which benefits measuring the errors in feedback loops.
State encoder. To begin with, we employ a multimodal encoder to transform raw pixel inputs into enriched visual representations, which comprises two ViT-based [41] encoders for RGB and depth respectively along with a multimodal feature fusion module. The feature fusion process uses squeeze-and ae----ndddeeeeteeee module [42] for channel-wise integration and selection. Subsequently, the token aggregator adaptively selects critical information pertaining to manipulation from the sequence of visual features, condensing them into compact state embedding. Expressly, the token aggregator is built upon a multi-head attention pooling [43] with a two-layer multi-layer perceptron (MIP) performing nonlinear projection. Given fused RGB-D features ( Egoal Rixd corresponding to current and goal inputs, respectively, this process can be specified as:
q = MLP MMult
q. - E)],
where è [gour:ggoa) } denotes state embeddings and q {qeur- guaat are queries initialized to extract visual features E' E Eeur. Egoal J. Here d is the hidden dimension size and / represents the visual token length. We employ shared weights for encoding both states in parallel, with the exception that two separately initialized queries are utilized to extract each state embedding. State encoder gives rise to an information bottleneck, prioritizing the encoding of manipulation--elest filtering out irrelevant background details to ensure informativeness.
Error measurement. In conventional closed-loop control systems, the controller generates control signals based on the error between the desired value and feedback signals [44]. Analogously, in our visuomotor control pipeline, we explicitly model the discrepancy between the current and goal states by performing element- wementtwise subtraction of the two corresponding state embeddings, Despite its simplicity, this approach has been proven effective in practice. induces strong prior that latent actions formulate the transitions between latent states [8]. More importantly, the transitions can be quantified and we note that not all embeddings hold the essential characteristies.
Notably, learning representations to distinguish among diverse instructions and visual states has been long-standing topic [45], while there exist few works exploring their quantitative metric for action. In CLOVER, the measurement capability of state embeddings is observed by learning to act from deviation signals, which is absent in pre-trained visual encoders or policy models learned based on current observations solely, i.e., behavior cloning. As illustrated in Figure 3(C), the cosine distance between state embeddings decreases together with the robot approaching each predicted sub-ge bob-goal. In the meantime, the distance to the previous sub-goal increases as proceeding to the next one. Besides, the numerical range of the distance spans approximately from 0.9, thereby providing a sufficient margin for distinguishing and identifying current states. On the contrary, concerning visual representations generated by pre-trained encoders (i.e., CLIP features in Figure 3(a),, there is a noticeable reduction in the range of value variations (within 6e-2) with pronounced fluctuations, although the curves show similar patterns in general. This result stands for all the pre-trained visual encoders we have studied, owing to the fact that manipulation-relevaa features will overwhelmed by immaterial background information. Furthermore, we also test employing state embeddings generated by our policy model but optimized without error measuring.
5

---

The resulting embeddings capture the state propagation evidenced by the significant numerical variability (Figure 3(b)); however, they lack the capability to measure interrelations among sub- sub-goals and exhibit poor monotonicity when reaching each sub-goal. Next, we introduce how to elevate the satisfactory error measuring feature for action decoding and adaptive feedback control autonomously.
Action decoder. To keep the framework concise and generalizable, we simply adopt an MLP to decode action outputs from error signals. We consider the action space of 7-DoF robotic arm, encompassing the position of the end-effector dgE E R and the gripper state Ogriper E RR. Our policy is learned with an Inverse Dynamics objective (a(ay Ou. Ox), where infers action ao based on current observation Oo and specified sub-goals O₃- To promote the transferability of our framework, we exploit third-view RGB-D images as inpuss only, with action labels as the training targets. State information such proprioception signals or gripper-view images are not applied to facilitate manipulation. Please refer Appendix B for further architectural and training details.
Algorithm 1: CLOVER: Test-time Execution
Input: Visual planner pe; Policy Tg. State encoding module g+):): Cosine distance Do(..). Hyper parameters Time limit T;; Distance threshold for replan and sub-goal transition (Dx, Ds}-
1 bub 0 while S do
Initialize the sub-goal selection index
if Replan or = 0 then
Ōn.K pa(O Ou,
Generate language-conditione sub-goals (Section 3.1)
b.nk- 11 (Do(ys(OL), da (o(()+1))) Da then
Replan True
D-Replan if sub-goals unreachable
clse
Replan + False
end
10
end
De (Oo)) go go()))))) Dg then Auu f- iub +
11
12
D Transition if the current sub-goal has been reached
13
end
14
Sample and Execute à ~ Tooool0o, Oswo) Oo Env(ā)
> Predict and execute action (Section 3.2)
Update current observation
f +
17 end
3.3 CLOVER
Equipped with the error quantification capability aforementioned, we have developed a closed-loop visuomotor control framework with feedback, illustrated in Algorithm 1. Notably, our framework distinguishes itself through two key aspects: 1) It can detect and address the instability of diffusion models by initiating replanning when sub-goals unreachable; 2) It achieves adaptive transitioning contrast to previous literature such as
between sub-goals based on the distance measurement. SuSIE [16] that sets up dataset-dependent hyperparameters to manually regulate sub-goal refreshing (usually required to be consistent with the frame intervals during training) and thus potentially limit their performance and scalability, to visual planners with varying intra- frame intervals. We provide illustrative examples in Appendix A to demonstrate the functionality of replanning and adaptive sub-goal transitions.
proposed CLOVER is agnostic training details and adaptable
4 Experiments
4.1 Experimental Setup
Simulation tasks. We conduct the majority of our experiments using CALVIN [54], an evaluation segguugge-conntionneee manipulation. CALVIN consists of four
benchmark designed for long-horizon, simulated environments (designated as A, B, C. and D), which differ in textures and object positions. Each environment comprises a Franka Emika Panda robot situated beside a desk equipped with various manipulable objects. We train policy models demonstrations collected from environments A, B, and C, and conduct zero evaluations in environment D. The evaluation protocol involves assessing model performance comprehensive set of 1,000 unique instruction chains, each

---

Table Long-horizon visuomotor control on CALVIN ABC-D. We report success rates along with the average length of completed tasks (out of the whole 5 tasks) per evaluation sequence. CLOVER outperforms all previous methods by notable margin. Lang and All denote whether models are trained only with the subset vision-language data. 'Results reported by [16].
Train episodes
Task completed (%)
Method
Type
Avg. Len.
MCIL [46]
All
30.4
HULC [47]
Language-condiing conditioned
All
41.8
1.9
0.67
RT-1 [48]
Lang
22.2
1.3
0.90
Bchaviour Cloning
RoboFlamingo [49]
Lang
82.4
61.9
46.6
33.1
23.5
2.48
GR-1 [50)
Lang
85.4
71.2
59.6
49.7
40.1
3.06
Diffuser Actor [51]
Diffusion Policy
Lang
92.2
78.7
51.2
41.2
3.27
UniPi" [15]
All
56.0
16.0
8.0
8.0
4.0
0.92
SuSIE [16]
Planner Exccutor
All
87.0
69.0
49.0
38.0
26.0
2.69
CLOVER (Ours)
Lang
96.0
83,5
70.8
57.5
45.4
3.53
Table 2:
Performances with real-world robot tasks. CLOVER achieves the best success and
superior generalization capability across the board.
Long-horizon task
Single task
Method
Sub-task
Sub-task 2
Sub-task 3
Avg. Len. f
Pour shrimp
Stack bowls
ACT [52]
46.7
13.3
0.0
0.6
33.3
46.7
R3M [53]
53.3
20.0
0.0
0.7
46.7
53.3
RT-1 [48]
66.7
40.0
0.0
1.1
80.0
66.7
CLOVER (Ours)
93.3
86.7
26.7
2.1
80.0
86.7
Long-horizon task
Sub-task 1: up the pot
Sub-task put the fish into the pot
Sub-task put the on the pot
Single
Task the strimp into the plate
Task B: stack two bowl
Figure 4: Real-world robot setting. We propose long- hang-horizon task encompassing three consecutive sub-tasks, where the failure of a prequel task will inevitably lead failure of subsequent tasks. The additional single tasks are designed to validate the generalizability of CLOVER of all aspects.
comprising five distinct tasks. The CALVIN benchmark provides extensive dataset paired with natural language annotations, thereby facilitating the training of generalized and reliable visual planner. Detailed implementation and training protocols provided in Appendix B.
Real-world experiments. The real-robot experiments are conducted on the AIRBOT Play robotic arm. We propose a long-horizor task comprising three consecutive sub-tasks and two additional single tasks ("Pour shrimp into plate" and "Stack two bowls", shown Figure 4). The fish and pot lid in sub-task 2 and sub-task 3, as well as the plate and bowl in two individual tasks, are randomly placed to reflect position generalizability. All metrics are evaluated with 15 independent runs.
4.2 Main Results
Visuomotor control on CALVIN. Table 1 indicates that CLOVER achieves state-of- tfate-of-the-an art perfor- mance on CALVIN, significantly outperforming previous methods with similar "Planner Executor" pipelines. Without using gripper view images and proprio signals, our approach exceeds methods employing GPT-style transformers with pretraining, such as RoboFlamingo [49] and GR-1 [50]. Note that all previous methods follow the CALVIN standard evaluation protocol [54], where the simulator
7

---

Table Generalization evaluation. CLOVER excels under visual distractions and dynamic scenes, while the success rate of baselines dramatically drops.
Long-horizon task
Setting
Method
Sub-task
Sub-task 2
Sub-task 3
Avg. Len.
ACT [52]
13.1
0
0.13
Visual
R3M [53]
20.0
0
0
0.20
Distraction
RT-1 [48]
40.0
6.7
0
0.47
CLOVER (Ours)
73.3
66.7
6.7
1.47
Dynamic
RT-1 | [48]
33.0
0
0
0.33
Scene
CLOVER (Ours)
80.0
53.3
20.0
1.53
returns the signal that marks the completion of the current task. However, such task completion sig- nals are not accessible in real-world environments. In fact, by leveraging the advantageous properties of our state embedding, we can determine the completion of a task autonomously without the signals. We leave this exploration to Appendix A.
Figure shows our diffusion model's proficiency in instruction- following, with "Slide down the switch" as representative task from the validation set and three others randomly proposed from the CALVIN's 's task pool. Our planner exhibits robust generaliz- ability, producing reliable action trajectories for the subsequent policy. More visualizations given in Appendix D.
Original setting
Visual distraction
Manipulation with real-world robots. We present the evalua- tions of real-world robotic tasks in Table 2. CLOVER surpasses all baseline models by a considerable margin, especially on the long-horizon manipulation metric (+1.0 on Avg. Len.). Note that the lid knob small and hard to grasp, which poses great challenges to policies' low-level precision. ACT [52] strug- gles to adjust the gripper to the right position before it should close in the first task, while CLOVER doubles the success rate. Moreover, all three baselines we test fail on the last task, which requires the robot to re-cover the pot lid that was previously placed down in Task 1. In this scenario with high uncertainty, CLOVER shows success rate of 26.7%, indicating its stronger robustness and position generalization capability.
Dynamic scene
Figure 5: Experiment setting of the generalization evaluation. We place entirely new objects absent from train- ing. alongside the interaction object to introduce visual distraction. We test policies under dynamic conditions by randomly placing and picking up doll to create unpredictable visual changes.
We further study the generalizability of CLOVER under visual distractions and dynamic environments, as inllustrated in Figure 5. Table 3 lists the results of the experiment. CLVOER remains performant while manipulating under distractors, with the performance gap baseline methods getting more pronounced. We provide qualitative analysis as shown in the Appendix Figure 17. The visual planner effectively disentangles background distractions with foreground movements and generates appropriate plans. Additionally, the feedback-driven policy proves robust to dynamic scene variations, yielding better generalizability over our leading baseline method, RT-1
4.3 Discussion on Closed-loop v.s. Open-loop
Preliminaries. In this section, we conduct the "open-loop" experiments with the same diffusion and policy model, but do not incorporate the feedback mechanism in Algorithm to facilitate adaptive replan and sub-goal transitioning, which is a common practice in previous works [16, 30, 15].
How does closed-loop work compared to open-loop? Figure 7(a) compares the manipulation performance of open- and closed-loop execution. The proposed CLOVER, which adaptively selects sub-goals by assessing the distance between current observations and given sub-goals, demonstrates significant performance improvement of +0.44 average completed task length on CALVIN. Incor- porating adaptive replanning further boosts performance by detecting unreliable visual plans and preventing error propagation. Notably, open-loop roll-out performance hinges on understanding the training specifics of both the planner and executor. Synchronizing the time interval for sub-goal transition with the frame intervals used during diffusion model training (Ai = 5 in our experiments) is needed for optimal performance. Figure 7(a) illustrates the performance deterioration when these
8

---

Slide down the switch
Open the drawer
Close the slider by pushing left
Pick up block on the table
Figure 6: Generated videos of diverse tasks conditioned on the same initial frame. CLOVER can generate precise visual plans corresponding to the tasks, facilitating low-level executor guidance. We downsample the video by 2 and exclude depth results in visualizations for simplicity.
3.53
3.45
3.01
Pulative Rug k.
Nug.
R.
AUg
2.3
2.4
Open-loap
2.0
VC-1
3.5
closed-loop
Closed-loop Rep
15
VIT (trom scratch)
2.
Time interval for transition
kumber at Steps
sttance Threshord
[a] Closed-oop OperHcop
(b) Distribution of action steps
[c] Robustness distance threshold
Figure 7: Analysis and comparisons on closed-loop and open-loop roll-out on CALVIN. (a) Comparative analysis of performance (Avg. Len.) through varying step lengths in open-loop control. Evaluations are conducted using identical models but employing different roll-out strategies. (b) The distribution of action steps taken in closed-loop roll-out to achieve each sub-goal. (c) Examination of the robustness of closed-loop control employing various visual encoders and distance thresholds.
settings not properly aligned. Its distribution pattern aligns with the closed-loop roll-out step distribution in Figure 7(b), supporting the necessity of adaptive steps for optimal performance.
How generalizable is our feedback mecha- nism? We investigate the effect of different visual encoders across varying distance thresh- olds (Ds in Algorithm used in policy models, as depicted in Figure 7(c). A larger Ds indi- cates a higher error tolerance.
Table 4: Error measurments with different rep- resentations. Our method shows exceptional cross-tasks robustness on CALVIN benchmark.
Task completed in row (%)
1
Avg.
Method
can be seen that there is consistent pattern across different distance thresholds for all encoders, with peak performance observed at Ds 0.02. Adopt- ing VC-1 [56] demonstrates the highest perfor-
4
Len.
CLIP [23]
72.4
46.8
25.0
13.7
5.1
1.63
LIV [55]
70.8
48.2
29.2
18.2
10.2
1.77
CLOVER
96.0
83.5
70.8
57.5
45.4
3,53
mance, while training ViT-Base [41] encoder from scratch yields exceptional stability, with the lowest result being 3.19. The results manifest the robustness of our feedback mechanism which is independent of specific encoders and does not require customized hyperparameters for different policy models. We compare the robustness of our error measurement scheme against other representations. specifically the dense reward learning framework LIV [55] and CLIP features
[23]. We maintain the same model for both the visual planner and the low-level policy, with the exception of the measure- ments used to determine sub-goal transitions and replanning. Our findings (as in Figure 3) indicate that CLIP features and LIV exhibit a narrower range of values, prompting us to set Ds to 2e 3. The results presented in Table show that unreliable measurements can lead performance that is even worse than in the open-loop setting. Moreover, we do not incorporate additional contrastive objectives in LIV, but investigate the inherent properties of the inverse dynamics- based policy.
4.4 Ablation Studies
Ablations on the diffusion model (visual planner). We conduct a comparative analysis of the high-level plan (video generation) quality. Results presented Table 5 reveal that our method outperforms AVDC [31] across all video generation metrics [57], both of which are built upon the

---

Table Comparison on video prediction. With optical flow-based regularization and architectural modifications, CLOVER generates videos of higher quality and more accurately. t: Reproduced.
Method
SSIM
PSNR f
LPIPS
FID 4
RMSE (Depth) 4
Avg. Len. (CALVIN)
AVDCT [31]
0.837
20.76
0,086
12.74
1.42
CLOVER (wlo Flow Reg.)
0.848
21.42
0.076
12.38
0.084
3.26
CLOVER (Ours)
0.858
22.19
0.062
12.00
0.063
3.53
120
100
100
Success Rate (85)
80-
Success Rate (M
20+
60
Avg. Lon.
20
Avg Lm
Success
Aod
40
100
D4. K40 bm
TM0
T
D-l-2
No. -00
Wio Depth
Deform Altn. Convitaoed
Lac
0wg
LCBCC
0urs
Cross Att
10 15
20
30
[a] Error Measuring
(b) Multimodal Fusion
(c) Robusinees to sampling steps
Figure 8: Ablations on the policy model: (a) error measuring mechanism and (b) multimodal fusion module, with discussions on (c) robustness to different sampling steps when generating sub-goals. We report success rates and the average length of completed tasks (divided by dash lines in each plot).
Imagen framework [33], alleviating the learning and generalization burden on the policy model. Notably, AVDC struggles to generate visual plans consistent with the task description, resulting in significant performance degradation on the CALVIN benchmark. Additionally, the optical flow- based regularization not only brings a generation quality improvement across all aspects, but also significantly accelerates the training convergence. Please refer to Appendix C for further analysis.
Ablations on the policy model. Besides the ablation of feedback mechanisms in Section 4.3, we additionally evaluate the performance of CLOVER using various error measurement approaches, illustrated Figure 8(a). Policies learned with behavior cloning (BC, Figure 3(b)) and LCBC [58] serve as baselines that do not employ error measurement, while CLOVER exceeds them by notable margin. Figure 8(b) demonstrates that incorporating geometry information from depth data leads to a not trivial improvement on CALVIN, with convolution- based multimodal fusion modules achieving the best performance. We also examine the robustness of our policy to generation quality by varying the sampling steps of the diffusion model. As shown in Figure 8(c), increasing the sampling steps generally provides generated videos with more details but shows diminishing returns. We set the sampling step 20 to strike balance between performance and efficiency.
5 Conclusion
In this paper, we present a generalized closed-loop visuomotor control framework, termed CLOVER. It comprises visual planner that specifies desired sub-goals, a policy that executes actions as planned, and feedback-driven control strategy to realize long- long-horizon robotic tasks. CLOVER excels in both simulation and real-world applications, showcasing the virtue of our feedback mechanism.
Limitation and future works. We validate CLOVER for simulation and real-world experiments by training the models heavily on the corresponding data. However, emerging evidence suggests both the diffusion models and IDM-based policies exhibit out-of-distributio generalizability [59, 60, 61]. Visual planner can be trained with actionless videos, and IDM can be learned data data-efficiently with random actions with corresponding observations. This points to the potential of our framework for performing few-shot and long-horizon manipulations by pre-training on web-scale datasets.
Acknowledgments
We would like to thank Jiazhi Yang, Chonghao Sima, Shenyuan Gao and all the members in Open eLab for fruitful discussions and support. We thank the anonymous reviewers for their insightful feed- back. Our hardware used in real-world experiments is partially supported by DISCOVER Robotics. This work was further supported by National Key R&D Program of China (2022/D016)) 60201), NSFC (62206172), Shanghai Committee of Science and Technology (23YFL462000), HK RIF (R7030-22),
10

---

HK ITF (GHP/169/00SZ) the Huawei Flagship Research Grants in 2021 and 2023, the HKU- SCF FinTech Academy R&D Funding Schemes in 2021 and 2022. Hong Kong RGC GRF (HKU 17208223).
References
[1] Siddharth Karamcheti, Suraj Nair, Annie S. Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Perey Liang. Language-driven representation learning for robotics. In RSS, 2023. 1
[2] Jia Zeng. Qingwen Bu, Bangjun Wang, Wenke Xia, Li Chen, Hao Dong, Haoming Song, Dong Wang. Di Hu, Ping Luo, Heming Cui, Bin Zhao, Xuelong Li, Yu Qiao, and Hongyang Li. Learning manipulation by predicting interaction. arXiv preprint vrXiv22406.00439 2024.
[3] Sudeep Dasari, Mohan Kumar Srirama, Unnat Jain, and Abhinav Gupta. An unbiased look at datasets for visuo-motor pre-training. In CoRL, 2023. 1
[4] Yufei Wang. Zhou Xian, Feng Chen, "sun-Hsuan Wang, Yian Wang, Katerina Fragkiadaki. Zackory Erickson, David Held, and Chuang Gan. RoboGen: Towards unleashing infinite data for automated robot learning via generative simulation. arXiv preprint 1rXxv:2311.01455: 2023.
[5] Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang. Jifeng Dai, Qiao, and Ping Luo. EmbodiedGPT: Vision-language pre-training via embodied chain thought. In NeurIPS, 2023.
[6] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. RT-2: Vssion-lagguage-cttinn models transfer web knowledge to robotic control. arXiv preprint arXiv:2307. 158188 2023.
[7] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikow, Gabriel Marto, Mai Gimenez, Yury Sulsky, Jackie Kay. Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.0617 06755 2022. 1
[8] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to Control: Learning behaviors by latent imagination. arXiv preprint
2019. 1,5
[9] Tuomas Haarnoja, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, and Sergey Levine. Compos- able deep reinforcement learning for robotic manipulation. In ICRA, 2018.
[10] Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej. Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino. Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv: /ttl::///0/ 1910.07/13. 2019. 1
[11] Yafei Hu, Quanting Xie, Vidhi Jain, Jonathan Francis, Jay Patrikar, Nikhil Keetha, Seungehan Kim, Yaqi Xie, Tianyi Zhang. Zhibo Zhao, et al. Toward general-purpont robots via foundation models: A survey and meta-analysis. arXiv preprint arXiv:2312.08782, 2023.
[12] Dingzhe Li, Yixiang Jin, Hongze Yu, Jun Shi, Xiaoshuai Hao, Peng Hao, Huaping Liu, Fuchun Sun, Bin Fang, et al. What foundation models can bring for robot learning in manipulation: survey, arXiv preprint arXiv:2404. 18201, 2024.
[13] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, et al. PaL.--: An embodied multimodal language model. In ICML, 2023.
[14] Linui Wang, Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, and Xiaolong Wang. GenSim: Generating robotic simulation tasks via large language models. In ICLR, 2024. 1
[15] Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text- lext-guided video generation. In NewrIPS, 2023. 1, 7, 8, 17
[16] Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Rich Walke, Chelsea Finn. Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pre-trained image-editing diffusion models. In ICLR, 2024. 3, 4, 6, 7,8 8
[17] Sherry Yang. Yilun Du Seyed Kamyar Seyed Ghasemipour, Jonathan Tompson, Leslie Pack Kaelbling. Dale Schuurmans, and Pieter Abbeel. Leamingg interactive woll-world simulators. In ICLR, 2024. 1,

---

[18] Wenlong Huang. Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence. Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner Monologue: Embodied reasoning through planning with language models. In CoRL., 2022. 1.3
[19] Kuo C. and Golnaraghi M E. Automatic control systems. Prentice hall, 1995.
[20] Frederik Ebert, Sudeep Dasari, Alex X Lee, Sergey Levine, and Chelsea Finn. Robustness via retrying: Closed-loop robotic manipulation with self-supervised learning. In CoRL, 2018. 2,3
[21] Connor Schenck and Dieter Fox. Visual closed-loop control for pouring liquids. In ICRA, 2017. 2,3
[22] Annie S Chen, Suraj Nair, and Chelsea Finn. Learning generalizable robotic reward functions from" in-the-wild" human videns. arXiv preprint arXiv:2113.. 168/7, 2021.
[23] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. 2, 4, 9, 17, 19, 21
[24] Paul Christiano, Zain Shah, Igor Mordatch, Jonas Schneider, Trevor Blackwell, Joshua Tobin, Pieter Abbeel, and Wojeiech Zaremba. Transfer from simulation real world through learning deep inverse dynamics model. arXiv preprint orXiv:1610.03518, 2016. 2
[25] Brijen Thananjeyan, Ashwin Balakrishna, Ugo Rosolia, Felix Li, Rowan McAllister, Joseph Gonza- Icz, Sergey Levine, Francesco Borrelli, and Ken Goldberg. Safcty augmented value estimation from demonstrations (saved); Safe deep model-based rl for sparse cost robotic tasks. RA-L, 2020. 3
[26] Zackory Erickson, Henry M Clever, Greg Turk, C Karen Liu, and Charles C Kemp. Deep haptic model predictive control for robot-assisted dressing. ICRA, 2018.
[27] Changyeob Shin, Peter Walker Ferguson, Sahba Aghajani Pedram, Ma, Erik P Dutson, and Jacob Rosen. Autonomous tissue manipulation via surgical robot using learning based model predictive control. In ICRA, 2019. 3
[28] Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In ICRA, 2017. 3
[29] Monroe Kennedy, Kendall Queen, Dinesh Thakur, Kostas Daniilidis, and Vijay Kumar. Precise dispensing of liquids using visual feedback. In IROS, 2017.
[30] Anurag Ajay, Seungwook Han, Yilun Du Shuang Li, Abhi Gupta, Tommi Jaakkola, Josh Tenenbaum. Leslie Kaelbling, Akash Srivastava, and Pulkit Agrawal. Compositional foundation models for hierarchical planning. In NeurlPS, 2023. 3,4, 8
[31] Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B. Tenenbaum. Learning to Act from Actionless Videos through Dense Correspondences. In ICLR, 2024. 3, 9, 10, 17,21
[32] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-A- Video: Text-tovvidco generation without text-video data. In ICLR, 2023. 3
[33] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-to- image diffusion models with deep language understanding. NeurIPS, 2022. 3, 4, 10, 17
[34] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06122 2022. 4
[35] Jonathan Ho and Tim Salimans. Classifier-fie free diffusion guidance. arXiv preprint arXiv:2207.1259% 2022.
[36] David Kent. Carl Saldanha, and Sonia Chernova. Leveraging depth data in remote robot teleoperation interfaces general object manipulation. IJRR, 2020.
[37] Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen. Pretraining is all you need for image-to-image translation. arXiv preprint arXiv:2205. /2952, 2022. 4
[38] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
12

---

[39] Abhishek Padalkar, Acom Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan. Alexander Khazatsky, Anant Rai, Anikait Singh. Anthony Brohan, et al. Open X-Embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.0886. 2023. 4
[40] Zachary Teed and Jia Deng. RAFT: Recurrent all all-pairs field transforms for optical flow. In ECCV, 2020.
4, 17
[41] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021. 5, 9. 17, 19
[42] Jie Hu, Li Shen, and Gang Sun. Squeeze-andeexciaation networks. In ICCV, 2018. 5, 17
[43] Juho Lee, Yoonho Lee. Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set Transformer:
A framework for attention-based crmmutation-invariinn neural networks. In ICML, 2019. 5
[44] Kangbeom Cheon, Jachoon Kim, Moussa Hamadache, and Dongik Lee. On replacing pid controller with deep learning controller for de motor system. JOACE, 2015.
[45] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell, Curiosity-driven exploration by self-supervised prediction. In ICML, 2017.
[46] Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. In RSS, 2021. 7
[47] Oier Mees, Lukas Hermann, and Wolfram Burgard. What matters in language conditioned rohotic imitation learning over unstructured data. RA-I,, 2022.
[48] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, al. RT-1: Robotics transformer for real- world control at scale. arXiv preprint arXiv:2212.06817, 2022. 7, 8
[49] Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing. Weinan Zhang, Huaping Liu, Hang Li, and Tao Kong. Vision-language foundation models effective robot imitators. In ICLR, 2024. 7
[50] Hongtao Wa, Ya Jing, Chilam Chcang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. In ICLR, 2024.7 7
[51] Tsung- Wei Ke, Nikolaos Gkanatsios, and Katerina Fragkiadaki. 3D Diffuser Actor: Policy diffusion with scene representations. arXiv preprint arXiv:2402.10885: 2024. 7
[52] Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipula- tion with low-cost hardware. In RSS, 2023. 7,8,2 2,
[53] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsca Finn, and Abhinav Gupta. R3M: A universal visual representation for robot manipulation. In CoRL, 2022. 7, 8, 21
[54] Oier Mees, Lukas Hermann, Erick Cosete-Beas, and Wolfram Burgard. CALVIN: A benchmark for language-conditioned policy learning for oong-horizon robot manipulation tasks. RA-L, 2022. 6. 7, 16, 18, 21
[55] Yecheng Jason Ma, Vikash Kumar, Amy Zhang, Oshert Bastani, and Dinesh Jayaraman. LIV: Language- image representations and rewards for robotic control. In ICMI,, 2023. 9, 16
[56] Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Jason Ma, Claire Chen, Sncha Silwal, Aryan Jain, Vincent-Piene Berges, Tingfan Wu, Jay Vakil, et al. Where are we in the search for an artificial visual cortex for embodied intelligence? In NeurIPS, 2023. 9, 17, 21
[57] Qiang Fan, Wang Luo, Yuan Xia, Guozhi Li, and Daojing He. Metrics and methods of video quality assessment: a brief review. Multimedia Tools and Applications, 2019. 9
[58] Homer Rich Walke, Kevin Black, Tony 7. Zhan, Quan Vuong, Chongyi Zheng, Philippe Hansen-Fstruckt Andre Wang Hc, Vivek Myers, Moo Jin Kim, Max Du, et al. BridgeData v2: A dataset for rohot learning at scale. In CoRI, 2023. 10
[59] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable Video Diffusion: Scaling latent video diffusion models to large datasets, arXiv preprint 1511722 2023. 10, 17
13

---

[60] David Brandfonbrener, Ofir Nachum, and Joan Bruna. Inverse dynamics pretraining learns good represen-
tations for multitask imitation. In NeurIPS, 2024. 10
[61] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion Policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.041377 2023.
10
[62] Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.0372: 037488 2018.
[63] Jiazhi Yang. Shenyuan Gao, Yihang Qiu, Li Chen, Tianyu Li, Bo Dai, Kashyap Chitta, Penghao Wu, Jia Zeng, Ping Luo, Generalized predictive model for autonomous driving. In CVPR, 2024. 17
[64] Lin Sun, Kui Jia, Yan Yeung, and Bertram E Shi. Human action recognition using factorized spatio- temporal convolutional networks. In ICCV, 2015. 17
[65] Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechuman, Ersin Yumer, and Ming-Hsuun Yang. Learning blind video temporal consistency. In ECCV, 2018. 17
[66] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez. Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without supervision. TMLR, 2024. 17, 19, 21
[67] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. In NewrlPS, 2017. 18
[68] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
18
[69] Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. In ICCV, 2023. 18
[70] Jiaming Song, Chenlin Meng, Stefano Ermon, Denoising diffusion implicit models. ICLR, 2021. 18
14

---

Supplementary
A Examples of Test-time Execution with Error Measurement
In this supplemental section, we provide examples of how our proposed deviation quantification helps the autonomous test-time execution. In particular, as mentioned in Section 3.3, can adopted for replanning when sub-goals are infeasible unrealistic, and transitioning to the next desired state when the robot reaches each sub-goal. In the meantime, we introduce the application of task completion assessment, though not utilized directly for performance comparisons in CALVIN.
Sub-goal replan. We first show how identify unreachable sub- sub-goals mitigate the uncertainty of diffusion based visual planners. Figure 9 gives examples of common cases showing the instability of the diffusion model. We measure the cosine distance between state embeddings of consecutive frames generated by the model. Specifically, consistent and reasonable plans exhibit stable intra- frame distance, typically below 0.2, whereas inconsistencies are characterized by significant increase, often exceeding 1. Therefore, we set the distance threshold for replanning as DR 1.0 under all circumstances, which proves to work effectively in practice. The distinctive variation verifies the sensitivity and virtuc of the measuring ability of state embeddings. By adaptively detecting erroncous plans and reinitializing generation, we prevent error propagation to the subsequent policy.
Case 1: Unreachablity Detection
Genercled Subedaaccs
CD.. Distance
Threahold
Sub-goa Indek
Case 2: Inconsisencyy Delection
Generoled Sub-pool
COs, Distance
Threshold
Sub-goa Index
Figure Automatic identification of unreliable sub-goals generated. We set the distance threshold for replanning Dn 1.0 under all circumstances. Ideally, the distances between generated frames remain consistent and relatively small, whereas significant variations occur during unstable generation. Examining the distance within two adjacent frames, we detect erroneous plans generated by the diffusion model before passing them to the subsequent policy model for execution.
Sub-goal transition. Distinguished from previous works that adopt fixed action steps mentioned in Section 4.3, our framework incorporates predefined distance threshold Dg (refer Algorithm to facilitate adaptive transitioning between sub-goals upon achievement. In Figure 10, we illustrate the distance variation between the current observation and selected goals during the roll-out process. In our experimental setup, CLOVER dynamically adjusts the number of steps for approaching each sub- goal, ranging from threshold Ds. Notably, the distribution of generated goals in the latent space is uneven, and the policy does not adhere to a fixed speed in reaching targets. This evident from the varying starting distances and changing slopes of the approach process for each sub-goal in Figure 10. These observations further underscore the necessity feedhack mechanism to enable the adaptive sub-goal transition to mitigate error accumulation.
7, and transitions to the next sub-goat once the distance falls below the
15

---

0.10
Distance goals
0.08
0.06
0.04
Steps
Figure 10: Adaptive sub-goal transitions. The cosine distance between the current observation and the selected sub-goals is plotted, with dashed gray lines indicating the transitions between sub-go sub-goals. The distance threshold Dy for sub-goal transitioning each assigned sub-gou aob-goal and minimizes errors through an adaptive number of action steps.
0.02. Our policy effectively reaches
Involve the switch turn the bulb
imagle squencee
task comp on
2.0.8
Place the red block in the drawer
Task Image squenne
0.8
Sanitur
0.6
Open the drawer
compltton
Sintaitt
0.6
Figure 11: Task completion score given by our value function. As the robot approaches each sub- goal and eventually completes the task described in the text, the learned value function monotonically increments, which could be helpful for task completion assessment.
Task completion assessment. All previous methods follow the standard evaluation protocol of CALVIN [54], where the simulator updates the observation after each action roll-out and returns signal marking the completion status the current task. The long- long-horizon manipulation tasks in CALVIN consist of sequence of five consecutive sub-tasks. Consequently, the completion status of the current sub-task serves as a valuable signal for policies to progress to the subsequent one. However, such signals are not accessible in real-world environments.
To fully exploit the advantageous properties of our state embedding as introduced in Section 3.2, we also explore using them to facilitate end-of-task assessment. Since we model latent actions for each step with deviation of current and sub-goal embeddings, it natural to exploit the deviation between the embeddings of the task's 's initial and final states to represent the latent action for the entire task. Inspired by LIV [55], we introduce a contrastive objective for language and state embeddings to establish a dense (value) reward, where a high reward for the current state represents approaching
16

---

status of the entire task. Specifically, we use the subtraction of the randomly sampled intermediate state and initial state embeddings as the negative example, while the subtraction of final and initial embeddings as the positive example. We then compute the InfoNCE loss [62] with the encoded text goal to align image and language goals. Intuitively, a current state closing to the ground-truth final state resembles the overall text descriptions. Notably, the training process is built upon the pretrained policy and text encoders, with only a lightweight projector trained to align modalities. Figure presents examples of using this reward design to assess task's completion status, Such characteristic demonstrates its potential as a task completion judge.
Denoising UNet
Attn. Block
Depth Maps
Spotial Self-attention
Residurl Block
Attention Block
Residual Block
Attention Biock
Res Block
Attm. Block
Ros. Blocks
Attn. Blocks
Concet &
Likes block
Add Nose
Cousual Temporal
Attention
Longuage
Conditoning
RGB Imoges
Correspondonce
Text Allenlion
Optical-flow
ContexINct in RAFT
Context
Estimation
Figure 12: The architecture of our visual planner. We augment the original UNet proposed in Imagen [33] with casual temporal attention to improve intra-frame consistency and additional cross-attention-basee language conditioning. Combining lightweight ContextNet introduced in RAFT [40], we estimate optical flows with a correspondence map of diffusion latent embeddings.
B Implementation Details
B.1 Model Architecture
Visual planner. Figure 12 depicts the detailed architecture of our visual planner. Following [15, 3 11], we adopt video diffusion model to generate synthesized sub-goals. Considering diffusion models are generally computationally expensive, we down-scale the model size to achieve a balance between the efficiency for robot manipulation tasks and high fidelity for video generation.
The visual planner needs to generate temporally coherent and consistent videos (sub-goals) to guide the subsequent executor. In line with previous studies on video generation [59, 31, 63], we concatenate the input condition frame Oo to all the future frames 0I.K to ensure coherency. Furthermore, to foster better temporal consistency and causal reasoning ability, causal temporal attention is adopted to encourage the full exploitation of historical information for predicting future interactions. Combining this with the factorized temiralll convolution [64] in AVDC [31]. our visual planner can faithfully reason about temporal causalities with improved training and inference efficiency.
To perform optical flow estimation F with diffusion latents we employ an additional context network to encode context information from the k-th frame following RAFT [40]. In contrast to utilizing an external flow estimation model directly [31, 65), our approach incurs a minimal parameter overhead of 0.7M during training, with the optical flow estimation component being discarded during test-time control.
(redback-driverr policy. We leverage the off-the-shelf pre-trained visual encoders (i.e., VC-1 [56]. CLIP [23], and DINO [66]) to imbue the training process with visual priors. As for depth, we intuitively adopt a transformer based encoder (ViT-Small [41]) to align mudti-modal information in the latent space. To ensure compatibility with various preitrained visual encoders, the input and patch size of the depth encoder are adapted to be consistent with the visual token length, thereby facilitating the widespread adoption of our policy. For fusing RGB (FRGB E 1 and depth features (F Depth (Rlxd we simply concatenate them on channels followed by a 3x3 convolutional layer to integrate spatial local information and reduce the channel dimension to dRCB. It then followed by Squeeze- -nd-xxitttiion module [42] as channel attention select important fused features.
17

---

0.6
20
0.00
who Flow Reg.
Flow
wio Flow Reg.
w/ Flow Reg.
w/ Flow Reg.
Fow Reg.
0,090
RMSE (Depth)
0.070
0.6
0.7 10k
100k
200c
10
50c
100K
2006
0.000 10c
50c
10ck
200k
Training Steps
Training Steps
Training Steps
[a] SSIM
[b] RDD
(c) RMSE [Depthl
Figure 13: Quantitative performance of video diffusion model with and without the flow-based regularization term. The optical flow-based regularization endows the video diffusion model with more efficient training convergence and notable performance improvement on CALVIN.
For the subsequent token aggregator, we set the number of attention heads [67] to 12 in the multi-head attention module, and multiply the channel width by 4 in the MLP to enhance the representation capacity. The action decoding head is three-layer MLP with ReLU as the activation function.
B.2 Training Protocol
Visual planner. Our diffusion model-based planner can be factorized as Po Op, qi), with o presenting the condition given by the language. During the training process, it acts as a denoising function é predicting noises applied on future video frames OI.K [68]. Given the noise scheduling Be, the training objective of the diffusion model is:
Laii Σ le lle-calvi- Be Oz VB (e..el
where the noise e E {ERGB Deppth } is drawn from a multivariate standard Gaussian distribution, and t represents randomly selected diffusion step. Specifically, sample noises separately for RGB and depth from two independent distributions. We further adopt the min-SNR weighting strategy [69] to speed up convergence. Combining the flow-based regularization term in Section 3.1, the final optimization objective of the visual planner can be formulated as:
Lplanerr = Laift + XLreg
(4)
where À is a balancing factor and is set 0.1 by default. In our experiments on the CALVIN [54] benchmark, we train the diffusion model for 300k iterations with learning rate of le-4, Models are trained on a system equipped with A100 GPUs with the batch size set as 32. We adopt the A optimizer without weight decay. Besides, we track an exponential moving average (EMA) of the model parameters with a decay rate of 0.999 and use the EMA parameters at test time. For real-world experiments, tune the diffusion model for 50,000 iterations on 50 collected demonstrations. Due to hardware limitations, we are not able to collect depth data in a real environment, so the model generates RGB images only.
For test-time execution, the DDIM sampler [70] employed, with 20 sampling steps to strike a balance between efficiency and quality. The text guidance weight is set to 4 for generating visual plans that align with linguistic descriptions.
policy. To optimize the policy model, we leverage mean squared error and binary cross-entropy loss to supervise the end-effector's position dgF E 6 and gripper state agripper R respectively. In each training episode, two frames with an interval ranging from 1 to kmmx = 5 are sampled as inputs
enhance the model's robustness. We train the policy on ABC training split of CALVIN for 10 epochs with batch size of 128. Only the relative cartesian action of a single timestamp is used for training. The training process takes around 10 hours on 8 A100 GPUs.
C Extended Ablation Studies
Optical flow based regularization. We investigate the effect of optical flow-based regularization in model convergence, besides the results reported in Table 5. As depicted in Figure 13, our analysis
18

---

Table 6: Modeling Inverse Dynamics with different visual encoders. CLVOER achieves promising
results on CALVIN with different visual encoders of varying sizes. " Trained from scratch.
Task completed in row (%)
f
Encoders
Params.
Avg. Len. 1
2
3
4
5
VIT-S" [41]
22M
96.0
80.2
64.4
52.6
41.4
3.35
VIT-B" [41]
86M
96.6
83,8
67.6
55.0
43.2
3.46
CLIP [23]
86M
94.0
81.2
68.0
56.2
43,4
3.43
DINOv2 [66]
307M
94.8
80.4
67.0
55.4
43.8
3.41
VC-1 [56]
86M
96.0
83.5
70.8
57.5
45.4
3.53
Figure 14: Visualization on attention maps of the policy model. The policy model demonstrates the ability to direct attention towards the end-effector and the object it interacts with, without the need for explicit supervision during the learning process.
illustrates the impact on model performance across different training phases. The regularization significantly boosts the model's convergence, evidenced by consistently improved metrics. Notably, the visual planner trained with 10k iterations outperforms its 100k iterations counterpart in terms of SSIM and RMSE. Furthermore, the final model, trained over 200k iterations, demonstrates superior video quality with 0.38 decrease in FID and 0.02 in RMSE. By leveraging the pixel-space correlation regularize the diffusion model's latent space representations, the regularization term encourages the model to focus movement and interaction dynamics, leading to improved convergence and performance on manipulation videos.
Generalization to different visual encoders. In Table 6, we report detailed statistics regarding experiments of different visual encoders for our feedback mechanism Figure 7(c), under the default distance threshold setting. Notably, training a 22M parameters ViT-Small model [41] from scratch already yields satisfactory results. VC-1 [56], which is trained tons of roboties data, provides the policy with a better visual prior and brings the best performance. In general, we hypothesize that the performance bottleneck of the framework still lies in the visual planner, while IDM-based policies are easier to learn and generalize.
D Extended Visualizations
Attention maps of our policy. As discussed in Section 3.3, the state embeddings extracted by the token aggregator distribute reasonably the latent space and exhibit decent measurement capabilities. We plot attention maps that show the tokens chosen by the token aggregator of the policy model in Figure 14 to provide additional explanation for the observation. As visualized, our model focuses on information essential for action execution, specifically the end-effector and the object it interacts with. One thing worth noting is that it is not explicitly supervised with any form of affordance.
Qualitative analysis. We also provide qualitative examples of the generated videos paired with corresponding tasks in both CALVIN and real-world environments, in Figure and Figure 16, respectively. Our proposed visual planner reason feasible sub-goal, with high temporal coherence.
19

---

push right the blue block
Open the drawer
Pick up the pink block from the sliding cabinet
Turn off the yellow light
Figure 15: Visualization on generated RGB-D visual plans. Our model can generate reliable sub-goals with high consistency between RGB and depth.
up the pod lid
Put fish into the pod
Put the on the pod
Figure 16: Visualization on visual plans in real-world environments.
20

---

pieckciad
detaction
(b.1) Visual detraction
Orginshiiiigg y
Ac-llo
Object vallation
la] Expertmentsseiihn
Object variatiai
Figure 17: The predicted visual plan and actual robot execution in real-world experiments. Our diffusion- based visual planner (up row in each case) can generate reasonable plans new scenarios to facilitate successful manipulation (bottom row in each case).
E Broader Impact
CLOVER contributes to the field of robotics by addressing the limitations of open-loop systems and providing simple yet robust baseline for closed-loop control. This advancement can inspire further research into adaptive control strategies, error modeling, and real-time feedback mechanisms, pushing the boundaries of robot learning and embodied intelligence. All our models trained on publicly available data that is free of private and sensitive information.
F License of Assets
CALVIN [54] is open-source simulator which is under the MIT License. The reimplemented methods for performance comparison including ACT [52], R3M [53], and AVDC [31] are all under the MIT License. The pretrained visual encoders adopted, i.e.. VC-1 [56], CLIP [23], and DINO [66]), are under the CC BY-NC-SA 3.0 US license, MIT license, and Apache License 2.0, respectively.
Our source code and trained models will be publicly available under Apache License 2.0.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the claims to reflect the contributions and scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer:
[NA]
Justification:
This paper does not include theory assumptions or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed implementation parameters and architectures in Sec- tion 4.1, Appendix B.I, and Appendix B.2. We will also open-source the code and models for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
23

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We will release the code and models.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss cttpssccccpbccccccceececcecccccc /ttpe/guuddeeeeeeuubiiiiiaeeee /Cooeeeooiissseeeeeee
cessionPoocyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
See Section 4.1 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The success rates of our algorithm in simulated environments and real-world robotic experiments are obtained after multiple trials as specified Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide experimental training details in Appendix B.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that wwwpuurpps.cpppulllpeeeee cctppblic//eiieeeececccccc ctwtee/ehhhhnheeellnnennnnnee
didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https : http///eerripp.epe
Answer: [Yes]
Justification: The authors have thoroughly reviewed and conformed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See discussions in Appendix E.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited the corresponding papers throughout our paper and specified their licenses in Appendix F.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We will release code and models under Apache License 2.0 (Appendix F),
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "PN7Zw4YAaP", "reviewer": "Reviewer_hfKW", "summary": "The authorsThe authors introduce CLOVER, a generalizable closed-loop visuomotor control framework that incorporates a feedback mechanism to improve adaptive robotic control. The method uses a text-conditioned video diffusion model to generate reference visual subgoals, an error measurement to quantify the difference between the current state and the planned sub-goal, and a feedback-driven controller via inverse dynamics model to ourput actions. Experiments on CAlVIN benchmark and ALOHA real robot shows the method outperforms all previous methods by a notable margin.", "questions": "1. How is your work different from AVDC?\n2. Could the authors elaborate more on the superiority of their design which use diffusion model generate reference frames and then use inverse dynamics model to get the actions, over the design like diffusion policy[60] which directly generate actions from diffusion model with a closed-loop style?\n3. What is the orange dash line meaning in Figure 6(a)?\n4. It seems author defines sub-goals as unreachable when the cosine distance between the state embeddings of consecutive are too far. \n5. Would there be circumstance when the embedding is close but the sub-goal is actually not reachable due to singularity or other robot limit? Especially when testing the framework on real robot?", "limitations": "Did not test on generalization ability.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The problem of using closed-loop visuomotor control to solve long-horizon tasks is very meaningful.\n2. The motivation and storyline is stated clearly. The paper is well-written and easy to follow.\n3. The experiments on both the simulation and the real robot are convincing.\n4. The ablation study of visual embedding, optical flow regularization, error measuring, multi modal fusion and sampling rate are very sufficient.", "weaknesses": "See questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful review and we really appreciate your comments. We address your questions below.\n\n>*${\\color{BrickRed}Question 1:}$* How are you different from AVDC?\n\nOur work differentiates from AVDC in the following aspects:\n- **Model structure:** For the visual planning part, we introduce a novel constraint term to enhance its temporal consistency and endow the model with additional depth generation ability.\n- **Action output mechanisms:** As discussed in the related work, AVDC infers actions from predicted video content with dense correspondence. It combines an off-the-shelf optical flow estimator with depth information to compute SE(3) transformations of the end-effector and transfer it into action control signals. However, CLOVER learns an inverse dynamic model to output actions and adaptively reach given goals. Paired with error measurement and feedback mechanism, CLOVER is more robust to the inherent instability of video diffusion models and can be adapted to broader scenarios.\n- **Feedback policy:** The AVDC framework lacks a feedback mechanism within its visuomotor control system. In contrast, our work aims to explicitly quantify errors and incorporate them into a unified framework for long-horizon manipulation.\n\nWe have added the above discussion to the revision.\n\n>*${\\color{BrickRed}Question 2:}$* Could the authors elaborate more on the superiority of their design which use diffusion model generate reference frames and then use inverse dynamics model to get the actions, over the design like diffusion policy which directly generate actions from diffusion model with a closed-loop style? \n\nThanks for the question. Our decoupled paradigm has two main benefits:\n- As mentioned in Section 4.2, our diffusion-based video model can effectively understand high-level instructions and generate corresponding plans. This alleviates the learning complexity of policy by decoupling the planning and control into a two-level hierarchy, where the low-level policy can be trained in a task-agnostic manner (no need for high-level instruction labels).\n- The inverse dynamic model (IDM) can be established based on the generated sub-goals, which maps the state transitions to actions. In contrast, diffusion policy uses the current observation only following the paradigm of behavior cloning (BC). Previous work [59] proves that IDM could be more performant and robust than BC,  and scales best with the pretraining dataset.\n\nWith the above benefits, we show the stronger performance of our method in Table 1, compared to the diffusion policy-based model \"3D diffusion actor\" [52], even though it utilizes more sensor input.\n\n>*${\\color{BrickRed}Question 3:}$* What is the orange dash line meaning in Figure 6(a)? \n\nWe intend to highlight the highest performance achieved by open-loop methods for clearer comparison to closed-loop counterparts. We have revised the subfigure.\n\n>*${\\color{BrickRed}Question 4:}$* It seems author defines sub-goals as unreachable when the cosine distance between the state embeddings of consecutive are too far. Would there be circumstances when the embedding is close but the sub-goal is actually not reachable due to singularity or other robot limits? Especially when testing the framework on real robots?\n\nThanks for the insightful question. Injecting physical constraints or considering potential robot limitations in video generation models (world models) remains a longstanding challenge in visual plan generation. Even models like Sora, which are trained with extensive data and computational resources, have been found to fail in fully adhering to physical laws. However, the reliability of visual planners can be significantly improved by training on large-scale, real-world robot demonstrations that are inherently physically plausible. The generalizability of such models to unseen embodiments during the training phase remains an area that requires further exploration.\n\nIn practice, the lack of physical feasibility in visual planning can result in scenarios where a robotic arm stops at a certain position for an extended period without the error reaching the predefined threshold. To empirically enhance the reliability of visual planning, we can implement human-defined rules to detect these conditions and trigger replanning across multiple rounds of generation, thereby addressing this limitation.\n\n>*${\\color{BrickRed}Question 5:}$* Did not test on generalization ability.\n\nThank you for the comment. In the current draft, the benchmark CALVIN ABC-> D itself certifies the scene generalization ability of the learned policy [45]. Specifically, the texture of the table and the position of buttons, drawers, and sliders in the tested environment are different from those in the training set, which challenges the model's generalization in unseen scenarios. In real-world robot experiments, we further validate the position generalization by putting the fish to be grasped at different positions for each rollout.\n\nDuring the rebuttal, we provide *additional* real-world results to verify CLOVER's robustness against background distraction and object variation. Please refer to Fig. R1 and R2 in *submitted PDF* for detailed experiment settings and the results given in the *Global Rebuttal* session Q1(1) and (2).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ny2LW8FHtU", "reviewer": "Reviewer_DjFL", "summary": "The paper presents a novel framework named CLOVER. The proposed system aims to enhance the adaptability and robustness of robotic manipulation in long-horizon tasks by incorporating closed-loop control principles. CLOVER consists of three main components: a text-conditioned video diffusion model for generating visual plans, a measurable embedding space for accurate error quantification, and a feedback-driven controller that refines actions based on real-time feedback. The framework shows significant improvement in real-world robotic tasks and sets a new state-of-the-art performance on the CALVIN benchmark.", "questions": "1. Can the authors provide more detailed training process used for the text-conditioned video diffusion model and the error measurement strategy, including training datas, network architecture?\n2. How does CLOVER handle highly dynamic environments where changes occur rapidly and unpredictably?\n3. How is the training data organized during the diffusion model generation phase of the visual plans? How many intermediate points are selected for each trajectory? I believe this needs to be explained, as it affects the performance of the final trajectory.\n4. What are the computational requirements for running CLOVER in real-time, and how does it scale with more complex tasks?", "limitations": "The authors have addressed some limitations, such as they validate CLOVER for simulation and real-world experiments by training the models heavily on the corresponding data. However, further discussion on the scalability of the framework and its adaptability to a wider range of robotic morphologies and environments would be beneficial.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Originality: CLOVER introduces a unique combination of closed-loop control principles with generative models for robotic manipulation. The use of a text conditioned video diffusion model to generate visual plans is a creative approach that leverages recent advances in generative AI.\n2. Quality: The methodology is well-articulated, detailing the design of each component in the framework. The inclusion of depth map generation and optical flow regularization to enhance the reliability of visual plans demonstrates the challenges in robotic manipulation.\n3. Clarity: The paper is clearly written, with a logical structure that guides the reader through the motivation, methodology, and experimental validation. Figures and diagrams are effectively used to illustrate complex concepts and the overall system architecture.\n4. Significance: The proposed framework addresses a critical challenge in robotics—improving the robustness and adaptability of robotic systems for longhorizon tasks. The notable performance improvements on the CALVIN benchmark and real-world tasks underscore the potential impact of CLOVER on the field of robotic manipulation.", "weaknesses": "1. Novelty of the error measurement approach: While the paper introduces a measurable embedding space for error quantification, it does not provide a detailed comparison with existing methods in terms of novelty and performance. Additional analysis or experiments comparing this approach with other state-of-the-art error measurement techniques would strengthen the contribution.\n2. Clarification of the effect of closed-loop system: The article seems to tackle the long horizon manipulation tasks, but in the article there are no obvious evidences to prove it.\n3. Scalability of the feedback-driven controller: The paper presents the feedback-driven controller as an effective solution for adaptive control. However, it lacks a discussion on the scalability of this approach for more complex and diverse task environments. Evaluating the controller's performance across a wider range of scenarios, particularly dynamic scenarios would provide a better understanding of its generalizability.\n4. Lacked methodology clarification: Some technical aspects, particularly the algorithms and specific implementation details, are not as thoroughly explained as they could be. This makes it harder to fully grasp the intricacies of the proposed system.\n5. Relatively limited experiments: The experiments conducted on the CALVIN dataset and real-world scenarios may not be sufficient to demonstrate the effectiveness of the proposed method. It is recommended to perform additional experiments in more long-horizon simulation environments such as RLBench and robosuite. Additionally, the real-world experiments are relatively limited, so further\nexperiments should be conducted to provide a more comprehensive evaluation.\n6. Computational complexity: The incorporation of a video diffusion model and feedback mechanisms might introduce significant computational overhead. The paper does not provide an in-depth analysis of the computational requirements and how they impact real-time performance, which is crucial for practical deployment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review. We address your questions below.\n\n> *${\\color{BrickRed}Question 1:}$* Novelty of the error measurement approach. Comparison with other state-of-the-art error measurement methods.\n\nAs discussed in Sec. 2, existing works rely on additional detection models with manually set rules or high-cost VLMs to understand tasks' completion state. These impose restrictions on their applicability and efficiency. \n\nCompared to works like LIV [61] which also tends to learn measurable embeddings, we do not incorporate additional contrastive objectives but investigate the inherent property of inverse dynamics-based policy. In the *new* results below, our method shows more robust performance for consecutive tasks. We will update the analysis in our revision.\n\n|Methods|Task 1|Task 2|Task 3|Task 4|Task 5|Avg. Len.|\n|-|-|-|-|-|-|-|\n|LIV [61] (newly added)|70.8|48.2|29.2|18.2|10.2|1.77|\n|State Embedding (Ours, Fig. 3(c))|**96.0**|**83.5**|**70.8**|**57.5**|**45.4**|**3.53**|\n\n> *${\\color{BrickRed}Question 2:}$* Clarification of the effect of closed-loop system (long-horizon manipulation).\n\nEach test rollout in CALVIN consists of five distinct instructions (sub-tasks), which requires the policy to succeed in **chained** sub-tasks. Our framework's improved performance on CALVIN and long-horizon real-world tasks demonstrates its effectiveness in such settings. We acknowledge that current settings are limited compared to tasks like \"making a cuisine\". We will work to develop more complex long-horizon manipulation tasks in the future.\n\n> *${\\color{BrickRed}Question 3:}$* Scalability of the feedback-driven controller. Evaluating the controller's performance across a wider range of scenarios.\n\nWe provide *additional* real-world generalization experiments, including background distraction, object variation, and dynamic scenarios in *Global Rebuttal*.\n\n**Scalability**: As discussed in Secs. 3.1 and 5, by decoupling planning and low-level control into a two-level hierarchy, the visual planner can learn from massive human videos to be robust world simulators [16]; our inverse dynamics-based policy offers greater robustness for multitask IL compared to BC-based policies [59]. Fig. R4 shows that the BC-based method RT-1 struggles with scene variation.\n\n> *${\\color{BrickRed}Question 4:}$* Lacked methodology clarification. Detailed training process used for the text-conditioned video diffusion model and the error measurement strategy.\n\nWe have detailed our model architecture and training protocol in Appendix B1 & B2, and we will publicly release all materials. In the rebuttal, we provide details below to address the question.\n- **Video diffusion model:** Extending Imagen [32], we add input/output channels and separate noise injection for RGB and depth, enabling depth generation. Optical flow-based regularization is introduced using diffusion latent embeddings to create cost volumes, with a lightweight CNN serving as ContextNet, inspired by RAFT [39]. Training data consists of 8 randomly sampled frames with fixed intervals (5 in CALVIN, 20 in real-world experiments).\n- **Feedback-driven policy:** We use MLPs for the action decoder, but Transformers or Diffusion Policy could be alternatives. The policy is trained with the inverse dynamics objective, using frames from random intervals in demonstrations to enhance its robustness.\n\n> *${\\color{BrickRed}Question 5:}$* Relatively limited experiments.\n\nThanks for the advice. RLBench and Robosuite focus on few-shot learning with tasks' horizons comparable to a single subtask in CALVIN. Thus, CALVIN is the most suitable benchmark for validating long-horizon capabilities. We will consider involving additional simulation experiments.\n\n Please also refer to *Global Rebuttal* Q1(3) for additional real-world experiments.\n\n> *${\\color{BrickRed}Question 6:}$* Computational complexity.\n\nWe provide detailed discussion in *Global Rebuttal* Q2.\n\n> *${\\color{BrickRed}Question 7:}$* How does CLOVER handle highly dynamic environments?\n\nMost existing works test in static environments where the interaction object remains stationary. Our work also focuses on similar settings and does not address environmental dynamics. We envision this as an important future direction and have incorporated the discussion below into the revision.\n- **How to adapt to dynamic environments:** CLOVER could be adapted for dynamic settings by using multi-frame conditioning to capture velocity and acceleration information. Besides, incorporating the replanning mechanism (Appendix A) at each inference step could help handle new scenarios.\n- **Additional experiments with unpredictable changes:** In the added tests, we randomly place and pick up a doll to create unpredictable visual changes. Thanks to the robustness of visual misalignment of our inverse dynamics-based policy, CLOVER surpasses RT-1 by a large margin.\n\n> *${\\color{BrickRed}Question 8:}$* How is the training data organized during the diffusion model generation phase of the visual plans?\n\nDuring training, CLOVER extracts 8 frames at five-frame intervals, covering key task segments. This ensures better task alignment and fewer generation rounds during a test (1-2 rounds per task). As shown in Figs. 6(a) and 9, CLOVER effectively achieves sub-goals with adaptive steps and generalizes to different visual planners without specific adjustments. Detailed protocol is in Appendix B2 and will be further clarified.\n\n> *${\\color{BrickRed}Question 9:}$* Computational requirements. How does it scale with more complex tasks?\n\nPlease refer to *Global Rebuttal* for the discussion on computational requirements.\n\nFor more complex tasks, we may introduce LLMs to decompose the tasks into more manageable subtasks similar to HiP [29]. CLOVER can then generate visual plans for simplified tasks and perform real-time low-level control. The computational cost of this system will potentially scale linearly with the number of subtasks derived.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U5iIuQsXBj", "reviewer": "Reviewer_hKyy", "summary": "This paper proposes CLOVER, a closed loop visuomotor control framework that incorporates feedback mechanisms to improve adaptive robotic control. The framework consists of a text-conditioned video diffusion model for generating visual plans as reference inputs, an error measurement module to model the discrepancy between the current and goal states in the embedding space, and a feedback-driven controller that refines actions from feedback and initiates replans as needed.", "questions": "1. In Figure 3, the authors mention is obtained during a roll-out, what task and what environment are you using? How many tasks and environments did you test?\n\n2. How long does it take to execute one task in simulation and real robot?\n\n3. Can you show video demos of your experiments not just screenshots in the paper?", "limitations": "1. High computation required for both training and real-time execution could limit practical applicability. The proposed framework's complexity might pose challenges in implementation. \n\n2. The scalability of the framework to diverse and complex tasks remains uncertain.\n\n3. The real-time execution of the feedback-driven controller might face latency issues, especially in dynamic and unpredictable environments.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written and organized, clear to read.\n\n2. Integrating closed loop feedback into robotic visuomotor control is interesting and useful, which can help deal with deviation and improve control accuracy.\n\n3. Combines visual planning, error measurement, and feedback-driven control in a cohesive system.", "weaknesses": "1. Limited evaluation to real robot environment. The authors only show one experiment comprising three consecutive sub-tasks, which is not enough to test the generalizability and robustness of the approach across different environments and tasks. \n\n2. The measurable embedding space for error quantification seems effective not be sufficiently validated. This component’s effectiveness could vary significantly in different scenarios, and the paper does not provide extensive evidence of its robustness.\n\n3. The framework seems need a lot of computation, each step in the loop the diffusion model needs to generate a video, then get embeddings through two ViT-based encoders for RGB and depth respectively. I guess the high computation required could limit its real-time execution and practical applicability.\n\n4. Further validation on a wider range of benchmarks and task environments is necessary to confirm the framework's generalizability.\n\n5. Minor, in Figure 5, the authors show generated videos of four tasks conditioned on the same initial frame, but the first figure in each task looks different, better to make them the same.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful review and valuable comments. We address each question below.\n\n>*${\\color{BrickRed}Question 1:}$* Limited evaluation to real robot environment. \n\nWe conduct *new* real-world experiments to further validate the effectiveness and generalizability of CLOVER. Please refer to the *Global Rebuttal* session Q1.(1) and (2) above, and the qualitative analysis in *Fig. R2* of our rebuttal PDF.\n\n>*${\\color{BrickRed}Question 2:}$* The measurable embedding space for error quantification seems effective not be sufficiently validated. This component’s effectiveness could vary significantly in different scenarios, and the paper does not provide extensive evidence of its robustness.\n\nWe address the concern on robustness in the following perspectives and will update them in our revision:\n- **Empirically robust to various tasks and scenes:** The CALVIN benchmark comprises 1,000 unique instruction chains across 34 different tasks. We select the most challenging scenario where the texture and position of objects to interact during testing is distinct from the training samples (ABC -> D).  As outlined in Algorithm 1, our sub-goal transitions and replanning depend entirely on measurable embeddings. Thus, the performance improvement on CALVIN can be a reflection of the robustness of error qualification.  For instance, using embeddings with poor robustness can cause the policy to get stuck on certain goals, as the measured distance keeps failing to meet the threshold required to trigger a sub-goal transition. \n- **Robustness comparison with different embeddings for error quantification:** In the table below, We provide quantitative results of LIV which introduces tailored training objectives for dense reward learning (error measurement). CLOVER yields notable robustness compared with previous works. Further qualitative illustrations are provided in Appendix A.\n\\indent\n|Methods|Task 1|Task 2|Task 3|Task 4|Task 5|Avg. Len.|\n|-|-|-|-|-|-|-|\n|LIV [61] (**newly added**)|70.8|48.2|29.2|18.2|10.2|1.77|\n|CLIP Feature (Fig. 3 (a))|72.4|46.8|25.0|13.7|5.1|1.63|\n|State Embedding (Ours, Fig. 3 (c))|**96.0**|**83.5**|**70.8**|**57.5**|**45.4**|**3.53**|\n\n- **Model and parameter insensitive:** Figure 6(c) shows that we do not need to carefully cherry-pick hyperparameters about the measurement even for models with different architectures.\n\n>*${\\color{BrickRed}Question 3:}$* Computational overhead.\n\nThanks for the question. We provide a detailed discussion in the Q2 of *Global Rebuttal* session above.\n\n>*${\\color{BrickRed}Question 4:}$* Further validation on a wider range of benchmarks and task environments is necessary to confirm the framework's generalizability.\n\nThanks for the comment. On the one hand, the tested simulation benchmark CALVIN ABC-> D considers certain scene generalizations. The texture of the table and the position of buttons, drawers, and sliders in the tested environment is different from what the model has seen in the training set, which is quite challenging to the generalizability of different policies.\n\nTo further validate the generalizability of our framework, we conduct more real-world experiments under certain perturbations. *Please refer to the *submitted PDF* for detailed experiment setting and in the *Global Rebuttal* session.*\n\n>*${\\color{BrickRed}Question 5:}$* Minor, in Figure 5, the authors show generated videos of four tasks conditioned on the same initial frame, but the first figure in each task looks different, better to make them the same.\n\nThanks for the suggestion. In Fig. 5, we show the 2nd, 4th, 6th, and 8th frames of the generated video for simplicity. We have revised our manuscript where the initial frame has been positioned at the forefront.\n\n>*${\\color{BrickRed}Question 6:}$* In Figure 3, the authors mention is obtained during a roll-out, what task and what environment are you using? How many tasks and environments did you test?\n\nThe visualization in Fig. 3 is based on *a randomly sampled task* (\"open the drawer\") within the CALVIN D environment. We only show one task in Fig. 3 for simplicity and clarity. However, we note that *similar patterns are observed across all tasks*, where our measurable embedding space consistently shows monotonicity when approaching each sub-goal, compared to the other two counterparts. **We also provide a similar plot to Fig. 3, but encompassing all possible tasks within over 300+ independent rollouts, in Fig. R3 of our rebuttal PDF.** As shown in the figure, the measurability remains and generalizes well under all scenarios.\n\n>*${\\color{BrickRed}Question 7:}$* How long does it take to execute one task in simulation and real robot?\n\n- **Simulation:** Our experiments are conducted using NVIDIA RTX 3090 GPUs. The low-level policy, with an input size of 196, operates at a frequency exceeding 70 Hz. The video diffusion model requires approximately 5 seconds to generate 8 video frames with a resolution of 128x128. It only takes around 11 seconds on average to complete a task in CALVIN simulation.\n- **Real-world:** We perform experiments on an NVIDIA RTX 5000 Ada GPU with the input size of 128*96, where the low-level policy runs at more than 25Hz and the visual plan generation takes around 4.3 seconds. Overall, it takes around 38 seconds to complete the 3 consecutive tasks with our real-world robot.\n\nWe have added the detailed execution time in the revised version. Please also refer to our reply in *Global Rebuttal* for further information and comparison with other methods.\n\n>*${\\color{BrickRed}Question 8:}$* Video demos?\n\nThanks for the advice. Regrettably, we are not allowed to provide an external link to present the demonstration videos according to the NeurIPS rebuttal guidelines. We will release a project page with video demos when publicly releasing the work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2OTDAMl1GE", "reviewer": "Reviewer_eUsr", "summary": "This paper introduces a novel framework that incorporates closed-loop feedback into vision-based control systems for robot manipulation tasks. The author proposes a text-conditioned video diffusion model for high-level reference path planning. The error measurement and feedback policy are obtained through an encoder-decoder architecture. This framework improves task performance over open-loop policies with a higher success rate and good instruction-following without the need for giant LLM models. The method is tested on proper benchmarks for language-based manipulation and has a thorough discussion of the method and results.", "questions": "1. How the dataset in CLOVER is obtained? What data helps train the CLOVER? More details will be welcomed.\n2. How does CLOVER perform in environments with significant variations from the training scenarios? Have you tested the framework on completely new tasks or objects not seen during training?", "limitations": "The author discussed the potential societal impact in the Appendix.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novelty: This paper presents a novel feedback policy for visual input. The proposed measurable embedded space can help identify the error between the reference plan and the executed plan, and the encoder-decoder bridges the input-feedback-output modules. This design helps in long-horizon planning where open-loop control easily fails.\n2. Presentation: This paper is well-written and provides a thorough discussion of the methods.\n3. Potential impact: A feedback system for visual-based control has a potential impact in other robotics systems other than manipulators.", "weaknesses": "1. Generalization: how this framework can be generalized across different task/robot platforms/datasets? Especially for the sub-goal replan/transition, which seems to be very specific for each task. The replanning threshold is also designed by hand. Can it be determined during training?\n2. The paper seems not to report the computational time for the model during testing, which is crucial for a close-loop policy in real-time. Especially with a diffusion-based policy, the sampling time could be long. Could provide more discussion on addressing this issue.\n3. Do you expect your proposed architecture to scale well when pre-trained on a large dataset; what additional design might be needed?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable review. We address your concerns below.\n\n>*${\\color{BrickRed}Question 1:}$* Generalization: how this framework can be generalized across different task/robot platforms/datasets? Especially for the sub-goal replan/transition, which seems to be very specific for each task. Can the threshold be determined during training?\n\n**Generalize across platforms and datasets:** The idea of CLOVER (the closed-loop visuomotor control framework) itself is embodiment-agnostic and can be generalized across different platforms. For instance, the robot in CALVIN simulation is Franka Emika Panda, whereas ALOHA is employed for conducting real-world experiments. Therefore, the generalization is more related to data. In our work, we train CLOVER on the respective data from the two platforms and successfully conduct verification. It is also expected that training on large-scale cross-embodiment datasets may lead to zero-shot generalization.\n\n**Sub-goal replan/transition generalization across tasks and models:** The thresholds for sub-goal replan/transition empirically generalize well across tasks. For the CALVIN benchmark, there are 34 different tasks in total and the evaluation suite involves 1,000 distinct instruction chains. We use the same set of hyperparameters for sub-goal replan and transition, which works well for all tasks and different visual encoders (Fig. 6(c)). Please also refer to *Fig. R3* in the rebuttal PDF.\n\n**Threshold determination during training:** In our current implementation, the visual planner and the subsequent inverse dynamics-based policy are trained separately. Consequently, the required threshold can only be measured during the inference stage, when the two components are integrated. Future research will focus on training these models jointly in an end-to-end manner, potentially facilitating the threshold determination process during training.\n\n\n>*${\\color{BrickRed}Question 2:}$* Discussions on computational overhead\n\nThanks for the question. Please refer detailed discussion in the Q2 of *Global Rebuttal* session.\n\n>*${\\color{BrickRed}Question 3:}$* Do you expect your proposed architecture to scale well when pre-trained on a large dataset; what additional design might be needed?\n\nThanks for the insightful question.\n- **Diffusion-based video generation models** have been proven to be scalable with the size of the dataset and network. All we need to do to scale up CLOVER is to collect more videos and extend the model with more channels and layers. Notably,  the videos can be free of action labels, and even human videos would help as well. Works like UniPi and UniSim [16] have made very successful attempts towards building world simulators by scaling up the pre-training dataset, and it would be similar for CLOVER to perform such extensions as well.\n- **Feedback-driven policy**: Its training is grounded in an inverse dynamics objective. While it necessitates action labels, it does not require high-level, task-specific knowledge for policy training. This characteristic facilitates the potential for training the policy on extensive, cross-embodiment datasets [38], thereby enabling few-shot cross-embodiment generalization.\n\n>*${\\color{BrickRed}Question 4:}$* How the dataset in CLOVER is obtained? What data helps train the CLOVER? More details will be welcomed.\n\nWe train our models exclusively on in-domain datasets. For simulation experiments, we utilize the ABC split of the CALVIN training dataset, which includes language instruction labels, to train the video diffusion model. Consecutive frames and their action labels are randomly sampled to train the low-level policy. Similar procedures are applied in real-world experiments. Additional training details are available in Appendix B2.\n\nBy decoupling the training of the visual planner from the feedback-driven policy, we allow the visual planner to be trained on diverse video datasets without action labels. As discussed in Question 3, this approach enables us to leverage internet-scale videos without action labels for training a robust and generalizable visual planner and to utilize large-scale cross-embodiment robot datasets (Open X-Embodiment [38]) for training a more effective low-level policy.\n\n\n>*${\\color{BrickRed}Question 5:}$* How does CLOVER perform in environments with significant variations from the training scenarios? Have you tested the framework on completely new tasks or objects not seen during training?\n\nThanks for the question. Zero-shot generalization to completely new tasks or objects stands as a significant challenge for current robotic research. Following the common experimental design, we mainly adopt certain variations during testing to demonstrate the generalization ability of our method. \n\nAs requested, we conduct *additional* real-world experiments in the rebuttal, by introducing entirely new objects — a yellow clay and a doll — alongside the primary interaction object. Please refer to the *Global Rebuttal* session for detailed results.\n\n**Additional Analysis:**\n\n- Our simulation experiment setup necessitates the policy's generalizability. In the CALVIN benchmark, the textures of the table and the positions of buttons, drawers, and sliders in the test environment differ from those in the training set, posing a substantial challenge to the generalizability of various policies. CLOVER demonstrates exceptional generalizability in this context.\n- CLOVER relies on a video generation model that must reliably follow task instructions to produce corresponding visual plans. Currently, video generation models struggle to generalize to completely novel tasks (instructions) outside the training set. This challenge could potentially be mitigated by extensive pre-training on large-scale internet datasets, which we hope future open-sourced video foundation models will provide to the community. Our inverse dynamics-based policy is inherently task-agnostic and capable of generalizing to new tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Area Chairs and Reviewers,

We thank all the Reviewers for their detailed and helpful comments on our work. We appreciate the Reviewers for acknowledging our strengths and contributions, such as a  **creative and novel feedback policy and cohesive system design** (DjFL, hKyy, eUsr), a useful and **meaningful research problem** of integrating closed-loop feedback into visuomotor control (hfKW, DjFL, hKyy, eUsr), **clear motivation and well-articulated methodology** (hfKW, DjFL), **sufficient ablations and notable improvements** (hfKW, DjFL), and **well-written** (hfKW, DjFL, hKyy, eUsr).

During the rebuttal phase, we have made diligent efforts to address the concerns raised by the Reviewers, add new ablation studies and real-world experiments, provide discussions on computational complexity, and add clarity and depth to address any ambiguities. Our responses to specific concerns are detailed below. We thank you all for the opportunity to improve our work with your constructive feedback.

Best regards,  
The Authors

***
**Here we refer to two general questions:**

>***${\color{BrickRed}Question 1}$ (Reviewer hfKW, Reviewer DjFL, Reviewer hKyy):* More real-world experiments, or experiments on generalization.**

We conduct additional generalization evaluation with our original long-horizon task and perform two more tasks to testify CLOVER against our baselines. From the results below, it can be observed that CLOVER greatly outperforms existing works under distractions and in more challenging tasks. Please refer to the submitted rebuttal PDF for the illustrative experiment settings. We will add the results to our revision.

- **(1)** (hfKW Q.5, DjFL Q.3, hKyy Q.1, eUsr Q.5) **Robustness evaluation with visual distractions.** (See Fig. R1(a) in the rebuttal PDF for a detailed setting)
|Methods|Task 1|Task 2|Task 3|Avg. Len.|
|-|-|-|-|-|
|ACT [53]|13.3|0|0|0.13|
|R3M [54]|20.0|0|0|0.20|
|RT-1 [49]|40.0|6.7|0|0.47|
|CLOVER (Ours)|**73.3**|**66.7**|**6.7**|**1.47**|

- **(2)** (DjFL Q.3, hKyy Q.4) **Robustness evaluation with dynamic scene variation.** (See Fig. R1(a) in the rebuttal PDF for a detailed setting; Due to limited time, we choose the competitive RT-1 for comparison only in this experiment.)
|Methods|Task 1|Task 2|Task 3|Avg. Len.|
|-|-|-|-|-|
|RT-1 [49]| 33.3|0|0|0.33|
|CLOVER (Ours)|**80.0**|**53.3**|**20.0**|**1.54**|

- **(3)** (DjFL Q.5, hKyy Q.1) Experiments with **two new tasks**: Pour shrimp into the plate & Stack bowl. (See Fig. R1(b) in the rebuttal PDF for a detailed setting)
|Methods|Pour Shrimp|Stack bowl|Avg.|
|-|-|-|-|
|ACT [53]|33.3|46.7|40.0|
|R3M [54]|46.7|53.3|50.0|
|RT-1 [49]|**80.0**|66.7|73.4|
|CLOVER (Ours)|**80.0**|**86.7**|**83.4**|

>***${\color{BrickRed}Question 2}$ (Reviewer DjFL, Reviewer hKyy, Reviewer eUsr):* Computational complexity**

**Statistics:** By default, a ViT-Base (86M) backbone is employed to encode RGB data, and a ViT-Small (22M) backbone is utilized for the extraction of depth features. As indicated by Table 4, substituting the ViT-Base encoder in the RGB branch with a ViT-Small encoder also results in comparably exceptional performance. When utilizing the aforementioned configuration and running on a Nvidia RTX 5000 Ada GPU, the inference time of the proposed policy model is **less than 0.04 seconds**, which means the policy operates in real-time at a frame rate **greater than 25Hz**. We conduct simulation experiments with a server equipped with an RTX 3090 GPU, where our policy can run at **over 70Hz**. In the rebuttal, we provide the following statistics to enable a full grasp of the computational complexity of CLOVER and other competitive methods.

|Methods|Performance|Video Generation (s)|Policy (s)|Avg. Time to complete a task (s)|# Params. in total (M)|
|-|-|-|-|-|-|
|RoboFlamingo [50]|2.48|/|0.072|**8**|3000|
|SuSIE [15]|2.69|9|0.15|49|400|
|CLOVER (Ours)|**3.53**|**5**|**0.013**|11|**200**|

Note that our diffusion model serves as a high-level planner, i.e., *activated only when subtask starts and replans*. Therefore, it does not require real-time inference. Unlike image editing models like SuSIE, our video generation model can cover a larger span of task rollouts, thus fewer inferences are needed. As shown in the table above, CLOVER achieves better performance with much lighter computational requirements.

**Analysis:** We agree that the main bottleneck is in video generation. Therefore, we made multiple designs to enhance efficiency:
  - Illustrated in Sec. 4.4, we use a 20-step DDIM sampler for balanced efficiency and performance. Yet, a 10-step process halves video generation time and remains competitive performance, achieving Avg. Len. 3.21 on CALVIN. In contrast, SuSIE uses a larger model with 50 denoising steps, while it yields inferior performance (Avg. Len. 2.69). Integrating advanced samplers like DPMSolver could further reduce the steps needed.
  - Based on Imagen, our diffusion model downscales the channel dimension and limits the attention blocks to only 1/8 and 1/16 downsampled feature maps, cutting the attention module's quadratic computational load. We build a compact diffusion network with merely 72M parameters to minimize latency.

We will add the above results and discussions to the revision.
***
*Please refer to the rebuttal modules below for our point-to-point responses to each reviewer.*

Meta-review: The detailed rebuttals by the authors were appreciated. All reviewers recommend accepting the paper.
Decision: Accept (poster)