Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models
Authors: Kevin Black
Mitsuhiko Nakamoto
Pranav Atreya
Homer Rich Walke
Chelsea Finn
Aviral Kumar
Sergey Levine
Venue: ICLR 2024 poster (2024)
Keywords: robot learning
diffusion model
Primary area: applications to robotics, autonomy, planning

Abstract: If generalist robots are to operate in truly unstructured environments, they need to be able to recognize and reason about novel objects and scenarios. Such objects and scenarios might not be present in the robot’s own training data. We propose SuSIE, a method that leverages an image-editing diffusion model to act as a high-level planner by proposing intermediate subgoals that a low-level controller can accomplish. Specifically, we finetune InstructPix2Pix on video data, consisting of both human videos and robot rollouts, such that it outputs hypothetical future “subgoal” observations given the robot’s current observation and a language command. We also use the robot data to train a low-level goal-conditioned policy to act as the aforementioned low-level controller. We find that the high-level subgoal predictions can utilize Internet scale pretraining and visual understanding to guide the low-level goal-conditioned policy, achieving significantly better generalization and precision than conventional language-conditioned policies. We achieve state-of-the-art results on the CALVIN benchmark, and also demonstrate robust generalization on real-world manipulation tasks, beating strong baselines that have access to privileged information or that utilize orders of magnitude more compute and training data. The project website can be found at http://rail-berkeley.github.io/susie.

ZERO-SHOT ROBOTIC MANIPULATION WITH PRETRAINED IMAGE-EDITING DIFFUSION MODELS
Kevin Black" Mitsuhiko Nakamoto*
Pranav Atreya' Homer Walke
Chelsea Finn Aviral Kumar
Sergey Levine
University of California, Berkeley
Stanford University "Google DeepMind
"Equal contribution
ABSTRACT
If generalist robots are to operate in truly unstructured environments, they need to able to recognize and reason about novel objects and scenarios. Such objects and scenarios might not be present in the robot's own training data. We propose SuSIE, method that leverages an image-editing diffusion model to act as high- level planner by proposing intermediate subgoals that a low-level controller accomplish. Specifically, we finetune InstructPix2Pix on video data, consisting of both human videos and robot rollouts, such that it outputs hypothetical fu- ture "subgoal" observations given the robot's current observation and a language command. We also use the robot data train a low-level goal-conditioned pol- icy to act as the aforementioned low-level controller. We find that the high-level subgoal predictions can utilize Internet-scale pretraining and visual understand- ing to guide the low-level goal-come conditiooe policy, achieving significantly bet- ter generalization and precision than conventional language-conditioned policies. We achieve state-of-the-an aft results on the CALVIN benchmark, and also demon- strate robust generalization on real-world manipulation tasks, beating strong base- lines that have access to privileged information that utilize orders of mag- nitude more compute and training data. The project website can be found at github. susiie..
INTRODUCTION
A useful generalist robot must be able to - much like a person recognize and reason about novel objects and scenarios it has never encountered before. For example, if user instructs the robot to "hand me that jumbo orange crayon," ought to be able to do so even if it has never interacted with jumbo orange crayon before. In other words, the robot needs to possess not only the physical capability to manipulate an object of that shape and size but also the semantic understanding to reason about an object outside of its training distribution. As much robotic manipulation datasets have grown in recent years, it is unlikely that they will ever include every conceivable instance of objects and settings, any more so than the life experiences of person ever include physical interactions with every type of object. While these datasets contain more than enough examples of manipulating elongated cylindrical objects, they lack the broad semantic knowledge necessary to ground the particular objects that robots will encounter during everyday operation.
How can we imbue this semantic knowledge into language-guides robotic control? One approach would be to utilize models pretrained on vision and language to initialize different components of the robotic learning pipeline. Recent efforts, for example, initialize robotic policies with pretrained dision-language encoders [7] apply pretrained models to semantic scene augmentation [11, 64]. While these methods bring semantic knowledge into robot learning, remains unclear if these ap- proaches realize the full potential of Internet pretraining in improving robotic policy execution at every level, or whether they simply improve the high-level visual generalization of the policy.
We propose approach for leveraging pretrained image-editing models to enable generalizable robotic manipulation. We finetune the image-editing model on video data such that, given the cur- rent frame and a language description of the current task, the model generates hypothetical future

---

"put the orange crayon the wooden bowl"
Generate
Execute
Generate
Execute
Generate
Generate
ElExccute
Execute
Real
Generated
"put the black marker in the metal pot"
Figure 1: (SuSIE) Our method leverages pretrained image-editing model to generate future subgoals based on language commands. A low-level goal-ree rea-hreching policy then executes the actions needed to reach each subgoal. Alternating this loop enables solve the task.
frame. This does require the model precisely understand the intricacies of the robot's 's low-level dynamics and therefore should facilitate transfer from other data sources (e.g.. human videos) where the low-level physical interactions and precise embodiment do match. At test time, we employ a low-level goal-reaching policy trained on robot data to reach this hypothetical future frame; this policy, in turn, only needs infer visuo-motor relationships determine the correct actuation and does not need to understand the overlying semantics. Furthermore, such subgoals can simplify the task by inferring likely poses for the arm at intermediate substeps, such as the pose corresponding to grasping an object (see Figure 1). In fact, we observe in our experiments that even when existing approaches possess sufficient semantic understanding to solve a task, they often fail due to impre- method to
cise localization of obstacles and objects; following the generated subgoals enables perform well such scenarios. Much like a person first constructs a high-level plan to complete a task before deferring to muscle memory for the low-level control, method can also be viewed as first running a high-level planner with integrated semantic reasoning and visual understanding before deferring low-level controller to execute the plan.
The main contribution of our work is SUbgoal Synthesis via Image Editing (SuSIE), a simple and scalable method for incorporating semantic information from pretrained models to improve robotic control. A pretrained image-editing model is used with minimal modification, requiring only finetuning on video data. The low-level coul-conditioned policy trained with standard supervised learning, and faces the comparatively casy problem of reaching nearby image subgoals; this typically only requires attending
single object or the arm position, ignoring most parts of the scene. Together, this approach achieves state-of-the-an results on the CALVIN benchmark and solves real robot control tasks involving novel objects, novel distractors, and even novel scenes. It outperforms all prior approaches on these real-world tasks, including oracle baselines with privileged information and RT-2-X [14] - a 55 billion parameter model trained on Internet-scale vision-language data as well as order of magnitude more robot data than SuSIE.
2 RELATED WORK
Incorporating semantic information from pretrained vision-language models. Prior works that incorporate semantic information from pretrained vision-language models (VLMs) into robot learn- ing can be classified into two categories. The first category aims to improve visual scene under- standing in robot policies with information from VLMs. For instance, GenAug [11], ROSIE [64], DALL-E-BE Bot [34], and CACTI [43] use toxt-to-image generative models to produce semantic aug- mentations of given scene with novel objects and arrangements and train the robot policy the augmented data to enable it to perform well a similar scene. MOO [57] uses a pretrained object detector to extract bounding boxes to guide the policy towards the target object. Other works di-
2

---

rectly train language and image-basee based policies [6, 7, 55], some using VLM initializations [15, 52], to produce action sequences.
While these approaches do utilize pretrained models, find in experiments that pretraining using VLMs [7] does not necessarily enhance low-level policy execution in the sense that learned policies often localize the object or move the gripper imprecisely (see Figure 3). On the other hand, our approach is able to incorporate benefits of pretraining into low-level control by synthesizing subgoals that carefully steer the motion of the low goal reaching policy, improving its precision. While our approach can be directly applied in unstructured open-world settings, applying methods from [11, 64, 43] requires additional prior information, such as object bounding boxes or 3D meshes. This significantly restricts their applicability to scenarios where this additional information is not available: for example, GenAug [11] is not applicable in our experiments since 3D object meshes for new target objects were not available. Distinct from our approach for utilizing generative models, other works design representation learning objectives for vision-language pretraining for control [49, 42, 35, 4]. but these methods still need to utilize limited amounts of data from the target task to learn a policy. In contrast, our approach solves the target task in zero-shot, without any task-specific data.
The second category of approaches incorporates semantic information from pretrained models for informing low-level controllers. Typically. these prior methods use pretrained models to imagine visual [16, 2] or textual plans [8. 28, 29, 39], which then inform low-level robot control policy. Of these, methods that train a low-level policies conditioned on text suffer from a grounding problem, which our approach eircumvents entirely since the low-level policy only sees images.
Perhaps the most related methods to our approach in this second category are UniPi [16] and HiP [2], which train video models to generate a sequence of frames achieving the target task, and then extract robot actions by training an inverse dynamics model. Our approach does not attempt to generate full videos (i.e., all frames a rollout), but only the next waypoint that low-level policy must achieve to solve the commanded task. While this difference might appear small, it has major implications: modeling an entire video puts a very high burden on the generative model, requiring the frames to obey strict physical consistency. Unfortunately, we find that open-source video models often produce temporally inconsistent frames, which confuse the low-level controller (see Appendix C). Our method provides more freedom to the low-level controller to handle the physical aspects of the task over a longer time interval while guiding at a level that is suitable to the diffusion model's ability to preserve physical plausibility. In our experiments, we find that our method significantly improves over an open-souree source reimplementation of UniPi [16].
Classical model-based RL and planning with no pretraining. The idea behind our approach is also related to several methods in the deep RL literature that do not use pretrained models and generally do not study Language-guided control. For instance, [21, 37, 63, 60, 53, 22] train action- conditioned dynamics models and run RL in the model. While our approach also models multi-step dynamics, our model not conditioned on an action input. Removing the dependency on an action input enables us to de-couple the finetuning of the (large) image-editing model from the policy entirely, improving simplicity and time efficiency. APV [54] trains an action agnostic dynamics model from videos but finetunes it in a loop with the policy, and hence, does not enjoy the above benefits. Finally, these model-based RL methods do not exhibit zero-shot generalization abilities to new tasks, which is an important capability that our method enjoys. Our approach is also related to several video prediction methods [17, 38, 3, 58] but utilizes a better neural network architecture (i.e., diffusion models instead of LSTMs and CNNs). Most related is to our method is hierarchical visual foresight (HVF) [48]: while HVF uses MPC to find an action, our approach uses a goal-reaching policy thereby eliminating the cost of running MPC with large dynamics models.
Our approach is also related to prior works that use generative models for planning in a single-task setting, with no pretraining. Trajectory transformer (TT) [31], decision transforeer (DT) [10], and their extensions condition the policy on the target return or goal. While diffusion-based variants of these methods [32, 1] use diffusion models like our approach, they still require training data from the target task to learn policy, unlike our zero-shot planning approach.
PRELIMINARIES AND PROBLEM STATEMENT
We consider the problem setting of language-conding condiiioee robotic control. Specifically, we want robot to accomplish a task described by a novel language command. We divide sources of data into 3 categories: labeled video clips D) that do not include robot actions; language-laheled
3

---

robot data Dr,a that include both language labels and robot actions; and unlabeled robot data Do that include only actions (c.g.. play data).
Formally, define Di,-- (1) (72. (T" (N)) where each trajectory 7" consists of a sequence of images (or states) " E and actions a.' E A that were exccuted while collecting this data, i.e. 7" = (sh. a s" at ...... following the standard assumptions of a Markov decision process. 1th is a natural language command describing the task accomplished in the trajectory. Dr and Da are organized similarly, but are missing either actions a, or language annotations 1", respec- tively. At test time, given a new scene SU and a new natural language description (test of a task, we evaluate method in terms of its success rate accomplishing this task starting from sp
SUSIE: SUBGOAL SYNTHESIS VIA IMAGE EDITING
Our goal to utilize semantic information from the Internet improve language- guided robot con- trol in the presence of novel environments, scenes, and objects. How can we do this when models trained on general-purpose Internet data do not provide guidance in selecting low-level actions? Our key insight is that can effectively leverage the capabilities of the pretrained model if we decouple the robot control problem into two phases: (1) generating subgoals that would need to be attained to succeed at the task, and (II) learning low-level control policies for reaching these generated sub- goals. Our method incorporates semantic information from both text-image pretraining on Internet data as well non-robot video data in Phase (1) by finetuning a text-guided image-editing model on Dj U "Dia- Phase (II) is accomplished via a goal-conditioned policy trained on the robot data Di, U Da. We describe each of these phases below and then summarize the resulting algorithm.
4.1 PIASE (I): SYNTHESIZING SUBGOALS FROM IMAGE-DDTING MODELS
The primary component of our method is generative model that, given target task specified by natural language, can guide a low-level controller to a state that advances the task. One way to accomplish this is train generative model to produce an immediate next subgoal image. We can then incorporate semantic information from the Internet into our algorithm by initializing this generative model with suitable pretrained initialization, followed by finetuning on multi-task, diverse video data consisting of robot rollouts as well as other videos from the Internet.
What is a good pretrained initialization for this model? Our intuition is that, since accomplishing a task is equivalent to "editing" the pixels of an image of the robot workspace under constraints prescribed by the language command, a good pretrained initialization may be provided by a text- guided image-editing model. We instantiate our approach with Instruct Pix2Pix [9], though other image-editing models could also be used. Formally, this model is given by pe (Sedited Sorig. l). Then, using the dataset D, U Di, of language-labeless video clips and robot trajectories, we finetune pe to produce valid subgoals Sedied given an initial image Sonig and a language label 7. Formally, the training objective is given by
max E ("",M)),,UD.. sP--" j-(0 [log (s)) [")]
(1)
where q(i i) is a distribution of our choosing that controls what subgoals the model is trained to produce. We want the diffusion model to generate subgoals that are a consistent distance in the future from the current state close enough to be reachable by the low-level policy, but far enough to achieve significant progress on the task. As such, we choose dataset-dependent hyperparameters kmin and kmax, and set q to select subgoals uniformly between kmin and kimax steps in the future, i.e.,
q 8) = Uj; [8 kimin i + kimax))
4.2 PHASE (II): REACHING GENERATED SUBGOALS WITH GOAL- COAL-CONDITIONED POLICIES
In order to utilize the finetuned image-editing model to control the robot, additionally need to train a low-level controller to select suitable robot actions. In this section, we present the design of low-level controller, followed by full description of our test-time control procedure. Since the image-editing model in SuSIE produces images of future subgoals conditioned on natural language task descriptions, our lowelevel controller can simply be a language-agnostic goal-reaching policy.
Training a goal-reaching policy, Our goal-reaching policy is parameterized as T s,; where 8y is future frame that the policy intends to reach by acting at s;. At test time, we only
4

---

need the low-level policy to be proficient at reaching close-by states that lie within kmax steps of the current state, since the image-editing model from Phase (1) is trained to produce subgoals within Amax steps of any state. To train this policy, run condiconnd behavioral cloning (GCBC) on the rohot data Di.a U Da. Formally, the training objective is given by
max E pm PPN.DL,LDD (e) Jerr (- (kmn [log ( |s" [8]]
where Ks is another hyperparameter that provides a small amount of overhead, since the image- editing model is not perfect and may not always produce subgoals that are reachable in less than kmax steps, especially for unseen tasks.
Test-time control with To and pe- Once both the goal-reaching policy as and the subgoal gen- eration model pe are trained, we utilize them together to solve new manipulation tasks based on user-specified natural language commands. Given new scene sh and language command grest SuSIE attempts to solve the task by iteratively generating subgoals and commanding the low-level policy to reach these subgoals. At the start, we sample the first subgoal S., ~ pa(s+ | si ) Once the subgoal is generated, we then roll out the goal-reaching policy Th conditioned kust timesteps where kiest test-time hyperparameter. After kust timesteps, we refresh the sub- goal by sampling from the subgoal generation model again and repeat the process. In practice, for computational efficiency, we set kigst to be similar to the corresponding Amax used with the robot data and found this to be sufficient for obtaining good performance. However, given unlimited computational budget, conventional wisdom would suggest that regenerating subgoals more often would lead more robust control. Pseudocode for test-time control provided in Algorithm
S., for
Algorithm 1 SuSIE: Zero-Shot, Test-Time Execution
Require: Subgoal model (s 8t, policy To (a St, 8+), time limit T, subgoal sampling interval keea, initial state spst, language command pust
1: 0
2: while & < T' do
3: Sample S ~ po (S+ s/ ((a)
> Generate a new subgoal every kistt steps
4: for 1 to kiesa do
5: Sample a, ~ Te (al s S+)
Predict the action from current state and subgoal
6: Execute a,
7:
slest <- robot observation
8:
t t
9:
end for
10:
end while
4.3 IMPLEMENTATION DETAILS
In Phase (I), we utilize the pretrained initialization from the InstructPix2Pix model [9]. We imple- ment Equation 1 using the standard variational lower bound objective for training diffusion models [25]. Our diffusion model and policy operate images of size 256 x 256. To ensure that this model pays attention to the input image and the language command, we apply classifier-free guidance [24] separately to both the language and the image, following InstructPix2Pixx To obtain robust goal- reaching policy in Phase (II), we use diffusion policy [12, 23] that predicts chunks of 4 actions and performs temporal averaging over these predictions at test time [65]. More details about the training hyperparameters and architecture are provided in Appendix A.
EXPERIMENTAL EVALUATION
The goal of our experiments evaluate the efficacy of SuSIE at improving generalization and the low-level policy execution in ooen-world robotic manipulation tasks. To this end, our experiments aim to study the following questions:
1. Can SuSIE solve task in novel environment, with novel objects, and with novel lan- guage command, in zero-shot?
2. Does SuSIE exhibit an elevated level of precision and dexterity compared to other ap- proaches that do not use subgoals?
5

---

3. How crucial is pretraining Internet data, well cotraining non-robot video data, for attaining zero-shot generalization?
To answer these questions, our experiments compare SuSIE to several prior methods including state- of-the-art approaches for training language-conditioned policies that leverage pretrained vision- language models in a variety of ways.
5.1 EXPERIMENTAL SCENARIOS AND COMPARISONS
Real-world experimental setup and datasets. We conduct our real-robot experiments Wid- owX250 robot platform. Our dataset BridgeData V2 [59], a large and diverse dataset of robotic manipulation behaviors designed for evaluating open-vocabulary instructions. The dataset contains over 60k trajectories, 45k of which are language-labeled, which we use as our language-laheled robot dataset Di.a. We use the remaining 15k trajectories as our action-only dataset Da.
Our video-only dataset Dr is the Gomething-Somethingg dataset [19], a dataset consisting of short video clips of humans manipulating various objects. We chose Something-SSmethiing because primarily contains examples of object manipulation with still camera frame and hence exhibits smaller domain gap to the robot data collected with an thehule-s camera compared other video datasets that contain substantial egocentric motion. We filter the Semething--Smethiigg dataset using its labels to remove trajectories that contain egocentric motion or otherwise lack significant manipulation behavior, producing final dataset size of approximately 75k video clips.
Our evaluations present three different scenes (Figure 2) designed specifically to test the ability of various methods at different levels of open- world generalization: Scene A: this scene includes environment and objects that are well-represented
BridgeData V2; Scene B: this scene situated tabletop but novel background and distractors, where the robot must
in environment with move a seen object (bell pepper) into a choice of seen container (orange pot) or unseen container (ceramic bowl); and Scene C: this scene includes table texture unlike anything in BridgeData V2 and requires manipulating both seen and unseen objects. Semantically, Scene is the most challenging since the robot needs carefully ground the language command identify the correct object while resisting its affinity for an object that is well-represented in the robot data (the spoon). Scene B also requires semantic grounding distinguish between the seen and unseen receptacles, while adding the additional challenge of manipulating the plastic bell pepper this object requires particular precision grasp due to being light, smooth, and almost as wide as the gripper.
Seen BridgeData Unseen BridgeData
Scene A
Scene
CALVIN: Train/Test Split
Touthe spaplant on Deattte
Env
Eav
porth comen on the plate
nut"
"purthe gaplaatt the por
"yur the bell pepper cenuaic bowd"
Train
Train
Backgruund
Scene C
Eav
Eαv
great spoce sasdan
wosten bowf"
Triin
Toot
"yogtran in the wooden
"masee the wooden howi to the tog
kavtronment:
Unseen Thble Testure
Figure 2: (Experimental setup) (L) We evaluate our method on 9 tasks across 3 real-world scenes. In terms of semantic generalization, the scenes become progressively more difficult, due to both an increasing visual departure from the robot training data and an incrcasingly confounding mixture of both seen and unseen objects. simulation, evaluate method the zero-shot setting of the CALVIN benehmark, which involves training on 3 environments (A, B, and C) and testing on a environment (D), The environments differ in table texture, positioning of furniture elements, and possible configurations various colored blocks. Each environment comes with 34 language-specified tasks.

---

Simulation tasks. We rum simulation experiments in CALVIN [45]. benchmark for long hori- zon, language-condition
manipulation. CALVIN consists of four simulated environments, labeled A through D, and each environment comes with dataset of human-collecter play trajectories. Ap- proximately 35% of the play data is annotated with language, which we use as our language- laheled robot dataset Di..- We use the remaining 65% of the play data and do not use any video-only dataset D; in our simulation experiments.
action-only robot dataset Da.
Each environment consists of Franka Emika Panda robot arm positioned next to a desk with various manipulatable objects, including a drawer, sliding cabinet, light switch, and various colored blocks. Environments are differentiated by their table textures, positions of the furniture objects, and con- figurations of colored blocks. With this benchmark, we study the most challenging zero-shot multi- environment scenario: training on A, B, and C. and testing on D. We follow the evaluation protocol from Mees et al. [45]. During evaluation, a policy is given 360 timesteps to complete a chain of five language instructions. We provide more details about our experimental setup in Appendix
Comparisons. Our experiments cover methods that utilize pretrained models of vision and language in language-guided robot control in a variety of ways. Several prior methods tackle language-based robotic control as discussed in Section 2. In our experiments, we choose to compare to a repre- sentative subset of these prior methods to maximally cover the possible set of comparisons. In our real-world experiments, we compare to the following methods:
RT-2-X [14], 55 billion parameter vision-language model finetuned on the full Open X- Embodiment dataset to produce robot actions. The robot training data includes all of Bridge- Data V2, as well as vast quantity of additional robot manipulation data totaling over 1.1 million trajectories.
MOO [57]. which utilizes pretrained object detectors obtain 2D localization information for the manipulation targets and trains with this extra information. We re-implement MOO by using large language model to extract the manipulation targets from the unstructured language labels in BridgeData V2, then OWL- ViT [46] to extract 2D bounding boxes for these objects. Unlike MOO, at test time we manually give the policy the ground truth 2D bounding boxes of the manipulation targets.
behavioral cloning (LCBC) policy
UniPi [16], which finetunes a language- condutione video prediction model on robot data. Since the original UniPi model utilized proprietary pretrained initializations that are not avail- able publicly, we replicated this method in two different ways: (1) using the UniPi model from HiP [2], which a PVDM [62] latent video diffusion model, and (2) by implementing our own image-space video diffusion model with a factorized spatial-temporal 3D UNet similar to Ho et al. [27]. We found that (1) was not able to produce reasonable-looking videos in the real robot scene (Appendix C),
only evaluated (2) in real-world experiments.
LCBC, which trains a low-level policy conditioned on the language labels in BridgeData V2. This method is broadly representative of prior language-conditioned behavioral cloning meth- ods such as RT-1 [6] and BC-Z [30]. We use the same architecture and hyperparameters as the low-level policy in SuSIE, as we found that they outperformed RT-1 on the BridgeData tasks.
We provide more details about the baselines and their implementations in Appendix A.2.
In our simulation experiments, we compare to additional methods previously studied on CALVIN. This includes methods that explicitly tackle long-horizon language-based control on CALVIN such as multi-context imitation (MCIL) [41], hierarchical universal
condutiioon policy (HULC) [44], and improved variants of HULC [18]. We also compare to other sfate-of-the-an meth- ods from Ge et al. [18] that employ an identical training and evaluation protocol as our experiments, namely MdetrLC [33] and AugLC [50].
5.2 CAN SUSIE PERFORM ZERO-SHOT ROBOTIC MANIPULATION?
Simulation results. We present performance in Table 1, in terms of success rates (out of 1.0) for completing chains of language instructions. SuSIE obtains superior zero-shot performance (train A, B. C- test D) compared to the previous sfate-of-the-aat We find that our reimplementation of UniPi achieves nontrivial success, but is still significantly outperformed by SuSIE.
Real-world results. We present performance real-world evaluations in Table 2. SuSIE achieves the best performance across the board, beating RT-2-X, a 55 billion parameter model trained on

---

Success Rate for No. of Instructions Row
Avg. No. of Instructions Completed
HULC [44]
0.43
0.14
0.04
0.01
0.00
0.62
MCIL [41]
0.20
0.00
0.00
0.00
0.00
0.20
MdetrLC [18]
0.69
0.38
0.20
0.07
0.04
1.38
AugLC [18]
0.69
0.43
0.22
0.09
0.05
1.43
LCBC
0.67
0.31
0.17
0.10
0.06
1.31
UniPi (HiP) [2]
0.08
0.04
0.00
0.00
0.00
0.12
UniPi (Ours) [16]
0.56
0.16
0.08
0.08
0.04
0.92
SuSIE (Ours)
0.87
0.69
0.49
0.38
0.26
2.69
Table 1: (CALVIN benchmark performance) SuSIE is able to chain together more instructions with higher success rate than all prior methods in the zero-shot (train A, B, C - test D) setting. We provide an example rollout in Appendix D.
Task
LCBC
MOO
UniPi
RT-2-X
SuSIE (Ours)
Eggplant on plate
0.9
0.4
0.0
0.3
1.0
Carrot plate
0.4
0.3
0.0
0.6
0.9
Scene A
Eggplant in pot
0.6
0.7
0.0
0.4
0.7
Average
0.63
0.47
0.0
0.43
0.87
Bell pepper in pot Bell pepper bowl
0.1
0.0
0.0
0.0
0.5
Scene
0.3
0.1
0.1
0.0
0.5
Average
0.20
0.05
0.05
0.00
0.50
Toothpaste in bowl
0.0
0.0
0.0
0.5
0.6
Crayon in bowl
0.0
0.0
0.0
0.9
1.0
Scene C
Spoon in bowl
0.1
0.3
0.1
0.7
0.9
Bowl top
0.6
0.1
0.1
0.9
1.0
Average
0.18
0.10
0.05
0.75
0.88
Table 2: (Real-world performance) SuSIE achieves the best success rate across the board, demonstrating both high precision as well as the ability to generalize to novel environments, objects, and language commands.
significantly more robot and Internet data. As expected, all methods perform well in Scene A, which is well-represented in the robot data. SuSIE performs uniquely well in Scene B, as it is the only method that can consistently grasp the bell pepper.
In Scene C, while SuSIE still achieves the best performance, RT-2-X comes in a close second. We hypothesize that this is because - contrast to Scene B all the objects in Scene C are easy to grasp. Therefore, the low-level precision of the policy is less important, which is the primary weakness of RT-2-X, as we discuss further in the next section. Qualitatively, we observed that the failure cases in Scene C for both SuSIE and RT-2-X were almost always imprecise manipulations (failed grasps or early dropping) rather than semantic misunderstanding: of the objects, the tooth- paste is the most difficult grasp, which is why its success rate is the lowest. That is, both methods solve the semantic understanding component of the tasks, but SuSIE's s improved low-level precision allows perform slightly better.
The performance of LCBC is as expected: since it is trained only on BridgeData V2, it has no way of grounding novel objects, and often puts the bell pepper the wrong receptacle (in Scene B) or attempts to grasp the wrong object (in Scene C). Surprisingly, it can recognize the wooden bowl in Scene C, even though that exact object does not appear in the training data; we hypothesize that the wooden bowl is distinctive enough to recognize from the word "bowl" alone combined with the many other bowls in BridgeData V2. MOO underperforms expectations, even with its privileged test-time information in the form of ground-truti bounding boxes. We hypothesize that, despite our best attempts, the target objects and bounding boxes extracted at train time are noisy due to the unstructured nature of BridgeData V2 and therefore only serve confuse the policy (see Appendix 1.2.1). UniPi's poor performance is due to the video model often producing temporally inconsistent frames (see Appendix C).
8

---

SUSIE (Ours)
Generated Subgoals
Executed Rollout
RT-2-X
Oracle GCBC
Executed Rollout
Ground-Truth Goal
Figure 3: (Visualizing rollouts) Visualized rollouts from SuSIE, RT-2-X, and Oracle GCBC on the task "put the yellow hell pepper in the ceramic howl" from Scene B. While RT-2-X and Oracle GCBC fail to grasp the object, the generated subgoals from SuSIE precisely guide the low-level controller, improving low-level skill execution for difficult manipulation tasks.
Task
Oracle GCBC
SuSIE (Ours)
Eggplant plate Carrot on plate
0.7
1.0
0.8
0.9
Scene A
Eggplant in pot
1.0
0.7
Average
0.83
0.87
Bell pepper pot Bell pepper in bowl
0.1
0.5
Scene B
0.0
0.5
Average
0.05
0.50
Toothpaste in bowl Crayon in bowl
0.7
0.6
1.0
1.0
Scene C
Spoon in bowl
0.8
0.9
Bowl to top
0.6
1.0
Average
0.78
0.88
Move slider right/left
0.52
0.86
Turn on/off light bulb
0.74
0.96
CALVIN
Open/close drawer
1.00
0.98
Turn on/off LED
0.36
1.00
Average
0.66
0.95
Table 3:
(Comparison to GCBC with oracle goals) Exccuting generated subgoals improves performance
even compared to executing privileged ground-truth final goals.
5.3 DOES SUSIE IMPROVE PRECISION AND LOW-LEVEL. SKILI EXECUTION?
Our real-world and simulated results clearly demonstrate the efficacy of SuSIE in executing novel language commands variety of scenes. In Section 5.2, we hypothesize that the advantage of SuSIE is twofold: semantic generalization due to Internet and video pretraining, and low-level pre- cision due to subgoal guidance. this section, we aim to validate the latter half of this hypothesis. train Oracle GCBC policy, which is identical to the low-level policy in SuSIE
To this end, except that it trained with a goal horizon that extends to the end of each trajectory. At test time, we provide the policy with privileged real goal image of the fully completed task, eliding the need for

---

Task
BridgeDData Only
BridgeData + Something-Sometee
Bell pepper in pot Bell pepper in bowl
0.2
0.5
Scene B
0.4
0.5
Average
0.30
0.50
Toothpaste in bowl Crayon in bowl
0.7
0.6
0.9
1.0
Scene C
Spoon in bowl Bowl to top
0.9
0.9
0.7
1.0
Average
0.80
0.88
Table (Comparison BridgeData-only subgoal generation model) Cotraining the subgoal generation model on the Something-Somellingg dataset improves performance Scenes and C, which are unseen in BridgeData V2.
any semantic understanding. Thus, the only advantage SuSIE over Oracle GCBC the subgoal guidance, while SuSIE is disadvantaged by additionally needing to interpret the language instruction and generate subgoals for the correct task.
As demonstrated in Table 3, although Oracle GCBC is a strong baseline, SuSIE is still the best- performing method on average. In particular, we observed that Oracle GCBC is still unable to grasp the bell pepper in Scene B due lack of low-level precision. Figure 3 provides a qualitative visualization of SuSIE grasping the bell pepper while Oracle GCBC and RT-2-X fail to do so. These results validate the hypothesis that, in addition enabling semantic generalization, SuSIE also improves low-level precision and dexterity by decomposing the problem into a two-level hierarchy.
5.4 ARE INTERNET AND VIDEO TRAINING CRUCIAL FOR ZERO-SHOT GENERALIZATION?
Finally, we conduct an experiment to understand the effect of both Internet-scale pretraining (in the form of InstructPix2Pix initialization) and video cotraining (in the form of training on BridgeData V2 and Something -Smmething simultaneously). We train two additional subgoal generation models: one robot-only model trained with InstructPix2Pix initialization but only on BridgeData V2, and one from-scratch model trained without InstructPix2Pix initialization at all. The from-scratch model uses the same frozen text encoder, image encoder, and UNet architecture as InstructPix2Pix. but initializes the UNet with random weights.
We present sample generations from these models in Figure 4. The generations of the from-scrated model are consistently low-quality, either misunderstanding the task, producing significant artifacts, failing to edit the image at all; this demonstrates the importance of initializing from Interne-- scale pretrained weights. As one might expect, the BridgeDaaaaonlyy model is on par with the full Something-Somethinn model in-distribution tasks, which come from the BridgeData V2 val- idation set. However, on out-of-distribution tasks, the BridgeData-only model has an increased propensity to hallucinate background items or misunderstand the task. We found that for moving the wooden bowl, in particular, the full Something-Sometting model is the only one that can produce fully correct subgoals.
As additional evidence, we present a quantitative evaluation of the Bridge model on the tasks from Scenes and C in Table 4. real-world task execution, the difference between the BridgeData-only model and the full Something-Somethiee qualitative samples. This because we find that the low-level goal-conditioned policy is fairly robust to hallucinations of background items, suggesting that it primarily attends to the gripper pose and the target object. However, the overall increased quality of the subgoals from the full model still produces small quantitative improvement average.
model is not as pronounced as in the
6 DISCUSSION AND FUTURE WORK
We presented a method for robotic control from language instructions that generates subgoals to guide a low-level goal-conditioned policy. The subgoals are generated by an image-editing diffusion model finetumed on video data. This system improves both zero-shot generalization to new objects and the overall precision of the policy because the subgoal model incorporates semantie benefits from pretraining and commands the low-level policy with fine-grained guidance. Our experiments show that SuSIE improves over prior techniques the CALVIN benchmark and improves visual
10

---

Bridge
Input
Smthn- Smthn
Bridge Only
From Scratch
"fold the cloth from bottom right to top left"
"open the cabinet"
In Distribution
"put the mushroom on the blue towel"
"sweep into pile"
"put the yellow bell pepper in the orange pot"
Out of Distribution
"put the yellow bell pepper in the ceramic bowl"
"move the wooden bowl to the top of the table"
"put the green spoon in the wooden bowl"
Figure (Comparison of subgoal quality) comparison between subgoal generation models: trained on both robot and video data (Bridge + only (Bridge Only): and one trained without InstructPix2Pix initialization (From Scratch). The top half (In Distribution) comes from the BridgeData V2 validation set, while the bottom half (Out of Distribution) comes from our evaluation Scenes B and C, unseen BridgeData V2. Both Internet-scale pretraining and video cotraining and important for generating high-quality subgoals, especially in the zero-shot generalization setting.
used throughout this paper; one trained on robot data
11

---

generalization and low-level control real-world manipulation tasks. In the real world, SuSIE outperforms
conditioned behavioral cloning, privileged goal-conditioned policy that gets access to ground-trute truth final goal, well the state-of-the-ard instruction-followiii approach, RT-2- X, which is trained on more than an order of magnitude more robot data (over 1.1 million trajectories for RT---XX, vs. 60k for ours).
Our method is simple and provides good performance, but it does have limitations that suggest promising directions for future work. For instance, the diffusion model and the low-level policy are trained separately. meaning the diffusion model is unaware of the capabilities of the low-level policy - is trained on the same data, but assumes that anything that reachable in the data can also be reached by the policy. We hypothesize that performance can be improved by making the diffusion model aware of the low-level policy's 's capabilities. More broadly, we found the performance of our method to often be bottlenecked by the performance of the low-level policy, suggesting that addressing either of these limitations might lead to a more performant method for importing Internet- scale knowledge into robotic manipulation.
ACKNOWLEDGMENTS
We thank Quan Vuong, Vincent Vanhoucke, Karl Pertsch, and Oleg Rybkin for their feedback on earlier versions of the paper. We thank Anurag Ajay, Zoey Chen, and Quan Vuong for their support with baseline approaches. Aviral Kumar thanks Abhishek Gupta, Sherry Yang, and Boyi Li for informative discussions. We also thank the TRC program from Google Cloud for their cloud TPU donations that were crucial for running experiments.
This research was partly supported by the Office of Naval Research through N00014-22-1-2777 AFOSR FA9550-22-1-0277 and NSF grant IIS-2150826. Kevin Black and Pranav Atreya are supported by the NSF Graduate Research Fellowship. Mitsuhiko Nakamoto is supported by the Nakajima Foundation Fellowship.
and
REFERENCES
[1] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations, 2023. URL https://
openreview.net. waaeet/fooonnnds11f22ee..
[2] Anurag Ajay, Seungwook Han, Yilun Du, Shaung Li, Abhi Gupta, Tommi Jaakkola, Josh Tenenbaum, Leslie Kaelbling, Akash Srivastava, and Pulkit Agrawal. Compositional founda- tion models for hierarchical planning. arXiv preprint 23Xi:2239008887 2023.
[3] Mohammad Babaeizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan. Fitvid: Overfitting in pixel-level video prediction. arXiv preprint
arXiv:2106.15 13195, 2020.
[4] Chethan Bhateja, Derek Guo, Dibya Ghosh, Anika Singh, Manan Tomar, Quan Ho Vuong, Yevgen Chebotar, Sergey Levine, and Aviral Kumar. Robotic offline rl from internet videos via valuc-function pre-training. 2023. URL https: //api. remanticscholas
CorpusID:2522177
[5] Greg Brockman, Atty Eleti. Elie Georges, Joanne Jang, Logan Kilpatrick, Rachel Lim, Luke Miller, and Michelle Pokrass. Introducing ChatGPT and Whis- https
per APIs. OpenAl Blog, 2023. URL
chatccinng-chtgpt--- wonddwhisper--pp..
[6] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06811 2022.
[7] Anthony Brohan, Noah Brown, Justice Carbajal Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision- Lanoon-lanuage-ctttoon models transfer web knowledge to robotic control. arXiv preprint 20ti::307.118/8 2023.
12

---

[8] Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. In Conference Robot Learning, pp. 287-318. PMLR, 2023.
[9] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 18992-18402, 2023.
[10] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, arXiv preprint (oXiv:2106001360 2021.
[11] Zoey Chen, Sho Kiami, Abhishek Gupta, and Vikash Kumar. Genaug: Retargeting behaviors to unseen situations via generative augmentation. arXiv preprint arXiv:2302.0667/ 2023.
[12] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint 1rXiv:2303.0413: 2023.
[13] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-iinttune language models. arXiv preprint arXiv:2210.1 11112211..111,,... 2022.
[14] Open X-Embodiment Collaboration, Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, Antonin Raffin, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Brian Ichter, Cewu Lu, Charles Xu, Chelsea Finn, Chenfeng Xu, Cheng Chi, Chen- guang Huang, Christine Chan, Chuer Pan, Chuyuan Fu, Coline Devin, Danny Driess, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Fed- erico Ceola, Fei Xia, Freek Stulp, Gaoyue Zhou, Gaurav Sukhatme, Gautam Salhotra, Ge Yan. Giulio Schiavi. Hao Su, Hao-Shu Fang. Haochen Shi, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky I. Liang, Jachyung Kim, Jan Schneider, Jasmine Hsu, Jeannette Bohg, Jeffrey Bingham, Jiajun Wu, Jialin Wu, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jitendra Ma- lik, Jonathan Tompson, Jonathan Yang. Joseph J. Lim, João Silvério, Junhyek Han, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Zhang, Keyvan Majd, Krishan Rana, Krishnan Srinivasan, Lawrence Yunliang Chen, Lerrel Pinto, Liam Tan, Lionel Ott, Lisa Lee, Masayoshi Tomizuka, Maximilian Du, Michael Ahn, Mingtong Zhang, Mingyu Ding. Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil Joshi, Niko Suenderhauf, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Pannag R Sanketi, Paul Wohlhart, Peng Xu, Pierre Sermanet, Priya Sundaresan, Quan Vuong. Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Martin-Martín, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Busta- mante, Sean Kirmani, Sergey Levine, Sherry Moore, Shikhar Bahl, Shivin Dass, Shuran Song, Sichun Xu, Siddhant Haldar, Simeon Adebola. Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Sudeep Dasari, Suneel Belkhale, Takayuki Osa, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Vidhi Jain, Vincent Vanhoucke. Wei Zhan, Wenxuan Zhou, Wol- fram Burgard, Xi Chen, Xiaolong Wang, Xinghao Zhu, Xuanlin Li, Yao Lu, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yueh hua Wu, Yujin Tang, Yuke Zhu, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zhuo Xu, and Zichen Jeff Cui. Open X-Embodiment: Robotic learning datasets and RT-X models. https: / htt//oootiiioo www.oooiisttraaneeee..ee wwwss-traasfoooee---err-e-
github. io, 2023.
[15] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.0 03378, 2023.
[16] Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. arXiv preprint arXiv:2302.001 2023.
13

---

[17] Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforeement learning for vision- based robotic control. arXiv preprint 1rXXi:18122000666 2018.
[18] Yuying Ge, Annabella Macaluso, Li Erran Li, Ping Luo, and Xiaolong Wang. Policy adaptation from foundation model feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 19059-19069, 2023.
[19] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pp. 5842-
5850, 2017.
[20] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18995-19012 2022.
[21] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: 1912.01603, 2019.
[22] Danijar Hafner. Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse do- mains through world models. arXiv preprint arXiv:2301.0m 104, 2023.
[23] Philippe
Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
[24] Jonathan Ho and Tim Salimans. arXiv:2207.12590 2022.
Classifier-free diffusion guidance.
arXiv preprint
[25] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Ad- vances in Neural Information Processing Systems, 33:6840-688 2020.
[26] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David Fleet, and Tim Salimans. Ima- gen video: High definition video generation with diffusion models, 2022.
[27] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. urXiv:2204.03458. 2022.
[28] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint arXiv:2201.07207, 2022.
[29] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05600, 2022.
[30] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Be-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pp. 991-1002. PMLR, 2022.
[31] Michael Janner, Qiyang Li, and Sergey Levine Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems, 2021.
[32] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning. 2022.
[33] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulatee detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1790-1790, 2021.
[34] Ivan Kapelyukh, Vitalis Vosylius, and Edward Johns. Dall-e-bot: Introducing web-scale diffu- sion models to robotics. 2023.
[35] Siddharth Karamcheti, Suraj Nair, Annie S Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-driven representation learning for robotics. arXiv preprint
arXiv:2302. /2766, 2023.
14

---

[36] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR), 2015.
[37] Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor- critic: Deep reinforcement learning with a latent variable model. Advances Neural Informa- fion Processing Systems, 33:741-752, 2020.
[38] Donghwan Lee and Niao He. Stochastic
primal-dual q-leamiig.
arXiv preprint
arXiv:1810.00290 2018.
[39] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 9493-9500 IEEE, 2023.
[40] Πya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 171 2017.
[41] Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint arXiv:2005.0766 2020.
[42] Yecheng Jason Ma, William Liang, Vaidehi Som, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. Liv: Language-image representations and rewards for robotic control. arXiv preprint 230.::2300000558, 2023.
[43] Zhao Mandi, Homanga Bharadhwaj. Vincent Moens, Shuran Song, Aravind Rajeswaran, and Vikash Kumar. Cacti: framework for scalable multi-task multi-scene visual imitation learn- ing. arXiv preprint arXiv:2212.05711, 2022.
[44] Oier Mees, Lukas Hermann, and Wolfram Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters, (4):11205- 1212, 2022.
[45] Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. Calvin: A benchmark for conuuage-onddttioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters (RA-L), 7(3)73277-7334, 2022.
[46] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open-vocabulary object detection. In European Conference on Computer Vision, pp. 728-755. Springer, 2022.
[47] Vivek Myers, Andre He, Kuan Fang, Homer Walke, Philippe Hansen-Estruch,, Ching-An Cheng, Mihai Jalobeanu, Andrey Kolobov, Anea Dragan, and Sergey Levine. Goal repre- sentations for instruction following: A semi-supervised language interface to control. arXiv preprint arXiv:2307.0011 2307.00117 2023.
[48] Suraj Nair and Chelsea Finn. Hierarchical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation. arXiv preprint arXiv: 2019.
[49] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhi Gupta. R3m: A universal visual representation for robot manipulation. ArXiv, abs/2203. 12601, 2022.
[50] Alexander Pashevich, Robin Strudel, Igor Kalevatykh, Ivan Laptev, and Cordelia Schmid. Learning to augment synthetic images for sim2real policy transfer. In 20/9 JEEE/RSJ Interna- lional Conference on Intelligent Robots and Systems (IROS), pp. 2651-2657. IEEE, 2019.
[51] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual Reasoning with a General Conditioning Layer, December 2017.
[52] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-88 8763. PMLR, 2021.
[53] Rafael Rafailov, Tianhe Yu, A. Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. Learning for Decision Making and Control (L4DC), 2021.
15

---

[54] Younggyo Sco, Kimin Lee, Stephen L James, and Pieter Abbeel. Reinforcement learning with action-free pre- tre-training from videos. In International Conference on Machine Learning, pp- 19561-19579. PMLR, 2022.
[55] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for
robotic manipulation. Conference on Robot Learning, pp. 894-906. PMLR, 2022.
[56] Jiaming Song. Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXtv:2010.02500, 2020.
[57] Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrishnan, Kuang-Huei Lee, Quan Vuong, Paul Wohlhart, Brianna Zitkovich, Fei Xia, Chelsea Finn, et al. Open-world object manipula- tion using pre-trained vision- laioonlanguage models. arXiv preprint PrXiv:2303300003 2023.
[58] Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoc V Le, and Honglak Lee. High fidelity video prediction with large stochastic recurrent neural networks. Advances in Neural Information Processing Systems, 32, 2019.
[59] Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim. Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning at scale. Conference on Robot Learning (CoRL), 2023.
[60] Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. Day- dreamer: World models for physical robot learning. In Conference on Robot Learning, pp- 2226-2240. PMLR, 2023.
[61] Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Her- nandez Abrego, Steve Yuan, Chris Tar, Yun- -ssun Sung, al. Multilingual Universal Sentence Encoder for Semantic Retrieval, July 2019.
[62] Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models projected latent space. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14556-18466, 2023.
[63] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. arXiv preprint arXiv:2102.08363, 2021.
[64] Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tompson, Anthony Brohan, Su Wang, Jaspiar Singh, Clayton Tan, Jodilyn Peralta, Brian Ichter, et al. Scaling robot learning with semanti- cally imagined experience. arXiv preprint arXiv:2302. 1550, 2023.
[65] Tony Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained biman- ual manipulation with low-cost hardware. arXiv preprint arXiv:2304.137000 2023.
16

---

CALVIN
Smthn-Smthn
Bridge
Kmin (Eq. 1)
20
11
Kmax (Eq. 1)
22
14
14
kg (Eq. 2)
2
N/A
6
CALVIN
Real-world
kiest (Alg. 1)
20
20
Table 5: (Goal sampling hyperparameters) The first two hyperparameters, Amin kmmx, are used for training the image-editing model. hi is used for training the low-level policy. All 3 of these correspond to a particular dataset. kie is test-time hyperparameter that corresponds to an evaluation setup.
A IMPLEMENTATION DETAILS
We provide implementation details for SuSIE and the baselines. Table documents goal sampling hyperparameters.
A.1 SUSIE IMPLEMENTATION DETAILS
A.1.1 IMAGE-EDITING DIFFUSION MODEL
We finetune InstructPix2Pix [9] using similar hyperparameter' to the initial InstructPix2Pix training. We use the AdamW optimizer [40] with a learning rate of le-4, a linear warmup of 800 steps, and weight decay of 0.01. We track an exponential moving average (EMA) of the model parameters with a decay rate of 0.999 and use the EMA parameters at test time. We train for 40k steps with batch size of 1024 on single v4-64 TPU pod, which takes 17 hours.
For simulation, we train the model exclusively on the CALVIN dataset. For real-world experiments, we cotrain with a sampling mixture of 60% Something- Soaeiiinnn and 40% BridgeData V2.
At test time, we use an image guidance weight of 2.5 and a text guidance weight of 7.5, We use the DDIM sampler [56] with 50 sampling steps.
A.1.2 GOAL-REACHING POLICY
use diffusion model for our goal-reaching policy since recent work has shown that diffusion- based policies can better capture multi modtiiymdalliy in robot data [12. 23], leading improved per- formance across a variety of tasks. our implementation (which follows Walke et al. [59]). the observation and goal image stacked channel-wise before being passed into a ResNet-50 im- age encoder. This image encoding is used to condition a diffusion process that models the action distribution. The diffusion process uses an MLP with 3 256-unit layers and residual connections. Following Chi et al. [12], rather than predicting a single action, we predict a sequence of 4 actions encourage temporal consistency. We use the Adam optimizer [36] with learning rate of 3e-4 and linear warmup of 2000 steps. We train with a batch size of 256 for 445k steps single v4-8 TPU VM, which takes 15 hours. We augment the observation and goal with random crops, random resizing, and color jitter. For goal sampling, we use ka 6, meaning we sample goals uniformly from the range| [0, 20), since the corresponding kimx used with BridgeData V2 is (see Table 5).
At test time, we have several options for how to predict and execute action sequences. Chi et al. [12] use receding horizon control, sampling k-length action sequences and only executing some of the actions before sampling However, we found that the robot behavior was quite jerky as the policy switched between different modes in the action distribution with each sample. Instead, similar to Zhao et al. [65]. We predict a new 4-action sequence at each timestep and execute the dimension- wise mean of the last 4 predictions for that timestep.
sequence. This strategy can make the policy more reactive.
temporal ensembling strategy
17

---

A.2 BASELINE IMPLEMENTATION DETAILS
A.2.1 MOO
MOO [57] utilizes mask to represent the target objects and incorporates it as an additional channel in the observation. Specifically, they train a sunguae-connitionn policy that takes a 4-channel image and a language command as inputs. To acquire the mask for target objects, the OWL-ViT [46] object detector is employed. This detector is an apen-vocabulary object detection model, pretrained on Intemet-scale datasets, and it is used to extract the bounding boxes of the objects of interest from the image. For tasks like "move X to Y," MOO extracts the bounding box for X, representing the object of interest, and Y, indicating the target place. A mask is then created where the pixel at the center of the predicted bounding box is assigned a value of 1.0 for and 0.5 for Y.
Extracting object entities from BridgeData V2 language annotations. In order to obtain the mask, it is necessary to extract the entities corresponding to the object of interest, denoted as X, and the target place, Y. from the language command. In the MOO paper, the authors assume that the lan- guage in their dataset is structured in a way that facilitates easy separation of X and Y. Specifically, they employ a dataset that exclusively consists of language annotations such as "pick X," "move X near Y," "knock X over," "place X upright," " and "place X into Y."
Given that the language annotations in BridgeData V2 are diverse and unstructured, it challenging to naively extract X and Y. We utilized OpenAl's 3. S--turbe-instrue model [5] to extract the object of interest and the target place (if any) from the language annotation and input them into OWL-ViT to create masks. It is worth noting that due to the unstructured nature of BridgeData V2, the extracted bounding boxes in training data are sometimes incorrect as shown in Figure 5. We then trained mask-conditioned I LCBC policy using the same architecture as described in Appendix A.2.3, Following MOO, we removed X and from the prompt and replaced the entity X with "object of interest" and the entity Y with "target place". For example, given the language prompt "put the eggplant in the pot", we use a modified prompt "put object of interest in target place" as the input the policy during both train and test time.
Test time. During test time, we manually give ground-truth masks to the policy. To enable this, we build simple interface on the robot machine, allowing the evaluator to create the masks by clicking on the initial camera image at the beginning of each trial.
Success Cases
opon
LUOI
banana
"orange cloth"
"spoon"
"banana"
"silver pot"
Fallure Cases
manggl MACK
Slwer Do!
spatila
"blue spoon"
"triangle block"
"silver pot"
"spatula"
Figure (Examples of extracted bounding boxes for MOO). Despite our best attempts, the bounding boxes extracted at train time are noisy due to the unstructured nature of BridgeData V2. This might have led to MOO's poor performance in our evaluations.
18

---

A.2.2 UNIPI
UniPi [16] trains video diffusion model pe (T so to generate a sequence of frames given a language command and an initial frame. The original paper employs the model architecture from Imagen Video [26, 27]. To achieve higher resolution and longer videos for their real-world results, the authors leverage a 1.7B 3D UNet and four pretrained super-resolution models from Imagen Video, with 1.7B, 1.7B, 1.4B, and 1.2B parameters, respectively. Since the original models and code are not publicly available, we tried to replicate their approach in two different ways.
UniPi (HiP, Ajay et al. [2I) For the second approach, we followed the UniPi replication in Ajay et al. [2]. We trained latent video diffusion model from PVDM [62], building upon the codebase https: : /ggithu... com/ sih sihuun- calyuuuuuuuua 2u/PVDM where we added first-frame conditioning. We first trained the video autoencoder to project a video of size 16 x 128 x 128 into latent space, followed by training a PVDM-L model that uses a 2D UNet architecture. We used a Flan-T5-Ban Ba-e [13] encoder to obtain the language embeddings.
UniPi (Ours). We implemented 3D UNet video diffusion model, following Ho et al. [27, 26] and adding UniPi's first-frame conditioning. Due to limited compute, we did not train spatial/temporal super-resolution models; instead, we trained a 3D UNet-based diffusion model to directly generate images with a resolution of 128 x 128. The model includes 4 residual blocks, with (input channels, output channels) as follows: (64, 64), (64, 128). (128, 256), and (256, 640). The model is trained to produce the trajectory with a fixed horizon of 10 frames T {st. S1+1,- ... $149} conditioned on the current frame 8 and language command. We used frozen pretrained CLIP [52] encoder to obtain the language embeddings.
Data and training details. Following the UniPi baseline from HiP [2), we utilized Ego4D [20] to instill the model with Internet-scale knowledge. For UniPi (Ours), we first pretrained the video model on Ego4D for 270K steps, and finetuned it on the robotics dataset (CALVIN for simulation and BridgeData V2 for the real world) for an additional 200K steps. We used a batch size of during training. For UniPi (HiP), we jointly trained single model on Ego4D, BridgeData V2, and CALVIN at the same time. The autoencoder was trained for 85K steps, and the PVDM-L model was trained for 200K steps. We used batch size of 8 during training.
Inverse model and test time control. To extract actions from generated videos, we trained a Gaus- sian inverse dynamies model n (a 8t+1) to predict the action from two adjacent frames. The model consists of a ResNet-50 image encoder and 256-unit MLP layers. During test time, given the current observation 8, and the language command I, we synthesize H/ image frames from the video model and apply the inverse dynamics model to obtain the corresponding H - 1 actions. The predicted actions are executed, and we generate new video from H and repeat the process.
Generated videos. While the quality of the video model trained on the simulation dataset is good enough for solving the tasks on the CALVIN benchmark as shown in Table we found that is nontrivial to obtain high-quality generations for the real-world scenes. We show examples of generations in Appendix C.
A.2.3
-ONDULTINNCD BEHAVIOR CLONING (LCBC)
For the LCBC baseline, we use the same architecture and hyperparameters as the low-level policy in SUSIE. We encode the language instruction using MUSE [61] and feed it into the ResNet-50 image encoder using FiLM conditioning
[51]. We found that this policy outperformed the LCBC policy
used in Walke et al. [59] and Myers et al. [47] (which was not a diffusion policy), as well as the RT-1 architecture [6] trained on BridgeData V2 (also appearing in Walke et al. [59]).
B EXPERIMENTAL SETUP
CALVIN. We follow the evaluation setup from Mees et al. [45]. During each trial, the policy gets 360 timesteps total to complete a chain of 5 instructions, only moving to the next instruction once the previous one is completed. Subsequent instructions in each language chain are chosen by the simulator based on environment affordances after completing the previous language instruction. Re- sults in Table are averaged over 100 trajectories, enabling direct comparison with results from Ge et al. [18), which are also averaged over 100 trajectories. Results for UniPi (HiP) and UniPi (Ours)
19

---

averaged over 25 trajectories, due to the computational cost associated with querying the video diffusion models, and the relatively short video generation horizon (10 timesteps) requiring frequent regeneration.
To obtain comparisons with an oracle GCBC policy Table 3, we sampled ground truth goal image states from the simulator feed to the oracle policy by manually resetting the environment internal state to the state corresponding to the goal being achieved. This manual resetting process was only amenable for certain tasks in CALVIN, namely the ones in Table 3. Results in the table correspond to total of 200 evaluation trajectories.
Real world. Figure 2 provides exhaustive list of all tasks evaluate in the real world. We conduct 10 trials per task and report the average success rates. We vary the positions and orientations of various objects between trials, but use the same of positions and orientations for evaluating all methods. In Scene A, we change the positions and orientations of the plate, eggplant, and carrot every trial; the pot always remains the drying rack. For "put carrot on plate", we start the carrot in the pot for 4 out of 10 trials, and in the sink otherwise. The eggplant always starts in the sink. In Scene B, we change the positions of the bell pepper, orange pot, and ceramic bowl every trial, performing 5 trials where the target container is closer to the target object and 5 trials where it is further away. In Scene C, for "put X in the wooden bowl," we change the positions of the objects every trial while the wooden bowl remains on the left; also randomize the rotations of the objects up to 45 degrees from vertical. For "move the wooden bowl to the top of the table," we change the horizontal position of the bowl every trial, but it always starts the bottom of the table.
C QUALTITATIVE EXAMPLES OF VIDEO GENERATION
Figure 6 shows the video frames generated by the UniPi (HiP) model. The model fails generate the robot's gripper accurately and consistently produces blurry results. For this reason, we did not evaluate this model the real robot. Instead, world tasks. Figure 7 shows examples of success and failure cases of the test trials. While the robot succeeded few trials, we observed that in most cases the generated videos suffered from hallucination and physical inconsistency, which led to UniPi's poor performance.
evaluated UniPi (Ours) model for the real-
Figure 6: (UniPi (HiP) example generations). Observe that the model not able to generate clear robot
gripper, and produces vidcos that blurry and visually poor.
20

---

Success Cases
"put the yellow bell pepper the ceramic bowl"
Generation
Actual
Rollout
"put the green spoon the wooden bow!"
Generation
Actual Rollout
Failure Cases
"put the yellow pepper ceramic bow!"
Generation
Actual
Rollout
"put the eggplant on the plate"
Generation
Actual Rollout
Figure 7: (UniPi (Ours) example generations and the resulting robot behaviors). While the robot suc- ceeded in few trials, observed that in most cases the generated videos suffered from hallucination and physical inconsistency, which confused the low-level policy.
21

---

D QUALITATIVE EXAMPLES OF SUSIE ON CALVIN
Instruction 1: block from the sliding cabinet"
Cenerated Subgoulls
Execuled Rollout
Instruction 2: "store the grasped block in the drawer"
Oer enreedddd
Execuled
Instruction 3: "press the button to turn on the led light"
Gennretert Subgoets
Execured Rollout
Instruction "push the sliding door to the right side"
Generated Subgosle
Execused Plollout
Instruction "take the block from the drawer"
Genernted Subgoals
D
Execured Rolluut
Figure 8: (Example rollouts in CALVIN.) Visualization of SuSIE-generated subgoals and trajectory rollouts successful 5-instruction language chain CALVIN.
22


Reviews:
[{"review_id": "mS5G2DN387", "reviewer": "Reviewer_rGNF", "summary": "SuSIE enables robots to recognize and reason about new objects and scenarios using a pre-trained image editing diffusion model, InstructPix2Pix. It proposes intermediate subgoals for the robot to reach, which are fine-tuned on robot data. A low-level policy is trained to reach these subgoals, resulting in robust generalization capabilities. The approach successfully solves real robot control tasks with novel objects, distractors, and environments in both real world and simulation settings.", "questions": "1. In Algorithm 1\nI was wondering how to determine the key frames that cannot be described in a single instruction. You mentioned sampling a new subgoal every K steps. How do you choose the goal sampling interval K to ensure the subgoals are plausible enough?  \n2. In 5.2 you mention \"an understanding of dynamics to predict how to move and rotate the gripper to grasp it.\"\nHow can you determine the plausibility of the prediction in terms of dynamics, given that the generating subgoals and learning low-level control policies are separate?  \n3. It seems that an extremely accurate pose of the end effectors is needed. I am somewhat concerned that the precision of the genarated subgoal frame may not be sufficient to achieve it.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "SuSIE uses a pre-trained image editing diffusion model as a high-level planner, along with a low-level goal-conditioned policy trained to achieve these subgoals.\nThe approach successfully solves real robot control tasks that involve new objects, distractions, and environments.\nThe method is compared to existing approaches in terms of incorporating semantic information from pre-trained models.\nThe paper is well-written and easy-to-read.", "weaknesses": "The diffusion model and low-level policy are trained separately, meaning the diffusion model is unaware of the low-level policy's capabilities.\nThere is no discussion on how to handle the situation if the generated subgoal is incorrect, nor are there any mechanisms for error detection and recover.\nMost of the tasks presented in the paper appear to be easily solvable using pre-trained perception modules and traditional planning methods.\nThe core of the method InstructPix2Pix can only infer from visual information and cannot infer any physical properties of the scene that certain tasks require.\nReal-world evaluation is conducted in just three scenes, which may not fully capture the complexity and variability of actual manipulation tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0FBFc134Lk", "reviewer": "Reviewer_UFZY", "summary": "This paper tackles the problem of zero-shot generalization to novel environments, objects, and scenarios in the context of robotics dexterous manipulation. The tasks are defined by a starting and end states, provided as visual observations. The goal is then to control a robotic arm to reach the goal state from the start state.\n\nAuthors propose to leverage large pre-trained language-guided image editing diffusion models to hallucinate the next subgoal or waypoint to reach to complete a given task. The diffusion model is thus used as a high-level planner whose goal is to output a close next sub-goal to find. A low-level policy is also trained to reach a sub-goal given its current state.\n\nThe diffusion model is fine-tuned on robotic data to learn to predict a waypoint given a language intersection, while the low-level policy is trained with behavior cloning on a dataset of trajectories.\n\nThe main claim of the paper is that the knowledge coming from Internet-scale pre-training of diffusion models allows a broader generalization to new scenarios and that such models are able to provide physically realistic new sub-goals. As all high-level planning is done by the generative model, the policy only has to perform low-level short-term control to reach a close waypoint, leading to more precise manipulation.\n\nExperiments are conducted both in simulation and on a real-world robotic platform, and the introduced method is compared against diverse relevant baselines.", "questions": "All questions and suggestions are already mentioned in the “Weaknesses” section as a list of numbered points.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* **S1**: The idea is straightforward and properly motivated in the paper. Indeed, defining two modules, i.e. high-level planning and low-level control, appears to be an interesting idea to simplify the task to learn. Moreover, framing high-level planning as an image editing task and focusing on only predicting the short-term future (instead of full video prediction) is an appealing approach.\n\n* **S2**: The paper is well-written and easy to follow.\n\n* **S3**: Experiments are conducted both in simulation and on a real-world platform, and the proposed method is compared against several challenging baselines.", "weaknesses": "* **W1**: **[Major]** Quality of the predicted sub-goals: Authors present a qualitative study of predicted sub-goals in Figure 3. They claim that the predicted sub-goal images are physically plausible and of higher quality when leveraging the *InstructPix2Pix* pre-training. I find it hard to assess the quality of the generated images only from looking at these examples. This leads to 2 concerns:\n    * **W1.1**: Is the pre-training improving the downstream policy performance? The paper only presents a qualitative comparison of predicted sub-goals. Authors should report quantitative results, controlling the low-level policy to reach subgoals generated by the 2 models (with and without InstructPix2Pix pre-training) to show that the Internet-scale pre-training brings a boost in performance.\n    * **W1.2**: Are the predicted subgoals really physically plausible, and is it a pre-requisite for the low-level policy to succeed in controlling the robotic arm? By looking at Figure 3, it is not easy to verify that sub-goals are physically plausible. Further experiments should be conducted to verify this quantitatively. Moreover, authors should conduct experiments to show that having physically plausible sub-goals improves the performance of the low-level policy. A possible scenario might be that sub-goals are not always physically plausible, but through its training, the low-level policy has learned to come as close as possible to sub-goals while avoiding ending up in any undesirable state.\n\n* **W2**: **[Major]** How are findings from Figure 4 generalizable? It shows qualitative rollouts from the proposed method and RT-2-X which seem to indicate that RT-2-X manipulates objects with less precision. However, quantitative results in Table 2 show that RT-2-X outperforms the introduced method in certain tasks, showing it might be more precise in some cases.\n\n* **W3**: **[Major]** RT-2-X outperforms the method on certain tasks. However, as mentioned by the authors, RT-2-X was trained on a large robotics dataset and the proposed method still showcases promising performance on most tasks. I thus believe this smaller performance compared with RT-2-X on scene C does not outweigh the interesting contributions of this work.\n\n* **W4**: **[Minor]** To the best of my knowledge, the policy action space is not described in the main paper. A description should be added.\n\n* **W5**: **[Minor]** Even if authors properly reference the paper of interest and refer to the appendix later, the following sentence is quite vague: “To obtain a robust goal-reaching policy in Phase (ii), we follow the implementation details prescribed by Walke et al. (2023).” Additional details should be added directly in the main paper following this sentence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rAzF8YUYdN", "reviewer": "Reviewer_rx11", "summary": "The paper discusses the challenge of enabling robots to recognize and respond to novel objects and situations that were not part of their training data. While robotic manipulation datasets have expanded, they can never encompass every potential object or context a robot might encounter. To address this, the paper proposes integrating semantic knowledge into language-guided robotic control by leveraging pre-trained vision and language models.\n\nIn essence, the proposed approach, SuSIE, decomposes the robotic control problem into two phases. In the first phase, an image-editing model is fine-tuned on robot data, allowing it to generate hypothetical future subgoals based on the robot's current observation and a natural language command. These subgoals serve as intermediate objectives that guide the robot towards accomplishing the user-specified task. Importantly, the image-editing model doesn't need to understand how to achieve these subgoals; it only generates them based on semantics. In the second phase, a low-level goal-reaching policy is employed to help the robot reach these hypothetical subgoals. This policy focuses on inferring visuo-motor relationships for correct actuation but doesn't require a deep understanding of semantics. By breaking down the task into smaller, semantically guided subgoals, the overall problem becomes more manageable. SuSIE is compared against a diverse set of approaches, and shown to achieve comparable or sometimes better performance.", "questions": "-\tThe figures depict a properly selected sub-goals for the scenes, such as a pre-grasp pose in figure 1. However, it is unclear if or how the sub-goal synthesis procedure is optimized to output such intuitive sub-goals, rather than less important/effective transitionary scenes that simply happen to follow the input pose. In essence, how do you select/optimize for the feasible/relevant subgoals?\n- Table 1 uses some success rates from another paper (Ge et al., 2023). However, the approach proposed in this cited paper achieves higher success rates (see 'Ours' row in Table 2 of Ge et al.) than the ones reported on this paper. Why did you omit this result? Ge et al. seem to also address out-of-distribution generalization. Other than 1-step chain case (theirs 72% vs. SuSIE 75%), the longer chains achieve better results than SuSIE.\n-\tIt was mentioned that Scene C should be a more difficult task than Scene B, however the results in 5.3 show that both MOO, RT-2-X and SuSIE perform significantly better on scene C than on B. Can you explain what the reason might be? Is it related to the VLM or the low-level control policy?\n-\tThe model is illustrated to depict sub-goals nicely as it is able to distinguish between particular objects to orient the gripper towards. But there are non-visual properties of these objects that low-level controllers need to consider to perform pick and place tasks, such as weight and friction. This model however seems to rely on visual properties only. Would the robot in this pipeline be able to pick up such challenging objects, such as a heavy and slippery object, even after a few tries?\n-\tInstructPix2Pix is not capable of performing viewpoint changes, how would this challenge be reflected in a real-life or research scenario where there are often multiple or moving cameras present?\n-\tOften, the gripper may be obscured in the camera view due to other objects, such as the rest of the robot. The first phase may output these subgoals as a suggestion of pre-grasp or grasp pose. However, due to decoupling, the low-level controller would have no way of knowing this is the case. In the image of the scene, any pose of the gripper would look the same. How would the scene play out, would the robot be able to recover or get stuck?\n-\tCan you elaborate on the role/effect of the conditional policy (Section 4.2)? It seems to enforce the robot to follow the trajectory sampled in the dataset when it encounters a similar state and subgoal. What happens when it encounters, say, a similar current state but a different subgoal (that was not in the dataset) during test time?\n-\tDoes your robot dataset consist of only completely successful trajectories? If not, could a subgoal be generated for failure (e.g., robot holds the desired object in the current state, but subgoal generates a state where it is dropped? If so, how do you handle such cases?\n-\tThere is a minor error in the table references. In Section 5.3 and A2.2.2, you mentioned \"Table 5.3,\" which refers to \"Table 1.\"", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Using VLMs to synthesize subgoals is a novel approach (however, it's unclear how these subgoals are trained - see questions).\n- Decoupling the subgoal generation and the low-level control policy learning components seems to be effective (but authors seem to have contradicting opinions, see my comments in the questions part).\n- Experiment setup was sound (different scenes with seen/unseen objects, with increasing difficulty by involving more unseen objects and other clutter)\n- A good amount of comparative analysis is presented. The authors present both simulation and real-life experiments. Furthermore, they have shown instances where their models perform well, but they have also demonstrated cases where their models perform worse than previous models and provided valid explanations for these outcomes.", "weaknesses": "- Subgoal training procedure is unclear - see questions.\n- Some parts could be reformulated/reworded better in Section 4 to reduce ambiguities and improve reproducibility.\n  - In 4.1, “k” is used for two different contexts (hyperparameter and index).\n  - In 4.1, “q” distribution appears to be a uniform discrete distribution in [i, i + k], but it remains undefined.\n- The information regarding the datasets used can be provided more comprehensively and at an earlier stage in the paper.\n- Some results are missing and/or unclear - see questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7IDd1MyinR", "reviewer": "Reviewer_choL", "summary": "This paper introduces the interesting approach to utilize the Internet-scale semantic knowledge for solving robotic tasks by leveraging pre-trained image-editing models. The main components consist of two part: 1) Image-editing model that generates future subgoals given the current state and task description in language. This model is first pre-trained with Internet-scale data and fine-tuned with in-domain robotic data. This guides low-level policy by specifying near future subgoals. 2) Low-level goal-conditioned policy that accomplishes generated subgoals. The method is evaluated on simulated (CALVIN) and real-world environments, where the proposed method shows better performance in generalization ability that deals with solving novel tasks, objects and environments.", "questions": "- It would be better if the authors could provide additional experiments with the baselines which utilize low-dimensional image embeddings as subgoals rather than high-dimensional raw images to show that image-editing model is effective as a subgoal generator.\n- When learning the goal-reaching policy, the model is learned to provide subgoals within k steps of any state. But I wonder if how much the change in k value makes a difference in performance. It would be nice to see additional experiments showing changes in performance according to the value of k.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper starts with the interesting questions such as \"how to effectively inject semantic knowledge into robot control problems?\" and \"how to better utilize the Internet-scale foundation model to solve the problem of robot learning?\", which are quite important problems these days.\n- This paper is well-written and easy to follow. The idea is very simple, yet consistently perform well in overall experiments and shows effectiveness.\n- The experimental settings are well designed for the purpose the authors want to show such as zero-shot generalization to novel objects and environments, so the experimental results seem to show effectively that the proposed method is better.", "weaknesses": "- I think the analysis of the results of the real-world experiment is a little insufficient. In the results shown in Table 2, RT-2-X performs much better on scene C, so why does it perform better only on scene C? RT-2-X is said to have been learned through larger model and more data, but that does not show such performance in scene A and B, but analysis on this part seems to be insufficient.\n- In order to generate subgoal well, an overall temporal understanding of the task is required. But there is not much clear analysis or intuition why the fine-tuned image-editing model performs better than the video prediction model.\n- The results from the simulated environment(CALVIN) don’t show much significant performance difference from other language-conditioned policy baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)