Generative Image as Action Models
Authors: Mohit Shridhar
Yat Long Lo
Stephen James
Venue: CoRL 2024 (2024)
Keywords: Diffusion Models
Image Generation
Behavior Cloning
Visuomotor
TL;DR: Stable Diffusion can be fine-tuned to draw joint-actions for visuomotor control.

Abstract: Image-generation diffusion models have been fine-tuned to unlock new capabilities such as image-editing and novel view synthesis. Can we similarly unlock image-generation models for visuomotor control? We present GENIMA, a behavior-cloning agent that fine-tunes Stable Diffusion to “draw joint-actions” as targets on RGB images. These images are fed into a controller that maps the visual targets into a sequence of joint-positions. We study GENIMA on 25 RLBench and 9 real-world manipulation tasks. We find that, by lifting actions into image-space, internet pre-trained diffusion models can generate policies that outperform state- of-the-art visuomotor approaches, especially in robustness to scene perturbations and generalizing to novel objects. Our method is also competitive with 3D agents, despite lacking priors such as depth, keypoints, or motion-planners.

slide book and up Generative Image as Action Models
Mohit Shridhar Yat Long , Stephen James* Dyson Robot Learning Lab, "Equal Contribution
genima-robot. github.io
target elbow
target
target base
target gripper
open the
slide block
up the toilet
the hanger
hang the scarf on the stand
touch the button with your elbow
lift transparent bag
Abstract: image-generatie diffusion models have been fine-tuned to unlock new capabilities such as image-editing and novel view synthesis. Can we similarly unlock mange-generation models for visuomotor control? We present GENIMA, a behavior-cloning agent that fine-tunes Stable Diffusion to "draw joint-actions" as targets on RGB images. These images are fed into a controller that maps the visual targets into a sequence of joint-positions. We study GENIMA on RLBench and 9 real-world manipulation tasks. We find that, by lifting actions into image-space, internet pre-trained diffusion models can generate policies that outperform state- of-the-an visuomotor approaches, especially in robustness to scene perturbations and generalizing to novel objects. Our method is also competitive with 3D agents, despite lacking priors such as depth, keypoints, or motion-planners.
Keywords: Diffusion Models, Image Generation, Behavior Cloning, Visuomotor
Introduction
Image-generation diffusion models [1, 2, 3] generalists in producing visual-patterns. From photo-realistic images [4] to abstract art [5], diffusion models can generate high-fidelity images by distilling massive datasets of captioned images [6]. Moreover, if both inputs and outputs are in image-space, these models can be fine-tuned to unlock new capabilities such as image-editing [7, 8), semantic correspondences [9, 10], or novel view synthesis [11, 12]. Can we similarly unlock image- generation models for generating robot actions?
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

"apen the box g
"open the box" g
at2
24
SD-Turbo ControiNet
Controller
reaches targets
draws targets
Tiled RGB O
image Targets A++K
Joint Positions A,
Figure J. GENIMA Overview. GENIMA is behavior-cloning agent that maps multi-view RGB observations and language goals joint- position actions. GENIMA is composed of two stages: (1) SD-Turbo [27] is fie-tuned with ControlNet [8] to draw target joint-position, which from the + timestep expert demonstrations. Each joint input into ACT [19, 23] controller, which translates them into sequence of K joint-positions The controller is trained to ignore background context by using random backgrounds (see Higure 2). Both stages are trained independently and used sequentially during inference.
as a uniquely colored sphere. (2) generated targets are
Prior works in roboties have used image-generation for subgoal generation [13, 14, 15, 16), data- augmentation [17, 18, 19], and features-extraction for 3D agents [20, 21J. Subgoal generation pre- dicts goal images as targets, however, producing exact pixel-level details about interactions with de- formable objects, granular media, and other chaotic systems, is often infeasible. Data-augmentation methods randomize scenes with image-generation outputs to improve robustness to lighting, tex- tures, and distractors, but they do not use image-generation models to directly generate actions. 3D agents use pre-trained features from diffusion models improve generalization, however, they rely on privileged information such depth, keypoints, task-specific scene bounds, and motion-planners.
In this work, we use image-gent genereteeerionnnn models in their native formulation: drawing images. We present GENIMA, a multi-task behavior-cloning agent that directly fine-tunes Stable Diffusion [1] to "draw joint-actions". To supervise the fine-tuning, we format expert demonstrations into an image-to-image dataset. The input an RGB image with a language goal, and the output is the same image with joint-position targets from future timestep in the demonstration. The targets are rendered colored spheres for each joint (as shown in the previous page). These visual targets are fed into controller that maps them to sequence of joint-positions. This formulation frames action retion-gennration as an sengee-gnrrttionn problem such that action action-patterns become visual-patterms
We study GENIMA on 25 simulated and 9 real-world tasks. In RLBench [22], GENIMA outperforms state-of-the-ar visuomotor approaches such as ACT [19, 23] in 16/25 tasks, and DiffusionPoli- cies [24] in 25/25 tasks. More than task performance, we show that GENIMA robust to scene perturbations like randomized object colors, distractors, and lighting changes, and also in generaliz- ing novel objects. We find that RGB-to-joint methods can approach the performance of privileged 3D next- next-best-por best-pose methods [25, 26] without using priors like depth. keypoints, or motion-plannen planners. We validate these results with real-world tasks that involve dynamic motions, full-body control, transparent and deformable objects. In summary, our contributions are:
A novel problem-oormmneeeeeeee formulation that frames joint-action action generation as image-generation.
A proof-of-concept system for drawing and executing joint-actions.
Empirical results and insights from simulated and real-world experiments.
Our code and pre-trained checkpoints are available at genima-robot. github... io.
2 GENIMA
GENIMA is behavior-cloning agent that maps RGB observations O, and language goal g to joint-position actions A The key objective is to lift actions into image-space such that internet- pretrained diffusion models can learn action-pant patterns
This is accomplished through a two-stage process: (1) fine-tuning Stable Diffusion [1] to draw target joint-positions A image on input images, K timesteps ahead, and (2) training a controller to translate these targets into to a sequence of executable joint-positions A, jouns (an, at,... atx This simple two-stage process offloads semantic and task-level reasoning to a generalist image-generatee model, while the con- troller reaches nearby joint-positions indicated by the visual targets. The sections below describe the two stages in detail, and Figure I provides an overview.
t+k

---

2.1 Diffusion Agent
The diffusion agent controls what to do next. The agent takes language goal g and multi-view RGB images O, as input, and outputs target joint-positions A smage
(+K on the same images. This problem formulation is a classic image-to-image setting, so any fine-tuning pipeline [7, 28, 29] can be used. We specifically use ControlNet [8] to preserve spatial-layoute and for data-efficient fine-tuning.
Fine-Tuning Data. To supervise the fine-tuning, we randomly sample observations and target joint-positions from + K timesteps in expert demonstrations. For that timestep, obtain 6- DoF' poses of each robot joint, which is available through robot-APIs (from forward-kinematies), We place spheres at those poses with pyrender', and render them on four camera observations O, = {o o,
: o,," righ with known intrinsics and extrinsics. On a 7-DoF Franka Panda, we only render four joints: base, elbow, wrist, and gripper, to avoid cluttering the image. Each joint is represented with an identifying color, with separate colors for gripper open and close. The spheres include horizontal stripes parallel the joint's 's rotation axis, acting as graduations indicating the degree of rotation. Only horizontal stripes are necessary each joint has only rotation axis. The stripes also asymmetric across the poles to help identify the sphere orientation.
Fine- Tine-Tuning with ControlNet. Given an image-to-image dataset, we finetune Stable Diffusion [27] with ControlNet [8] to draw targets on observation images. ControlNet is a two-stream architecture: one stream with frozen Stable Diffusion UNet that gets noisy input and language descriptions, and trainable second stream that gets conditioning image modulate the output. This architecture re- tains the text-to-ine to-image capabilities of Stable Diffusion, while fine-tuning outputs to spatial layouts in the conditioning image. GENIMA uses RGB observations as the conditioning image to draw precise targets. We use SD-Turbo [27] a distilled model that can generate high-quality images within I to 4 diffusion steps - as the base model for fine-tuning. We use the Hugging implementation [30]² of ControlNet without modifications. See Appendix D for more details on fine-tuning.
Tiled Diffusion. Fine-tuning Stable Diffusion robot data poses three key challenges. Firstly, Stable Diffusion models work best with image resolutions of 151225511 512 or higher due to their training data. In robotics, large images increase inference latency. Secondly, multi-view generation suffers from inconsistencies across viewpoints, However, multi-view setups are crucial to avoid occlusions and improve spatial-robustnet Thirdly, diffusion is quite slow, especially for generating four target images at every timestep. Inspired by view-synthesis works [31, 32], we solve all three challenges with a simple solution: tiling. We tile four observations of size 256 256 into single image of 512 512. Tiling generates four images Hz on an NVIDIA A100 or 4 Hz on RTX 3090.
2.2 Controller
The controller translates target images A A+K into executable coint-positions A. The controller can be implemented with any visuomotor policy that maps RGB observations to joint-positions. We specifically use ACT [23, 33] -a a Transformer-bast speed and training stability. However, in theory, any framework like Diffusion Policies [24] or RL-based methods [35] can be used. Even classical controllers can be used if pose-estimates of target spheres are provided, but in our implementation, opt for learned controllers for simplicity.
image
policy architecture [34] - for its fast inference-
Training. During training, the controller receives current joint- positions, the language goal, and RGB images with ground-truth targets overlaid on random backgrounds. The random back- grounds, as shown in Figure 2, force ACT follow targets and ignore any contextual information in the scene. We use the same hyperparameters and settings from the original ACT codebase" with minor modifications. To improve robustness to fuzzy diffu- sion outputs, we augment images with random-crops [36], color
training
inference
Figure During tmining, the controller's input are ground-truth targets with random backgrounds (left). During inference, gets are from the diflusion agent (right).
html
3

---

jitters, elastic transforms, and Gaussian noise. We use L1 loss for joint-actions, and cross-entropy loss for gripper open and close actions. For liguuageconnitiooing we use FiL. [37] layers follow- ing MT-ACT [19]. The controller is trained independently from the diffusion agent. See Appendix for hyperparameters and Appendix E for controller details.
Inference. During inference, the controller gets target images from the diffusion agent. The con- troller predicts a sequence of K joint-actions, and executes K actions or less before querying the diffusion agent in closed-loop fashion. The controller runs at ~ 50 Hz on an NVIDIA RTX 3090.
Experiments
We study GENIMA in both simulated and real-world environments. Specifically, we are interested in answering the following questions:
3.1 How does GENIMA compare against Mate-of-art -rt visuomotor policies and 3D baselines?
3.2 What are the benefits of drawing actions with internet-pretraing image-generation models?
3.3 Which factors affect GENIMA's 's performance?
3.4 How well does GENIMA perform on real-world tasks?
We start with benchmarking method in simulated environments for reproducible and fair com- parisons. The following sections describe our simulation setup and evaluation methodology.
Simulation Setup. All simulated experiments are set in CoppeliaSim [38] interfaced through PyRep [39]. The robot is 7-DoF Franka Emika Panda placed behind a tabletop. Observations are captured from four RGB cameras: front, left beft_shoulder right_ shoulder, and wrist, each with resolution of 256 256. The robot is commanded with joint-position actions via PID control, end-effector actions via IK solver.
25 RLBench Tasks. We choose 25 (out of 100) tasks from RLBench [22]. While most RLBench tasks are suited for discrete, quasi-static motions that benefit 3D next-best-pose agents [25, 26], we pick tasks that are difficult to execute with end-effector control. Tasks such as open box and open microvave require smooth non-linear motions that sampling- solvers struggle with. Each RLBench task includes several variations, but only use variationd to reduce training time with limited resources. However, method should be applicable to multi- variation settings without any modifications. We generate two datasets: 50 training demos and 50 evaluation episodes per task. For both datasets, objects are placed randomly, and each episode is sanity-checked for solvability. Language goals are constructed from instruction templates. See Appendix A for details on individual tasks.
motion-plant paation-llanners and IK
Evaluation Metric. Multi-task agents trained all 25 tasks, and evaluated individually each task. Scores either 0 for failures 100 for successes, with no partial successes. We report average success rates on 50 evaluation episodes across the last three epoch checkpoints: 50x3 = 150 episodes per task, which adds up to 150 x 25 3750 in total. We use single set of checkpoints for all tasks without any -aec-fpe optimizations or cherry-picking.
Visuomotor Baselines. We benchmark GENIMA against three state-of-the-art visuomotor ap- proaches: ACT [19, 23), DiffusionPPllicess [24), and SuSIE [13]. ACT is transformer-based policy that has achieved compelling results in bimanual manipulation. Although GENIMA uses ACT as the controller, our controller has never seen RGB observations from demonstrations, just sphere targets with random backgrounds. DiffusionPPlliies is a widely adopted visuomotor approach that generates
trajectories through diffusion. SuSIE is the closest approach to GENIMA, instead of drawing target actions, SuSIE generates target RGB observations as goal images. We adapt SuSIE to our setting by training a controller that maps target and current RGB observations to joint- position actions. All multi-task baselines conditioned with language goals. ACT and DiffusionPoli Poliiiss use FiLM conditioning [37], whereas SuSIE uses the goal prompt.

---

Diff. Policy
Diff.
Task
GENIMA
ACT
SuSIE
Actor
GENIMA
baeketball hoop
50.0
32.7
5.3
100
insert usb
26.0
18.0
0.0
29.3
Absolute
nove banger
open box
79.3
94.0
69.3
42.0
21.3
36.0
0.0
76.0
6.0
Action Modee
Delta
26.7
open
26.6
open draver
77.3
82.7
67.3
71.3
grill
48.7
40.0
26.0
0.0
93.3
10 aciiions
22.9
open nicrowave
46.7
22.6
10.0
58.0
Prediction Horizo
actions
open
vindow
69.3
8.0
24.6
96.7
phone en base
18.7
13.3
1.0
2.0
94.0
pick up cup
36.0
43.3
24.6
92.7
10 10
35.0
play jenga
90.0
92.0
Execution Horizon
octions
26.7
sxitch
72.7
65.3
74.7
22.7
83.3
Tuction
push button
76.7
31.3
7.3
2.7
46.7
put books on shelt
14.7
44.0
1.0
36.7
SDAL-Turbo
put knife on board
12.7
77.3
Diffusion Sottings
put rubbish bin
26.7
15.3
6.7
96.0
Non-Tlled
$9,9
acoop with spatula
11.3
22.7
1.0
0.0
66.0
No Stripes
34,4
slide black plate off
12.7
22.0
0.0
0.0
99.3
72.0
Coll-Co ACT
56.4
21.3
37.3
4.0
72.7
Controller Settings
Full-Goneect
toilet seat up
93.3
45.3
50.0
2.0
94.0
Randon Back
burn on lamp turn tap
12.0
19.3
6.0
4.0
4.0
average
49.6
39.6
21.8
2.9
73.8
Table I. Visuomotor and 3D Baselines on 25 RLBencc tasks, Success rates opisedes per task. We report average scores across the last three checkpoints. The four methods best pose
Figure Ablations and Sensitivity Analyses. We study factors that affect GENIMA'S performance by training multi-usk agent on 3 tasks: take lid off, open success rates across the tasks.
for multi-takk agents trained with
ovaluated
RGB to joint agents. rightmost method agent with extra priors: depth, keypoints, scene bounds, and motion- palionns
report
3D Baseline. We also benchmark GENIMA against 3D Diffuser Actor [40] state-of-the-ee agent RLBench. 3D Diffuser Actor uses CLIP [41] to extract vision and language features, and diffuses end-effector poses with 3D transformer. We 3D Diffuser Actor as the best-performing representative of 3D next-best-pose agents [25, 26] such as PerAct [42], Hiveformer [43], RVT [44], Act3D [45], DNAct [20], and GNFactor [21]. These works rely on several priors: depth cameras, motion-planners to reach poses, keypoints that segment trajectories into bottlenecks, task-specific scene bounds, and quasi-static assumption for motions.
3.1 Visuomotor and 3D Baselines
Our key result is that we show GENIMA an image-genen -eneeeteeeriionnn model fine-tuned to draw actions works at all for visuomotor tasks. the sections below, we go beyond this initial result and quantify GENIMA'S performance against state-of-the-ar visuomotor and 3D baselines.
GENIMA outpeforms ACT, DiffusionPoliciee and SuSIE. Table presents results from RLBench evaluations. GENIMA outperforms ACT [19, 23] in 16/25 tasks, particularly in tasks with occlusions (e.g., open window) and complex motions (e.g., turn tap). Against SuSIE [13], GENIMA per- forms better on 23/25 tasks, as SuSIE struggles to generate exact pixel-level details for goals. Dif- fusionPolicy [24] performs poorly in multi-task settings with joint-position control. We ensured that our implementation correct by training DiffusionPolie on just take lid off, which achieved reasonable success rate of 75%. but we could not scale 25 tasks.
RGB-to-joint agents approach the performance of 3D next-best-pose agents. Without 3D input and motion-planners most prior works [42, 46] report zero-performance for RGB-only agents in RLBench. However, our results in Table show that RGB-to-joint agents like GENIMA and ACT can be competitive with 3D next-best-pose agents. GENIMA outperforms 3D Diffuser Actor in 6/25 tasks, particularly tasks with non-linear trajectories (e.g., open box, open door), and tiny objects (e.g., turn on lamp). GENIMA also performs comparably (within 3%) on more tasks: insert usb, play jenga, toilet seat up, despite lacking priors. 3D Diffuser Actor performs better on most tasks, but training GENIMA for longer or with more data could bridge this gap.

---

Object Color
Distractors
Lighting
Table Texture
Background Texture
Camera Pose
GENIMA
33%%
-14%
-7%
-10%
96%
ACT
-74%
-72%
-39%
-92%
0%
-94%
Figure 4. Performance drops from Colosseum [46] perturbations. We evaluate GENIMA and ACT on perturaatonn categories: randomized object part colors, distractoor objects, lightnn color brightness variations, randomized textures, randomized backgrounds, camera pose changes. We report success rates from 150 evaluation episodes per tusk, where perturbations are randomly sampled episodically. ACT overfits objects and lighring conditions, whereas GENIMA more robust to perturbations See supplementary video examples.
3.2 Semantic and Spatial Generalization
While all evaluations in Section 3.1 train and test on the same environment, the key benefit of using image-generation models is in improving generalization of visuomotor policies. In this section, we examine semantic and spatial generalization aspects of visuomotor policies.
GENIMA is robust to semantic perturbations on Colosseum tasks. We evaluate the same multi- lask GENIMA and ACT agents (from Section 3.1) on 6 perturbation categories in Colosseum [46]: randomized object and part colors, distractor objects, lighting color and brightness variations, ran- domized table textures, randomized scene backgrounds, and camera pose changes. Figure presents results from these perturbation tests. Despite being initialized with a pre-trained ResNet [47], ACT overfits and significantly drops in performance with changes object color, distractors, lighting, and table textures. Whereas GENIMA has minimal drops in performance from emergent property that reverts scenes to canonical textures and colors from the training data. See supplementary videos for examples. However, both methods fail to generalize to unseen camera poses.
GENIMA extrapolates to spatial loca- tions with aligned image-action spaces. By drawing actions images, GEN- IMA keeps the image-space and action- space aligned. This alignment has been shown to improve spatial generalization and dfff-effic in prior works [42, 48, 49]. We observe similar benefits in Fig- 5, where ACT struggles in the upper- right region with minimal training exam- ples, but GENIMA succeeds in extrapolat- ing to those locations.
Toin
62
93
44 25 06 O7
00
02
Secuzo
hallrz
Sarren
0.50
060
GENIMA
ACT
Higure S. Spatial Generalization. Train and test saucepan positions (from a top down view of tabletop) evaluation on take gles to extrapolate to the upper-right region, whereas GENIMA uses aligned image-action spaces for better spatial generalization.
strug-
3.3 Ablations and Sensitivity Analyses
We investigate factors that affect GENIMA'S performance. We report average success rates from multi-task GENIMA trained on tasks: take lid off, open box, and slide block. Our key results are presented in Figure 3, and the sections below summarize our findings.
Absolute joint-position is the best performing action-mode. Delta action-modes accumulate er- rors, and end-effector control through IK struggles with non-linear trajectories. Joint-position ac- tions are also more expressive, allowing for full-body control and other embodiments.
Longer action sequence predictions are erucial. In line with prior works [23, 24], modeling trajectory distributions requires predicting longer action sequences. We find that predicting K 20 actions is optimal, since observations are recorded at 20Hz in RLBench.
Longer execution horizons avoid error accumulation. Shorter execution horizons lead to jerky motions that put the robot in unfamiliar states. We find that executing all 20 actions works best.
SDXL improves performance over SD. Larger base models such SDXL-Turbo [27] have more capacity to model action-patie pattens. Newer Transformer-based models [50] might scale even better.

---

Tiled diffusion improves generation speed while keeping performance. Tiled generation of four target images takes 0.2 seconds, whereas generating individual images takes 0.56 seconds. Both methods achieve similar performance, however tiled generation is more multi-view consistent across a wider set of tasks. See Appendix G for reference.
Without stripes on spheres, performance drops in half. These stripes act as graduation indicators for joint-angles. Including stripes helps Stable Diffusion learn joint rotations as a visual-pattern.
Full-context controllers overfit to observations. Instead of random backgrounds, controllers are trained with target spheres overlaid on RGB observations from demos, they tend to ignore the targets, and just use observations for action prediction. This hurts performance and generalization.
ACT works better than DiffusionPolicy the controller. Similar to results in Section 3.1, we find that ACT is better at joint-action prediction than DiffusionPollicy (DP). ACT also has faster inference speed of 0.02 seconds, whereas DiffusionPolicy takes 0.1 seconds (for 20 diffusion steps).
GENIMA data-efficient. We study data-efficiency by constraining poses of objects in training demos. Follow- ing R&D [49], we sample poses in grid-style that maximizes workspace coverage based on the number of de- mos. GENIMA achieves 80% of the peak performance with 25 demos.
Data Efficiency
Diffusion Spoed
82-
Succosss Batt
Rate
take
ppon box
siide block
open
humber Danus
Figure 6. Data-Efficiency and Diffusion Speed of GENIMA.
GENIMA works with 5 diffusion steps or more. With SD-Turbo [27] the base model, GENIMA can generate target images with just 5 diffusion steps within 0.2 seconds. Future works can use better schedulers and distillation methods to further improve generation speed and quality.
3.4 Real-robot Evaluations
In-Distribution
Out-of-Distrriattion
Task
GENIMA
ACT
GENIMA
ACT
Category
We validate our results by benchmark- ing GENIMA and ACT on real-robot setup. Our setup consists of a Franka Emika Panda with 2 external and wrist cameras. We train multi-task agents from scratch on tasks with 50 demos per task. These tasks involve dynamic be- haviors (e.g., slide book), transparent objects (e.g., lift bag), deformable ob- jects (e.g., hang scarf), and full-body control (e.g., elbow touch). See Figure 7 and supplemen- tary videos for examples. Appendix B covers task details. When comparing GENTMA and ACT, we ensure that the initial state is exactly the same by using an image-overlay tool to position objects. Table reports average success rates from evaluation episodes per task. We also report out- of-distribution performance with unseen objects and scene perturbations. In line with Section 3.2, GENIMA is better than ACT at generalizing to -uttof-distribution tasks. GENIMA also exhibits some recovery behavior from mistakes, but DAgger-style [51] training might improve robustness.
lid off place teddy
60
20
new saucepan
100
80
100
60
new toy
new background
hang acart
40
60
60
20
new scarf
put marker
40
20
moving objects
slide book
100
20
100
darker lightngg
avoid lamp
0
new abject
lift hag
40
40
distractios
flip cup
20
80
20
40
new cup
Table 2, Real-robol Results. Success rales multi task GENIMA and ACT om real-world tasks, evaluated on episodes per
4 Related Work
Visuomotor agents map images to actions in an end-to-end manner [52, 53, 54]. ACT [23] uses a transformer-basee policy architecture [34] to encode ResNet [47] features and predict action-chunks. MT-ACT [19] extends ACT to the multi-task settings with anguuge-conditioning. MVP [55] uses self-supervised visual pre-training on in-the-wild videos and fine-tunes for real-world robotic tasks. Diffusion Policy [24] uses the diffusion process learn multi-modas trajectories. RT-2 [56] fine- tune vision-language models to predict tokenized actions. Octo [57] can be adapted to new sensory
7

---

IC
(d)
Figure 7. Real-world Tasks. 5 of tasks: (a) rlip cup, hang scart, (c) albox touch, 1id off, and (e) slide book.
inputs and action spaces by mapping inputs into a common tokenized format. All these methods adapt paradigms from vision and language like tokenization and pre-training for predicting actions.
In contrast, GENIMA lifts actions back into image-space use mmage-generation models natively.
3D next-best-pon agents encode 3D input and output 6-DoF poses [25, 26]. These poses are keypoints (or bottlenecks in the trajectory) that are executed with a motion-planner or IK-solver. C2F-ARM [26] and PerAct [42] use calibrated multi-camera setups to voxelize scenes into 3D grids. Voxel grids are computationally expensive, so Act3l [45] and 3D Diffuser Actor [40] replace voxels with sampled 3D points. RVT [44] renders RGB-D input into orthographic projections, and detects actions in image-space. GNFactor [21] and DNAct [20] lift Stable Diffusion features into 3D to improve generalization. Chained Diffuser [58] and HDP [59] use diffusion-based policies as low- level controllers to reach keypoints predicted by 3D next-best-
agents. All these 3D next-best- pose agents rely on several priors: depth cameras, keypoints, task-specific scene bounds, and/or motion-planie ponion-planners.. Whereas GENIMA is simple RGB-to-joint agent without any of these priors.
Diffusion models for robot learning. In addition to modeling policies [60, 61, 62, 63, 64] with diffusion [65, 66), diffusion models have been used in robot learning in various ways. This includes offline reinforcement learning [61, 63, 67], imitation-learning [68, 69, 70], subgoal generation [16, 71, 72]. and planning [58, 59, 73, 74, 75, 76]. Others include affordance prediction [77], skill acquisition [78, 79] and chaining [80]. reward functions [81], grasping [82]. and sim-to-real [83].
Image-generation models in robot learning have been used for out-of-distribution data genera- tion [69], and video-conditioned policies [15, 84, 85]. ROSIE [18] and GenAug [17] use image- generation outputs to augment datasets. SuSIE [13], RT-Sketch [86], and RT-Trajectory [87], condi- tion policies on goals generated by image generation models in the form of observations, observation sketches, and trajectory sketches, respectively. GENIMA does not use image-generation models to generate observations, videos, or trajectories, but instead draws joint-actions as targets.
Representing actions in images has been shown improve apatial-robusten and generalization. PIVOT [88] annotates observations with markers to query vision- laniuagl models for end-effector actions. RoboTTap [89], ATM [90], and Track2Act [91] track dense points on images to learn end- effector policies. R&D [49] renders grippers on images to aid the diffusion process. Yang et al. [92] plan by in-painting navigation actions. C3DM [93] iteratively zooms-into images to predict 6-DoF poses. In comparison, GENIMA does not track points, and learns joint-actions in image-space.
Conclusion and Limitations
We presented GENIMA, a multi-task agent that fine-tunes Stable Diffusion to draw joint-actions. Our experiments both in simulation and real-world tasks indicate that fine-tuned image-genene eegerggenerttion models are effective in visuomotor control. While this paper is proof-of-concept, GENIMA could be adapted to other embodiments, and also draw physical attributes like forces and accelerations.
GENIMA is quite capablc, but not without limitations. Like all BC-agents, GENIMA only distills ex- pert behaviors and does not discover new behaviors. GENIMA also uses camera calibration to render targets, assuming the robot is always visible from some viewpoint. We discuss these limitations and offer potential solutions in Appendix But overall, we are excited about the potential of pre-traines trained diffusion models in revolutionizing robotics, akin to how they revolutionized image- genge-ggneration.

---

Acknowledgments
Big thanks the members of the Dyson Robot Learning Lab for discussions and infrastructure help: Abdi Abdinur, Nic Backshall, Nikita Chemyadev, lain Haughton, Yunfan Lu, Xiao Ma, Sumit Patidar, Younggyo Seo, Sridhar Sola, Eugene Teoh, Jafar Uruç, and Vitalis Vosylius. Special thanks to Tony Z. Zhao and Cheng Chi for open-sourcing ACT and Diffusion Policies, and the HuggingFace Team for diffusers.
References
[1] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image syn- thesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[2] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot texi-to-image generation. In International conference on machine learning, pages 8821-8831. Pmlr, 2021.
[3] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall. Dreamfusion: Text-to-3d using 2d diffu- sion. arXiv preprint arXiv:2209. /4988, 2022.
[4] A. Gupta, L. Yu, K. Sohn, X. Gu, M. Hahn, L. Fei-Fei, I. Essa, L. Jiang, and J. Lezama. Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06622 2023.
[5] Podell, Z. English, K. Lacey, A. Blattmann, Dockhom, J. Müller, J. Penna, and R. Rom- bach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
[6] Schuhmann, Beaumont, Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, Katta, C. Mullis, M. Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 25278-25294, 2022.
[7] T. Brooks, A. Holynski, and A. A. Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392-18402, 2023.
[8] L. Zhang, A. Rao, and M. Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[9] L. Tang, M. Jia, Q. Wang, C. Phoo, and B. Hariharan. Emergent correspondence from image diffusion. Advances Neural Information Processing Systems, 36: 1363-1389, 2023.
[10] E. Hedlin, G. Sharma, S. Mahajan, H. Isack, A. Kar, A. Tagliasacchi, and K. M. Yi. Un- supervised semantic correspondence using stable diffusion. Advances in Neural Information Processing Systems, 36, 2024.
[11] R. Liu, R. Wu, B. Van Hoorick, P. Tokmakov, S. Zakharov, and C. Vondrick. Zero-1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
[12] Y. Shi, P. Wang, J. Ye, M. Long, K. Li. and X. Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. 2.0...611... 2023.
[13] K. Black, M. Nakamoto, Atreya, H. Walke, C. Finn, A. Kumar, and S. Levine. Zero- shot robotic manipulation with pretrained image-editing diffusion models. arXiv preprint arXiv:2310.10633 2023.

---

[14] Y. Du, M. Yang, P. Florence, F. Xia, A. Wahid, B. Ichter, P. Sermanet, T. Yu, P. Abbeel, J. B. Tenenbaum, et al. Video language planning. arXiv preprint arXiv:2310. 10625, 2023.
[15] Y. Du, S. Yang, B. Dai, H. Dai, O. Nachum, J. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation. Advances in Neural Information Processing Systems, 36, 2024.
[16] I. Kapelyukh, V. Vosylius, and E. Johns. Dall-e-hot: Introducing web-scale diffusion models to robotics, IEEE Robotics and Automation Letters, 2023.
[17] Z. Chen, S. Kiami, A. Gupta, and V. Kumar. Genaug: Retargeting behaviors to unseen situations via generative augmentation. arXiv preprint arXiv:2302.06671 2023.
[18] T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang. J. Singh, C. Tan, J. Peralta, B. Ichter, et al. Scaling robot learning with semantically imagined experience. arXiv preprint arXiv:2302. 1550, 2023.
[19] H. Bharadhwaj, J. Vakil, M. Sharma, A. Gupta, S. Tulsiani, and V. Kumar. Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking. arXiv preprint arXiv:2309.019/6 2023.
[20] G. Yan, Y.H.. Wu, and X. Wang. Dnact: Diffusion guided multi-task 3d policy learning. arXiv preprint 24Xiv:2433044155 2024.
[21] Y. Ze, G. Yan, Y.-H. Wu, A. Macaluso, Y. Ge, J. Ye, N. Hansen, L. E. Li, and X. Wang. Gn- factor: Multi-task real robot learning with generalizable neural feature fields. In Conference on Robot Learning, pages 284-301. PMLR, 2023.
[22] James, Z. Ma, D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark & learning environment. JEEE Robotics and Automation Letters, 5(2):3019 2020.
[23] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint 20o::334.13755. 2023.
[24] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
[25] S. James and A. Davison. Q-attention: Enabling efficient learning for vision-based robotic manipulation. IEEE Robotics and Automation Letters, 7(2::612-1619, 2022.
[26] S. James, Wada, T. Laidlow, and A. Davison. Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13739-13748, 2022.
[27] A. Sauer, D. Lorenz, Blattmann, and Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:2311.1744 2023.
[28] Mou, X. Wang. L. Xie, Wu, Zhang. Z. Qi, and Y. Shan. T2i-adapter: Learning adapters dig out controllable ability for text-to-image diffusion models. In Proceedings the AAAI Conference on Artificial Intelligence, volume 38, pages 4296-4304, 2024.
[29] H. Ye, Zhang, S. Liu, X. Han, and W. Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308. 067221 2023.
[30] P. von Platen, Patil, A. Lozhkov, P. Cuenea, N. Lambert, K. Rasul, M. Davaadorj. Nair, S. Paul, W. Berman, Y. Xu, S. Liu, and T. Wolf. Diffusers: State-of-thhe diffusion models.
https://githut
cow/huggingfacc// httph//ggggignnaaaaafou...eeeeae /iwfffferffeee.o/eo//e// 2022.
[31] M. Zhao, C. Zhao, X. Liang, L. Li, Z. Zhao, Z. Hu, Fan, and X. Yu. Efficientdreamer: High-fidelity and robust 3d creation via orthogonal-vict diffusion prior. arXiv preprint arXiv:2308. 13228,322,,000 2023.
10

---

[32] J. Li, H. Tan, K. Zhang, Z. Xu, F. Luan, Y. Xu, Y. Hong, K. Sunkavalli, G. Shakhnarovich, and S. Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. arXiv preprint arXiv:2311.062 1.6214, 2023.
[33] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, t.. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[34] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In European conference computer vision, pages 213- 229. Springer, 2020.
[35] P. Arm, M. Mittal, H. Kolvenbach, and M. Hutter. Pedipulate: Enabling manipulation skills using quadruped robot's leg. arXiv preprint 17Xi::2402.10837 2024.
[36] Yarats, R. Fergus, Lazaric, and L.. Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint vcXiv:2107.09645, 2021.
[37] E. Perez, Strub, H. De Vries, V. Dumoulin, and A. Courville. Film: Visual reasoning with a general conditioning layer. Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[38] E. Rohmer, S. P. N. Singh, and M. Freese. Coppeliasim (formerly v-rep): a versatile and scalable robot simulation framework. In Proc. The International Conference on Intelligent Robots and Systems (IROS), 2013. www.coppelaaroboticscom
[39] S. James, M. Freese, and A. J. Davison. Pyrep: Bringing v-rep to deep robot learning, arXiv preprint arXiv: 17,, 2019.
[40] T.W.. Ke, N. Gkanatsios, and K. Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations. arXiv preprint 2488. 2024.
[41] A. Radford, W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, al. Learning transferable visual models from natural language super- vision. In International conference machine learning, pages 8748-8763 PMLR, 2021.
[42] M. Shridhar, Manuelli, and D. Fox. Perceiver-actor: multi-task transformer for robotic manipulation. In Conference Robot Learning, pages 785-799. PMLR, 2023.
[43] P.-L. Guhur, S. Chen, R. G. Pinel, M. Tapaswi, Laptev, and C. Schmid. Instruction-driven hisee policies for robotic manipulations. In Conference on Robot Learning, pages 175-187, PMLR, 2023.
[44] A. Goyal, J. Xu, Y. Guo, V. Blukis, Y.W. Chao, and D. Fox. Rvt: Robotic view transformer for 3d object manipulation. Conference on Robot Learning, pages 694-710. PMLR, 2023.
[45] T. Gervet, Z. Xian, N. Gkanatsios, and K. Fragkiadaki. Act3d: Infinite resolution action detection transformer for robotic manipulation. arXiv preprini urXiv:2306.178 2023.
[46] W. Pumacay, I. Singh, J. Duan, R. Krishna, J. Thomason, and D. Fox. The colos- seum: A benchmark for evaluating generalization for robotic manipulation. arXiv preprint OrXiv:2402.08191, 2024.
[47] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770- 778, 2016.
[48] A. Zeng, Florence, Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong. I. Krasin, D. Duong, V. Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference Robot Learning, pages 726-747, PMLR, 2021.
11

---

[49] V. Vosylius, Y. Seo, J. Urug, and S. James. Render and diffuse: Aligning image and action spaces for diffusion- based behaviour cloning, 2024.
[50] P. Esser, Kulal, A. Blattmann, R. Entezari, Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer,
F. Boesel, al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint Seti:::40030322000 2024.
[51] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured predic- tion to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Pro- ceedings, 2011.
[52] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1-4) 2016.
[53] S. James and E. Johns. 3d simulation for robot arm control with deep q-learning. NIPS 2016 Workshop (Deep Learning for Action and Interaction), 2016.
[54] S. James, A. J. Davison, and E. Johns. Transferring end-to-end visuomotor control from simulation to real world for a multi-stage task. In Conference on Robot Learning, pages 334-343. PMLR, 2017.
[55] I. Radosavovic, T. Xiao, S. James, P. Abbeel, J. Malik, and T. Darrell. Real-world robot learning with masked visual pre-training. In Conference on Robot Learning, pages 416-426. PMLR, 2023.
[56] Brohan, N. Brown, J. Carbajal, Chebotar, X. Chen, K. Choromanski, Ding, D. Driess, A. Dubey, C. Finn, al. Rt-2: Vision-language-actin models transfer web knowledge to robotic control. arXiv preprint arXiv:2307. 2023.
[57] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint vrXiv:2405.12213. 2024.
[58] Z. Xian, N. Gkanatsios, T. Gervet, T.W.. Ke, and K. Fragkiadaki. Chaineddiffuser: Uni- fying trajectory diffusion and keypose prediction for robotic manipulation. In 7th Annual Conference on Robot Learning, 2023.
[59] X. Ma, Patidar, Haughton, and James. Hierarchical diffusion policy for kinematics- aware multi-task robotic manipulation. arXiv preprint arXiv:2403.03890, 2024.
[60] H. Ha, P. Florence, and S. Song. Scaling up and distilling down: Language-guice guided robot skill acquisition. In Conference on Robot Learning, pages 3766-3777. PMLR, 2023.
International Conference on Machine Learning, pages 20035-20064. PMLR, 2023. [61] W. Li, X. Wang, B. Jin, and H. Zha. Hierarchical diffusion for offline decision making. In
[62] X. Li, V. Belagali, J. Shang, and M. S. Ryoo. Crossway diffusion: Improving diffusion- based visuomotor policy via self-supervised learning. arXiv preprint arXiv:2307.01849, 2023.
[63] Z. Wang, Hunt, and M. Zhou. Diffusion policies as an expressive policy class for offline
reinforcement learning. arXiv preprint arXiv:2208.06/933 2022.
[64] X. Liu, F. Weigend, Y. Zhou, and H. B. Amor. Enabling stateful behaviors for diffusion-based policy learning. arXiv preprint arXiv:2404.1253 12539, 2024.
[65] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learn- ing using nonequilibrimm thermodynamies. In International conference machine learning, pages 2256-2265. PMLR, 2015.
12

---

[66] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:684006851, 2020.
[67] A. Ajay, Y. Du, A. Gupta, J. Tenenbaum, T. Jaakkola, and P. Agrawal. Is conditional genera- tive modeling all you need for decision-makingg? arXiv preprint arXiv:2211.15657, 2022.
[68] M. Reuss, M. Li, Jia, and Lioutikov. Goal-conditioned imitation learning using score- based diffusion policies. arXiv preprint arXiv:2304.02532 2023.
[69] X. Zhang, M. Chang, P. Kumar, and S. Gupta. Diffusion meets dagger: Supercharging eye- in ha-d imitation learning. arXiv preprint arXiv:2402.17766 2024.
[70] Y. Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu. 3d diffusion policy. arXiv preprint arXiv:2403.033 03955, 2024.
[71] Chen, F. Deng. K. Kawaguchi, C. Gulcehre, and S. Ahn. Simple hierarchical planning with diffusion. arXiv preprint arXiv:2401.02644, 2024.
[72] W. Liu, Y. Du, T. Hermans, S. Chernova, and C. Paxton. Structdiffusion: Language- guided creation physically-valid structures using unseen objects. arXiv preprint arXiv:2211 1.04604, 2022.
[73] M. Janner, Y. Du, J. B. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205. 09991, 2022.
[74] Z. Liang, Y. Mu, M. Ding, F. Ni, M. Tomizuka, and P. Luo. Adaptdiffuser: Diffusion models adaptive seol-evolving planners. arXiv preprint arXiv:2302.01877, 2023.
[75] Z. Liang, Y. Mu, H. Ma, M. Tomizuka, M. Ding, and P. Luo. Skilldiffuser: Interpretable hierarchical planning via skill abstractions in diffusion- based task execution. arXiv preprint arXiv:2312. 1598, 2023.
[76] Z. Wang, T. Oba, T. Yoneda, R. Shen, M. Walter, and B. C. Stadie. Cold diffusion on the replay buffer: Learning to plan from known good states. In Conference on Robot Learning, pages 3277-3291, PMLR, 2023.
[77] Y. Ye, X. Li, A. Gupta, S. De Mello, S. Birchfield, J. Song, S. Tulsiani, and S. Liu. Af- fordance diffusion: Synthesizing hand-object interactions. In Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22479-22489, 2023.
[78] L. Chen, S. Bahl, and D. Pathak. Playfusion: Skill acquisition via diffusion from language- annotated play. In Conference on Robot Learning, pages 2012-2029, PMLR, 2023.
[79] M. Xu, Z. Xu, Chi, M. Veloso, and Song. Xskill: Cross embodiment skill discovery. In Conference on Robot Learning, pages 3536-3555 PMLR, 2023.
[80] U. A. Mishra, S. Xue, Y. Chen, and D. Xu. Generative skill chaining: Long-horizon skill planning with diffusion models. In Conference on Robot Learning, pages 2905-2925. PMLR, 2023.
[81] F. Nuti, T. Franzmeyer, and J. F. Henriques. Extracting reward functions from diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[82] J. Urain, N. Funk, J. Peters, and G. Chalvatzaki. Se (3)ddiffusionfield:: Learning smooth cost functions for joint grasp and motion optimization through diffusion. In 2023 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pages 9923-5930 IEEE, 2023.
[83] Y. Li, Z. Wu, H. Zhao, T. Yang, Z. Liu, P. Shu, J. Sun, R. Parasuraman, and T. Liu. Aldm- grasping: Diffusion-aidee zero-shot sim-to-real transfer for robot grasping. arXiv preprint arXiv:2403.1 1114599..111,,..... 2024.
13

---

[84] H. He, Bai, L. Pan, W. Zhang, B. Zhao, and X. Li. Large-scale actionless video pre-training via discrete diffusion for efficient policy learning. arXiv preprint arXiv:2402. 14407, 2024.
[85] A. Ajay, S. Han, Y. Du, Li, A. Gupta, T. Jaakkola, J. Tenenbaum, L. Kaclbling, A. Srivas- tava, and Agrawal. Compositional foundation models for hierarchical planning. Advances in Neural Information Processing Systems, 36, 2024.
[86] P. Sundaresan, Q. Vuong. J. Gu, P. Xu, T. Xiao, S. Kirmani, T. Yu, M. Stark, A. Jain, K. Haus- man, D. Sadigh, J. Bohg, and S. Schaal. Rt-sketch: Goal-conditioned imitation learning from hand-drawn sketches, 2024.
[87] J. Gu, S. Kirmani, Wohlhart, Y. Lu, M. G. Arenas, K. Rao, W. Yu, C. Fu, K. Gopalakrish- nan, Xu, al. Ri-trajectory: Robotie task generalization via hindsight trajectory sketches. arXiv preprint arXiv:2311.0197. 2023.
[88] S. Nasiriany, F. Xia, W. Yu, T. Xiao, J. Liang, I. Dasgupta, A. Xie, D. Driess, A. Wahid, Z. Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint ar 24X1::2402.07872, 2024.
[89] M. Vecerik, C. Doersch, Y. Yang. T. Davchev, Y. Aytar, G. Zhou, R. Hadsell, L. Agapito, and
J. Scholz. Robotap: Tracking arbitrary points for few-shot visual imitation. arXiv preprint arXiv:2308. 15975, 2023.
[90] C. Wen, X. Lin, J. So, K. Chen, Q. Dou, Y. Gao, and P. Abbeel. Any-point trajectory modeling for policy learning. arXiv preprint arXiv:2401.00025, 2023.
[91] H. Bharadhwaj. R. Mottaghi, A. Gupta, and S. Tulsiani. Track2act: Predicting point tracks from internet videos enables diverse zero-shot robot manipulation. arXiv preprint urXiv:2405.01527 2024.
[92] C.-F. Yang, H. Xu, T.-L. Wu, X. Gao, K.-W. Chang, and Gao. Planning as in-painting: A cratusioo-bbsedd embodied task planning framework for environments under uncertainty. arXiv preprint arXiv:2312.0100
2023.
[93] V. Saxena, Y. Koga, and D. Xu. Constrained-contex conditional diffusion models for imita- tion learning. arXiv preprint 1xXi1:211..0141, 2023.
[94] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manip- ulation. In Conference on robot learning, pages 894-906. PMLR, 2022.
[95] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 181000080 2018.
[96] K. Black, M. Janner, Y. Du, I. Kostrikov, and S. Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301 2023.
[97] T. Karras, M. Aittala, T. Aila, and S. Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35:26565-26577, 2022.
[98] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCAl 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part !!I 18, pages 234-241. Springer, 2015.
[99] T. E. Lee, J. Tremblay, To, J. Cheng, T. Mosier, O. Kroemer, D. Fox, and S. Birchfield. Camera-to-robo pose estimation from single image. In 2020 IEEE International Confer- ence Robotics and Automation (ICRA), pages 9426-9432. IEEE, 2020.
14

---

[100] A. Kodaira, C. Xu, T. Hazama, T. Yoshimoto, K. Ohno, S. Mitsuhori, S. Sugano, H. Cho, Z. Liu, and K. Keutzer. Streamdiftusion: pipeline-level solution for real-time interactive generation. arXiv preprint arXiv:2312.12491, 2023.
[101] B. Wallace, M. Dang, R. Rafailov, L. Zhou, A. Lou, S. Purushwalkam, S. Ermon, C. Xiong, S. Joty, and N. Naik. Diffusion model alignment using direct preference optimization. arXiv preprint ArXiv:231 12908, 2023.
[102] L. Yang, Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. arXiv preprint arXiv:2401.10891, 2024.
[103] S. Luccioni, C. Akiki, M. Mitchell, and Jernite. Stable bias: Analyzing societal repre- sentations in diffusion models. arXiv preprint arXiv:2303.11408, 2023.
[104] A. Birhane, V. U. Prabhu, and E. Kahembwe. Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arX09:2110000196,, 2021.
[105] P. Schramowski, M. Brack, B. Deiscroth, and K. Kersting, Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22522 -22311, 2023.
15

---

A RLBench Tasks
We select 25 out of 100 tasks from RLBench [22] for our simulation experiments. Only variationO used reduce training time with limited resources. In the following sections, we describe each of the 25 tasks in detail, including any modifications from the original codebase.
A.1
Basketball in Hoop
Task: Pick up the basketball and dunk it in the hoop. basketball in hoop-py
filename:
Modified: No.
Success Metric: The basketball goes through the hoop.
A.2 Insert USB in Computer
Task: Pick up the USB and insert it into the computer.
filename: insert.usb.re www..in.coomuuerr.. py
Modified: No.
Success Metric: The USB tip is inserted into the USB port on the computer.
A.3 Move Hanger
Task: Move the hanger from one rack to another.
filename: move geve_hanger.pp py
Modified: No.
Success Metric: The hanger is hung on the other rack and the gripper is not grasping anything.
A.4 Open Box
Task: Grasp the lid and open the box.
filename: open box.py
Modified: For the data efficiency experiment in Figure 6, we perform grid sampling of box poses for both training and evaluation following R&D [49]. A grid size of 5cm x 20cm is used, with yaw rotation range of 45 around the axis. All other experiments use the default random sampling.
Success Metric: The joint between the and the box is at 90%
A.5 Open Door
Task: Grip the handle and push the door open.
filename: open_door.pp
Modified: No.
Success Metric:
The door opened with the door joint at 25"
A.6 Open Drawer
Task: Open the bottom drawer.
filename: open drawer. .p
Modified: No.
Success Metric: The prismatic joint of the button drawer is fully extended.
A.7 Open Grill
Task: Grasp the handle and raise the lid to open the grill.
filename: open-grill.pp py
Modified: No.
Success Metric: The lid joint of the grill cover reaches 50%.
16

---

A.8
Open Microwave
Task: Pull open the microwave door.
filename: open open_mi cic.miie...weppp. py
Modified: No.
Success Metric:
The microwave door open with its joint reaches 80".
A.9 Open Washer
Task: Pull open the washing machine door.
filename: been_washinggaaee machine........ py
Modified:
No.
Success Metric: The washing machine door is open with its joint reaches 40
A.10 Open Window
Task: Rotate the handle unlock the left window, then open it.
filename: open_window, py
Modified: No.
Success Metrie: The window is open with its joint reaches 30%.
A.11
Phone On Base
Task: Put the phone on the holder base
filename: phone on base. py
Modified: No.
Success Metric: The phone is placed on the base and the gripper is not holding the phone.
A.12 Pick Up Cup
Task: Pick up the red cup.
filename: pick up cup.pt py
Modified: No.
Success Metric: The red cup is picked up by the gripper and held within the success region.
A.13 Play Jenga
Task: Take the protruding block out of the Jenga tower without the tower toppling.
filename: play-jenge
Modified: No.
Success Metric:
The protruding block no longer on the Jenga tower, the rest of the tower remains
standing.
A.14 Press Switch
Task: Flick the switch.
filename: press_switch.py
Modified: No.
Success Metric: The switch is turned on.
A.15 Push Button
Task: Push down the maroon button.
filename: push_button.p py
Modified: No.
Success Metric: The maroon button is pushed down.
17

---

A.16 Put Books on Shelf
Task: Pick up books and place them on the top shelf.
filename: put wtt.books_oom on an.bookshheff py
Modified: No.
Success Metric:
The books are on the top shelf.
A.17 Put Knife on Board
Task:
Pick up the knife and put it on the chopping board.
filename: put knife cnchping bow.ddoord.... py
Modified: No.
Success Metric:
The knife is on the chopping board and the gripper is not holding it.
A.18 Put Rubbish in Bin
Task:
Pick up the rubbish and place it in the bin.
filename: put rubbish in bin py
Modified: No.
Success Metric: The rubbish is inside the bin.
A.19 Scoop with Spatula
Task: Scoop up the block and lift it up with the spatula.
filename: scoop_wite wtoop..it.sspatlla.
Modified: No.
Success Metric:
The cube is within the success region, lifted up by the spatula.
A.20 Slide Block Target
Task:
Slide block towards the green square target.
filename:
block_to_targer py
For the data efficiency experiment in Figure 6, perform grid sampling of block poses for both training and evaluation following R&D [49]. A grid size of 15cm 40cm is used, with yaw rotation range of 90 around the axis. All other experiments use the default random sampling. The block is in side the green target area.
Modified:
Success Metric:
A.21 Take Lid off Saucepan
Task: Take the off the saucepan
filename: take lid off saucepan.py
Modified: For the data efficiency experiment in Figure 6, we perform grid sampling of saucepan poses for both training and evaluation following R&D [49]. A grid size of 35em 44em is used, with yaw rotation range of 90" around the axis. All other experiments use the default random sampling.
Success Metric: The lid is lifted off from the saucepan to the success region above it.
A.22 Take Plate off Colored Dish Rack
Task: Take the plate off the black dish rack and leave it on the tabletop.
filename: take plate off colored dish rack.py
Modified: No.
Success Metric: The plate is lifted off the black disk rack and placed within the success region the tabletop.
18

---

A.23
Toilet Seat Up
Task: Lift the lid of the toilet seat to an upright position.
filename: toilet_seat.up py
Modified: No.
Success Metric: The joint is at 90".
A.24 Turn on Lamp
Task: Press the button to turn on the lamp.
filename: lamp-on.2
Modified: No.
Suecess Metric: The lamp is turned on by pressing the button.
A.25 Turn Tap
Task: Grasp the left tap and turn it.
filename: turn_tap-py
Modified: No.
Success Metric: The left tap is rotated by 90º from the initial position.
B World Tasks
We evaluate 9 real-world tasks. In the following sections, describe each of 9 tasks in detail, including tests we perform assess out-of-distribution generalization. Figure 8 shows objects and scene perturbations.
B.1 Lid Off
Task: Take the lid off the saucepan.
In-Distribution: black saucepan with an oval-shaped lid handle seen during training.
iut-of-Distributiin: An unnseen smaller saucepan with round-shaped lid handle.
Success Metrie: The lid is picked up from the saucepan and placed on the right side.
B.2 Place Teddy
Task: Place the teddy into the drawer In-Distribution: beige-color teddy bear toy seen during training. Out-of-Distribution: An unseen blue plush toy.
Success Metric: The toy is inside the drawer.
B.3
Elbow Touch
Task: Touch the red button with the elbow joint. In-Distribution:
The button is placed over blue cloth seen during training. :istri-Dittii The button is placed over unseen pink cloth.
Success Metric: The robot touches the button with its elbow joint.
B.4 Hang Scarf
Task: Hang the searf on the hanger.
In-Distribution: seen green-and-blaad checkered scarf seen during training. Out-of-Distributte An unseen red checkered scarf with different thickness.
Success Metric: The scarf hangs still the lowest peg of the hanger.
19

---

lid off
place teddy
elbow press
hang scarf
flip cup
avoid lamp
slide book
lift bag
put marker
ID
0OD
Figure lask objeects.
abjects distribution duit-i-ddittiiuttio (OOD) evaluations.
B.5
Put Marker
Task:
Put the highlighter into the mug.
In-Distribution: highlighter and mug seen during training.
:ittr--Dittiittion:: The same highlighter and mug is moved around during execution. Success Metric: The highlighter inside the mug.
B.6 Slide Book
Task: Slide the book to the drawer's edge and pick it up from the side. In-Distribution: A book seen during training.
Out-of-Distribution: The same book and scene but with darker lighting conditions. Success Metric: The book is lifted up from the drawer.
B.7
Avoid Lamp
Pick up the sponge and place it inside the drawer without bumping into the obstacle. In-Distribution: sponge and lamp (as the obstacle) seen during training.
Task:
Out-of Distribitri:::::::: The same sponge, but with either cup stand water bottle the obstacle.
Success Metric: The sponge is placed into the drawer without bumping into the obstacle.
B.8 Lift Bag
Task: Lift up the plastic bag.
In-Distribution: plastic bag seen during training.
:ittr--Distribution: The same plastic bag, but placed on distractors of different heights.
Success Metric: The bag is lifted up from the drawer.
B.9 Flip Cup
Task: Pick up the cup. rotate it, and place it in an upright position. In-Distribution: A plastic wine glass seen during training.
Distribbiti::::::::: unseen ceramic coffee cup. Success Metric:
The cup standing upright on the drawer.
20

---

C
Hardware Setup
C.1
Simulation
Our simulated experiments four-camera setup: front, left shoulder, right shoulder, and wrist. All cameras are set to default camera poses from RLBench [22] without any modifica- tions, except for the perturbation tests in Section 3.2.
C.2 Real-Robot
Hardware Setup. Real-robot experiments use a 7-DoF Franka Emika Panda equipped with Robotiq 2F-140 gripper. We use four RealSense D415 cameras to capture RGB images. Two cameras on the end-effector (upper wrist, lower wrist) to provide a wide field-of-view, and two external cameras (front, right shoulder) that are fixed on the base. We use TARION camera mount for the right shoulder camera. The extrinsics between the cameras and robot base-frame are calibrated with the easy handeye package² in ROS.
Data Collection. We collect demonstrations for real- world tasks using joint-mirroring setup similar to ALOHA [23]. Figure 9 shows the data collection setup. A Leader Franka is moved by the operator and the Follower Franka mirrors the Leader's movement in joint space. Vi- sual observations and joint states are recorded at 30 FPS. When training controllers, we set the action prediction horizon to match the data recording frequency avoid big jumps slow trajectory execution.
Leader
Follower
C.3 Training and Evaluation Hardware
Higure 9. Joint-mirroring setup used for data-collection
The diffusion agents of GENIMA and SuSIE [13], and the 3D Diffuser Actor [40] baseline were trained on a single NVIDIA A100 GPU with 80GB VRAM. The controllers were trained on a single NVIDIA L4 GPU with 24GB VRAM. Evaluation inference for real-world agents was done on an NVIDIA GeForce RTX 3090 GPU with 24GB VRAM.
D ControlNet Overview
ControlNet [8] is a fine-tuning architecture that preserves the text-to-image capabilities of Stable Diffusion while following the spatial layout conditioning image. ControlNet has achieved com- pelling results in several image-to-image domains such sketch-to-image, normal-map-to-imag- depth-to-image, soun-eddge-o-imeen ularly, the method preserves spatial structures and can be trained small datasets. This is achieved through a two-stream architecture similar to prior works like CLIPort [94].
segmentations-to-ttt to-imag,, and Human-pose-to-imedt Partic-
Two-stream architecture. ControlNet's architecture composed of two streams: frozen and train- able. The frozen-stream is pre-trained copy of Stable Diffusion's UNet, whose parameters kept frozen throughout fine-tuning. The trainable-stream is another copy of the UNet's downsampling encoder, whose parameters are fine-tuned. The frozen-strenn gets sampled latents, prompt, and time embeddings as input. The trainable-stream gets latents of the conditioning image (that is en- coded with frozen autoencoder), a prompt, and time embeddings as input. The two streams are connected through zero-convolution layers where outputs from each layer of the trainable-stream are added to the decoder layers the frozen-stream.
'https://github Coc.:llCaMc.OLLICCCAAA/AAAAC handeye
21

---

Zero-convolution connections regulate the flow of information from the trainable-stream to the frozen-stream. Zerooconvolution layers are x convs initialized with zeroed weights and biases. At the start of the fine-tuning process, the trainable-stream stream makes no contribution to the final output because of the zero-intialization. But after fine-tuning, the trainable layers modulate the output to follow the spatial layout the conditioning image.
Training Loss. ControlNet is trained with a standard diffusion loss that predicts noise added to noise image. This is implemented as an L2-loss on the latents. For more details on the training process, refer to the original ControlNet paper [8].
E
ACT Overview
Action Chunking with Transformers (ACT) [23] predicts action chunks (or sequences) to reduce the effective horizon of long-horizon tasks. This helps alleviate compounding errors in behavior- cloning when learning from human demonstrations. The chunk size is fixed at length K, Given an observation, the model outputs the next K actions to be executed sequentially.
Images are encoded with pre-trained ResNet-18 18 [47]. The vision features from each camera and proprioceptive features are then fed into a conditional variational autoencoder (CVAE). The CVAE consists of a BERT-like [95] transformer encoder and decoder. The encoder takes in the current joint position and target action sequence to predict the mean and variance of style variable 2. This style variable z helps in dealing with condition the action decoder during training and is discarded at test time by zeroing out. The action decoder is based on DETR [34], and is trained maximize the log-likelihood of action chunks from human demonstrations using two losses: an action reconstruction loss and a KL regularization term to encourage Gaussian prior for z.
Architecture.
demonstrations. is only used to
Temporal Smoothing. To avoid jerky robot motions, ACT [23] uses temporal ensembling each timestep. An exponential weighted scheme w; exp(-mn is applied obtain weighted average of actions from overlapping predictions across timesteps, where wi is the coefficient for the oldest action and controls the speed for incorporating new observations.
Our modifications to ACT, We made several modifications to ACT [23] in our implementation to improve data-efficiency and robustness:
Data augmentation: we use random-crops [36], color jitters, elastic transforms and Gaus- sian noise from Torchvision". The original ACT [23] does not use any data augmentation.
Sliding window sampling: we apply a sliding window along each demonstration trajectory to obtain action chunks. The original ACT [23] samples action chunks sparsely from each trajectory. The sliding-window ensures full data-coverage every epoch.
Discrete gripper loss: we use cross entropy loss for gripper open and close actions instead an Ll-loss. This makes the prediction closer to how the data was collected.
we do not use temporal smoothing for our ACT [23] con- trollers. It oversmoothens trajectories, which reduces precision and recovery behaviors.
Temporal ensemble smoothing:
Transformer decoder features: the original implementation conditions action predictions on only the first decoder layer, leaving some unused layers?. We replace it with the last decoder feature instead.
html
22

---

Hyperparameters
In this section, provide training and evaluation hyperparameters for GENIMA and other baselines. Note that GENTMA' 's controller, SuSIE [96], and the ACT [23] baseline all share the same hyperpa- rameters and augmentation settings for fair one-to-one comparisons. Real-world GENIMA and ACT agents also use the same hyperparameters (except for the camera setup).
Base Model
SD-Turbo [27]
Target K timestep
20
Learning rate
le
Weight decay
le -2
Epochs
200
Batch size
24
Image size
512 x 512 (tiled)
Image augmentation
color jitter, random crop
Train scheduler
DDPM [66]
Test scheduler
Euler Ancestral Discrete [97] constant
Learning rate scheduler Learning rate warm-up steps Inference diffusion steps Joints with rendered spheres Sphere radius for each camera (wrist, front, right, left)
500
10 (for RLBench)
base, elbow, wrist, gr gripper
3cm. 8cm. 6.5cm, 6.5em
Table 3. Diffusion Agent hyperparameters for GENIMA and SuSIE [13].
Backbone
ImageNet-trained ResNet18 [47]
Action dimension
(7 joints gripper open)
Cameras
wrist, front, right shoulder, left shoulder
Learning rate
1e
Weight decay
le
Image size
256 x 256
Action sequence K
20
Execution horizon
20
Observation horizon
# encoder layers
# decoder layers
6
# heads
Feedforward dimension
2048
Hidden dimension
256
Dropout
0.1
Epochs
1000
Batch size
96
Temporal ensembling
False
Action Normalization
zero mean. variance color jitter, random crop. elastic, and Gaussian noise
Image augmentation
Table 4. Controller hyperparameters for GENIMA, SuSIE [13), and ACT [23] baseline.
23

---

Backbone
ImageNet-trained R8sNet18 [47] UNet [98]
Noise Predictor
Action Dimension
(7 joints gripper open)
Cameras
wrist, front, right shoulder, left shoulder le
Learning rate
Weight decay
1e-
Image size
256 x 256
Observation horizon
Action sequence K
16
Execution horizon
16
Train, test diffusion steps
50.
Hidden dimension
512
Epochs
1000
Batch size
Scheduler
DDPM [66]
Action Normalization
[-1, I]
color jitter, random crop. elastic, and Gaussian noise
Image augmentation
Table s. Diffusion Policy [24] hyperparameters.
Learning rate
le
Weight decay
5e
Action history length
Train, test diffusion steps
100
Embedding dimension
120
Training iterations
550000
Batch size
Position scheduler
scaled lincar
Rotation scheduler
squared cosine noise
Loss weight wn
30
Loss weight W2
10
Table 6. 3D Diffuser Actor [40] hyperparameters.
G Tiled vs. Non-Tiled Generation
Figure 10 shows an example of tiled vs non-tiled generation. Non-tiled generation only gets one camera-view input time during diffusion. Without the full scene context, non-tiled generation tends produce inconsistent and ambiguous predictions like duplicate elbow targets.
Tiled
Non-Tiled
Figure 10. Non-tiled generation makes inconsistent predictions like duplicate
how Largets highlighted with red circles on right.

---

H Base Diffusion Models and Fine- Tune-Tuning Pipelines
GENIMA's formulation agnostic to the choice base Stable Diffusion model and also the fine- tuning pipeline. The SD-Turbo [27] base model used in GENIMA can be replaced with a bigger base model like SDXI--Turbo [27, 5] that trained on larger images. Likewise, instead of fine-tuning with ControlNet [8], can also use Instruct-pix2pix [7]. See Figure 11 for examples.
SD-Turbo
SDXL-Turbo
Instruct Pix2Pix
Figure 11. Drawing joint-actions with SD-Turbo [27]. SDXL-Turo [27. 5), and Instruct-pix2pix [7].
I SuSIE Goal Predictions
Figure 12 shows examples of goal images generated by SuSIE [13] with fined-tuned ControlNet [8]. In general. SuSIE struggles to precisely predict pixel-level details dynamic scenes with complex object interactions such as turn tap and take plate off.
open grill
turn on ammp
Input
Predicted Goal
Input
Predicted Goal
take plate
turn tap
m ihut
WEL Levezd
HiDI
tereer
LAXERE
Input
Predicted
Input
Predicted
Figure Examples of guals predicted by SuSIE [13].
25

---

J Limitations and Potential Solutions
While GENIMA is quite capable, is not without limitations, In the following sections, we discuss some these limitations and potential solutions.
Camera extrinsics during training. To create fine-tuning dataset, GENIMA relies calibrated cameras with known extrinsics to render target spheres. While the calibration process is quick, it can be difficult to obtain extrinsics for pre-existing datasets or in- in-the-wild data. A simple solution could be to use camera cose-estimation methods like DREAM [99]. DREAM takes single RGB image of a known robot and outputs extrinsics with comparable error rates to traditional hand-eye calibration.
Robot visibility in observations. One strong assumption method makes is that the robot is always visible from some viewpoint in order to draw actions near joints. This assumption might not always hold, especially in camera setups with heavy occlusion or wrist-only input. A potential solution could be to provide a virtual rendering of the robot- state, which commonly available from visualization and debugging tools like RViz". The virtual rendering can be tiled with observations such that the diffusion agent can incorporate both the virtual robot-state and observations. See Figure 13 for an illustration.
Figure 13, Tiled prediction with vistual rebot-state (bottom lett).
Slow speed of diffusion agent. The diffusion agent runs at a consid- crably lower frequency (5 Hz) than the controller (50 Hz). This makes GENIMA less reactive and prone to error accumulation. But diffusion speed unlikely to be a major issue in the future with rapid-advances from the image-generation community. Recent works like StreamDiffusion [100] run at 91.07 Hz on a single NVIDIA RTX 4090.
Jerky motions. The actions generated by GENIMA, especially right after generating a new target image, can sometimes result in jerky motions. This behavior is noticeable for some tasks in the supplementary videos. Such behavior could be result of the agent not being trained enough. We also tried temporal ensembling [23] smoothen outputs, but this hurt the ability recover from mistakes. Future works could experiment with other smoothing techniques.
Controller fails to follow targets. Sometimes the controller visibly fails to reach the target provided by the diffusion agent. This could be because the controller does not know how to reach the target from the current state, given its limited training data. One solution could be to pre-train the controller to reach arbitrary cobot-confgurration to maximize the workspace coverage.
Object rotations. All RGB-to-join agents in Section 3.1 struggle with tasks that randomize initial object poses with a wide-range of rotations. For instance. in phone on base, GENIMA achieves 19%, whereas 3D Diffuser Actor [40] achieves 94%. A small change to the phone's rotation, re- sults widely different trajectories for picking and placing it on the base. This effect could make behavior-cloning difficult. A potential solution could be to pre-train RGB-to-joint agents on tasks that involve heavy rotations such that they acquire some rotation -qquiain-quivvariant behavior.
Discovering new behaviors. Like all behavior-cloning agents, GENIMA only distills behaviors from human demonstrations, and does not discover new behaviors. It might be possible to fine-tune GENIMA on new tasks with Reinforcement- Learning (RI.). Furthermore, advances in preference optimization [96, 101] for Stable Diffusion models could be incorporated to shape new behaviors.
Hallucinations in predictions. As with any image-generation framework, GENIMA'S diffusion agent is prone to hallucinating visual artifacts. These hallucinations are limited to joint-action spheres since the background observations remain unchanged. A potential solution could be to use a classifier to detect out-of-distribution actions and then execute recovery behaviors.
26

---

K Things that did not work
In this section, we briefly describe things we tried but did not work in practice.
Predicting target spheres fixed intervals. Instead of predicting spheres continuously at K timesteps, tried fixed intervals of K e.g.. 20, 40, 60 etc. These fixed intervals act as waypoints, where the spheres hover in place until the target is reached (instead of always being K steps ahead). In this setting, controllers cannot be trained with random backgrounds, because the trajectory be- tween fixed intervals is offloaded to the controller without any visual context. So we trained con- trollers with full context, but found that these controllers tend to ignore target spheres, and directly use the visual context for predicting actions.
Other sphere shapes and textures. We experimented with few vari- ations of target spheres. Our goal was to design simple shapes that are easy to draw with image-generen geneeeenrrtiinn models, without having to draw the full robot with proper links and joints. Figure 14 illustrates an example which we made the visual appearance more asymmetric. The sphere's s surface divided into octants with different colors and black dots. The dots indicate gripper open and close actions. But in practice, we found that simple sphere with horizontal stripes works the best. Future works could improve GENIMA'S performance by simply iterating on the tar- get's appearance.
Figure Sphere Octants.
Controller with discrete actions. We tried training controllers that output discrete joint actions similar to RT-2's [56] discrete end-effector actions. We discretized joint actions into bins. Each joint has its own prediction head and is trained with cross entropy loss. We found that the discrete controllers tend to generate smoother trajectories but lacks lack the precision for fine- fine-grained tasks like manipulating small door handles pressing tiny buttons.
Observations with rendered current joint-states. Instead of just RGB observations as input to the diffusion agent, we experimented with ren- dering the current joint-states with spheres as visual reference. This led agents with significantly worse performance, likely due the in- put spheres occluding important visual information in the scene.
Segmenting target spheres for the controller. We experimented with providing binary segmentation masks of the target spheres to the con- troller. They had negligible effect on success. Training with random backgrounds seems sufficient make the controller focus on targets.
Figure 15. Depth by DepthÁny- thing [102] from RGB input.
Depth-based ControlNet. We tried conditioning ControlNet on depth input instead of RGB. We used DeythAny [102] to generate depth maps from RGB observations as shown in Figure 15. The performance with depth was worse or comparable to RGB input. Future works could experiment with fusing RGB and depth-based ControlNets [8].
Safety Considerations
Real-robot systems controlled with Stable Diffusion [1] requires thorough and extensive safety eval- uations. Internet pre-trained models exhibit harmful biases [103, 104], which may affect models fine-tuned for action prediction. This issue is not particular to Stable Diffusion, and com- monly used ImageNet-pretrained ResNets [47] exhibit similar biases. Potential solutions include safety guidance [105] and detecting out-of-distribution or inappropriate generations with classifiers to pause action prediction and ask for human assistance. Keeping humans-in-the-logg with live vi- sualizations of action predictions, and incorporating language-driven feedback, could further help in mitigating issues.
'nteps:/ ai/
27


Reviews:
[{"review_id": "Hir4DhLtwe", "reviewer": "Reviewer_LDdg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a novel approach using diffusion-based image generation for visuomotor control. The authors propose to fine-tune a pre-trained diffusion model with ControlNet to draw target joints (represented as striped spheres) onto observed RGB images. These synthesized joint images are then used to generate actions through an ACT controller. Experiments conducted in both simulation and real-world demonstrate the effectiveness of the proposed method compared to baseline methods.", "summary_of_recommendation": "Overall, I think this is an interesting paper, and the experimental results demonstrate the effectiveness of the proposed method. Therefore, I recommend accepting this paper.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Questions:\n- I am curious why the authors chose striped spheres instead of other shapes, such as keypoints or axis.\n- Regarding the stripes on the spheres, is the northern hemisphere asymmetric to the southern hemisphere?\n- What is the direct output of the diffusion model (the output of the VAE decoder)? Is it only the spheres, or are the spheres blended with the background image of the robot? If it is the latter, how is it ensured that the background part matches the input image?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jbs0dCckTA", "reviewer": "Reviewer_u2x6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes to finetune text to image diffusion model as a visuomotor policy for planning robot actions during task execution. Specifically, the proposed method parameterized robot actions as four patterned spheres representing the 6 dof locations, orientations, and states of the robot joints and use a dataset to finetune a stable diffusion model with controlnet. Real worlds experiments were conducted to validate the effectiveness of the proposed method.", "summary_of_recommendation": "I like the idea and the method of the paper and I think it could potentially work better in the future if the limitations are addressed. However, the justification of using large-scale pretrained image diffusion model is not well-established. I would believe a smaller model with less weights can achieve similar results with higher inference frequency thus potentially be able to handle dynamic tasks.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- How will the results look like if the diffusion model is trained from scratch with the same data? Same goes to if the generative model is a much smaller one.\n\n- Why do you need 4 textured spheres to represent the robot body? For most of the tasks, only the end effector poses are relevant to its success. Given the 4 colored spheres, how do you resolve the inconsistency between the generated colored spheres?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UfJKsGoQIq", "reviewer": "Reviewer_oFJa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a method to enable a robot to perform tasks conditioned on text. Their approach uses a pre-trained image generator. Rather than training the image generator to generate future goal states as images of the robot, it generates images of the joint states.", "summary_of_recommendation": "The paper introduces a novelty in a recent trend of using image generators as robot planners. I recommend weak accept due to the clarity of the paper.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "In the 3D Diffusers Actor paper, SuSIE and 3D Diffuser Actor perform relatively similarly (Fig. 1 and Table IV). SuSIE is definitely inferior but the results are close. In your experiments, SuSIE and 3D Diffuser Actor are not even remotely close in performance. Could you please explain why?\n\nYou mention that 3D Diffuser Actor is not comparable exactly to your approach because it uses extra priors of depth, keypoints, scene bounds, and motion-planners. Are these pieces of information that you could extract with your current real-world setup? Do you think that if you were to incorporate these extra priors into your method, it could perform better or worse?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Strengths:
- Clear presentation;
- Interesting and novel idea.

Weaknesses:
- Potential limitations inherent to the construction of the method;
- Additional comparisons and ablations may further strengthen the experiment section.

Post rebuttal:
All three reviewers recommended acceptance, despite some valid concerns from reviewer u2x6.  Weighing all the pros and cons, the AC recommends acceptance, but encourages the authors to acknowledge the limitations in the camera ready as pointed out by reviewers.
Decision: Accept