Generative Image as Action Models

Authors: mohit  shridhar
yat long lo
stephen  james
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-05-31 17:13:45

Abstract: Image-generation diffusion models have been fine-tuned to unlock new capabilities such as image-editing and novel view synthesis. Can we similarly unlock image-generation models for visuomotor control? We present GENIMA, a behavior-cloning agent that fine-tunes Stable Diffusion to “draw joint-actions” as targets on RGB images. These images are fed into a controller that maps the visual targets into a sequence of joint-positions. We study GENIMA on 25 RLBench and 9 real-world manipulation tasks. We find that, by lifting actions into image-space, internet pre-trained diffusion models can generate policies that outperform state- of-the-art visuomotor approaches, especially in robustness to scene perturbations and generalizing to novel objects. Our method is also competitive with 3D agents, despite lacking priors such as depth, keypoints, or motion-planners.

Main paper: 
 introduction: 
 Image-generation diffusion models [1, 2, 3] are generalists in producing visual-patterns. From photo-realistic images [4] to abstract art [5] , diffusion models can generate high-fidelity images by distilling massive datasets of captioned images [6] . Moreover, if both inputs and outputs are in image-space, these models can be fine-tuned to unlock new capabilities such as image-editing [7, 8] , semantic correspondences [9, 10] , or novel view synthesis [11, 12] . Can we similarly unlock imagegeneration models for generating robot actions? 8th Conference on Robot Learning (CoRL 2024), Munich, Germany. "open the box"withEQUATION...Figure 1 . GENIMA Overview. GENIMA is a behavior-cloning agent that maps multi-view RGB observations and language goals to jointposition actions. GENIMA is composed of two stages: (1) SD-Turbo [27] is fine-tuned with ControlNet [8] to draw target joint-positions, which are from the t + K timestep in expert demonstrations. Each joint is rendered as a uniquely colored sphere.(2) The generated targets are input into an ACT [19, 23] controller, which translates them into a sequence of K joint-positions. The controller is trained to ignore background context by using random backgrounds (see Figure 2 ). Both stages are trained independently and used sequentially during inference.Prior works in robotics have used image-generation for subgoal generation [13, 14, 15, 16] , dataaugmentation [17, 18, 19] , and features-extraction for 3D agents [20, 21] . Subgoal generation predicts goal images as targets, however, producing exact pixel-level details about interactions with deformable objects, granular media, and other chaotic systems, is often infeasible. Data-augmentation methods randomize scenes with image-generation outputs to improve robustness to lighting, textures, and distractors, but they do not use image-generation models to directly generate actions. 3D agents use pre-trained features from diffusion models to improve generalization, however, they rely on privileged information such as depth, keypoints, task-specific scene bounds, and motion-planners.In this work, we use image-generation models in their native formulation: drawing images. We present GENIMA, a multi-task behavior-cloning agent that directly fine-tunes Stable Diffusion [1] to "draw joint-actions". To supervise the fine-tuning, we format expert demonstrations into an image-to-image dataset. The input is an RGB image with a language goal, and the output is the same image with joint-position targets from a future timestep in the demonstration. The targets are rendered as colored spheres for each joint (as shown in the previous page). These visual targets are fed into a controller that maps them to a sequence of joint-positions. This formulation frames action-generation as an image-generation problem such that action-patterns become visual-patterns.We study GENIMA on 25 simulated and 9 real-world tasks. In RLBench [22] , GENIMA outperforms state-of-the-art visuomotor approaches such as ACT [19, 23] in 16/25 tasks, and DiffusionPolicies [24] in 25/25 tasks. More than task performance, we show that GENIMA is robust to scene perturbations like randomized object colors, distractors, and lighting changes, and also in generalizing to novel objects. We find that RGB-to-joint methods can approach the performance of privileged 3D next-best-pose methods [25, 26] without using priors like depth, keypoints, or motion-planners.We validate these results with real-world tasks that involve dynamic motions, full-body control, transparent and deformable objects. In summary, our contributions are:• A novel problem-formulation that frames joint-action generation as image-generation.• A proof-of-concept system for drawing and executing joint-actions.• Empirical results and insights from simulated and real-world experiments.Our code and pre-trained checkpoints are available at genima-robot.github.io. 
genima: 
 GENIMA is a behavior-cloning agent that maps RGB observations O t and a language goal g to joint-position actions A joints t . The key objective is to lift actions into image-space such that internetpretrained diffusion models can learn action-patterns as visual-patterns. This is accomplished through a two-stage process: (1) fine-tuning Stable Diffusion [1] to draw target joint-positions A image t+K on input images, K timesteps ahead, and (2) training a controller to translate these targets into to a sequence of executable joint-positions A joints t = {a t1 , a t2 ... a t K }. This simple two-stage process offloads semantic and task-level reasoning to a generalist image-generation model, while the controller reaches nearby joint-positions indicated by the visual targets. The sections below describe the two stages in detail, and Figure 1 provides an overview.The diffusion agent controls what to do next. The agent takes language goal g and multi-view RGB images O t as input, and outputs target joint-positions A image t+K on the same images. This problem formulation is a classic image-to-image setting, so any fine-tuning pipeline [7, 28, 29 ] can be used. We specifically use ControlNet [8] to preserve spatial-layouts and for data-efficient fine-tuning.Fine-Tuning Data. To supervise the fine-tuning, we randomly sample observations and target joint-positions from t + K timesteps in expert demonstrations. For that timestep, we obtain 6-DoF poses of each robot joint, which is available through robot-APIs (from forward-kinematics). We place spheres at those poses with pyrender1 , and render them on four camera observations O t = {o front t , o wrist t , o left t , o right t } with known intrinsics and extrinsics. On a 7-DoF Franka Panda, we only render four joints: base, elbow, wrist, and gripper, to avoid cluttering the image. Each joint is represented with an identifying color, with separate colors for gripper open and close. The spheres include horizontal stripes parallel to the joint's rotation axis, acting as graduations indicating the degree of rotation. Only horizontal stripes are necessary as each joint has only one rotation axis. The stripes are also asymmetric across the poles to help identify the sphere orientation.Fine-Tuning with ControlNet. Given an image-to-image dataset, we finetune Stable Diffusion [27] with ControlNet [8] to draw targets on observation images. ControlNet is a two-stream architecture: one stream with a frozen Stable Diffusion UNet that gets noisy input and language descriptions, and a trainable second stream that gets a conditioning image to modulate the output. This architecture retains the text-to-image capabilities of Stable Diffusion, while fine-tuning outputs to spatial layouts in the conditioning image. GENIMA uses RGB observations as the conditioning image to draw precise targets. We use SD-Turbo [27] -a distilled model that can generate high-quality images within 1 to 4 diffusion steps -as the base model for fine-tuning. We use the HuggingFace implementation [30] 2 of ControlNet without modifications. See Appendix D for more details on fine-tuning. Tiled Diffusion. Fine-tuning Stable Diffusion on robot data poses three key challenges. Firstly, Stable Diffusion models work best with image resolutions of 512×512 or higher due to their training data. In robotics, large images increase inference latency. Secondly, multi-view generation suffers from inconsistencies across viewpoints. However, multi-view setups are crucial to avoid occlusions and improve spatial-robustness. Thirdly, diffusion is quite slow, especially for generating four target images at every timestep. Inspired by view-synthesis works [31, 32] , we solve all three challenges with a simple solution: tiling. We tile four observations of size 256 × 256 into a single image of 512 × 512. Tiling generates four images at 5 Hz on an NVIDIA A100 or 4 Hz on an RTX 3090.The controller translates target images A image t+K into executable joint-positions A joints t . The controller can be implemented with any visuomotor policy that maps RGB observations to joint-positions. We specifically use ACT [23, 33] -a Transformer-based policy architecture [34] -for its fast inferencespeed and training stability. However, in theory, any framework like Diffusion Policies [24] or RL-based methods [35] can be used. Even classical controllers can be used if pose-estimates of target spheres are provided, but in our implementation, we opt for learned controllers for simplicity.training inference Training. During training, the controller receives current jointpositions, the language goal, and RGB images with ground-truth targets overlaid on random backgrounds. The random backgrounds, as shown in Figure 2 , force ACT to follow targets and ignore any contextual information in the scene. We use the same hyperparameters and settings from the original ACT codebase3 with minor modifications. To improve robustness to fuzzy diffusion outputs, we augment images with random-crops [36] , color jitters, elastic transforms, and Gaussian noise. We use L1 loss for joint-actions, and cross-entropy loss for gripper open and close actions. For language-conditioning, we use FiLM [37] layers following MT-ACT [19] . The controller is trained independently from the diffusion agent. See Appendix F for hyperparameters and Appendix E for controller details.Inference. During inference, the controller gets target images from the diffusion agent. The controller predicts a sequence of K joint-actions, and executes K actions or less before querying the diffusion agent in a closed-loop fashion. The controller runs at ∼ 50 Hz on an NVIDIA RTX 3090. 
experiments: 
 We study GENIMA in both simulated and real-world environments. Specifically, we are interested in answering the following questions: § 3.1 How does GENIMA compare against state-of-art visuomotor policies and 3D baselines? § 3.2 What are the benefits of drawing actions with internet-pretrained image-generation models? § 3.3 Which factors affect GENIMA's performance? § 3.4 How well does GENIMA perform on real-world tasks?We start with benchmarking our method in simulated environments for reproducible and fair comparisons. The following sections describe our simulation setup and evaluation methodology.Simulation Setup. All simulated experiments are set in CoppeliaSim [38] interfaced through PyRep [39] . The robot is a 7-DoF Franka Emika Panda placed behind a tabletop. Observations are captured from four RGB cameras: front, left shoulder, right shoulder, and wrist, each with a resolution of 256 × 256. The robot is commanded with joint-position actions via PID control, or end-effector actions via an IK solver.25 RLBench Tasks. We choose 25 (out of 100) tasks from RLBench [22] . While most RLBench tasks are suited for discrete, quasi-static motions that benefit 3D next-best-pose agents [25, 26] , we pick tasks that are difficult to execute with end-effector control. Tasks such as open box and open microwave require smooth non-linear motions that sampling-based motion-planners and IK solvers struggle with. Each RLBench task includes several variations, but we only use variation0 to reduce training time with limited resources. However, our method should be applicable to multivariation settings without any modifications. We generate two datasets: 50 training demos and 50 evaluation episodes per task. For both datasets, objects are placed randomly, and each episode is sanity-checked for solvability. Language goals are constructed from instruction templates. See Appendix A for details on individual tasks.Evaluation Metric. Multi-task agents are trained on all 25 tasks, and evaluated individually on each task. Scores are either 0 for failures or 100 for successes, with no partial successes. We report average success rates on 50 evaluation episodes across the last three epoch checkpoints: 50×3 = 150 episodes per task, which adds up to 150 × 25 = 3750 in total. We use a single set of checkpoints for all tasks without any task-specific optimizations or cherry-picking.Visuomotor Baselines. We benchmark GENIMA against three state-of-the-art visuomotor approaches: ACT [19, 23] , DiffusionPolicies [24] , and SuSIE [13] . ACT is a transformer-based policy that has achieved compelling results in bimanual manipulation. Although GENIMA uses ACT as the controller, our controller has never seen RGB observations from demonstrations, just sphere targets with random backgrounds. DiffusionPolicies is a widely adopted visuomotor approach that generates multi-modal trajectories through diffusion. SuSIE is the closest approach to GENIMA, but instead of drawing target actions, SuSIE generates target RGB observations as goal images. We adapt SuSIE to our setting by training a controller that maps target and current RGB observations to joint-position actions. All multi-task baselines are conditioned with language goals. ACT and DiffusionPolicies use FiLM conditioning [37] , whereas SuSIE uses the goal as a prompt. 3D Baseline. We also benchmark GENIMA against 3D Diffuser Actor [40] -a state-of-the-art agent in RLBench. 3D Diffuser Actor uses CLIP [41] to extract vision and language features, and diffuses end-effector poses with a 3D transformer. We use 3D Diffuser Actor as the best-performing representative of 3D next-best-pose agents [25, 26] such as PerAct [42] , Hiveformer [43] , RVT [44] , Act3D [45] , DNAct [20] , and GNFactor [21] . These works rely on several priors: depth cameras, motion-planners to reach poses, keypoints that segment trajectories into bottlenecks, task-specific scene bounds, and quasi-static assumption for motions.Our key result is that we show GENIMA -an image-generation model fine-tuned to draw actionsworks at all for visuomotor tasks. In the sections below, we go beyond this initial result and quantify GENIMA's performance against state-of-the-art visuomotor and 3D baselines.GENIMA outpeforms ACT, DiffusionPolicies, and SuSIE. Table 1 presents results from RLBench evaluations. GENIMA outperforms ACT [19, 23] in 16/25 tasks, particularly in tasks with occlusions (e.g., open window) and complex motions (e.g., turn tap). Against SuSIE [13] , GENIMA performs better on 23/25 tasks, as SuSIE struggles to generate exact pixel-level details for goals. Dif-fusionPolicy [24] performs poorly in multi-task settings with joint-position control. We ensured that our implementation is correct by training DiffusionPolicy on just take lid off, which achieved a reasonable success rate of 75%, but we could not scale it to 25 tasks.RGB-to-joint agents approach the performance of 3D next-best-pose agents. Without 3D input and motion-planners, most prior works [42, 46] report zero-performance for RGB-only agents in RLBench. However, our results in Figure 4 . Performance drops from Colosseum [46] perturbations. We evaluate GENIMA and ACT on 6 perturbation categories: randomized object and part colors, distractor objects, lighting color and brightness variations, randomized table textures, randomized backgrounds, and camera pose changes. We report success rates from 150 evaluation episodes per task, where perturbations are randomly sampled episodically. ACT overfits to objects and lighting conditions, whereas GENIMA is more robust to such perturbations. See supplementary video for examples.While all evaluations in Section 3.1 train and test on the same environment, the key benefit of using image-generation models is in improving generalization of visuomotor policies. In this section, we examine semantic and spatial generalization aspects of visuomotor policies.GENIMA is robust to semantic perturbations on Colosseum tasks. We evaluate the same multitask GENIMA and ACT agents (from Section 3.1) on 6 perturbation categories in Colosseum [46] : randomized object and part colors, distractor objects, lighting color and brightness variations, randomized table textures, randomized scene backgrounds, and camera pose changes. Figure 4 presents results from these perturbation tests. Despite being initialized with a pre-trained ResNet [47] , ACT overfits and significantly drops in performance with changes to object color, distractors, lighting, and table textures. Whereas GENIMA has minimal drops in performance from an emergent property that reverts scenes to canonical textures and colors from the training data. See supplementary videos for examples. However, both methods fail to generalize to unseen camera poses. GENIMA extrapolates to spatial locations with aligned image-action spaces.Train Success Failure Success FailureBy drawing actions on images, GEN-IMA keeps the image-space and actionspace aligned. This alignment has been shown to improve spatial generalization and data-efficiency in prior works [42, 48, 49] . We observe similar benefits in Figure 5 , where ACT struggles in the upperright region with minimal training examples, but GENIMA succeeds in extrapolating to those locations.We investigate factors that affect GENIMA's performance. We report average success rates from multi-task GENIMA trained on 3 tasks: take lid off, open box, and slide block. Our key results are presented in Figure 3 , and the sections below summarize our findings.Absolute joint-position is the best performing action-mode. Delta action-modes accumulate errors, and end-effector control through IK struggles with non-linear trajectories. Joint-position actions are also more expressive, allowing for full-body control and other embodiments.Longer action sequence predictions are crucial. In line with prior works [23, 24] , modeling trajectory distributions requires predicting longer action sequences. We find that predicting K = 20 actions is optimal, since observations are recorded at 20Hz in RLBench.Longer execution horizons avoid error accumulation. Shorter execution horizons lead to jerky motions that put the robot in unfamiliar states. We find that executing all 20 actions works best.SDXL improves performance over SD. Larger base models such as SDXL-Turbo [27] have more capacity to model action-patterns. Newer Transformer-based models [50] might scale even better.Tiled diffusion improves generation speed while keeping performance. Tiled generation of four target images takes 0.2 seconds, whereas generating individual images takes 0.56 seconds. Both methods achieve similar performance, however tiled generation is more multi-view consistent across a wider set of tasks. See Appendix G for reference.Without stripes on spheres, performance drops in half. These stripes act as graduation indicators for joint-angles. Including stripes helps Stable Diffusion learn joint rotations as a visual-pattern.Full-context controllers overfit to observations. Instead of random backgrounds, if controllers are trained with target spheres overlaid on RGB observations from demos, they tend to ignore the targets, and just use observations for action prediction. This hurts performance and generalization.ACT works better than DiffusionPolicy as the controller. Similar to results in Section 3.1, we find that ACT is better at joint-action prediction than DiffusionPolicy (DP). ACT also has a faster inference speed of 0.02 seconds, whereas DiffusionPolicy takes 0.1 seconds (for 20 diffusion steps). GENIMA is data-efficient. We study data-efficiency by constraining poses of objects in training demos. Following R&D [49] , we sample poses in a grid-style that maximizes workspace coverage based on the number of demos. GENIMA achieves 80% of the peak performance with 25 demos.GENIMA works with 5 diffusion steps or more. With SD-Turbo [27] as the base model, GENIMA can generate target images with just 5 diffusion steps within 0.2 seconds. Future works can use better schedulers and distillation methods to further improve generation speed and quality.In We validate our results by benchmarking GENIMA and ACT on a real-robot setup. Our setup consists of a Franka Emika Panda with 2 external and 2 wrist cameras. We train multi-task agents from scratch on 9 tasks with 50 demos per task. These tasks involve dynamic behaviors (e.g., slide book), transparent objects (e.g., lift bag), deformable objects (e.g., hang scarf), and full-body control (e.g., elbow touch). See Figure 7 and supplementary videos for examples. Appendix B covers task details. When comparing GENIMA and ACT, we ensure that the initial state is exactly the same by using an image-overlay tool to position objects. Table 2 reports average success rates from 5 evaluation episodes per task. We also report outof-distribution performance with unseen objects and scene perturbations. In line with Section 3.2, GENIMA is better than ACT at generalizing to out-of-distribution tasks. GENIMA also exhibits some recovery behavior from mistakes, but DAgger-style [51] training might improve robustness. 
related work: 
 Visuomotor agents map images to actions in an end-to-end manner [52, 53, 54] . ACT [23] uses a transformer-based policy architecture [34] to encode ResNet [47] features and predict action-chunks. MT-ACT [19] extends ACT to the multi-task settings with language-conditioning. MVP [55] uses self-supervised visual pre-training on in-the-wild videos and fine-tunes for real-world robotic tasks. Diffusion Policy [24] uses the diffusion process to learn multi-modal trajectories. RT-2 [56] finetune vision-language models to predict tokenized actions. Octo [57] can be adapted to new sensory inputs and action spaces by mapping inputs into a common tokenized format. All these methods adapt paradigms from vision and language like tokenization and pre-training for predicting actions.In contrast, GENIMA lifts actions back into image-space to use image-generation models natively.3D next-best-pose agents encode 3D input and output 6-DoF poses [25, 26] . These poses are keypoints (or bottlenecks in the trajectory) that are executed with a motion-planner or IK-solver. C2F-ARM [26] and PerAct [42] use calibrated multi-camera setups to voxelize scenes into 3D grids.Voxel grids are computationally expensive, so Act3D [45] and 3D Diffuser Actor [40] replace voxels with sampled 3D points. RVT [44] renders RGB-D input into orthographic projections, and detects actions in image-space. GNFactor [21] and DNAct [20] lift Stable Diffusion features into 3D to improve generalization. Chained Diffuser [58] and HDP [59] use diffusion-based policies as lowlevel controllers to reach keypoints predicted by 3D next-best-pose agents. All these 3D next-bestpose agents rely on several priors: depth cameras, keypoints, task-specific scene bounds, and/or motion-planners. Whereas GENIMA is a simple RGB-to-joint agent without any of these priors.Diffusion models for robot learning. In addition to modeling policies [60, 61, 62, 63, 64] with diffusion [65, 66] , diffusion models have been used in robot learning in various ways. This includes offline reinforcement learning [61, 63, 67] , imitation-learning [68, 69, 70] , subgoal generation [16, 71, 72] , and planning [58, 59, 73, 74, 75, 76] . Others include affordance prediction [77] , skill acquisition [78, 79] and chaining [80] , reward functions [81] , grasping [82] , and sim-to-real [83] .Image-generation models in robot learning have been used for out-of-distribution data generation [69] , and video-conditioned policies [15, 84, 85] . ROSIE [18] and GenAug [17] use imagegeneration outputs to augment datasets. SuSIE [13] , RT-Sketch [86] , and RT-Trajectory [87] , condition policies on goals generated by image generation models in the form of observations, observation sketches, and trajectory sketches, respectively. GENIMA does not use image-generation models to generate observations, videos, or trajectories, but instead draws joint-actions as targets.Representing actions in images has been shown to improve spatial-robustness and generalization. PIVOT [88] annotates observations with markers to query vision-language models for end-effector actions. RoboTap [89] , ATM [90] , and Track2Act [91] track dense points on images to learn endeffector policies. R&D [49] renders grippers on images to aid the diffusion process. Yang et al. [92] plan by in-painting navigation actions. C3DM [93] iteratively zooms-into images to predict 6-DoF poses. In comparison, GENIMA does not track points, and learns joint-actions in image-space. 
conclusion and limitations: 
 We presented GENIMA, a multi-task agent that fine-tunes Stable Diffusion to draw joint-actions. Our experiments both in simulation and real-world tasks indicate that fine-tuned image-generation models are effective in visuomotor control. While this paper is a proof-of-concept, GENIMA could be adapted to other embodiments, and also to draw physical attributes like forces and accelerations.GENIMA is quite capable, but not without limitations. Like all BC-agents, GENIMA only distills expert behaviors and does not discover new behaviors. GENIMA also uses camera calibration to render targets, assuming the robot is always visible from some viewpoint. We discuss these limitations and offer potential solutions in Appendix J. But overall, we are excited about the potential of pre-trained diffusion models in revolutionizing robotics, akin to how they revolutionized image-generation.Zero-convolution connections regulate the flow of information from the trainable-stream to the frozen-stream. Zero-convolution layers are 1 × 1 convs initialized with zeroed weights and biases. At the start of the fine-tuning process, the trainable-stream makes no contribution to the final output because of the zero-intialization. But after fine-tuning, the trainable layers modulate the output to follow the spatial layout in the conditioning image.Training Loss. ControlNet is trained with a standard diffusion loss that predicts noise added to a noise image. This is implemented as an L2-loss on the latents. For more details on the training process, refer to the original ControlNet paper [8] . 
 
 Appendix: Action Chunking with Transformers (ACT) [23] predicts action chunks (or sequences) to reduce the effective horizon of long-horizon tasks. This helps alleviate compounding errors in behaviorcloning when learning from human demonstrations. The chunk size is fixed at length K. Given an observation, the model outputs the next K actions to be executed sequentially.Architecture. Images are encoded with a pre-trained ResNet-18 [47] . The vision features from each camera and proprioceptive features are then fed into a conditional variational autoencoder (CVAE).The CVAE consists of a BERT-like [95] transformer encoder and decoder. The encoder takes in the current joint position and target action sequence to predict the mean and variance of a style variable z. This style variable z helps in dealing with multi-modal demonstrations. It is only used to condition the action decoder during training and is discarded at test time by zeroing it out. The action decoder is based on DETR [34] , and is trained to maximize the log-likelihood of action chunks from human demonstrations using two losses: an action reconstruction loss and a KL regularization term to encourage a Gaussian prior for z.Temporal Smoothing. To avoid jerky robot motions, ACT [23] uses temporal ensembling at each timestep. An exponential weighted scheme w i = exp(-m * i) is applied to obtain a weighted average of actions from overlapping predictions across timesteps, where w 0 is the coefficient for the oldest action and m controls the speed for incorporating new observations.Our modifications to ACT. We made several modifications to ACT [23] in our implementation to improve data-efficiency and robustness:• Data augmentation: we use random-crops [36] , color jitters, elastic transforms and Gaussian noise from Torchvision6 . The original ACT [23] does not use any data augmentation.• Sliding window sampling: we apply a sliding window along each demonstration trajectory to obtain action chunks. The original ACT [23] samples action chunks sparsely from each trajectory. The sliding-window ensures full data-coverage every epoch.• Discrete gripper loss: we use cross entropy loss for gripper open and close actions instead of an L1-loss. This makes the prediction closer to how the data was collected.• Temporal ensemble smoothing: we do not use temporal smoothing for our ACT [23] controllers. It oversmoothens trajectories, which reduces precision and recovery behaviors.• Transformer decoder features: the original implementation conditions action predictions on only the first decoder layer, leaving some unused layers 7 . We replace it with the last decoder feature instead. G Tiled vs. Non-Tiled Generation GENIMA's formulation is agnostic to the choice of base Stable Diffusion model and also the finetuning pipeline. The SD-Turbo [27] base model used in GENIMA can be replaced with a bigger base model like SDXL-Turbo [27, 5] that is trained on larger images. Likewise, instead of fine-tuning with ControlNet [8] , we can also use Instruct-pix2pix [7] . See Figure 11 for examples.Figure 11 . Drawing joint-actions with SD-Turbo [27] , SDXL-Turbo [27, 5] , and Instruct-pix2pix [7] .Figure 12 shows examples of goal images generated by SuSIE [13] with fined-tuned ControlNet [8] .In general, SuSIE struggles to precisely predict pixel-level details of dynamic scenes with complex object interactions such as turn tap and take plate off. Predicted Goal Input Predicted Goal Input Predicted Goal Input Predicted GoalIn this section, we briefly describe things we tried but did not work in practice.Predicting target spheres at fixed intervals. Instead of predicting spheres continuously at t + K timesteps, we tried fixed intervals of K e.g., 20, 40, 60 etc. These fixed intervals act as waypoints, where the spheres hover in place until the target is reached (instead of always being K steps ahead).In this setting, controllers cannot be trained with random backgrounds, because the trajectory between fixed intervals is offloaded to the controller without any visual context. So we trained controllers with full context, but found that these controllers tend to ignore target spheres, and directly use the visual context for predicting actions. Other sphere shapes and textures. We experimented with a few variations of target spheres. Our goal was to design simple shapes that are easy to draw with image-generation models, without having to draw the full robot with proper links and joints. Figure 14 illustrates an example in which we made the visual appearance more asymmetric. The sphere's surface is divided into octants with different colors and black dots. The dots indicate gripper open and close actions. But in practice, we found that a simple sphere with horizontal stripes works the best. Future works could improve GENIMA's performance by simply iterating on the target's appearance.Controller with discrete actions. We tried training controllers that output discrete joint actions similar to RT-2's [56] discrete end-effector actions. We discretized joint actions into bins. Each joint has its own prediction head and is trained with cross entropy loss. We found that the discrete controllers tend to generate smoother trajectories but lacks lack the precision for fine-grained tasks like manipulating small door handles or pressing tiny buttons. Observations with rendered current joint-states. Instead of just RGB observations as input to the diffusion agent, we experimented with rendering the current joint-states with spheres as a visual reference. This led to agents with significantly worse performance, likely due to the input spheres occluding important visual information in the scene.Segmenting target spheres for the controller. We experimented with providing binary segmentation masks of the target spheres to the controller. They had a negligible effect on success. Training with random backgrounds seems sufficient to make the controller focus on targets.Depth-based ControlNet. We tried conditioning ControlNet on depth input instead of RGB. We used DepthAnything [102] to generate depth maps from RGB observations as shown in Figure 15 .The performance with depth was worse or comparable to RGB input. Future works could experiment with fusing RGB and depth-based ControlNets [8] .Real-robot systems controlled with Stable Diffusion [1] requires thorough and extensive safety evaluations. Internet pre-trained models exhibit harmful biases [103, 104] , which may affect models fine-tuned for action prediction. This issue is not particular to Stable Diffusion, and even commonly used ImageNet-pretrained 9 ResNets [47] exhibit similar biases. Potential solutions include safety guidance [105] and detecting out-of-distribution or inappropriate generations with classifiers to pause action prediction and ask for human assistance. Keeping humans-in-the-loop with live visualizations of action predictions, and incorporating language-driven feedback, could further help in mitigating issues.https://pyrender.readthedocs.io/en/latest/examples/quickstart.htmlhttps://huggingface.co/docs/diffusers/en/using-diffusers/controlnethttps://github.com/tonyzhaozh/acthttps://amzn.eu/d/7xDDfJHhttps://github.com/IFL-CAMP/easy_handeyehttps://pytorch.org/vision/0.15/transforms.htmlhttps://github.com/tonyzhaozh/act/issues/25https://github.com/ros-visualization/rvizhttps://excavating.ai/ 


Decision: Strengths:
- Clear presentation;
- Interesting and novel idea.

Weaknesses:
- Potential limitations inherent to the construction of the method;
- Additional comparisons and ablations may further strengthen the experiment section.

Post rebuttal:
All three reviewers recommended acceptance, despite some valid concerns from reviewer u2x6.  Weighing all the pros and cons, the AC recommends acceptance, but encourages the authors to acknowledge the limitations in the camera ready as pointed out by reviewers.

Reviews: review: title: title: Interesting Idea
paper_summary: paper_summary: This paper proposes a novel approach using diffusion-based image generation for visuomotor control. The authors propose to fine-tune a pre-trained diffusion model with ControlNet to draw target joints (represented as striped spheres) onto observed RGB images. These synthesized joint images are then used to generate actions through an ACT controller. Experiments conducted in both simulation and real-world demonstrate the effectiveness of the proposed method compared to baseline methods.
main_review: main_review: Strength:
- The idea of using diffusion model to draw joints on images is an interesting and novel
- The proposed method offers better interpretability compared to other methods
- Sufficient experiments in both sim and real, demonstrating that the proposed method outperforms the baseline methods.

Weaknesses:
- I think the authors may consider including a metric for the accuracy of the predicted sphere poses. Although it might be tricky as the generation is in image space and poses are not explicitly represented (maybe through a pose estimator, or a differentiable renderer). This metric could help identify the performance bottleneck, whether it stems from errors in the image generator or the controller, and facilitate better diagnosis when the model fails. It could also be used to evaluate whether the generated joint poses are feasible and consistent across different views (see below).
- It would be good to include an additional ablation experiment with an open-loop trajectory follower, where the robot directly follows the estimated joints (sphere) poses, if these poses are available.
- The method may fail if parts of the joints are out of the field of views, and it is unclear if the generated configurations are consistent across different views.
questions: questions: Questions:
- I am curious why the authors chose striped spheres instead of other shapes, such as keypoints or axis.
- Regarding the stripes on the spheres, is the northern hemisphere asymmetric to the southern hemisphere?
- What is the direct output of the diffusion model (the output of the VAE decoder)? Is it only the spheres, or are the spheres blended with the background image of the robot? If it is the latter, how is it ensured that the background part matches the input image?
limitations: limitations: 3
review_summary: review_summary: Overall, I think this is an interesting paper, and the experimental results demonstrate the effectiveness of the proposed method. Therefore, I recommend accepting this paper.
title: title: Important problem, interesting method, unclear motivation of method
paper_summary: paper_summary: The paper proposes to finetune text to image diffusion model as a visuomotor policy for planning robot actions during task execution. Specifically, the proposed method parameterized robot actions as four patterned spheres representing the 6 dof locations, orientations, and states of the robot joints and use a dataset to finetune a stable diffusion model with controlnet. Real worlds experiments were conducted to validate the effectiveness of the proposed method.
main_review: main_review: Strengths:
- The paper is well presented. The figures are well-made, and the presentation of the method, experimental setup, training data, etc., are very clear. I had no trouble understanding the idea with different level of details.

- The proposed method is innovative. The core of the idea is essentially parameterizing a robot's action as an image, which can be learned by a text-to-image generative model. It might not achieve SOTA results on hardcore manipulation tasks but it is an innovative attempt and the results seem promising.

- The use of SD-Turbo is neat. Being able to achieve 5 hz inference frequency is quite impressive. This could allow lots of potentially more impressive tasks to be done by the proposed method.

- The discussions of limitations are very thorough.

Weaknesses:
- Parameterizing robot actions as images have many limitations by construction. For example, different spheres can occlude each other, which leads to ambiguity in robot joint actions.

- Many prior works use large-scale internet-pretrained for image-editing or 3D tasks because of the impressive image priors or geometry priors learned by these large models. A similarity across all these tasks is that the output of the diffusion model is natural images so the task can benefit from the priors learned by the pretrained diffusion model. In this case, however, the images, made of four textured spheres with random background images, are not natural images. Therefore, it's hard to understand why pretrained diffusion models are needed. It's very likely a trained-from-scratch stable diffusion can perform equally well. It's also likely that a image generative model with much smaller architecture and less number of parameters can perform equally well but much more efficiently, allowing faster operation time. In summary, the reason to fine-tune from a pretrained large-scale diffusion model like stable diffusion is unclear here.

- Image generative models tend to hallucinate physically incorrect images. How does the method cope with strange output by the Genima model? Can the learned policy recover itself from incorrect prediction in a closed-loop setting? Are there example videos showing the recovery behavior?
questions: questions: - How will the results look like if the diffusion model is trained from scratch with the same data? Same goes to if the generative model is a much smaller one.

- Why do you need 4 textured spheres to represent the robot body? For most of the tasks, only the end effector poses are relevant to its success. Given the 4 colored spheres, how do you resolve the inconsistency between the generated colored spheres?
limitations: limitations: 3
review_summary: review_summary: I like the idea and the method of the paper and I think it could potentially work better in the future if the limitations are addressed. However, the justification of using large-scale pretrained image diffusion model is not well-established. I would believe a smaller model with less weights can achieve similar results with higher inference frequency thus potentially be able to handle dynamic tasks.
title: title: Using an image generator to predict future joint states for a robot to perform a task conditioned on text.
paper_summary: paper_summary: This paper introduces a method to enable a robot to perform tasks conditioned on text. Their approach uses a pre-trained image generator. Rather than training the image generator to generate future goal states as images of the robot, it generates images of the joint states.
main_review: main_review: Overall, the paper is clearly written, presents a decently novel approach, and sufficient evaluation.  There are multiple recent approaches to manipulation using image generators as priors. The paper compares against what it states as the most similar method, SuSIE, which generates images of the robot as sub-goals in contrast to GENIMA which predicts images of proxies for joint states. Based on their evaluations, it appears that it could be a better approach to generate these joint states rather than depictions of the robot.

Strengths:
The paper is fairly well organized and written.
It is an interesting idea to predict images of the joint states rather than the RGB appearance of the robot.
The evaluation is thorough. They compared against a similar method (SuSIE) and a state-of-the-art method (3D Diffuser Actor).
The experiment section poses interesting questions.

Weaknesses:
Many of the figures can be challenging to read based on font size. Figure 5 is particularly challenging.
The method does not outperform all methods. Though, the presented method requires less information that the superior methods which require more equipment to operate the robot.
perturbations_tests.mp4 is difficult to understand.

Typo Line 161: "outpeforms"
The title may read better as "Generative Images as Action Models"
questions: questions: In the 3D Diffusers Actor paper, SuSIE and 3D Diffuser Actor perform relatively similarly (Fig. 1 and Table IV). SuSIE is definitely inferior but the results are close. In your experiments, SuSIE and 3D Diffuser Actor are not even remotely close in performance. Could you please explain why?

You mention that 3D Diffuser Actor is not comparable exactly to your approach because it uses extra priors of depth, keypoints, scene bounds, and motion-planners. Are these pieces of information that you could extract with your current real-world setup? Do you think that if you were to incorporate these extra priors into your method, it could perform better or worse?
limitations: limitations: 3
review_summary: review_summary: The paper introduces a novelty in a recent trend of using image generators as robot planners. I recommend weak accept due to the clarity of the paper.