Can Pre-Trained Text-to-Image Models Generate Visual Goals for Reinforcement Learning?
Authors: Jialu Gao
Kaizhe Hu
Guowei Xu
Huazhe Xu
Venue: NeurIPS 2023 poster (2023)
Keywords: Visual Reinforcement Learning
Large Generative Models
Image Editing
Robotics

Abstract: Pre-trained text-to-image generative models can produce diverse, semantically rich, and realistic images from natural language descriptions. Compared with language, images usually convey information with more details and less ambiguity. In this study, we propose Learning from the Void (LfVoid), a method that leverages the power of pre-trained text-to-image models and advanced image editing techniques to guide robot learning. Given natural language instructions, LfVoid can edit the original observations to obtain goal images, such as "wiping" a stain off a table. Subsequently, LfVoid trains an ensembled goal discriminator on the generated image to provide reward signals for a reinforcement learning agent, guiding it to achieve the goal. The ability of LfVoid to learn with zero in-domain training on expert demonstrations or true goal observations (the void) is attributed to the utilization of knowledge from web-scale generative models. We evaluate LfVoid across three simulated tasks and validate its feasibility in the corresponding real-world scenarios. In addition, we offer insights into the key considerations for the effective integration of visual generative models into robot learning workflows. We posit that our work represents an initial step towards the broader application of pre-trained visual generative models in the robotics field. Our project page: https://lfvoid-rl.github.io/.

Can Pre-Trained Text-to-lmage Models Generate Visual Goals for Reinforcement Learning?
Jialu Gaol Kaizhe Hu 12.34 Guowei Xu', Huazhe Xu 1,2,3
Tsinghua University Shanghai Qi Zhi Institute
Shanghai Al Lab
gaojialululu@@mail. com, muazhhe_u@@ail_ www.gnus...eeeee edu. edu. cn
Abstract
Pre-trained text-to-image generative models can produce diverse, semantically rich, and realistic images from natural language descriptions. Compared with language, images usually convey information with more details and less ambiguity. In this study, propose Learning from the Void (LfVoid), method that leverages the power of pre-trained text-to-image models and advanced image editing techniques to guide robot learning. Given natural language instructions, LfVoid can edit the original observations to obtain goal images, such as "wiping" stain off table. Subsequently, LfVoid trains an ensembled goal discriminator on the generated image to provide reward signals for reinforcement learning agent, guiding it to achieve the goal. The ability of Lffoid learn with zero in-domain training expert demonstrations or true goal observations (the void) is attributed to the utilization knowledge from web-scale generative models. We evaluate LfVoid across three simulated tasks and validate its feasibility in the corresponding real- world scenarios. addition, we offer insights into key considerations for the effective integration of visual generative models into robot learning workflows. We posit that our work represents an initial step towards the broader application of pre-trained visual generative models in the robotics field. Our project page: LfVoid. github. io
Introduction
What is the simplest way provide guidance goals robot? This question is answered multiple times: a set of expert demonstrations 2 goal images 5 6 77. or natural language
instructions 8 99. However, these answers either require laborious and sometimes prohibitive effort to collect data, or contain ambiguity across modalities. The desire to alleviate the challenges begs the question: can we generate goal images from natural language instructions directly, without physically achieving the goals for robots?
13 have achieved exciting breakthroughs, demon- strating an unprecedented ability to generate plausible images that are semantically aligned with given text prompts. Trained on extensive datasets, these models are thought to possess a basic understanding of the world [14]. Therefore, we aim to harness the power these large generative models to provide unambiguous visual goals for robotic tasks without any in-domain training.
Large text-to- text-to-image image generative models 10
In the past, common approach to leverage off the-shelf large generative models in robot learning involves using these models for data augmentation expert datasets to improve policy general-
equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

ization 15 16 17 Despite the success, these methods still rely on human demonstrations, not generative models, as the main source of guidance.
DALL-EE-Bot 18), an early attempt to utilize web-scale generative models in a zero-shot manner for guiding manipulation tasks, utilizes DALL.-E 2 [0] to generate goal images for object rearrangement tasks. However, the generated images are often too diverse and largely disagree with real-world scenarios, requiring segmentation masks for object matching and a rule-based transformation planner to close the visual discrepancy.
The recent uprise of language-bun based image editing methods [9 20 21 provides new paradigm:
editing the generated images to align with language descriptions, such as "wiping stain off a table", while keeping the visual appearance of the other objects and the background roughly unchanged. However, the edited images are usually examined by visual appearance rather than embodied tasks.
In this work, we introduce Learning from the Void (L.foidd), method that uses image editing techniques on pre-trained diffusion models to generate visual goals for Reinforcement Learning (RL.). LíVoid contains a unique image editing pipeline capable of performing appearance-basee and structure-based editing to generate goal images according to language instructions. With these goal images, LíVoid improves upon example-based RL methods [5] 6 to solve several robot control tasks without the need for any reward function or demonstrations.
We evaluate LfVoid on three simulated environment tasks and the corresponding real-world scenarios. Empirical results show that Vfoo can generate goal images with higher fidelity to both the editing instructions and the source images when compared to existing image editing techniques, leading to better downstream control performance comparison to tanguage-guiden and other image-editing- based methods. We also validate the feasibility of Vf'd in real- weal-world settings: LfVoid can retrieve meaningful reward signals from generated images comparable to those from true goal images. Based on these observations, we discuss key considerations for the current generative model research when aiming for real-world robotic applications.
First, we propose an effective approach to leverage the knowledge encupsulated in large pre-trainet generative models and apply in zero-shot manner to guide robot learning tasks. Second, we provide empirical evidence that suggests dVVoid-editer images provide guidance more effectively than other methods including the direct usage of text prompts. Lastly, we identify the existing gap between the capabilities of current text-conditioned image generation models and the demands robotic applications, thus shedding light on a potential direction for future research in this domain.
Our contributions are threefold:
2 Related work
Image editing with diffusion models. Recent work in text-to-image diffusion models has demonstrated a strong ability to generate images that are semantically aligned with given text I1 12. 24, 25). Based on these generative models, Prompt-to-Promp proposes to perform cexticonditioned editing through the injection of the cross-attention maps. Imagic [20] proposes to perform editing by using interpolation between source and target embeddings to gener- ate images with edited effects. InstructPix2Pix [26) demonstrates another approach for training a diffusion model on the paired source and target images and the corresponding editing instructions. Apart from editing techniques, several methods focusing on controlled editing have been proposed. Inversioo (27 and DreamBooth 28 both aim to learn special token corresponding to user-specified object and at inference time can generate images containing that object with a diverse background. Directed Diffusion [21] focuses object placement generation, which control the location of a specified object to reside in a given area.
prompts 22 23 10
Visual RL with implicit rewards. Standard reinforcement learning methods require an explicit reward function to guide the agent towards desired goal states, yet these reward functions may not always be available or may not capture the essence of the task hand. One possible way of learning without access to explicit reward functions is using superviede learning to learn representation where the distances can be used as reward signals 22 30), Another line work uses example-based RL methods that aim to utilize the goal states to guide the learning process. VICE, proposed by Fu et al. [5), establishes general framework for learning from goal observations: trains a discriminator with the observations in the replay buffer as negative samples and goal images

---

as positive samples. The positive logits for new observations are used as reward to guide the agent toward the goal observation. Building upon this, Singh et al. [6] introduce VICE-RAQ, integrating the VICE algorithm with active querying methods through periodically asking human to provide labels for ambiguous observations. In addition, it proposes lahel-smoothing technique for a better reward shaping. Other works [3 32 also explore ways to improve the reward shaping of VICE as well. In a parallel direction, Eysenbach et al. 77 develop Recursive Classification of Examples (RCE), a method that learns the Q function of actor-critic methods directly from the goal observations.
Large generative models for robot control. Current application of large generative models in 34 35| or
robotic tasks mainly focuses on using Large Language Models for planning [8 9 33
learning a language conditioned policy [36 37], while the application of pre-trained image generative models for robotics tasks are limited. A common way of leveraging large-scale diffusion models for robot learning by using diffusion models to perform data augmentation on training data, as suggested in CACTI [1], GenAug [16] and ROSIE 17]. Another line of work suggests using
diffusion models to generate plans to solve robotics tasks 38 39 40 More recently, video generation and editing techniques have also been used with imitation learning to guide robot policies [41 42] More specifically, Uni trains a text-to-video generation model on web-scale datasets and expert demonstrations generate image sequences for planning and inverse modeling. The most related work is DALL-E-Boo [18), which uses the DALL-E [ model to generate goal plans and performs object rearrangement according to these goals. However, unlike work, DALL-E- Bot does not apply image editing techniques for goal generation and requires rule-based matching and predefined pick-and-place actions bridge the visual gap between generated images and true observations.
3 Method
In this study, we aim to provide zero-shot visual goals to reinforcement learning agents for manipula- tion tasks using only text prompts. Our approach utilizes the information grounded in large-scale pre-trained visual generative models to create semantically meaningful visual goals. We first generate synthetic goal image dataset from raw observations using image editing techniques, then employ example-based visual reinforcement learning that is optimized for our task with the generated dataset.
The structure of this section unfolds as follows: Section 3.1 elucidates the image editing techniques and adjustments applied to create goal images from text prompts and initial image observations, as depicted in Figure [a). [). Section 3.2 discusses the execution of example-ber based visual reinforcement learning using the generated goal images, as outlined in Figure bb).
3.1 Visual goal generation
In the first phase of our methodology, we edit and generate goal images from raw observations based on natural language guidance provided by humans. Given source prompt P. a source image Tsre, and an editing instruction, LfVoid synthesizes a target image Zigt using a pre-trained Latent Diffusion Model (LDM) [12]. We consider two types of editing instructions: either target prompt Pt describing the appearance changes in the target image, or a bounding- box region B and a set of tokens I corresponding to the object be relocated when structural changes are needed.
To provide sufficient visual guidance for downstream reinforcement learning tasks, the generated goal images should highlight the visual changes while preserving the irrelevant scene as much as possible, which a challenging requirement even for state-of-the-art image editing techniques. To this end, we integrate a number of different techniques in the visual goal generation pipeline of LfVoid, consisting of a feature extracting module, an inversion module, and an editing module, as outlined in Figure T a). Following conventional notations, we denote I; the synthesized LDM image at time step & E {T,T- - 1, .....),, 0}, where is the total diffusion time steps. Specifically, xr denotes the Gaussian noise, and To denotes the generated image. The techniques used in each module and their purpose are described as follows.
3.1.1 Feature extracting module
To ensure high fidelity to the source image &src. we learn unique token sks that encapsulates the visual features of objects within &sre as in DreamBooth (28). This process involves optimizing the diffusion model parameters along with the special token sks, using a set of images that contain the
3

---

Goal Image Generation
Example- Basee-Baeedd Visual
Agwet
von the robot diann the
markings the mible.
Bdling prompt A white sible wifh aothing on
(a) Goal Image Generation
(b) Example Based Visual RL
AA white rable win merkings on
'morkings" "exehing"
Generated Visual Goals
Prompt-to Promp: Appexcance Based Blang
Festuze Extrctiig Medile
Invensio Module
Ediing Moshlle
Generated Visual Goals
Classifer-Ensemlle Rewaed
Obvervations
Medhiee
PIP-DD Sanstar-Hooo Ediring
"A ploto cube white rable"
Figure 1: An overview of LfVoid. The goal of LfVoid is to learn robot policies requiring only language descriptions from humans. LfVoid consists of two parts: (a) Goal image generation, where we apply image editing on the initial observations according to different editing instructions to obtain visual goal dataset; (b) Example-Based Visual RL, where we perform reinforcement learning on the generated dataset to achieve the desired goal image in various environments.
target object. As result, details of Zarc. such as the color and texture of a cube to be positioned, or the shape of a Franka robot arm for manipulation. As later demonstrated in our experiments, this module substantially boosts the resemblance details of our edited images to the corresponding source images.
able to derive specialized model that can accurately retain key
3.1.2 Inversion module
After employing the special token sks to capture the essential features of the source scene, we invert the provided source image Tare diffusion process. A simple inversion technique using the Denoising Diffusion Implicit Model (DDIM) 233 sampling scheme calculates the samples TT, Frr: To by reversing the ODE process. However, the inverted image To obtained this way often deviates from Esrc due to cumulative errors. Null-text inversion [3] aims to mitigate this discrepancy by fine-tuning the "null-text" embedding used in the classifier-free guidance [44) &, for each E [T, T I.... 1} to control the generation process. Thus, with the initial noise sample 2T obtained through DDIM inversion and the optimized "null-text" embedding @T, @T-1,-.. P1, the diffusion process is able to generate the image x z'ree approximation of the source image Terc- Based on the diffusion process obtained with the inversion module, LfVoid can perform precise and detailed editing control on the source image, which will be discussed in Section 3.1.3
3.1.3 Editing module
Depending on the specific requirements, we divide the editing tasks into two distinet categories: appearance-bras based and structure- -based image editing. Appearance-basse editing involves maintaining the structural layout of the image while altering certain visual aspects, such cleaning the surface of a table or lighting up an LED bulb, In contrast, structure- streture-based editing involves changes in the layout of the image, such as relocating objects within the image from one area another.
Appearance- based editing. tasks involving appearance-bant based editing, we employ the Prompt-to- Prompt editing control technique. When the Latent Diffusion Model (L.M)) generates an image zo conditioned on text prompt P. the information encapsulated in influences the diffusion process via the cross-attention layers [12 [9] Prompt-to-Proee Prompt reveals that the spatial configuration

---

XY-W
4
y-1
P2P
P2P
X XO
Xarc
A
gripper white
ky
X--1
Xy-N
4by
Py-1
Qy-N
Directed Diffusion Attention Editing
Conventional Denoising
Xtgt
x
Figure 2: Image editing with structural changes. The details of P2P-DD technique used in the editing module of LfVoid perform structural changes. LfVoid performs combination of P2P attention map injection with Directed Diffusion attention editing on the diffusion process generating the target image Zigt, based on the Gaussian noise FT and the optimized null-text embeddings @r, Pr-1..., 21 obtained through the Inversion Module of LfVoid.
of the image To largely depends the cross-attention maps Mt within these cross-attention layers, especially during the initial diffusion steps. Consequently, it suggests replacing the attention maps Mž of the target image diffusion process with the attention maps Mf from the source image diffusion process for the first N time steps, beginning at time step T:
M.
f otherwise
P2
M;
This method ensures that the structure the source image encapsulated in the attention maps M, conserved in the target image in more controlled manner, which is crucial in downstream RL tasks.
Structure-based editing. notable limitation of Crompt-to-Proppe editing the incapacity to spatially move existing objects across the image, i.e., implement structural changes. Therefore, we introduce novel approach termed P2P-DD for structure-based editing tasks. P2P-DD combines Prompt-to-Prompt control with the idea of Directed Diffusion (21), extending its capability to perform object replacement.
Directed Diffusion (11 illustrates the possibility of achieving object replacement through direct editing of attention maps during the first N time steps of the generation process. The method performs attention strengthening on the cross-attention maps corresponding to the tokens in I (recall that Z is the token sets representing the object of interest), through calculating a Gaussian strengthening mask (SM) of the bounding-bie box region B. It also performs attention annealing to the remaining area B by applying constant weakening mask (WM), and the two masks are weighted by a scalar c:
M, WM(B.I) SM(BII) if T N M
DDEdit(M,, B,I) =
otherwise
Our proposed P2P-DD method aims to encourage the background and other details of image generated by Directed Diffusion with higher resemblance the source image. P2P-DD first injects cross-attention maps Mj into Mt as suggested by Hompt-to-Prompe This will preserve the structure and background information from the source image, providing decent starting point for attention map editing. Next, P2P-DD performs pixel value strengthening and weakening on the attention maps M, according to the bounding- B and the tokens in I. suggested in the Directed Diffusion, to achieve the desired object placement:
P2P-DDEdit(M,, M; B, DDEdi M B,),
This combined editing control only applied the first N diffusion time steps, i.e. (T, N-1+ After the first N time steps, cease any control attention maps and allow the target diffusion
5

---

Wipe (Sim)
LED (Sim)
Push (Sim)
Wipe (Real)
LED (Real)
Push (Real)
Figure 3: Visualization of robot manipulation tasks. We evaluate LfVoid on three simulation tasks: Wipe, Push, and LED, as well as three real-robot environments correspondingly.
process to complete in conventional denoising manner. The details of P2P-DD are shown in Figure Attention visualization of the generation process can be found in Appendix A.. and we provide a detailed description of the full algorithm in Appendix A.2
3.2 Example-based visual reinforcement learning
We modify and extend the approach of VICE 5 to devise our method for example-based visual reinforcement learning. At the core of process, employ the image editing methods described in Section 3.1 on observations gathered from randomly initialized environments, producing a dataset of 1024 target images for each task. During training, these target images serve as positive samples, while the images sampled from the agent's replay buffer after number of random exploration steps are treated as negative samples. We train discriminator using these instances with the binary cross-entropy loss and use the output of the discriminator for new observations for the positive class as the agent reward.
The discriminator shares the CNN encoder with the reinforcement learning agent and performs classification the output latent representations. To improve the reward shaping, utilize the label mixup method [6], which performs random linear interpolation of the 0-1 labels and their corresponding hidden vectors obtain continuous labels between and 1. We discover that restricting the negative instances from the recent portion of the replay buffer (the last 5%) promotes the discriminator's ability to discern subtle differences between the target and current observations. Additionally, we find that our method enjoys an ensemble of discriminators for classification results (i.e., RL rewards), which gives the agent more representative rewards.
For the reinforcement learning backbone algorithm, we use DrQ-v2 45 for visual RL training. Based on Twin Delayed DDPG (TD3) (46), DrQ-v2 enhances image representation applying random crop augmentation to the input images. Collectively, these refinements constitute our approach to example-based visual reinforcement learning, making the "learning" from initial observation and language prompts possible.
4 Experiments
this section, we present a comprehensive evaluation LíVoid across both simulated environments and real-world robotic tasks. An illustration of each task can be found in Figure
The environments we use are: LED, where the robot reaches for switch or touches the LED light directly to turn the light from red to green; 2) Wipe, where the robot needs to wipe out stains from the table; 3) Push, where the robot needs push red eube to a goal position indicated by a green dot. The simulated tasks are developed based on the Robosuite benchmark, while we provide corresponding real-world tasks for each environment. A full description of the environments provided in Appendix B.1
4.1 Goal generation
In this section, we evaluate the ability of L.Void and other baseline methods to generate goal images according to two types of editing instructions: appearance-based editing and structure-based editing.

---

Input
Imagic
InstruciPix2Pix D-L
DDIM P2P DreamBooth P2P Nall-text P2P
LfVoid (Ours)
Wape
(Sim)
Wipe
(Real)
LED
(Sim)
(Real)
Figure 4: Appearance- goal image editing results. We compare Li'oid against three recent image editing methods: Imagic, InstructPix2Pix, and DALL E-2, as well as three ablations of LfVoid. Note that InstructPix2Pix has different prompts because it requires editing instruction rather than a description of the image. In both simulation and real-world settings, LfVoid can generate images that better aligned with given text prompts while preserving the remaining source scene. See Appendix C for more examples.
Baselines and ablations. We select three recent image editing methods as baselines: Imagic (0) InstructPix2Pix [26], and mask-based DALL E-2 editing used in DALL-E-Bol (with masks provided by humans), For ablations, DDIM P2P removes both the feature extracting module and the inversion module, and DDIM DD additionally removes P2P in the editing module structure-based editing. On top of DDIM DD, Null-text DD adds the inversion module back. For both appearance- based and structure- based editing tasks, DreamBooth P2P/P2PDDD ablates the inversion module, and Null-text P2P/P2P-DS DDD ablates the feature extracting module.
Appearance-based goal image editing results. In Figure we present the visual goal generation results of the Wipe and LED tasks, focusing on editing object appearances in both simulated and real-world environments. Imagic struggles preserve the structure and details of the source image, and InstructPix2Pix frequently fails to perform local color editing, unable to locate the specific area indicated by the editing instruction. DALL E-2 is able preserve the scenes outside the masks but not capable of performing the desired edits within the masked area. For ablations, DDIM P2P and DreamBooth P2P-DD
mostly carry out the desired edits but often diverge largely from the source image. This demonstrates the importance of the inversion module in preserving the details and structures of the source image. Null-text P2P performs worse in the Wipe tasks compared to Lffoid, demonstrating the gains provided by the DreamBooth token. Overall, L.Void demonstrates superior performance in terms of maintaining the appearance of unrelated parts while effectively editing specified regions: through directly applying control and editing on the attention maps, L/Void capable of locating the target area and performing the desired edit based on language instructions.
Structure-based goal image editing results. In Figure we report the performance of LiVoid in editing tasks with structural changes (the Push task) in both simulated and real-world settings. For baselines, Imagic distorts the robot arm and table significantly and often introduces multiple objects into the image, rather than moving the existing one. InstructPix2Pix only changes the overall color of the image or replaces the gripper with geometric body, failing to achieve the required object displacement. DALL E-2 also fails to relocate the object to the desired place. For ablative results, while DDIM and Null-text DD can successfully move objects to the lower-left corner, the background and the robot arm nearly disappear. Null-text P2P-DD improves the preservation of the source image's 's background, but the shape the moved object inconsistent with its original form. DrcamBooth P2P-DD can preserve the shape of the moved object and relocation is successful, but
7

---

Null-text P2P-DD
Dremmoottt P2P-DD
Inpor
Imagie
InstractPis2Pix
DALLER
DDIM DD
Null-teat Di
I. L'Vddd (Ours)
Push
(Real)
Push
(Sim)
tube
"A phulo gripper photo cube and white table"
the Bonom left corner corner.."
Figure 5: Structure- goal image editing results. We compare LfVoid against Imagic, Instruct- Pix2Pix, and DALL E-2, as well as four ablations of LfVoid. Note that InstructPix2Pix has different prompts because requires editing instruction rather than description of the image. LfVoid can successfully perform object displacement and preserve the background and details of the source image, while other methods fail to do so. See Appendix C for more examples.
the background is not consistent with the source image. Our method, when compared to all baselines, can perform the object displacement task successfully while better preserving the visual features of the object and the remaining scene. These experiments reveal the importance of each component in our method. The creative integration of different image editing techniques provides a powerful tool for image editing with structural changes, demonstrating significant improvement over existing methods.
Quantitative results. In Table we present the quantitative results of goal image generation. We manually sample 500 real goal images per task for evaluation purposes) and calculate the pair-wise LIPIS distance between the generated images and the real goal images. The lower distance represents higher image editing quality, and results show that LiVoid outperforms all the baselines and ablations. Additionally, we present an ablation study of the feature extracting module on the Wipe and Push task Table showing the performance gain brought by the DreamBooth token.
Wipe(Sim)
LED(Sim)
Push(Sim)
Wipe(Real)
LED(Real) Push(Real)
InstructPix2Pix
0.23+0.11
0.41=0.06
0.34+0.17
0.18+0.04
0.45+0.06
0.53#0.08
Imagic 0.23+0.05
0.42+0.08
0.43+0.11
0.34+0.13
0.35+0.1
0.49=0.
DALL-E
0.10#0.04
0.25+0.07
0.20#0.06
0.39+0.08
0.2020.07
0.38#0.03
DreamBooth+Editing
0.53+0.05
0.62+0.05
0.42+0.15
0.58+0.06
0,43+0.08
0.3620.06
DDIM+DD/P2P
0.26#0.07
0.37+0.06
0.43+0.04
0.34+0.04
0,4020.03
0.50#0.03
Null-text+DD
N/A
N/A
0.36+0.05
N/A
N/A
0.51
LfVoid
0.08=0.04
0.17+0.06
0.14+0.07
0.12+0.05
0.19#0.05
0.32+0.03
Table Quantitative results on goal image generation. We calculate the pair-wise LIPIS distance between generated images and the manually created real goal images for evaluation purposes.
Wipe(Sim)
Push(Sim)
Wipe(Real)
Push(Real)
LfVoid w/o DreamBooth
0. 00.000.04
0.24+0.06
0.1840.06
0.39+0.06
LiVoid
0.0820.04
0.1420.07
0.1240.05
0.32+0.03
Table 2: Ablation study on DreamBooth. We evaluate the improvement brought by the DreamBooth token through pair-wise LIPIS distance between generated images and manually created real goal images for evaluation purposes.
4.2 Example-base based visual reinforcement learning
We now dive the results obtained from applying example-based visual reinforcement learning using the goal images generated by LfVoid. We select three basclines for comparison: CLIP, IP2P,

---

and Real Goal. CLIP relies only on language guidance and does not require any synthesized goal images. It leverages the Contrastive Language-Ine -mnage Pre-training score [47] of the task prompt and observations as a reward, and it aims to verify the effectiveness of translating prompts into images against raw language hints. IP2P uses InstructPix2Pix for generating the goal images and then runs the same example-based RL as in LfVo... It allows us to compare the performance of LfVoid with a straightforward image editing technique. Real Goal is an upper-bound performance of our pipeline, where we manually create ideal goal images, such erasing the stains directly in the simulator for the Wipe task. This strategy illustrates the potential of our algorithm, indicating the gap between the generated goals and the real ones, and demonstrates how the quality of generated images will influence the performance of downstream RL.
Simulation results. We report both episode reward (Figure 6 and numerical metrics (Table 3) three simulation tasks. The training details can be found in Appendix [B3]] We observe that the CLIP baseline achieves low rewards in all the tasks, showing that directly using language guidance through CLIP embeddings is not sufficient. Moreover, LfVoid consistently outperforms the IP2P baseline and gains comparable performance with the Real Goal upper bound, providing evidence that generated goal images with high fidelity to both the original scene and the editing instructions is crucial for successful deployment of downstream example-based RI..
Real-world results. In the more complex real-world settings, we aim to validate the feasibility of LíVoid through visualization of reward functions. As in our previous experiment, we generate goal images using InstructPix2Pix (IP2P) and LiVoid, and also manually collect a real goal dataset as an upper-bound (Real Goal). We then train discriminators on these datasets separately and use the outputs of the discriminators as rewards. In addition, we compare with language- based guidance (CLIP), which uses the distance of the CLIP embeddings between the task prompt and the observation images as rewards. We manually record five successful trajectories for each task and use these reward functions to obtain the reward curves. As illustrated in Figure 7 LfVoid can provide near monotonic dense reward signals comparable to those from Real Goal. On the contrary, the curves obtained by IP2P and CLIP fail to accurately track the progress of each task. This demonstrates that our method provides more instructive learning signals for real-world robotic tasks than either IP2P or CLIP.
CLIP
InstructPix2Pix
Real Goal (Oracle)
Ours
Wipe (Cleaned Stains Patch) )
0.0+0.0
1.7+1.9
22.0+5.8
21.3+5.8
LED (Success Rate / %)
10.0+20.0
0.0+0.0
13.3
75.0+50.0
Push (Success Rate %)
12.5+19.1
3.3+3.5
30.0+13.7
27.9+12.7
Table 3: Numerical metrics of simulation tasks. We report the success rate for LED and Push, and the number of stain patches cleaned for Wipe. Please refer Appendix B. for details.
Comparison to RL with imagined goals. In a parallel direction, RL with imagined goals can also solve robot manipulation tasks without the need for human-specified reward functions and expert demonstations 29 30). These methods first leverage self-supervised learning to learn a representation
15.0
P2P
IR2P
1.00
F2P
Episode Reward Reward
12.5
Real
QLIP
Episode Reward
Rui Goil CUP
Episode Reward
0.75
CUP
Real Doa
10.0
L'ord
0.50
Lividd
7.5
0.25
6.0
0.00
2.5
0226
0.0
-0.00
2
Frame
x1o
Frame
x10"
Frame
x10
(a) Wipe Episode Reward
(b) LED Fpisode Reward
(c) Push Episode Reward
Figure 6: Episode reward of simulation tasks. The results of the CLIP baseline (CLIP), Instruct- Pix2Pix baseline (IP2P), using real goal image as upper bound (Real Goal), and LfVoid (Ours).

---

CUP Real Gool IP2P Livid
CLP Real Goa P2P LPWiid
10
CUP Rest Gos! F2P
Reward Score
Reward Score
Livid
Reward
8
12 14 10
8
10
14
2
4
6
10
12
14
16
Trajectory Step
Trajectory Step
Trajectory Step
Wipe Reward Curve
(h) LED Reward Curve
(c) Push Reward Curve
Figure 7: Reward curve on successful trajectories for real-robot tasks. Visualization of the reward function of LíVoid (Ours) compared with baselines. See Appendix for implementation details.
space of observations, and then sample imagined goals to train a coal-conditioned policy. However, unlike LfVoid, a real goal image is still required at test time to run the noal-conditioned policy. In Table 4 we compare the performance of LfVoii with two algorithms: Visual RL with Imagined Goals (VIG) [29] and Goal-Aware Prediction (GAP) [30], on the three simulation tasks, We observe that only LfVoid can solve the tasks with positive rewards, the other two methods fail in all tasks. Because both these methods use random policy for exploration, they may never reach the user- specified goal states during training. Therefore, they lack the knowledge required to fulfill the goals. This further demonstrates the significance of leveraging web-scale text-to-image models to provide informative visual guidance during training. Please refer to Appendix C.2
for more analysis.
Push(Sim)
Wipe(Sim)
LED(Sim)
Visual with Imagined Goals (VIG)
10.39+1.79
-6.03+2.02
-1.77+0.55
Goal-Aware Prediction (GAP)
-0.07=0.91
-5.84+1.00
-0,39+0.
Learning from the Void (Li'ooid)
0.35#0.51
12.8614.31
4.47+2.10
Table 4: Comparison with RL using imagined goals. We report the episode reward during test time and compare LfVoid with existing methods of visual RL using imagined goals.
Discussion
In this work, we present LfVoid, an effective approach for leveraging the knowledge of large-scale text-to-image models and enabling zero-shot reinforcement learning from text prompts and raw observation images. We have identified and tackled numerous challenges that arise when applying ffute-of-the-ar image editing technologies example-base based RL methods. Our work highlights the potential for the adaptation and application of saage-generation techniques in the realm of robotics. Our findings not only enhance the understanding of image editing for robotic applications but also provide a clear direction for the image generation community to address real-world challenges.
Although our proposed method has shown promising results, we acknowledge that it is not without limitations. While LfVoid has succeeded in generating goal images to train example- based RL policies, certain level of prompt tuning is needed in order to achieve optimal editing performance (see Appendix B.2). and we refer the readers to Appendix for an analysis of failure cases.
We would also like to highlight that there exists a gap between the ability of text-to-image generation models and the need for robot learning. For example. large diffusion models exhibit poor under- standings of the spatial relationships between objects [48]; therefore, both the generative models and the editing methods struggle to handle object displacement solely through language prompts. The Directed Diffusion technique, while being able to perform movements of objects, requires a user- defined bounding box to achieve precise control. Furthermore, large generative models sometimes struggle to generate images that are considered valid under physics laws. When asked to pick up bottle with a robot arm, model simply stretches the shape of the arm to reach the bottle rather than changing the joint position. Lastly, Al-generated images inevitably introduce alterations to the details of objects. The robustness of current visual reinforcement learning algorithms to such changes remains an open question.
10

---

Acknowledgment
HX is supported by National Key R&D Program of China (2022/D0111100))
References
[1] Georgios Papagiannis and Yunpeng Li. Imitation learning with sinkhorn distances. In Machine Learning and Knowledge Discovery Databases: European Conference, ECMI. PKDD 2022, Grenoble, France, September 19-23, 2022, Proceedings, Part IV, pages 116-131. Sprinerr 2023.
[2] Robert Dadashi, Léonard Hussenot, Matthieu Geist, and Olivier Pietquin. Primal wasserstein imitation learning. arXiv preprint Siii:2006.046678, 2020.
[3] Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. bnseriminator-acto--rriiii Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint 1809.02809.0292. 2018.
[4] Siddhant Haldar, Vaibhav Mathur, Denis Yarats, and Lerrel Pinto. Watch and match: Super- charging imitation with regularized optimal transport. In Conference on Robot Learning, pages 32-43. PMLR, 2023.
[5] Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control with events: A general framework for data-driven reward definition. Advances in neural information processing systems, 31, 2018.
[6] Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-en robotic reinforcement learning without reward engineering, arXiv preprint arXiv:1904.077 07854, 2019.
[7] Ben Eysenbach, Sergey Levine, and Russ Salakhutdinov. Replacing rewards with examples: Example-based policy search via recursive classification. Advances in Neural Information Processing Systems, 34:11541-11552 2021.
[8] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
[9] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.0688) 2022.
[10] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 2022.
[11] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton. Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic exxtttoimage diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 25:36479-36490 2022.
[12] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models, 2021.
[13] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Benton C. Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. ArXiv, abs/2206.1072 2022.
[14] Stephen Adams, Tyler Cody, and Peter A Beling. A survey of inverse reinforcement learning. Artificial Intelligence Review, 55(6):3377-45 2022.
[15] Zhao Mandi, Homanga Bharadhwaj. Vincent Moens, Shuran Song, Aravind Rajeswaran, and Vikash Kumar. Cacti: framework for scalable multi-task multi-scene visual imitation learning. arXiv preprint
2022.

---

[16] Zoey Chen, Sho Kiami, Abhishek Gupta, and Vikash Kumar. Genaug: Retargeting behaviors to unseen situations via generative augmentation. arXiv preprint Xiix:2302.06671, 2023.
[17] Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tompson, Anthony Brohan, Su Wang, Jaspiar Singh, Clayton Tan, Jodilyn Peralta, Brian Ichter, et al. Scaling robot learning with semantically imagined experience. arXiv preprint arXiv:2302.11550 2023.
[18] Ivan Kapelyukh, Vitalis Vosylius, and Edward Johns. Dall-e-bot: Introducing web-scale diffusion models to roboties. arXiv preprint arXiv:2210.024538 2022.
[19] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yacl Pritch, and Daniel Cohen-Or. Prompt-to-promp image editing with cross attention control. arXiv preprint arXiv:2208.01622 2022.
[20] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. arXiv preprint arXiv:2210. 09276,, 2022.
[21] Wan-Duo Kurt Ma, JP Lewis. W Bastiaan Kleijn, and Thomas Leung. Directed diffusiio: Direct control of object placement through attention guidance. arXiv preprint arXiv:2302.13/55. 2023.
[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6855 2020.
[23] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500, 2020.
[24] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[25] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 2 2021.
[26] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09000 2022.
[27] Parsa Mahmoudieh, Deepak Pathak, and Trevor Darrell. Zero-shot reward specification via grounded natural language. In Kamalika Chaudhuri. Stefanie Jegelka, Le Song, Csaba Szepes- vari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 14743-14752. PMLR, 17-23 Jul 2022.
[28] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuiing text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208. 12242, 2022.
[29] Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. CoRR, abbs/180.00474 2018.
[30] Suraj Nair, Silvio Savarese, and Chelsea Finn. Goal-aware prediction: Learning to model what matters, 2020.
[31] Kevin Li, Abhishek Gupta, Ashwin Reddy, Vitchyr H Pong, Aurick Zhou, Justin Yu, and Sergey Levine. Mural: Meta-leaming uncertainty-aware rewards for outcome-driven reinforcement learning. In International conference on machine learning, pages 6346-6356 PMLR, 2021.
[32] Daesol Cho, Scungjae Lee, and H Jin Kim. cntcome-directed reinforcement learning by uncer- tainty& & temporal distance-aware curriculum goal generation. arXiv preprint arXiv:2301.1174) 11741. 2023.
[33] Maria Attarian, Advaya Gupta, Ziyi Zhou, Wei Yu, Igor Gilitschenski, and Animesh Garg. See, plan, predict: Language-gun guided cognitive planning with video prediction. arXiv preprint arX.v:220.033825, 2022.
12

---

[34] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint Sitiv::2207056000 2022.
[35] Yan Ding, Xiaohan Zhang, Chris Paxton, and Shiqi Zhang. Task and motion planning with large language models for object rearrangement. arXiv preprint 2rX3v:2303.06444 2023.
[36] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning, pages
PMLR, 2022.
[37] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang. Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:2210.0309A 2022.
[38] Weiyu Liu, Yilun Du, Tucker Hermans, Sonia Chernova, and Chris Paxton. Structdiffusion: Language-guided creation of physically-valid structures using unseen objects, 2023.
[39] Nikolaos Gkanatsios, Ayush Jain, Zhou Xian, Yunchu Zhang. Christopher Atkeson, and Katerina Fragkiadaki. Energy-based models as zero-shot planners for compositional scene rearrangement. arXiv preprini arXiv:2304. 14391, 2023.
[40] Utkarsh A Mishra and Yongxin Chen. Reorientdiff: Diffusion model based reorientation for object manipulation. arXiv preprint arXiv:2303. 2023.
[41] Shikhar Bahl, Abhinav Gupta, and Deepak Pathak. Human-to-robo imitation in the wild, 2022.
[42] Yilun Dai, Mengjiao Yang. Bo Dai, Hanjun Dai, Ofrr Nachum, Josh Tenenbaum, Dale Schuur- mans, and Pieter Abbeel. Learning universal policies via text-guided video generation. arXiv preprint arXiv:2302.001111 2023.
[43] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or, Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.09794, 2022.
[44] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint WiXi::2207.12598, 2022.
[45] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regu- larizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021.
[46] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. ArXiv, abs/1802.094777, 2018.
[47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 7748-8766 PMLR, 2021.
[48] Colin Conwell and Tomer Ullman. Testing relational understanding in text-guided image generation. arXiv preprint arXiv:2208.0000. 2022.
[49] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 2017.
[50] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2207.0990 2021.
[51] Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009.12299, 2020.
13

---

[52] Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, and Wojeiech 7aremba. Multi-goal reinforcement learning: Challenging robotics environments and request for research, 2018.
[53] Mark E. Glickman. The glicko-2 system. hetp://www.glickt htw.icko.nntggliooggliiieee pdf 2013.
14


Reviews:
[{"review_id": "NjQmX29r09", "reviewer": "Reviewer_jYPU", "summary": "In this paper, authors propose a new method called LfVoid to tackle RL tasks. LfVoid can generate consistent visual goal frames through using pretrained LDM and subsequently train a discriminator to output rewards for downstream RL task. In order to improve editing consistency, DreamBooth, null-text inversion, P2P and Directed diffusion are incorporated into the pipeline. ", "questions": "1. The evaluation benchmark is a little simple which more or less wastes the strong expressive power of pretrained LDM. What is the advantage of using LDM in your application compared with training a latent VAE to generate the goal frame except for less training frames?\n2. As mentioned in summary, 4 techniques, including DreamBooth, null-text inversion, P2P and Directed diffusion, are incorporated but the ablation study only compares null-text P2P-DD and null-text P2P. In my opinion, the effects of DreamBooth and null-text inversion are overlapped. Can you provide any experiment to validate the improvement brought by DreamBooth token? Also, I understand the difficulty of ablation study in this work due to the lack of effective quantitative comparison. Is it possible to design some new metrics that only focus on the region of interest since the bounding boxes are provided?\n3. Comparison with existing RL methods leveraging goal frame generation, for example, Goal-Aware Prediction[1], was not provided. Can you compare LfVoid  with these baselines or explain why this is not meaningful? \n\n[1]. Nair, S., Savarese, S. &amp; Finn, C.. (2020). Goal-Aware Prediction: Learning to Model What Matters. <i>Proceedings of the 37th International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 119:7207-7219 Available from https://proceedings.mlr.press/v119/nair20a.html.\n\n", "limitations": "This paper solved the image editing problem required for goal frame generation. A solid contribution was made. I would increase my rating if my concerns are properly addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well written, clear and easy to understand.\n2. An organic integration of several techiniques for controlled generation  \n3. Compared with existing editing methods, only the proposed LfVoid correctly modifies the desired objects and makes no obvious changes to other irrelevant things.\n4. The improved generative quality subsequently enhance the performance of example-based RL.", "weaknesses": "1. LDM is kind of overkilling in this benchmark.\n2. Lack comparison with other RL methods leveraging goal frame generation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear **Reviewer jYPU**, we thank you for your detailed and thorough review. In the following section, we seek to address each of your concerns.\n\n---\n\n**Q**: \"LDM is kind of overkilling in this benchmark.\" and the advantage of LDM over VAE to generate the goal frame. \n\n**A**: In LfVoid, the LDM bears the task of **editing** the current observation based on a **text description**, while keeping the irrelevant parts of the image **unchanged**. While there exist works like [1] that utilize CVAEs to generate text-conditioned images or use VAE along with GAN for text-conditioned image editing [2], their ability to fulfill all three requirements of LfVoid is limited. To the best of our knowledge, the pipeline proposed by LfVoid that combines LDMs with inversion, attention control, and object specification is the only method that makes the downstream example-based RL procedure possible. We attribute this to large LDM models’ better understanding of the semantic meaning of the world. \n\nWe also explored the line of work that uses VAEs to generate random frames and train a goal-conditioned policy to fulfill a user-provided goal image during test time. The generated image (randomly sampled from the latent space and decoded) can be seen in Figure 6 of the attachment, it's clear that those images are blurry and lose much of the details, which is unbearable for downstream example-based RL methods.\n\n**Q**: \"Lack comparison with other RL methods leveraging goal frame generation, for example, Goal-Aware Prediction[3], was not provided.\"\n\n**A**: Thank you for your advice. This is indeed reverant to our method and we do think the comparison with such methods is valuable. We have conducted two additional experiments, Goal-Aware Prediction(GAP) and its predecessor work Visual Reinforcement Learning from Imaginary Goals (VIG)[4].\n\nGAP first collects transitions from the environment using a random policy, then trains a dynamic model in latent space on these data. During test time, GAP uses model predictive control to achieve a **user-provided** goal image. VIG collects a random dataset as well and trains a VAE over the image observations, then trains a goal-conditioned RL agent to fulfill sampled goals from the VAE.  \n\nWe want to point out that the goal image (at test time) in both settings is user-provided, which means the user has to figure out one way to achieve the goal state before ever having a feasible policy to achieve it. What's more, since both these method uses random policy to collect data, it's hard for them to have a faithful knowledge of the dynamics or distribution over the goal image since it can be difficult for the random policy to ever reach such a goal state (like wipe all the markings off the table)\n\nIn the additional results, we provide an oracle goal image (as contrust to the edited images from LfVoid) to both methods to make them fulfill the goal state. The rewards of the three simulated tasks can be found in Table 5 in the attachment. It's clear that they can not achieve any of the tasks despite providing the oracle goal observation.\n\n\n**Q**: \"As mentioned in summary, 4 techniques, including DreamBooth, null-text inversion, P2P and Directed diffusion, are incorporated but the ablation study only compares null-text P2P-DD and null-text P2P\"..., \"Can you provide any experiment to validate the improvement brought by DreamBooth token?\"\n\n**A**: We thank the reviewer for this question and have provided more ablation studies in Figure 1 and Figure 2 of the attachment. In particular, we report the results of removing the DreamBooth module (Null-text P2P and Null-text P2P-DD) and observe that the DreamBooth token contributes significantly to preserving the background details and performing the desired edit. Additionally, we report quantitative ablation results in Table 3 and Table 4 of the attachment.\n\nWe kindly refer reviewers to the general response section for common questions. If there's anything unclear in the above, we're more than glad to further discuss the details.\n\n[1] Zhang, Chenrui, and Yuxin Peng. \"Stacking VAE and GAN for context-aware text-to-image generation.\" 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM). IEEE, 2018.\n\n[2] Pernuš, Martin, et al. \"Fice: Text-conditioned fashion image editing with guided gan inversion.\" arXiv preprint arXiv:2301.02110 (2023).\n\n[3] Nair, Ashvin V., et al. \"Visual reinforcement learning with imagined goals.\" Advances in neural information processing systems 31 (2018).\n\n[4] Nair, Suraj, Silvio Savarese, and Chelsea Finn. \"Goal-aware prediction: Learning to model what matters.\" International Conference on Machine Learning. PMLR, 2020."}, {"review_id": "vpQ4ahYZie", "reviewer": "Reviewer_pQmz", "summary": "This paper demonstrates a novel way to utilize existing large-scale text-to-image models for robot learning. Specifically, they modify existing pre-trained text-to-image models to produce visual goals for example-based reinforcement learning, before learning policies from the generated images. Experimentation in both simulation and the real world shows that the proposed modifications to pre-trained text-to-image models are able to generate higher-fidelity images than the baselines, and that this also leads to gains in downstream performance.", "questions": "- Qualitative Results. While the language instructions for Imagic and the proposed method are the same, ensuring a fair comparison, why are they different for InstructPix2Pix? Furthermore, why is the ablation only done for Figure 5, and not for Figure 4?\n\n- RL Experiments. Were all hyper-parameters in the RL training the same across methods? In Figure 7, the number of trajectory steps seems to vary across tasks -- why is this the case? Are the trends different if more trajectory steps were allowed?", "limitations": "Yes, the authors have adequately addressed the limitations of the work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Well-written. The paper is well-written, and the figures aid in the understanding of the methodology.\n- Novelty. The proposed approach seems to be a novel one which builds on ideas from the literature.\n- Real world experiments. The inclusion of real world experiments for both the image goal generation and the RL training make the strength of the proposed approach a lot more convincing than otherwise.\n- Strong results. The proposed approach outperforms baselines from prior work as well as the ablations. The qualitative results provided in Figures 4 and 5 display impressive performance of the model.", "weaknesses": "- Evaluation. While both qualitative and quantitative metrics for image goal generation are included, the worry is that the qualitative examples may be cherry-picked and that the user-study done for a quantitative assessment may not be extensive enough. Would it be possible to use the standard metrics which conditional image generation papers utilize to quantitatively evaluate the proposed approach? For instance, once the ideal goal image has been generated, could the output of the proposed method be compared to using either MSE or LPIPS [1]? Furthermore, would it be possible to provide more qualitative examples? There are about three more examples per task in the appendix, but these all seem to be very similar to one another and limited in diversity. Particularly, more visualizations of the real world images would be appealing as this setting is much harder than the simulated images.\n\n- Prompt tuning. Ideally, at test-time, you are just given the current image, and a text-based goal prompt, nothing else. Line 296 says that \"a certain level of prompt tuning is needed in order to achieve optimal editing performance.\" How much prompt tuning was done for the method? Was the same done for the baselines?\n\n[1] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conf. Comput. Vis. Pattern Recog., 2018.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear **Reviewer pQmz**, we thank you for your detailed and thorough review. In the following sections, we seek to address each of your concerns.\n\n---\n\n**Q**: Would it be possible to provide more qualitative examples? There are about three more examples per task in the appendix, but these all seem to be very similar to one another and limited in diversity. \n\n**A**: We have provided more diverse qualitative results of LfVoid in Figure 7 of the attachment. Moreover, we add three additional real-world environments with a UR5 robot arm and one simulation task with a distractor to further increase the diversity. The results are shown in Figure 3 of the attachment and we observe that LfVoid can successfully perform all the desired edits in various environments.\n\n\n**Q**: Why is the ablation only done for Figure 5, and not for Figure 4?\n\n**A**: We thank the reviewer for this question and have added additional ablations for both Figure 5 and Figure 4. Please refer to Figure 1 and Figure 2 in the attachment. It should be noted that LED(Sim) and LED(Real) can perform the desired editing without the DreamBooth token and therefore we did not use DreamBooth when generating the goal images of these two environments.\n\n\n**Q**: While the language instructions for Imagic and the proposed method are the same, ensuring a fair comparison, why are they different for InstructPix2Pix? \n\n**A**: Both LfVoid and Imagic expect the text prompt to directly describe the goal image, while InstructPix2Pix expects the text prompt to describe the editing instructions to perform on the input image. This results in a slightly different prompt for these methods.\n\n**Q**: Were all hyper-parameters in the RL training the same across methods? \t\n\n**A**: Yes, the hyper-parameters are the same and reported in Table 6 in the Appendix.\n\n\n**Q**: In Figure 7, the number of trajectory steps seems to vary across tasks -- why is this the case? Are the trends different if more trajectory steps were allowed?\n\n**A**: This is because different tasks require different numbers of steps to finish when humans are collecting the demonstrations. For Wipe-Real, we found that it needs about 18 steps to completely wipe off the markings. And for LED-Real the number is 16, for Push-Real the number is 14.\n\n\nWe kindly refer reviewers to the general response section for common questions. If there's anything unclear in the above, we're more than glad to further discuss the details."}, {"review_id": "Q43Bga8YPz", "reviewer": "Reviewer_NX6j", "summary": "Post rebuttal updating score to 5\n\nThis work proposes to use generated visual goals for RL. A diffusion model based approach is used to edit visual observations based on text prompts. The proposed image editing approach is shown to be better than prior text-based image editing approaches based on a human evaluation. Visual goals predicted using the proposed method shows improved RL performance compared to baselines that use other image editing mechanisms. ", "questions": "* Many details about the setting are unclear. Need better clarity on the following. \n  - Where do the text prompts/instructions for editing come from? \n  - What are the inputs and the outputs of the components of the approach?\n  - How does learning work? \n* Is it possible to use any quantitative metrics for image editing experiments (e.g., based on CLIP encoders)?\n* For the manually defined visual goals how many images were manually created? How many task instances are there altogether? \n* line 297 mentions ‘a certain level of prompt tuning is needed’, but such details are missing from the main text. \n", "limitations": "See above", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Leveraging pre-trained models for image editing is an interesting idea\n* Proposed approach seems to work better than some prior editing approaches\n* Some indication that the generated visual goals help learning\n", "weaknesses": "Originality: Imagining visual goals for RL has been explored in prior work (e.g., [1] https://arxiv.org/pdf/1807.04742.pdf), which limits the originality of this work. The use of diffusion models could be new. \n\nSignificance: \n* Experimental results do not demonstrate the need for visual goals. For instance, there are no baselines in the RL experiments which are not based on imagined goals. \n* Very few tasks are tested\n* Baseilnes are inadequate\n* Approach is too specific to the tasks considered and generality of approach is questionable \n\nPresentation/Clarity\n* Need better clarity on motivation, background on methods, problem formulation and approach overview\n* There needs to be a clear motivation/overview section before section 3. Technical details lack context and are difficult to follow. For instance, section 3 talks about latent diffusion models without providing any background/motivation. There needs to be a technical overview section on diffusion models.\n* Method description is vague and mathematical details are missing. \n  - 3.1.1 Description of the optimization approach is vague.\n  - 3.1.2 What is inversion and why is it necessary? \n  - 3.1.3 Assumes prior knowledge and familiarity with diffusion models. What is x_0? What does it mean to replace attention maps? \n\nExperiments\n* Is the comparison on image editing fair? The proposed method is fairly specific to the tasks considered while it is compared against general purpose image generation/editing techniques such as pix2pix. \n* Inadequate baselines for the RL experiment: There seem to be no baselines that are not based on imagined goals/image editing. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear **Reviewer NX6j**, we thank you for your detailed and thorough review. In the following sections, we seek to address each of your concerns.\n\n---\n\n**Q**: Experimental results do not demonstrate the need for visual goals: no baseline in the RL experiments that are not based on imagined goals.\n\n**A**: We would like to point out that the CLIP baseline mentioned in line 260 is not based on imagined goals. It only requires a user-specified text prompt and the CLIP distance between the text prompt and observation image is used as a reward to train RL. Results show that LfVoid clearly outperforms it, demonstrating the advantage of providing visual guidance.\n\n**Q**: There needs to be a clear motivation/overview section before section 3. Technical details lack context and are difficult to follow. \n\n**A**: We appreciate your advice and will polish the paper to improve the delivery in the final version. Particularly, we will improve the corresponding part of the paper. Due to the length limit, it's hard to go into too much detail about the technical backgrounds of the diffusion model, and it's common practice to refer the readers to relevant papers for more detail, including the image editing baselines used in LfVoid [1,2,3], nevertheless, we agree that it's always better to provide sufficient background in the paper itself.\n\n**Q**: Descriptions of the optimization approach is vague.\n\n**A**: For the optimization method used in goal generation, we use the exact optimization methods used in the released code of DreamBooth and Null-text Inversion: AdamW for DreamBooth and Adam for Null-text Inversion. The other parameters are all the default values provided by the official code.\n\n**Q**: What is inversion and why is it necessary?\n\n**A**: We need to invert the provided source image to a diffusion process so that our editing module can generate the target image conditioning on both the source image and the text prompt. Diffusion models could only synthesize a target image conditioning on a text prompt. Through inversion, we can extrinsically condition the generation process on both the source image and the text prompt[3]. We provide results where we eliminate the inversion module completely: Dreambooth P2P in Figure 1 and Dreambooth P2P-DD Figure 2 in the attachment, and observe that the generated target image differs greatly from the source image.\n\n**Q**: What is x_0? What does it mean to replace attention maps?\n\n**A**: $x_0$ denotes the final generated image at the last diffusion step. During each generation step, the diffusion model uses cross-attention to process the image and the text prompt, and attention maps are produced. We can replace the attention maps between the diffusion processes that separately generate the source and target image. Please refer to line 130 and the original paper[1] for more details.\n\n**Q**: Where do the text prompts/instructions for editing come from?\n\n**A**: They are provided by humans, which is a description of the desired goal image.\n\n**Q**: What are the inputs and outputs of the components of the approach?\n\n**A**:  For the Feature Extracting Module, the input is several initial visual observations and a prompt describing the object we want to remember. The output is a fine-tuned model using the <sks> token to represent that object. \n\nFor the Inversion Module, the input is the source image and the text prompt description. The text prompt can contain the <sks> token to better describe the scene. The output is a noise vector and a series of optimized null-text embeddings for each diffusion timestep.\n\nFor the Editing Module, the input includes both the input and output of the Inversion Module, as well as the text prompt describing the target image(for appearance-based editing), or the bounding box and trailing tokens(for structure-based editing). The output is the edited image.\n\nWe will include this in Section 3.1. \n\n**Q**: How does learning work?\n    \n**A**: We assume this is asking about the procedure of the RL. Generally speaking, it consists of two iterative procedures. Given a set of generated goal images, a classifier is trained to distinguish the goal image and observation encountered by an RL agent from a replay buffer. Meanwhile, the RL agent is optimized to gain more reward from the classifier to achieve the states that assemble the goal image, granting it the ability to reach the desired goal during test time.\n    \n**Q**: For the manually defined visual goals how many images were manually created? \n\n**A**: In the pipeline of LfVoid, there are no manually defined visual goals since LfVoid can synthesize the visual goals requiring only a user-defined text prompt and/or bounding box. For evaluation purposes, we sample 1024 visual goals to test the upper bound of LfVoid and report this result as a baseline.\n    \n**Q**: Unfair comparison to general purpose methods.\n    \n**A**: We would like to argue that the goal-generation method is a general editing method that can perform various editing tasks that are not limit to robotics context and therefore the comparison on image editing is fair. Secondly, the example-based learning of LfVoid is also a general-purpose method compatible with any visual RL setting. The design choices and algorithms of LfVoid are not specific to those tasks.\n\nWe kindly refer reviewers to the general response section for common questions. If there's anything unclear in the above, we're more than glad to further discuss the details.\n  \n[1] Hertz, Amir, et al. \"Prompt-to-prompt image editing with cross attention control.\" arXiv preprint arXiv:2208.01626 (2022).\n\n[2] Kawar, Bahjat, et al. \"Imagic: Text-based real image editing with diffusion models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[3] Mokady, Ron, et al. \"Null-text inversion for editing real images using guided diffusion models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n"}, {"review_id": "k4Ds2etrlw", "reviewer": "Reviewer_WQFw", "summary": "The objective of this paper is to harness the capabilities of pre-trained text-to-image models and image editing techniques in order to facilitate robot learning. This is achieved by leveraging these tools to modify the current scene towards the intended image objective.", "questions": "I hope the authors can address all the questions and concerns i have in the weakness section.", "limitations": "Yes, the authors have addressed all the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "The paper presents a novel approach that harnesses the knowledge embedded in large pre-trained generative models to facilitate zero-shot guidance for robot learning tasks. Additionally, the paper asserts that its edited image functionality outperforms existing methods. Lastly, the paper identifies a gap in current text-conditioned image generation models.", "weaknesses": "1. Figure 1 can be enhanced to effectively communicate the problem the author intends to address, while also succinctly highlighting their contribution and providing an overview of their proposed method.\n2. In recent years, significant advancements in diffusion models have led to the emergence of notable works such as CACTI, GenAug, and ROSIE. These works have demonstrated considerable improvements in generative modeling and learning. However, none of these works were used as baselines in the paper, including DALL-E-Bot, which bears the closest resemblance to this work. While acknowledging technical differences, it would be valuable to compare and contrast the proposed method with these related works.\n3. I appreciate the author's effort to compare various methods of image editing with their proposed approach. However, many of these methods are pretrained on different datasets, necessitating the evaluation of their performance on a comprehensive image editing dataset for a more equitable and meaningful comparison.\n4. The author's inclusion of both simulation and real-world demonstrations is commendable. However, to enhance the persuasiveness of the findings, it would be beneficial to introduce slightly more complex tasks or incorporate additional distractors during evaluation to assess the robustness of the proposed method.\n5. The author may also consider exploring the line of work that involves generating demonstrations using video editing techniques for robot learning, as exemplified in \"Human-to-Robot Imitation in the Wild,\" RSS 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear **Reviewer WQFw**, we thank you for your detailed and thorough review. In the following sections, we seek to address each of your concerns.\n\n---\n\n**Q**: Figure 1 can be enhanced to effectively communicate the problem the author intends to address, while also succinctly highlighting their contribution and providing an overview of their proposed method.\n    \n**A**: We thank the reviewer for this suggestion and the updated Figure 1 is shown in Figure 5 of the attachment. We added a subfigure picturing what the problem is that we are trying to address.\n    \n**Q**: In recent years, significant advancements in diffusion models have led to the emergence of notable works such as CACTI, GenAug, and ROSIE. These works have demonstrated considerable improvements in generative modeling and learning. However, none of these works were used as baselines in the paper, including DALL-E-Bot, which bears the closest resemblance to this work. While acknowledging technical differences, it would be valuable to compare and contrast the proposed method with these related works.\n\n**A**: CACTI, GenAug, and ROSIE use diffusion models to perform data augmentation on existing expert demonstrations, improving the robustness and generalization ability of the manipulation policy. It should first be noted that LfVoid aims to solve a different problem when compared to this line of work: we utilize the large-scale diffusion model to translate language instructions into corresponding goal images. Therefore, the outcome of solving the task is not provided by the demonstrations but by the large-scale pre-trained model. That being said, the distinction between LfVoid and previous works is clear. While acknowledging the differences, the use of diffusion models to perform image editing is similar in these works and LfVoid. The editing modules in this line of work mainly perform localized image inpainting conditioned on human-specified or automatically generated object masks. Since the models used in these works are not open-sourced, we cannot directly compare LfVoid with them.\n\nDALL-E-Bot is a task-specific method that can only solve object rearrangement tasks, and thus its policy cannot generalize to our manipulation tasks. However, we are able to compare the goal generation method proposed by DALL-E-Bot with LfVoid. More specifically, we provide human-specified masks to DALL-E 2 model and the text prompt describing the edited image, and report the qualitative results in Figure 1 and Figure 2 and the quantitative results in Table 1 and Table 2 of the attachment. Despite DALLE-2 having the advantage of a user-specified mask (the region outside the mask will remain unchanged), LfVoid still has better or comparable performance in all the tasks.\n\n    \n**Q**: I appreciate the author's effort to compare various methods of image editing with their proposed approach. However, many of these methods are pre-trained on different datasets, necessitating the evaluation of their performance on a comprehensive image editing dataset for a more equitable and meaningful comparison.\n    \n**A**: We would like to point out first that both LfVoid and Imagic are **fine-tuning** methods, not models, that can be used on any pre-trained diffusion models. To ensure a fair comparison, we use the same open-source StableDiffusion model(v1-4) in our experiments when evaluating LfVoid and Imagic. \n\nSecondly, while Instruct-pix2pix is indeed a released model, it also builds upon the open-source StableDiffusion model: it fine-tunes the model parameters with generated paired data so that the model can perform editing based on an input image and a text instruction.\n \nTherefore, we believe that our comparison between these methods is reasonable since all three editing methods build upon the same pre-trained StableDiffusion Model.\n\n\n**Q**: The author's inclusion of both simulation and real-world demonstrations is commendable. However, to enhance the persuasiveness of the findings, it would be beneficial to introduce slightly more complex tasks or incorporate additional distractors during evaluation to assess the robustness of the proposed method.\n\n**A**: We have added a distractor object to push-sim, and have introduced three real-world tasks using the UR5 robot arm and reported the goal generation results in Figure 3 in the attachment. \n\n**Q**: The author may also consider exploring the line of work that involves generating demonstrations using video editing techniques for robot learning, as exemplified in \"Human-to-Robot Imitation in the Wild,\" RSS 2022.\n    \n**A**: LfVoid uses image editing methods to generate the goal frame for RL learning, while WHIRL uses video inpainting methods to align human interaction videos and robot interaction videos, and therefore can measure how good the robot performs compared with target human videos. The editing methods create a representation space for this objective function which selects the best robot interactions to perform imitation learning. The comparison with this line of work is very interesting and we will add this line of work to the references. \n    \nWe kindly refer reviewers to the general response section for common questions. If there's anything unclear in the above, we're more than glad to further discuss the details."}]

Author rebuttal:
Dear reviewers, we appreciate all your helpful feedback. In this response, we address the common questions and comments. We welcome further discussion with each reviewer to address any remaining concerns.

We’d like to thank Reviewer jYPU for acknowledging that our method is “An organic integration of several techniques for controlled generation” that “subsequently enhance the performance of example-based RL”, and Reviewer pQmz for acknowledging our work is "A lot more convincing" since the introduction of real world experiments and demonstrates "impressive performance".

Our work (LfVoid) explored the possibility of utilizing the intrinsic knowledge embedded in pertained latent diffusion models (LDM) to provide visual guidance for robotic reinforcement learning. Starting from the current observation and language instructions on the desired goal, LfVoid trains an RL agent to achieve the desired goal with **zero need** for **expert demonstrations** or human-designed **reward functions**.

In the following part, we address some of the common concerns of the reviewers:

**Quantitative evaluation over generated images**

We thank the reviewers for this suggestion. We have reported the LIPIS distance and L2 distance between the edited images and real goal images in Table 1 and Table 2 of the attachment. We also report several more ablation tasks to make the ablation study more concrete. It's clear from the results that LfVoid is not only superior to the other baselines but also better with all its components combined.

**The ability of LfVoid for general-purpose image editing**

We show that the goal-generation method proposed by LfVoid is not limited to robotics environments and can be used to perform general-purpose editing. We include several general editing results in Figure 4 of the attachment. The results show that LfVoid can perform more **localized** editing as well as **preserve the background** according to only text instructions when compared to existing editing methods. We only choose to study LfVoid’s performance in the robotics context because the ability to perform localized editing with high fidelity to the original image is important when using large-scale text2image models to guide robot learning.


**Prompt Tuning for the editing instructions**

Prompt tuning is very lightweight for LfVoid as we simply visualize the attention maps and see which prompt results in an attention map that better associates each token with its region of interest. We only searched about 5 different prompts for LfVoid, and we also performed the **same** amount and type of prompt tuning for InstructPix2Pix and Imagic. We will include this description in the revised paper.


**Reinforcement Learning with generated visual goal baselines**

At the heart of this work, LfVoid is leveraging knowledge in large-scale diffusion models to generate realistic goals to guide robot learning without any in-domain training. Although the mentioned works [1][2] also use imagined visual goals, these visual goals are generated from a random policy and sampled, and thus cannot be explicitly aligned with human intentions. Furthermore, a human-designed goal image is needed at test time. In comparison, LfVoid allows users to directly describe the goal state through language, and can clearly visualize the goal state to avoid ambiguity. 

We thank the reviewers for pointing out this line of work and will add it to the related work section in our final version. We provide an additional comparison with these works and the results can be found in table 5. Since the random exploration policy never reached the desired goal, these works failed to accomplish any of the simulated tasks.


**Additional tasks and results**

Our conducted experiments cover a wide range of manipulation tasks and visual appearances. For example, the Push task and the Wipe task require distinct skills to achieve the goal image. We also include both simulated environments and real robot tasks to demonstrate LfVoid's feasibility. 

During the rebuttal, we have added a distractor object to push-sim and introduced three real-world tasks using the UR5 robot arm and reported the goal generation results in Figure 3 in the attachment. 


We kindly refer reviewers to the attached PDF for additional figures and tables. 

[1] Nair, Ashvin V., et al. “Visual reinforcement learning with imagined goals.” Advances in neural information processing systems 31 (2018).

[2] Nair, Suraj, Silvio Savarese, and Chelsea Finn. “Goal-aware prediction: Learning to model what matters.” International Conference on Machine Learning. PMLR, 2020.

Meta-review: After discussion with the authors, all reviewers recommend acceptance and the AC agrees. Authors are encouraged to include clarifications and details about hyper-parameter / prompt tuning protocols in the evaluation. The comparison to existing RL with visual goals work is useful exposition that should be included as well even though most of these require the goal image to be specified. Likewise, the additional real-world experiments are valuable.
Decision: Accept (poster)