Title: Can Pre-Trained Text-to-Image Models Generate Visual Goals for Reinforcement Learning?
Venue: NeurIPS 2023
Accepted: yes
Keywords: Visual Reinforcement Learning

Large Generative Models

Image Editing

Robotics
Abstract: Pre-trained text-to-image generative models can produce diverse, semantically rich, and realistic images from natural language descriptions. Compared with language, images usually convey information with more details and less ambiguity. In this study, we propose Learning from the Void (LfVoid), a method that leverages the power of pre-trained text-to-image models and advanced image editing techniques to guide robot learning. Given natural language instructions, LfVoid can edit the original observations to obtain goal images, such as "wiping" a stain off a table. Subsequently, LfVoid trains an ensembled goal discriminator on the generated image to provide reward signals for a reinforcement learning agent, guiding it to achieve the goal. The ability of LfVoid to learn with zero in-domain training on expert demonstrations or true goal observations (the void) is attributed to the utilization of knowledge from web-scale generative models. We evaluate LfVoid across three simulated tasks and validate its feasibility in the corresponding real-world scenarios. In addition, we offer insights into the key considerations for the effective integration of visual generative models into robot learning workflows. We posit that our work represents an initial step towards the broader application of pre-trained visual generative models in the robotics field. Our project page: https://lfvoid-rl.github.io/.

Can Pre-Trained Text-to-Image Models Generate Visual Goals for Reinforcement Learning?

Abstract

Pre-trained text-to-image generative models can produce diverse, semantically rich, and realistic images from natural language descriptions. Compared with language, images usually convey information with more details and less ambiguity. In this study, we propose Learning from the Void (LfVoid), a method that leverages the power of pre-trained text-to-image models and advanced image editing techniques to guide robot learning. Given natural language instructions, LfVoid can edit the original observations to obtain goal images, such as 'wiping' a stain off a table. Subsequently, LfVoid trains an ensembled goal discriminator on the generated image to provide reward signals for a reinforcement learning agent, guiding it to achieve the goal. The ability of LfVoid to learn with zero in-domain training on expert demonstrations or true goal observations (the void) is attributed to the utilization of knowledge from web-scale generative models. We evaluate LfVoid across three simulated tasks and validate its feasibility in the corresponding realworld scenarios. In addition, we offer insights into the key considerations for the effective integration of visual generative models into robot learning workflows. We posit that our work represents an initial step towards the broader application of pre-trained visual generative models in the robotics field. Our project page: LfVoid.github.io .

1 Introduction

What is the simplest way to provide guidance or goals to a robot? This question is answered multiple times: a set of expert demonstrations [1, 2, 3, 4], goal images [5, 6, 7], or natural language instructions [8, 9]. However, these answers either require a laborious and sometimes prohibitive effort to collect data, or contain ambiguity across modalities. The desire to alleviate the challenges begs the question: can we generate goal images from natural language instructions directly, without physically achieving the goals for robots?

Large text-to-image generative models [10, 11, 12, 13] have achieved exciting breakthroughs, demonstrating an unprecedented ability to generate plausible images that are semantically aligned with given text prompts. Trained on extensive datasets, these models are thought to possess a basic understanding of the world [14]. Therefore, we aim to harness the power of these large generative models to provide unambiguous visual goals for robotic tasks without any in-domain training.

In the past, a common approach to leverage off-the-shelf large generative models in robot learning involves using these models for data augmentation on expert datasets to improve policy general-

ization [15, 16, 17]. Despite the success, these methods still rely on human demonstrations, not generative models, as the main source of guidance.

DALL-E-Bot [18], an early attempt to utilize web-scale generative models in a zero-shot manner for guiding manipulation tasks, utilizes DALL-E 2 [10] to generate goal images for object rearrangement tasks. However, the generated images are often too diverse and largely disagree with real-world scenarios, requiring segmentation masks for object matching and a rule-based transformation planner to close the visual discrepancy.

The recent uprise of language-based image editing methods [19, 20, 21] provides a new paradigm: editing the generated images to align with language descriptions, such as 'wiping a stain off a table', while keeping the visual appearance of the other objects and the background roughly unchanged. However, the edited images are usually examined by visual appearance rather than embodied tasks.

In this work, we introduce Learning from the Void (LfVoid), a method that uses image editing techniques on pre-trained diffusion models to generate visual goals for Reinforcement Learning (RL). LfVoid contains a unique image editing pipeline capable of performing appearance-based and structure-based editing to generate goal images according to language instructions. With these goal images, LfVoid improves upon example-based RL methods [5, 6] to solve several robot control tasks without the need for any reward function or demonstrations.

We evaluate LfVoid on three simulated environment tasks and the corresponding real-world scenarios. Empirical results show that LfVoid can generate goal images with higher fidelity to both the editing instructions and the source images when compared to existing image editing techniques, leading to better downstream control performance in comparison to language-guided and other image-editingbased methods. We also validate the feasibility of LfVoid in real-world settings: LfVoid can retrieve meaningful reward signals from generated images comparable to those from true goal images. Based on these observations, we discuss key considerations for the current generative model research when aiming for real-world robotic applications.

Our contributions are threefold: First, we propose an effective approach to leverage the knowledge encapsulated in large pre-trained generative models and apply it in a zero-shot manner to guide robot learning tasks. Second, we provide empirical evidence that suggests LfVoid-edited images provide guidance more effectively than other methods including the direct usage of text prompts. Lastly, we identify the existing gap between the capabilities of current text-conditioned image generation models and the demands in robotic applications, thus shedding light on a potential direction for future research in this domain.

2 Related work

Image editing with diffusion models. Recent work in text-to-image diffusion models has demonstrated a strong ability to generate images that are semantically aligned with given text prompts [22, 23, 10, 11, 12, 24, 25]. Based on these generative models, Prompt-to-Prompt proposes to perform text-conditioned editing through the injection of the cross-attention maps. Imagic [20] proposes to perform editing by using interpolation between source and target embeddings to generate images with edited effects. InstructPix2Pix [26] demonstrates another approach for training a diffusion model on the paired source and target images and the corresponding editing instructions. Apart from editing techniques, several methods focusing on controlled editing have been proposed. Textual-Inversion [27] and DreamBooth [28] both aim to learn a special token corresponding to a user-specified object and at inference time can generate images containing that object with a diverse background. Directed Diffusion [21] focuses on object placement generation, which can control the location of a specified object to reside in a given area.

Visual RL with implicit rewards. Standard reinforcement learning methods require an explicit reward function to guide the agent towards desired goal states, yet these reward functions may not always be available or may not capture the essence of the task at hand. One possible way of learning without access to explicit reward functions is using self-supervised learning to learn a representation where the distances can be used as reward signals [29, 30]. Another line of work uses example-based RL methods that aim to utilize the goal states to guide the learning process. VICE, proposed by Fu et al. [5], establishes a general framework for learning from goal observations: it trains a discriminator with the observations in the replay buffer as negative samples and goal images

as positive samples. The positive logits for new observations are used as a reward to guide the agent toward the goal observation. Building upon this, Singh et al. [6] introduce VICE-RAQ, integrating the VICE algorithm with active querying methods through periodically asking a human to provide labels for ambiguous observations. In addition, it proposes a label-smoothing technique for a better reward shaping. Other works [31, 32] also explore ways to improve the reward shaping of VICE as well. In a parallel direction, Eysenbach et al. [7] develop Recursive Classification of Examples (RCE), a method that learns the Q function of actor-critic methods directly from the goal observations.

Large generative models for robot control. Current application of large generative models in robotic tasks mainly focuses on using Large Language Models for planning [8, 9, 33, 34, 35] or learning a language conditioned policy [36, 37], while the application of pre-trained image generative models for robotics tasks are limited. A common way of leveraging large-scale diffusion models for robot learning is by using diffusion models to perform data augmentation on training data, as suggested in CACTI [15], GenAug [16] and ROSIE [17]. Another line of work suggests using diffusion models to generate plans to solve robotics tasks [38, 39, 40]. More recently, video generation and editing techniques have also been used with imitation learning to guide robot policies [41, 42]. More specifically, UniPi[42] trains a text-to-video generation model on web-scale datasets and expert demonstrations to generate image sequences for planning and inverse modeling. The most related work is DALL-E-Bot [18], which uses the DALL-E 2[10] model to generate goal plans and performs object rearrangement according to these goals. However, unlike our work, DALL-E-Bot does not apply image editing techniques for goal generation and requires rule-based matching and predefined pick-and-place actions to bridge the visual gap between generated images and true observations.

3 Method

In this study, we aim to provide zero-shot visual goals to reinforcement learning agents for manipulation tasks using only text prompts. Our approach utilizes the information grounded in large-scale pre-trained visual generative models to create semantically meaningful visual goals. We first generate a synthetic goal image dataset from raw observations using image editing techniques, then employ example-based visual reinforcement learning that is optimized for our task with the generated dataset.

The structure of this section unfolds as follows: Section 3.1 elucidates the image editing techniques and adjustments applied to create goal images from text prompts and initial image observations, as depicted in Figure 1(a). Section 3.2 discusses the execution of example-based visual reinforcement learning using the generated goal images, as outlined in Figure 1(b).

3.1 Visual goal generation

In the first phase of our methodology, we edit and generate goal images from raw observations based on natural language guidance provided by humans. Given a source prompt P , a source image x src , and an editing instruction, LfVoid synthesizes a target image x tgt using a pre-trained Latent Diffusion Model (LDM) [12]. We consider two types of editing instructions: either a target prompt P ∗ describing the appearance changes in the target image, or a bounding-box region B and a set of tokens I corresponding to the object to be relocated when structural changes are needed.

To provide sufficient visual guidance for downstream reinforcement learning tasks, the generated goal images should highlight the visual changes while preserving the irrelevant scene as much as possible, which is a challenging requirement even for state-of-the-art image editing techniques. To this end, we integrate a number of different techniques in the visual goal generation pipeline of LfVoid, consisting of a feature extracting module, an inversion module, and an editing module, as outlined in Figure 1(a). Following conventional notations, we denote x t as the synthesized LDM image at time step t ∈ { T, T -1 , ..., 0 } , where T is the total diffusion time steps. Specifically, x T denotes the Gaussian noise, and x 0 denotes the generated image. The techniques used in each module and their purpose are described as follows.

3.1.1 Feature extracting module

To ensure high fidelity to the source image x src , we learn a unique token sks that encapsulates the visual features of objects within x src as in DreamBooth [28]. This process involves optimizing the diffusion model parameters along with the special token sks , using a set of images that contain the

Editing

Module

Generated

Visual Goals

(a) Goal Image Generation

(b) Example-Based Visual RL

Classifier-Ensemble

Based Reward

...

...

...

...

Generated Visual Goals

"A

sks

white table with m arkings  on it"

Inversion

Module

Prompt-to-Prompt

Editing

P2P-DD

Editing

Feature

Extracting

Module

"A photo of a

sks

cube and a gripper

on a white table"

I nitial

Observations

"markings "          "nothing"

"s k s  c u b e "

Feature

Extracting

Module

Generation

Example-Based

Visual RL

I want the robot to clean the

markings on the table.

Editing prompt: A white

table with nothing on it

AGENT

000

AGENT

+

000

ENVIRONMENT

■

target object. As a result, we are able to derive a specialized model that can accurately retain key details of x src , such as the color and texture of a cube to be positioned, or the shape of a Franka robot arm for manipulation. As later demonstrated in our experiments, this module substantially boosts the resemblance in details of our edited images to the corresponding source images.

3.1.2 Inversion module

After employing the special token sks to capture the essential features of the source scene, we invert the provided source image x src to a diffusion process. A simple inversion technique using the Denoising Diffusion Implicit Model (DDIM) [23] sampling scheme calculates the samples x T , x T -1 , ..., x 0 by reversing the ODE process. However, the inverted image x 0 obtained this way often deviates from x src due to cumulative errors. Null-text inversion [43] aims to mitigate this discrepancy by fine-tuning the "null-text" embedding used in the classifier-free guidance [44] Φ t for each t ∈ { T, T -1 , ..., 1 } to control the generation process. Thus, with the initial noise sample x T obtained through DDIM inversion and the optimized "null-text" embedding Φ T , Φ T -1 , ..., Φ 1 , the diffusion process is able to generate the image x ′ src , an approximation of the source image x src . Based on the diffusion process obtained with the inversion module, LfVoid can perform precise and detailed editing control on the source image, which will be discussed in Section 3.1.3.

3.1.3 Editing module

Depending on the specific requirements, we divide the editing tasks into two distinct categories: appearance-based and structure-based image editing. Appearance-based editing involves maintaining the structural layout of the image while altering certain visual aspects, such as cleaning the surface of a table or lighting up an LED bulb. In contrast, structure-based editing involves changes in the layout of the image, such as relocating objects within the image from one area to another.

Appearance-based editing. In tasks involving appearance-based editing, we employ the Prompt-toPrompt editing control technique. When the Latent Diffusion Model (LDM) generates an image x 0 conditioned on a text prompt P , the information encapsulated in P influences the diffusion process via the cross-attention layers [12, 11, 19]. Prompt-to-Prompt reveals that the spatial configuration

P2P

P2P

...

...

...

...

Directed Diffusion Attention Editing

Conventional Denoising

A

red cube

and a

gripper on a white

table.

口

←xo

XT

XT-1

ΦT

ΦT-1

Xsrc

XT

XT-N

x2

X1

口

x-N

x

x*

of the image x 0 largely depends on the cross-attention maps M t within these cross-attention layers, especially during the initial diffusion steps. Consequently, it suggests replacing the attention maps M ∗ t of the target image diffusion process with the attention maps M t from the source image diffusion process for the first N time steps, beginning at time step t = T :

This method ensures that the structure of the source image encapsulated in the attention maps M t is conserved in the target image in a more controlled manner, which is crucial in downstream RL tasks.

Structure-based editing. A notable limitation of Prompt-to-Prompt editing is the incapacity to spatially move existing objects across the image, i.e., implement structural changes. Therefore, we introduce a novel approach termed P2P-DD for structure-based editing tasks. P2P-DD combines Prompt-to-Prompt control with the idea of Directed Diffusion [21], extending its capability to perform object replacement.

Directed Diffusion [21] illustrates the possibility of achieving object replacement through direct editing of attention maps during the first N time steps of the generation process. The method performs attention strengthening on the cross-attention maps corresponding to the tokens in I (recall that I is the token sets representing the object of interest), through calculating a Gaussian strengthening mask (SM) of the bounding-box region B . It also performs attention annealing to the remaining area ¯ B by applying a constant weakening mask (WM), and the two masks are weighted by a scalar c :

Our proposed P2P-DD method aims to encourage the background and other details of the image generated by Directed Diffusion with a higher resemblance to the source image. P2P-DD first injects cross-attention maps M t into M ∗ t as suggested by Prompt-to-Prompt. This will preserve the structure and background information from the source image, providing a decent starting point for attention map editing. Next, P2P-DD performs pixel value strengthening and weakening on the attention maps M t according to the bounding-box B and the tokens in I , as suggested in the Directed Diffusion, to achieve the desired object placement:

This combined editing control is only applied in the first N diffusion time steps, i.e. { T, ..., T -N +1 } . After the first N time steps, we cease any control on attention maps and allow the target diffusion

Wipe (Sim)

Wipe (Real)

LED (Real)

Push (Real)

process to complete in a conventional denoising manner. The details of P2P-DD are shown in Figure 2. Attention visualization of the generation process can be found in Appendix A.1, and we provide a detailed description of the full algorithm in Appendix A.2.

3.2 Example-based visual reinforcement learning

We modify and extend the approach of VICE [5] to devise our method for example-based visual reinforcement learning. At the core of our process, we employ the image editing methods described in Section 3.1 on observations gathered from randomly initialized environments, producing a dataset of 1024 target images for each task. During training, these target images serve as positive samples, while the images sampled from the agent's replay buffer after a number of random exploration steps are treated as negative samples. We train a discriminator using these instances with the binary cross-entropy loss and use the output of the discriminator for new observations for the positive class as the agent reward.

The discriminator shares the CNN encoder with the reinforcement learning agent and performs classification over the output latent representations. To improve the reward shaping, we utilize the label mixup method [6], which performs a random linear interpolation of the 0-1 labels and their corresponding hidden vectors to obtain continuous labels between 0 and 1. We discover that restricting the negative instances from the recent portion of the replay buffer (the last 5%) promotes the discriminator's ability to discern subtle differences between the target and current observations. Additionally, we find that our method enjoys an ensemble of discriminators for classification results (i.e., RL rewards), which gives the agent more representative rewards.

For the reinforcement learning backbone algorithm, we use DrQ-v2 [45] for visual RL training. Based on Twin Delayed DDPG (TD3) [46], DrQ-v2 enhances image representation by applying random crop augmentation to the input images. Collectively, these refinements constitute our approach to example-based visual reinforcement learning, making the 'learning' from an initial observation and language prompts possible.

4 Experiments

In this section, we present a comprehensive evaluation of LfVoid across both simulated environments and real-world robotic tasks. An illustration of each task can be found in Figure 3.

The environments we use are: 1) LED, where the robot reaches for a switch or touches the LED light directly to turn the light from red to green; 2) Wipe, where the robot needs to wipe out stains from the table; 3) Push, where the robot needs to push a red cube to a goal position indicated by a green dot. The simulated tasks are developed based on the Robosuite benchmark, while we provide corresponding real-world tasks for each environment. A full description of the environments is provided in Appendix B.1.

4.1 Goal generation

In this section, we evaluate the ability of LfV oid and other baseline methods to generate goal images according to two types of editing instructions: appearance-based editing and structure-based editing.

InstructPix2Pix

Input

LfVoid (Ours)

DDIM P2P

Null-text P2P

Wipe

Wipe

(Real)

LED

LED

(Real)

DALL E-2

Baselines and ablations. We select three recent image editing methods as baselines: Imagic [20], InstructPix2Pix [26], and mask-based DALL E-2 editing used in DALL-E-Bot [18](with masks provided by humans). For ablations, DDIM P2P removes both the feature extracting module and the inversion module, and DDIM DD additionally removes P2P in the editing module for structure-based editing. On top of DDIM DD, Null-text DD adds the inversion module back. For both appearancebased and structure-based editing tasks, DreamBooth P2P/P2P-DD ablates the inversion module, and Null-text P2P/P2P-DD ablates the feature extracting module.

Appearance-based goal image editing results. In Figure 4, we present the visual goal generation results of the Wipe and LED tasks, focusing on editing object appearances in both simulated and real-world environments. Imagic struggles to preserve the structure and details of the source image, and InstructPix2Pix frequently fails to perform local color editing, unable to locate the specific area indicated by the editing instruction. DALL E-2 is able to preserve the scenes outside the masks but is not capable of performing the desired edits within the masked area. For ablations, DDIM P2P and DreamBooth P2P-DD can mostly carry out the desired edits but often diverge largely from the source image. This demonstrates the importance of the inversion module in preserving the details and structures of the source image. Null-text P2P performs worse in the Wipe tasks compared to LfVoid, demonstrating the gains provided by the DreamBooth token. Overall, LfVoid demonstrates superior performance in terms of maintaining the appearance of unrelated parts while effectively editing specified regions: through directly applying control and editing on the attention maps, LfV oid is capable of locating the target area and performing the desired edit based on language instructions.

Structure-based goal image editing results. In Figure 5, we report the performance of LfVoid in editing tasks with structural changes (the Push task) in both simulated and real-world settings. For baselines, Imagic distorts the robot arm and table significantly and often introduces multiple objects into the image, rather than moving the existing one. InstructPix2Pix only changes the overall color of the image or replaces the gripper with a geometric body, failing to achieve the required object displacement. DALL E-2 also fails to relocate the object to the desired place. For ablative results, while DDIM DD and Null-text DD can successfully move objects to the lower-left corner, the background and the robot arm nearly disappear. Null-text P2P-DD improves the preservation of the source image's background, but the shape of the moved object is inconsistent with its original form. DreamBooth P2P-DD can preserve the shape of the moved object and relocation is successful, but

InstructPix2Pix

Input

"

...re d  c u b e  at th e

bottom  left

corner...

"

"

Move the red cube

to the bottom  left

corner...

"

"

A photo of a

red cube

and a gripper on a

white table

"

LfVoid (Ours)

Null-text DD

Null-text

P2P-DD

Push

(Real)

Push

DDIM DD

DreamBooth

P2P-DD

DALL-E 2

"

A photo of a

sks  cube

and a gripper

on a white table

"

the background is not consistent with the source image. Our method, when compared to all baselines, can perform the object displacement task successfully while better preserving the visual features of the object and the remaining scene. These experiments reveal the importance of each component in our method. The creative integration of different image editing techniques provides a powerful tool for image editing with structural changes, demonstrating a significant improvement over existing methods.

Quantitative results. In Table 1, we present the quantitative results of goal image generation. We manually sample 500 real goal images per task(only for evaluation purposes) and calculate the pair-wise LIPIS distance between the generated images and the real goal images. The lower distance represents higher image editing quality, and results show that LfVoid outperforms all the baselines and ablations. Additionally, we present an ablation study of the feature extracting module on the Wipe and Push task in Table 2, showing the performance gain brought by the DreamBooth token.

4.2 Example-based visual reinforcement learning

We now dive into the results obtained from applying example-based visual reinforcement learning using the goal images generated by LfVoid. We select three baselines for comparison: CLIP, IP2P,

and Real Goal. CLIP relies only on language guidance and does not require any synthesized goal images. It leverages the Contrastive Language-Image Pre-training score [47] of the task prompt and observations as a reward, and it aims to verify the effectiveness of translating prompts into images against raw language hints. IP2P uses InstructPix2Pix for generating the goal images and then runs the same example-based RL as in LfVoid. It allows us to compare the performance of LfVoid with a straightforward image editing technique. Real Goal is an upper-bound performance of our pipeline, where we manually create ideal goal images, such as erasing the stains directly in the simulator for the Wipe task. This strategy illustrates the potential of our algorithm, indicating the gap between the generated goals and the real ones, and demonstrates how the quality of generated images will influence the performance of downstream RL.

Simulation results. We report both episode reward (Figure 6) and numerical metrics (Table 3) on three simulation tasks. The training details can be found in Appendix B.3. We observe that the CLIP baseline achieves low rewards in all the tasks, showing that directly using language guidance through CLIP embeddings is not sufficient. Moreover, LfVoid consistently outperforms the IP2P baseline and gains comparable performance with the Real Goal upper bound, providing evidence that generated goal images with high fidelity to both the original scene and the editing instructions is crucial for successful deployment of downstream example-based RL.

Real-world results. In the more complex real-world settings, we aim to validate the feasibility of LfVoid through visualization of reward functions. As in our previous experiment, we generate goal images using InstructPix2Pix (IP2P) and LfVoid, and also manually collect a real goal dataset as an upper-bound (Real Goal). We then train discriminators on these datasets separately and use the outputs of the discriminators as rewards. In addition, we compare with language-based guidance (CLIP), which uses the distance of the CLIP embeddings between the task prompt and the observation images as rewards. We manually record five successful trajectories for each task and use these reward functions to obtain the reward curves. As illustrated in Figure 7, LfV oid can provide near monotonic dense reward signals comparable to those from Real Goal. On the contrary, the curves obtained by IP2P and CLIP fail to accurately track the progress of each task. This demonstrates that our method provides more instructive learning signals for real-world robotic tasks than either IP2P or CLIP.

Comparison to RL with imagined goals. In a parallel direction, RL with imagined goals can also solve robot manipulation tasks without the need for human-specified reward functions and expert demonstations [29, 30]. These methods first leverage self-supervised learning to learn a representation











×





























(a) Wipe Episode Reward











×























(b) LED Episode Reward











×





























(c) Push Episode Reward

0

2

4

6

8

10

12

14

16

18

Trajectory Step

1

0

1

2

3

4

5

Reward Score

CLIP

Real Goal

IP2P

LfVoid

(a) Wipe Reward Curve

0

2

4

6

8

10

12

14

Trajectory Step

0

2

4

Reward Score

CLIP

Real Goal

IP2P

LfVoid

(b) LED Reward Curve

0

2

4

6

8

10

12

14

16

Trajectory Step

0

2

4

6

8

10

Reward Score

CLIP

Real Goal

IP2P

LfVoid

(c) Push Reward Curve

space of observations, and then sample imagined goals to train a goal-conditioned policy. However, unlike LfVoid, a real goal image is still required at test time to run the goal-conditioned policy. In Table 4, we compare the performance of LfVoid with two algorithms: Visual RL with Imagined Goals (VIG) [29] and Goal-Aware Prediction (GAP) [30], on the three simulation tasks. We observe that only LfVoid can solve the tasks with positive rewards, the other two methods fail in all tasks. Because both these methods use random policy for exploration, they may never reach the userspecified goal states during training. Therefore, they lack the knowledge required to fulfill the goals. This further demonstrates the significance of leveraging web-scale text-to-image models to provide informative visual guidance during training. Please refer to Appendix C.2 for more analysis.

5 Discussion

In this work, we present LfVoid, an effective approach for leveraging the knowledge of large-scale text-to-image models and enabling zero-shot reinforcement learning from text prompts and raw observation images. We have identified and tackled numerous challenges that arise when applying state-of-the-art image editing technologies to example-based RL methods. Our work highlights the potential for the adaptation and application of image-generation techniques in the realm of robotics. Our findings not only enhance the understanding of image editing for robotic applications but also provide a clear direction for the image generation community to address real-world challenges.

Although our proposed method has shown promising results, we acknowledge that it is not without limitations. While LfVoid has succeeded in generating goal images to train example-based RL policies, a certain level of prompt tuning is needed in order to achieve optimal editing performance (see Appendix B.2), and we refer the readers to Appendix D for an analysis of failure cases.

We would also like to highlight that there exists a gap between the ability of text-to-image generation models and the need for robot learning. For example, large diffusion models exhibit poor understandings of the spatial relationships between objects [48]; therefore, both the generative models and the editing methods struggle to handle object displacement solely through language prompts. The Directed Diffusion technique, while being able to perform movements of objects, requires a userdefined bounding box to achieve precise control. Furthermore, large generative models sometimes struggle to generate images that are considered valid under physics laws. When asked to pick up a bottle with a robot arm, the model simply stretches the shape of the arm to reach the bottle rather than changing the joint position. Lastly, AI-generated images inevitably introduce alterations to the details of objects. The robustness of current visual reinforcement learning algorithms to such changes remains an open question.

Acknowledgment

HX is supported by National Key R&D Program of China (2022ZD0161700).

References

Georgios Papagiannis and Yunpeng Li. Imitation learning with sinkhorn distances. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2022, Grenoble, France, September 19-23, 2022, Proceedings, Part IV , pages 116-131. Springer, 2023.

Robert Dadashi, Léonard Hussenot, Matthieu Geist, and Olivier Pietquin. Primal wasserstein imitation learning. arXiv preprint arXiv:2006.04678 , 2020.

Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint arXiv:1809.02925 , 2018.

Siddhant Haldar, Vaibhav Mathur, Denis Yarats, and Lerrel Pinto. Watch and match: Supercharging imitation with regularized optimal transport. In Conference on Robot Learning , pages 32-43. PMLR, 2023.

Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control with events: A general framework for data-driven reward definition. Advances in neural information processing systems , 31, 2018.

Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering. arXiv preprint arXiv:1904.07854 , 2019.

Ben Eysenbach, Sergey Levine, and Russ R Salakhutdinov. Replacing rewards with examples: Example-based policy search via recursive classification. Advances in Neural Information Processing Systems , 34:11541-11552, 2021.

Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691 , 2022.

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817 , 2022.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems , 35:36479-36494, 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021.

Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Benton C. Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. ArXiv , abs/2206.10789, 2022.

Stephen Adams, Tyler Cody, and Peter A Beling. A survey of inverse reinforcement learning. Artificial Intelligence Review , 55(6):4307-4346, 2022.

Zhao Mandi, Homanga Bharadhwaj, Vincent Moens, Shuran Song, Aravind Rajeswaran, and Vikash Kumar. Cacti: A framework for scalable multi-task multi-scene visual imitation learning. arXiv preprint arXiv:2212.05711 , 2022.

Zoey Chen, Sho Kiami, Abhishek Gupta, and Vikash Kumar. Genaug: Retargeting behaviors to unseen situations via generative augmentation. arXiv preprint arXiv:2302.06671 , 2023.

Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tompson, Anthony Brohan, Su Wang, Jaspiar Singh, Clayton Tan, Jodilyn Peralta, Brian Ichter, et al. Scaling robot learning with semantically imagined experience. arXiv preprint arXiv:2302.11550 , 2023.

Ivan Kapelyukh, Vitalis Vosylius, and Edward Johns. Dall-e-bot: Introducing web-scale diffusion models to robotics. arXiv preprint arXiv:2210.02438 , 2022.

Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 , 2022.

Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. arXiv preprint arXiv:2210.09276 , 2022.

Wan-Duo Kurt Ma, JP Lewis, W Bastiaan Kleijn, and Thomas Leung. Directed diffusion: Direct control of object placement through attention guidance. arXiv preprint arXiv:2302.13153 , 2023.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021.

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 , 2021.

Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800 , 2022.

Parsa Mahmoudieh, Deepak Pathak, and Trevor Darrell. Zero-shot reward specification via grounded natural language. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 14743-14752. PMLR, 17-23 Jul 2022.

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208.12242 , 2022.

Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. CoRR , abs/1807.04742, 2018.

Suraj Nair, Silvio Savarese, and Chelsea Finn. Goal-aware prediction: Learning to model what matters, 2020.

Kevin Li, Abhishek Gupta, Ashwin Reddy, Vitchyr H Pong, Aurick Zhou, Justin Yu, and Sergey Levine. Mural: Meta-learning uncertainty-aware rewards for outcome-driven reinforcement learning. In International conference on machine learning , pages 6346-6356. PMLR, 2021.

Daesol Cho, Seungjae Lee, and H Jin Kim. Outcome-directed reinforcement learning by uncertainty & temporal distance-aware curriculum goal generation. arXiv preprint arXiv:2301.11741 , 2023.

Maria Attarian, Advaya Gupta, Ziyi Zhou, Wei Yu, Igor Gilitschenski, and Animesh Garg. See, plan, predict: Language-guided cognitive planning with video prediction. arXiv preprint arXiv:2210.03825 , 2022.

Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608 , 2022.

Yan Ding, Xiaohan Zhang, Chris Paxton, and Shiqi Zhang. Task and motion planning with large language models for object rearrangement. arXiv preprint arXiv:2303.06247 , 2023.

Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning , pages 894-906. PMLR, 2022.

Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:2210.03094 , 2022.

Weiyu Liu, Yilun Du, Tucker Hermans, Sonia Chernova, and Chris Paxton. Structdiffusion: Language-guided creation of physically-valid structures using unseen objects, 2023.

Nikolaos Gkanatsios, Ayush Jain, Zhou Xian, Yunchu Zhang, Christopher Atkeson, and Katerina Fragkiadaki. Energy-based models as zero-shot planners for compositional scene rearrangement. arXiv preprint arXiv:2304.14391 , 2023.

Utkarsh A Mishra and Yongxin Chen. Reorientdiff: Diffusion model based reorientation for object manipulation. arXiv preprint arXiv:2303.12700 , 2023.

Shikhar Bahl, Abhinav Gupta, and Deepak Pathak. Human-to-robot imitation in the wild, 2022.

Yilun Dai, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. arXiv preprint arXiv:2302.00111 , 2023.

Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.09794 , 2022.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations , 2021.

Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. ArXiv , abs/1802.09477, 2018.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Colin Conwell and Tomer Ullman. Testing relational understanding in text-guided image generation. arXiv preprint arXiv:2208.00005 , 2022.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 , 2017.

Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645 , 2021.

Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009.12293 , 2020.

Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, and Wojciech Zaremba. Multi-goal reinforcement learning: Challenging robotics environments and request for research, 2018.

Mark E. Glickman. The glicko-2 system. http://www.glicko.net/glicko/glicko2.pdf , 2013.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe objective of this paper is to harness the capabilities of pre-trained text-to-image models and image editing techniques in order to facilitate robot learning. This is achieved by leveraging these tools to modify the current scene towards the intended image objective.\n\nSTRENGTHS:\nThe paper presents a novel approach that harnesses the knowledge embedded in large pre-trained generative models to facilitate zero-shot guidance for robot learning tasks. Additionally, the paper asserts that its edited image functionality outperforms existing methods. Lastly, the paper identifies a gap in current text-conditioned image generation models.\n\nWEAKNESSES:\n1. Figure 1 can be enhanced to effectively communicate the problem the author intends to address, while also succinctly highlighting their contribution and providing an overview of their proposed method.\n2. In recent years, significant advancements in diffusion models have led to the emergence of notable works such as CACTI, GenAug, and ROSIE. These works have demonstrated considerable improvements in generative modeling and learning. However, none of these works were used as baselines in the paper, including DALL-E-Bot, which bears the closest resemblance to this work. While acknowledging technical differences, it would be valuable to compare and contrast the proposed method with these related works.\n3. I appreciate the author's effort to compare various methods of image editing with their proposed approach. However, many of these methods are pretrained on different datasets, necessitating the evaluation of their performance on a comprehensive image editing dataset for a more equitable and meaningful comparison.\n4. The author's inclusion of both simulation and real-world demonstrations is commendable. However, to enhance the persuasiveness of the findings, it would be beneficial to introduce slightly more complex tasks or incorporate additional distractors during evaluation to assess the robustness of the proposed method.\n5. The author may also consider exploring the line of work that involves generating demonstrations using video editing techniques for robot learning, as exemplified in \"Human-to-Robot Imitation in the Wild,\" RSS 2022.\n\nQUESTIONS:\nI hope the authors can address all the questions and concerns i have in the weakness section.\n\nLIMITATIONS:\nYes, the authors have addressed all the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPost rebuttal updating score to 5\n\nThis work proposes to use generated visual goals for RL. A diffusion model based approach is used to edit visual observations based on text prompts. The proposed image editing approach is shown to be better than prior text-based image editing approaches based on a human evaluation. Visual goals predicted using the proposed method shows improved RL performance compared to baselines that use other image editing mechanisms.\n\nSTRENGTHS:\n* Leveraging pre-trained models for image editing is an interesting idea\n* Proposed approach seems to work better than some prior editing approaches\n* Some indication that the generated visual goals help learning\n\nWEAKNESSES:\nOriginality: Imagining visual goals for RL has been explored in prior work (e.g., [1] https://arxiv.org/pdf/1807.04742.pdf), which limits the originality of this work. The use of diffusion models could be new. \n\nSignificance: \n* Experimental results do not demonstrate the need for visual goals. For instance, there are no baselines in the RL experiments which are not based on imagined goals. \n* Very few tasks are tested\n* Baseilnes are inadequate\n* Approach is too specific to the tasks considered and generality of approach is questionable \n\nPresentation/Clarity\n* Need better clarity on motivation, background on methods, problem formulation and approach overview\n* There needs to be a clear motivation/overview section before section 3. Technical details lack context and are difficult to follow. For instance, section 3 talks about latent diffusion models without providing any background/motivation. There needs to be a technical overview section on diffusion models.\n* Method description is vague and mathematical details are missing. \n  - 3.1.1 Description of the optimization approach is vague.\n  - 3.1.2 What is inversion and why is it necessary? \n  - 3.1.3 Assumes prior knowledge and familiarity with diffusion models. What is x_0? What does it mean to replace attention maps? \n\nExperiments\n* Is the comparison on image editing fair? The proposed method is fairly specific to the tasks considered while it is compared against general purpose image generation/editing techniques such as pix2pix. \n* Inadequate baselines for the RL experiment: There seem to be no baselines that are not based on imagined goals/image editing.\n\nQUESTIONS:\n* Many details about the setting are unclear. Need better clarity on the following. \n  - Where do the text prompts/instructions for editing come from? \n  - What are the inputs and the outputs of the components of the approach?\n  - How does learning work? \n* Is it possible to use any quantitative metrics for image editing experiments (e.g., based on CLIP encoders)?\n* For the manually defined visual goals how many images were manually created? How many task instances are there altogether? \n* line 297 mentions ‘a certain level of prompt tuning is needed’, but such details are missing from the main text.\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper demonstrates a novel way to utilize existing large-scale text-to-image models for robot learning. Specifically, they modify existing pre-trained text-to-image models to produce visual goals for example-based reinforcement learning, before learning policies from the generated images. Experimentation in both simulation and the real world shows that the proposed modifications to pre-trained text-to-image models are able to generate higher-fidelity images than the baselines, and that this also leads to gains in downstream performance.\n\nSTRENGTHS:\n- Well-written. The paper is well-written, and the figures aid in the understanding of the methodology.\n- Novelty. The proposed approach seems to be a novel one which builds on ideas from the literature.\n- Real world experiments. The inclusion of real world experiments for both the image goal generation and the RL training make the strength of the proposed approach a lot more convincing than otherwise.\n- Strong results. The proposed approach outperforms baselines from prior work as well as the ablations. The qualitative results provided in Figures 4 and 5 display impressive performance of the model.\n\nWEAKNESSES:\n- Evaluation. While both qualitative and quantitative metrics for image goal generation are included, the worry is that the qualitative examples may be cherry-picked and that the user-study done for a quantitative assessment may not be extensive enough. Would it be possible to use the standard metrics which conditional image generation papers utilize to quantitatively evaluate the proposed approach? For instance, once the ideal goal image has been generated, could the output of the proposed method be compared to using either MSE or LPIPS [1]? Furthermore, would it be possible to provide more qualitative examples? There are about three more examples per task in the appendix, but these all seem to be very similar to one another and limited in diversity. Particularly, more visualizations of the real world images would be appealing as this setting is much harder than the simulated images.\n\n- Prompt tuning. Ideally, at test-time, you are just given the current image, and a text-based goal prompt, nothing else. Line 296 says that \"a certain level of prompt tuning is needed in order to achieve optimal editing performance.\" How much prompt tuning was done for the method? Was the same done for the baselines?\n\n[1] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conf. Comput. Vis. Pattern Recog., 2018.\n\nQUESTIONS:\n- Qualitative Results. While the language instructions for Imagic and the proposed method are the same, ensuring a fair comparison, why are they different for InstructPix2Pix? Furthermore, why is the ablation only done for Figure 5, and not for Figure 4?\n\n- RL Experiments. Were all hyper-parameters in the RL training the same across methods? In Figure 7, the number of trajectory steps seems to vary across tasks -- why is this the case? Are the trends different if more trajectory steps were allowed?\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, authors propose a new method called LfVoid to tackle RL tasks. LfVoid can generate consistent visual goal frames through using pretrained LDM and subsequently train a discriminator to output rewards for downstream RL task. In order to improve editing consistency, DreamBooth, null-text inversion, P2P and Directed diffusion are incorporated into the pipeline.\n\nSTRENGTHS:\n1. This paper is well written, clear and easy to understand.\n2. An organic integration of several techiniques for controlled generation  \n3. Compared with existing editing methods, only the proposed LfVoid correctly modifies the desired objects and makes no obvious changes to other irrelevant things.\n4. The improved generative quality subsequently enhance the performance of example-based RL.\n\nWEAKNESSES:\n1. LDM is kind of overkilling in this benchmark.\n2. Lack comparison with other RL methods leveraging goal frame generation\n\nQUESTIONS:\n1. The evaluation benchmark is a little simple which more or less wastes the strong expressive power of pretrained LDM. What is the advantage of using LDM in your application compared with training a latent VAE to generate the goal frame except for less training frames?\n2. As mentioned in summary, 4 techniques, including DreamBooth, null-text inversion, P2P and Directed diffusion, are incorporated but the ablation study only compares null-text P2P-DD and null-text P2P. In my opinion, the effects of DreamBooth and null-text inversion are overlapped. Can you provide any experiment to validate the improvement brought by DreamBooth token? Also, I understand the difficulty of ablation study in this work due to the lack of effective quantitative comparison. Is it possible to design some new metrics that only focus on the region of interest since the bounding boxes are provided?\n3. Comparison with existing RL methods leveraging goal frame generation, for example, Goal-Aware Prediction[1], was not provided. Can you compare LfVoid  with these baselines or explain why this is not meaningful? \n\n[1]. Nair, S., Savarese, S. &amp; Finn, C.. (2020). Goal-Aware Prediction: Learning to Model What Matters. <i>Proceedings of the 37th International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 119:7207-7219 Available from https://proceedings.mlr.press/v119/nair20a.html.\n\nLIMITATIONS:\nThis paper solved the image editing problem required for goal frame generation. A solid contribution was made. I would increase my rating if my concerns are properly addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}