SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Appearance-based goal image editing results. In Figure 4, we present the visual goal generation results of the Wipe and LED tasks, focusing on editing object appearances in both simulated and real-world environments. Imagic struggles to preserve the structure and details of the source image, and InstructPix2Pix frequently fails to perform local color editing, unable to locate the specific area indicated by the editing instruction. DALL E-2 is able to preserve the scenes outside the masks but is not capable of performing the desired edits within the masked area. For ablations, DDIM P2P and DreamBooth P2P-DD can mostly carry out the desired edits but often diverge largely from the source image. This demonstrates the importance of the inversion module in preserving the details and structures of the source image. Null-text P2P performs worse in the Wipe tasks compared to LfVoid, demonstrating the gains provided by the DreamBooth token. Overall, LfVoid demonstrates superior performance in terms of maintaining the appearance of unrelated parts while effectively editing specified regions: through directly applying control and editing on the attention maps, LfVoid is capable of locating the target area and performing the desired edit based on language instructions.
Structure-based goal image editing results. In Figure 5, we report the performance of LfVoid in editing tasks with structural changes (the Push task) in both simulated and real-world settings. For baselines, Imagic distorts the robot arm and table significantly and often introduces multiple objects into the image, rather than moving the existing one. InstructPix2Pix only changes the overall color of the image or replaces the gripper with a geometric body, failing to achieve the required object displacement. DALL E-2 also fails to relocate the object to the desired place. For ablative results, while DDIM DD and Null-text DD can successfully move objects to the lower-left corner, the background and the robot arm nearly disappear. Null-text P2P-DD improves the preservation of the source image's background, but the shape of the moved object is inconsistent with its original form. DreamBooth P2P-DD can preserve the shape of the moved object and relocation is successful, but
Figure 5: Structure-based goal image editing results. We compare LfVoid against Imagic, InstructPix2Pix, and DALL E-2, as well as four ablations of LfVoid. Note that InstructPix2Pix has different prompts because it requires an editing instruction rather than a description of the image. LfVoid can successfully perform object displacement and preserve the background and details of the source image, while other methods fail to do so. See Appendix C for more examples.
the background is not consistent with the source image. Our method, when compared to all baselines, can perform the object displacement task successfully while better preserving the visual features of the object and the remaining scene. These experiments reveal the importance of each component in our method. The creative integration of different image editing techniques provides a powerful tool for image editing with structural changes, demonstrating a significant improvement over existing methods.
Quantitative results. In Table 1, we present the quantitative results of goal image generation. We manually sample 500 real goal images per task(only for evaluation purposes) and calculate the pair-wise LIPIS distance between the generated images and the real goal images. The lower distance represents higher image editing quality, and results show that LfVoid outperforms all the baselines and ablations. Additionally, we present an ablation study of the feature extracting module on the Wipe and Push task in Table 2, showing the performance gain brought by the DreamBooth token.
Wipe(Sim)	LED(Sim)	Push(Sim)	Wipe(Real)	LED(Real)	Push(Real)
InstructPix2Pix	0.23±0.11	0.41±0.06	0.34±0.17	0.18±0.04	0.45±0.06	0.53±0.08
Imagic	0.23±0.05	0.42±0.08	0.43±0.11	0.34±0.13	0.35±0.16	0.49±0.11
DALL-E 2	0.10±0.04	0.25±0.07	0.20±0.06	0.39±0.08	0.20±0.07	0.38±0.03
DreamBooth+Editing	0.53±0.05	0.62±0.05	0.42±0.15	0.58±0.06	0.43±0.08	0.36±0.06
DDIM+DD/P2P	0.26±0.07	0.37±0.06	0.43±0.04	0.34±0.04	0.40±0.03	0.50±0.03
Null-text+DD	N/A	N/A	0.36±0.05	N/A	N/A	0.51±0.06
LfVoid	0.08±0.04	0.17±0.06	0.14±0.07	0.12±0.05	0.19±0.05	0.32±0.03
Table 1: Quantitative results on goal image generation. We calculate the pair-wise LIPIS distance between generated images and the manually created real goal images for evaluation purposes.
Wipe(Sim)	Push(Sim)	Wipe(Real)	Push(Real)
LfVoid w/o DreamBooth	0.10±0.04	0.24±0.06	0.18±0.06	0.39±0.06
LfVoid	0.08±0.04	0.14±0.07	0.12±0.05	0.32±0.03
Table 2: Ablation study on DreamBooth. We evaluate the improvement brought by the DreamBooth token through pair-wise LIPIIS distance between generated images and manually created real goal images for evaluation purposes.
4.2 Example-based visual reinforcement learning
We now dive into the results obtained from applying example-based visual reinforcement learning using the goal images generated by LfVoid. We select three baselines for comparison: CLIP, IP2P,
and Real Goal. CLIP relies only on language guidance and does not require any synthesized goal images. It leverages the Contrastive Language-Image Pre-training score [47] of the task prompt and observations as a reward, and it aims to verify the effectiveness of translating prompts into images against raw language hints. IP2P uses InstructPix2Pix for generating the goal images and then runs the same example-based RL as in LfVoid. It allows us to compare the performance of LfVoid with a straightforward image editing technique. Real Goal is an upper-bound performance of our pipeline, where we manually create ideal goal images, such as erasing the stains directly in the simulator for the Wipe task. This strategy illustrates the potential of our algorithm, indicating the gap between the generated goals and the real ones, and demonstrates how the quality of generated images will influence the performance of downstream RL.
Simulation results. We report both episode reward (Figure 6) and numerical metrics (Table 3) on three simulation tasks. The training details can be found in Appendix B.3. We observe that the CLIP baseline achieves low rewards in all the tasks, showing that directly using language guidance through CLIP embeddings is not sufficient. Moreover, LfVoid consistently outperforms the IP2P baseline and gains comparable performance with the Real Goal upper bound, providing evidence that generated goal images with high fidelity to both the original scene and the editing instructions is crucial for successful deployment of downstream example-based RL.
Real-world results. In the more complex real-world settings, we aim to validate the feasibility of LfVoid through visualization of reward functions. As in our previous experiment, we generate goal images using InstructPix2Pix (IP2P) and LfVoid, and also manually collect a real goal dataset as an upper-bound (Real Goal). We then train discriminators on these datasets separately and use the outputs of the discriminators as rewards. In addition, we compare with language-based guidance (CLIP), which uses the distance of the CLIP embeddings between the task prompt and the observation images as rewards. We manually record five successful trajectories for each task and use these reward functions to obtain the reward curves. As illustrated in Figure 7, LfVoid can provide near monotonic dense reward signals comparable to those from Real Goal. On the contrary, the curves obtained by IP2P and CLIP fail to accurately track the progress of each task. This demonstrates that our method provides more instructive learning signals for real-world robotic tasks than either IP2P or CLIP.
CLIP	InstructPix2Pix	Real Goal (Oracle)	Ours
Wipe (Cleaned Stains / Patch )	0.0±0.0	1.7±1.9	22.0±5.8	21.3±5.8
LED (Success Rate / %)	10.0±20.0	0.0±0.0	93.3±11.5	75.0±50.0
Push (Success Rate / %)	12.5±19.1	3.3±3.5	30.0±13.7	27.9±12.7
Table 3: Numerical metrics of simulation tasks. We report the success rate for LED and Push, and the number of stain patches cleaned for Wipe. Please refer to Appendix B.1 for details.
Comparison to RL with imagined goals. In a parallel direction, RL with imagined goals can also solve robot manipulation tasks without the need for human-specified reward functions and expert demonstrations [29, 30]. These methods first leverage self-supervised learning to learn a representation
(a) Wipe Episode Reward
(b) LED Episode Reward
Figure 6: Episode reward of simulation tasks. The results of the CLIP baseline (CLIP), Instruct-Pix2Pix baseline (IP2P), using real goal image as upper bound (Real Goal), and LfVoid (Ours).
(c) Push Episode Reward
(a) Wipe Reward Curve
(b) LED Reward Curve
Figure 7: Reward curve on successful trajectories for real-robot tasks. Visualization of the reward function of LfVoid (Ours) compared with baselines. See Appendix B.4 for implementation details.
(c) Push Reward Curve
space of observations, and then sample imagined goals to train a goal-conditioned policy. However, unlike LfVoid, a real goal image is still required at test time to run the goal-conditioned policy. In Table 4, we compare the performance of LfVoid with two algorithms: Visual RL with Imagined Goals (VIG) [29] and Goal-Aware Prediction (GAP) [30], on the three simulation tasks. We observe that only LfVoid can solve the tasks with positive rewards, the other two methods fail in all tasks. Because both these methods use random policy for exploration, they may never reach the user-specified goal states during training. Therefore, they lack the knowledge required to fulfill the goals. This further demonstrates the significance of leveraging web-scale text-to-image models to provide informative visual guidance during training. Please refer to Appendix C.2 for more analysis.
Push(Sim)	Wipe(Sim)	LED(Sim)
Visual rl with Imagined Goals (VIG)	-0.39±1.79	-6.03±2.02	-1.77±0.55
Goal-Aware Prediction (GAP)	-0.07±0.91	-5.84±1.00	-0.39±0.11
Learning from the Void (LfVoid)	0.35±0.51	12.86±4.31	4.47±2.10
Table 4: Comparison with RL using imagined goals. We report the episode reward during test time and compare LfVoid with existing methods of visual RL using imagined goals.
In this work, we present LfVoid, an effective approach for leveraging the knowledge of large-scale text-to-image models and enabling zero-shot reinforcement learning from text prompts and raw observation images. We have identified and tackled numerous challenges that arise when applying state-of-the-art image editing technologies to example-based RL methods. Our work highlights the potential for the adaptation and application of image-generation techniques in the realm of robotics. Our findings not only enhance the understanding of image editing for robotic applications but also provide a clear direction for the image generation community to address real-world challenges.
Although our proposed method has shown promising results, we acknowledge that it is not without limitations. While LfVoid has succeeded in generating goal images to train example-based RL policies, a certain level of prompt tuning is needed in order to achieve optimal editing performance (see Appendix B.2), and we refer the readers to Appendix D for an analysis of failure cases.
We would also like to highlight that there exists a gap between the ability of text-to-image generation models and the need for robot learning. For example, large diffusion models exhibit poor understandings of the spatial relationships between objects [48]; therefore, both the generative models and the editing methods struggle to handle object displacement solely through language prompts. The Directed Diffusion technique, while being able to perform movements of objects, requires a user-defined bounding box to achieve precise control. Furthermore, large generative models sometimes struggle to generate images that are considered valid under physics laws. When asked to pick up a bottle with a robot arm, the model simply stretches the shape of the arm to reach the bottle rather than changing the joint position. Lastly, AI-generated images inevitably introduce alterations to the details of objects. The robustness of current visual reinforcement learning algorithms to such changes remains an open question.
HX is supported by National Key R&D Program of China (2022ZD0161700).
[52] Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, and Wojciech Zaremba. Multi-goal reinforcement learning: Challenging robotics environments and request for research, 2018.
[53] Mark E. Glickman. The glicko-2 system. http://www.glicko.net/glicko/glicko2.pdf, 2013.