Embodied Scene Cloning: Solving Generalization in Embodied AI via Visual-Prompt Image Editing

Authors: Binyuan Huang, Zihan Lan, Yucheng Zhao, Yaosi Hu, Weixin Mao, Tiancai Wang, Chang Wen Chen, Haoqiang Fan, Zhenzhong Chen
Primary area: generative models
Keywords: Embodied Intelligence, Data Augmentation

Abstract:
Recent advancements in robotic learning have enabled robots to perform a wide range of tasks. However, generalizing policies from training environments to deployment environments remains a major challenge, and improving these policies by collecting and annotating demonstrations in target environments is both costly and time-consuming. To address this issue, we propose Embodied Scene Cloning, a novel visual-prompt-based framework that generates visual-aligned trajectories from existing data by leveraging visual cues from the specific deployment environment. This approach minimizes the impact of environmental discrepancies on policy performance. Unlike traditional embodied augmentation methods that rely on text prompts, we propose to "clone" source demonstrations into the target environment and edit it with visual prompt to effectively improve the generalization ability on specific embodied scene. Experimental results demonstrate that samples generated by Embodied Scene Cloning significantly enhance the generalization ability of policies in the target deployment environments, representing a meaningful advancement in embodied data augmentation.

Decision: reject
Reviewer scores: 5, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730314666988

thread_type:
official_review

<thread_object>
creator: reviewer_xu12

<content>
summary:
The paper introduces a visual-prompt-based framework that enhances policy generalization in robotics by prompting with visual cues from deployment environments. It clones scenes with visual alignment, improving performance in novel environments.

strengths:
Originality: applying diffusion based image editing to change foreground background in robotics
Quality: good ablation study on elements of environment
Clarity: explaining context methods
Significance: improve policy domain transfer

weaknesses:
The main weakness is the tested environment is relatively simple. Only changing the color/texture of one object in foreground / background. To demonstrate real potential, needs more extensive experiments.

questions:
1. The papers refers "traditional augmentation methods" as text based. Meanwhile, there are other traditional augmentation such as color jitter, random crop etc. Suggest directly refer them as text based. In fact, it would be nice see if simple color jitter improve the performance.
2. In line 323, it says including baselines of RoboFlamingGo(Li et al., 2023) and 3D Diffusion Actor. Meanwhile, they are policy learning method not augmentation method. Therefore, it's confusing to call them baselines (which should be GreenAug for example).
3. More papers in image translation for robotics should be discussed in related works. For example: 
Li, Yiwei et al. “ALDM-Grasping: Diffusion-aided Zero-Shot Sim-to-Real Transfer for Robot Grasping.” ArXiv abs/2403.11459 (2024): n. pag.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730432636028

thread_type:
official_review

<thread_object>
creator: reviewer_i9yn

<content>
summary:
This paper proposes a novel data augmentation method called Embodied Scene Cloning for addressing the generalization challenge in robotic manipulation. The proposed method incorporates Conditional Feature Extraction, Progressive Masked Fusion, and Visual-Prompt Guided Image Editing to fulfill visual-instructed source demonstration cloning. As a result, the proposed method allows precise visual control over both the generated objects and the background environments compared to text-prompt-based data augmentation methods. Experimental results in both simulation and real-world environments demonstrate the superiority of the proposed approach over the state-of-the-art generative data augmentation methods, especially GreenAug.

strengths:
1 The proposed approach provides a potential solution to seamlessly integrate of new scene elements while maintaining the consistency and fidelity of the background areas. This will inspire the community to design visually-controlled data augmentation strategies in the future.

2 The effectiveness of the proposed approach is demonstrated in both simulation and real-world environments.

weaknesses:
1 My main concern is the actual generalization ability of the proposed method in real-world applications and the fairness of comparing the proposed method with existing methods. Generally, it is defaulted that the agent in the trained environment cannot access any information in unseen environments. And augmentation for a specific new embodied scene may not be a scalable way of generalization since the unseen scenarios are infinite. Moreover, the proposed policy seems to be relatively effective for texture or color generalization for objects, but its generalization to more common situations such as objects of different types, stacks, and occlusions is not verified. 

2 Additionally, the proposed method mainly focuses on the robotic manipulation task while the title claims a large topic by “Solving Generalization in Embodied AI”. I suggest the author give a more precise claim in both the title and the main text.

3 There lack the ablation studies to analyze the effect of specific design for different method components. Moreover, the failure cases should be given for a more complete analysis of the proposed approach.

4 Can the proposed method support sim2real augmentation? The additional experiments on this are appreciated. It will further demonstrate the generalization ability and practicality of the proposed approach. 

5 Visual prompting is a widely used strategy in both VLMs and text-to-image generation approaches. The authors should add an independent subsection to analyze existing methods and discuss the comparison in the Related Work section.

questions:
See details in the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693458570

thread_type:
official_review

<thread_object>
creator: reviewer_55tj

<content>
summary:
This paper aims to enhance the generalization of robotic policies to new environments by visually "cloning" source demonstrations using scene-specific visual prompts. 
To tackle the generalization challenge, it introduces a new framework, Embodied Scene Cloning that utilizes image-editing techniques to generate environment-aligned trajectories. 
By maintaining coherence between edited and non-edited regions, the model achieves smooth, consistent visual elements that better represent target deployment scenarios, both forground and background. 
Empirical results show significant performance improvements over existing methods, such as GreenAug, in simulated environments, indicating the approach's effectiveness in augmenting training data for policy generalization. 
The authors also conduct real-world experiments to showcase that the framework has the potential for real-world application.

strengths:
1. The experimental results on CALVIN Benchmark are promising. The results of Embodied Scene Cloning are significantly higher than the baseline, demonstrating that this method is effective and can effectively improve the model’s generalization ability to unseen foreground and background.
2. The goal of this paper is to address the generalization problem on test tasks, which is crucial for embodied AI. The authors provide some insights on this issue.

weaknesses:
1. Technique Contribution is limited. This work is a combination and application of Diffusion Model, DDIM, SAM2, and Depth-Anything V2, without a novel techniqle contribution.
2. The real-world experiments are insufficient. The authors conducted extensive experiments on the CALVIN benchmark, validating the method’s effectiveness in this setting. Although they included real-world experiments, the results are rather limited. The experiments focus solely on the foreground scenario with a single setup, addressing only the object-grasping task. The authors should consider expanding the real-world experiments to further verify the generalization capability of the proposed framework.
3. The practicality of the Embodied Scene Cloning method is not very high. In real applications, manual observation and comparison of the differences between the training and test environments are required,(e.g. identify the blue box to yellow box transition) followed by the manual design of visual and text prompts based on these differences, which introduces additional complexity. If there are differences in both the background and multiple objects between the test and training environments, this method will be difficult to apply, limiting the scope of the pipeline’s usability.
4. Minor concerns:
There are some typos and mistakes in the paper that need to be addressed:

     (i) In section 4.2, Line 332-334 Among the five research questions, the third one, “As the amount of cloned data... ”  is a statement, not a question. The authors should change that into a proper question.

     (ii) In section 4.2, Line 428-430 check this sentence, it’s been repeated.

     (iii) In section 4.2, Line 421, 427  The authors use three patterns of reference to the table, namely “Table 3”, “Table III”, “Table.4”. The authors are encouraged to change these into unified reference.

     (iiii) In section 4.2, Table 3. For lift_table task, there are two bold value.

     The authors should check grammar, typos, and phrasing to improve the quality of the paper.

questions:
1. I have some questions about the authors' real-world experiments. The success rates in the table are 0%, 20%, and 60%. Could the authors clarify how many trials were conducted to determine these success rates? If the number of trials was relatively low, could the authors increase the number of experiments to obtain more accurate success rates and better validate the effectiveness of the method?
2. Could the authors increase the variety of tasks in the real-world experiments? Currently,the real-world experiments only involve the robotic arm clamping cubes from a table and placing them inside a basket. Could the authors expand the objects being grasped (e.g. type, color...) and add different task types to better validate the effectiveness of Embodied Scene Cloning in real-world scenarios?
3. The formula in "Progressive Masked Fusion" seems to have some issues. First, for the formula $\tilde{z_t} = M_t \cdot z_t + (1 - M_t) \cdot \tilde{z_t}$​, according to Fig. 2, it seems that $\tilde{z_{t-1}}$should be computed from $z_t$​ and $\tilde{z_t}$​. Maybe the formula should be changed to ​$\tilde{z_{t-1}} = M_t \cdot z_t + (1 - M_t) \cdot \tilde{z_t}$​? Second, how is $M_0$​, the initial mask, obtained?
4. I have some doubts about the experiment of Embodied Scene Cloning dataset sizes (Table 3). For some tasks, fewer amount of cloning data may have higher success rate, why would this happen?
5. The authors only conducted experiments on the CALVIN dataset. Could the authors test the effectiveness of Embodied Scene Cloning on other datasets and tasks, such as RT-1, RT-2 and BEHAVIOR-1K?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730832046533

thread_type:
official_review

<thread_object>
creator: reviewer_6cxy

<content>
summary:
This paper targets solving the visual gap in generalizing robot tasks. The main motivation is to augment an image in embodied scenes which leads to the technique called Embodied Scene Cloning. A visual prompt is used to augment the object or the background environment in data. Extensive experiments demonstrate the proposed augmentation has achieved performance improvements in visuomotor baselines on CALVIN. A few real-world experiments are given.

strengths:
1. The paper is well written and easy to read.
2. The visual prompt based augmentation is interesting and novel.
3. The experiments demonstrate superior results over the baseline on CALVIN.

weaknesses:
1. The task setting sounds somehow tricky. An instance image is required for unseen task deployment. How to get the specific visual prompt without background？
2. The proposed technique is limited to tasks with similar trajectory patterns, only differing in visual appearance. I am not sure how to work in a truly unseen environment. Can the system still work when other physical attributes like geometry or weight change?
3. Following the above, the author only conducts experiments on CALVIN of patterned objects and tasks. No evidence on boarder robotic tasks is given.
4. The real-world experiment is weak. Only two sequences of real-world trajectories are given.

questions:
Some introductions on the baseline GAug are appreciated.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732703964362

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revisions

comment:
We deeply appreciate the valuable and constructive feedback from all reviewers. We have thoroughly revised the manuscript in response to the raised concerns, with **all changes highlighted in blue**. Below are the specific revisions:

- **@Reviewer i9yN**: We have updated the terminology to "Robotic Manipulation" in both the title and the main text for greater clarity.  
- **@Reviewer 6CXY**: In Section 2.2, we have enhanced our introduction to the baseline model *GreenAug*.  
- **@Reviewer i9yN**: Section 2.3 now includes applications of visual prompts in generative models and large language models, and compares these to our work.  
- **@Reviewer XU12**: Section 2.4 has been expanded to include a comprehensive discussion on image translation methods in the robotics domain, contrasting these with our method.  
- **@Reviewer 55TJ**: In Section 3.3, we have revised the formula issue. In Section 4.1, we have added more details regarding our real-world experiments.  
- **@Reviewer 6CXY, @Reviewer 55TJ, @Reviewer i9yN, @Reviewer XU12**: In Appendix A.2, we added additional visual experimental results, showcasing visual outcomes in more dataset scenarios, results from multiple visual prompt injections, outcomes in our real-world experiment scene, and the potential for simulation-to-reality transfer, demonstrating the significant potential of Embodied Scene Cloning for real-world applications.  
- **@Reviewer i9yN**: In Appendix A.3, we have included an analysis of ablation experiments, verifying the effectiveness of the different modules of Embodied Scene Cloning in embodied scenarios.  
- **@Reviewer i9yN**: In Appendix A.4, we provided an analysis of failure cases, offering a more comprehensive display of Embodied Scene Cloning.  
- **@Reviewer XU12**: In Appendix A.5, we have presented comparative experiments against traditional image enhancement methods, highlighting the significant advantages of Embodied Scene Cloning compared to traditional approaches.  

If you have any further questions or require additional clarification, please do not hesitate to reach out. We welcome an open discussion and look forward to your feedback.

Sincerely,  
The Authors
</content>
</thread_object>

</thread>


Metareview: