UIP2P: Unsupervised Instruction-Based Image Editing via Cycle Edit Consistency

Authors: Enis Simsar, Yongqin Xian, Alessio Tonioni, Thomas Hofmann, Federico Tombari
Primary area: generative models
Keywords: Unsupervised learning, Diffusion models, Cycle edit consistency, Instruction-based image editing

Abstract:
We propose an unsupervised model for instruction-based image editing that eliminates the need for ground-truth edited images during training. Traditional supervised approaches depend on datasets containing triplets of input image, edited image, and edit instruction, often generated by either existing editing methods or human-annotations, which introduce biases and limit their generalization ability. Our model addresses these challenges by introducing a novel editing mechanism called Cycle Edit Consistency (CEC). We propose to apply a forward and backward edit in one training step and enforce consistency in both the image and attention space. This allows us to bypass the need for ground-truth edited images and unlock training on datasets comprising either real image-caption pairs or image-caption-edit triplets. We empirically show that our unsupervised method achieves better performance across a wider range of edits with high fidelity and precision. By eliminating the need for pre-existing datasets of triplets, reducing biases associated with supervised methods, and introducing CEC, our work represents a significant advancement in unblocking scaling of instruction-based image editing.

Decision: reject
Reviewer scores: 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1729782883970

thread_type:
official_review

<thread_object>
creator: reviewer_gokn

<content>
summary:
This method presents a new method for instruct-based image editing. Different from the previous method, this method does not require paired editing images for training via cycle edit consistency. By training on the real images only, the overall results show much better performance than the previous supervised methods.

strengths:
- Editing the images via instruction is natural. The dataset collection is expensive. This method provides an unsupervised method to achieve this goal and several loss functions are utilized to make it happen.
- The experiment results are detailed. Different datasets are trained to show the differences. Single-turn and multi-turn editing are considered.

weaknesses:
- This method involves several loss functions for training, which might make it hard to find the best checkpoint.
- There are many loss functions in UIP2P. However, the author only does the ablations on the L_sim and  L_attn. What about other loss functions?

questions:
1. Are there multi-turn results contained in the paper?
2. UIP2P is trained based on instruct-pix2pix. Can UIP2P be trained from scratch (e.g. Stable Diffusion checkpoint)? Or Does this method require paired instructions pre-trained?
3. It is interesting to see that UIP2P can edit the image in fewer DDIM steps. Why? Any possible discussions?
4. In Fig.3. I am confused about clip Loss. The clip loss is added between the input caption, edited caption, input image, and edited image. Why this loss is added only between the input image and the edited image? What about the clip loss on the reconstructed image?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730444789949

thread_type:
official_review

<thread_object>
creator: reviewer_mjqy

<content>
summary:
This paper proposes an unsupervised model for instruction-based image editing. Existing supervised methods rely on datasets consisting of input images, edited images, and editing instructions, which limits their generalization. This paper introduces Cycle Edit Consistency (CEC) to avoid this heavy dependency. Specifically, the authors apply a forward and backward edit in one training step and enforce consistency in both the image and attention space with CLIP.

strengths:
1. This paper is well written and easy to follow.
2. The task is interesting and the problem setting is reasonable.
3. Although the idea of cycle consistency has been seen in CycleGAN, proving that it can be successfully applied to instruction-based image editing also represents a certain contribution.

weaknesses:
Due to the lack of GroundTruth, it is difficult to make an accurate comparison of the experimental results. Looking at the edited image results, the proposed method does not seem to have a clearly superior advantage compared to the methods against which it was benchmarked. For example, in the first row of Figure 4, I prefer the result of HIVE; in the second row, I favor MagicBrush; in the third row, I like the results of MagicBrush and MGIE; in the sixth row, I am more inclined towards MGIE's result; in the eighth row, I prefer MAIE's outcome; and in the last row, I favor the results of InstructPix2Pix and SmartEdit.

questions:
1. One thing I am curious about is whether the proposed method generates different editing results for the same input during inference. Is there a need to manually select the better images? Similarly, do the compared methods also select one result from multiple outcomes for comparison?
2. I noticed that the backbone structure of the proposed method is InstructPix2Pix. Therefore, during training, does this method use the pre-trained weights of InstructPix2Pix, or does it start training from scratch?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730556148024

thread_type:
official_review

<thread_object>
creator: reviewer_atp8

<content>
summary:
The paper is motivated to remove the dependency on the triplet data of before-and-after images and editing instructions in training an image editing model and also aims to remove the bias introduced by the automatically created triplet data, which is meaningful and objective. To achieve this, the author proposes the cycle edit consistency training that does not require the edited image. Specifically, the author designs several consistency losses in various aspects such as images, pixels, and attention. The author evaluates the method on magic brush test data.

strengths:
1. The motivation for removing the triplet is very meaningful and reasonable in instruction-based image editing. The bias of training data is also an important problem. 

2. The presentation is easy to understand and the discussion for the motivation is reasonable and convincing.

3. There are several experiment evaluation types including quantitative, qualitative, and user study.

weaknesses:
Weaknesses and Questions:

1.	Backpropagation efficiency: The author proposes several consistency losses. Considering the diffusion process InsP2P includes multiple steps to get the final image, what is the backpropagation process of the loss constructed on the image? I think this will be a recursive procedure since the loss will have to backpropagate from the last step of diffusion to the very first step which is insanely computationally expensive and requires large GPU memory.  The same questions apply to attention map consistency, etc.

2.	Validation for removing bias in InsP2P apart from inconsistency. I think the proposed method can maintain the consistency of forward and reversed edits but cannot guarantee that the edited results are good, disentangled, and do not include any bias from the training data. For example, if InsP2P edited results are not good, they can still be edited back to the original image. How does the method avoid this? Besides, which loss contributes most to the model? The table 4 does not clearly show this.

3.	The structure preservation. I think the CLIP similarity loss makes the model have the editing ability. However, this does not indicate the model can preserve the structure since the edited image can be very different from the original but have the same semantics and be edited back to the original. The author may elaborate on this.

4.	Evaluation metrics and results. To validate the editing ability, I think the author should evaluate more diverse editing types and datasets such as PIE. 

5.	I am also interested in the attention map visualization in forward and reversed editing. I think the attention can be different at the same timestep since the forward and reverse process are opposite.  Which attention map at which timestep is enforced with consistency? More clarifications and visualization evidence on the loss should be provided.

6.	Besides, the equations are not numbered.

questions:
Please see the weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732525305861

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
**Dear Reviewers and ACs,**

Thank you for taking the time to review our submission. Your thoughtful and constructive feedback has been instrumental in refining our work. We have carefully addressed each concern point by point and incorporated the corresponding changes into the revised manuscript, with updates **highlighted in green** for clarity.

The revisions focus on key areas, including computational efficiency, structure preservation, subjective evaluations, and scalability. Additionally, we have emphasized the practical advantages of our approach, particularly its ability to deliver strong performance without relying on pre-trained supervision or ground-truth edited images, distinguishing it from competing methods.

We hope the revised manuscript meets your expectations and addresses all concerns effectively. **We are happy to address any further concerns.**



Best regards,

Paper 1784 Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper aims to address the limited generalization and the bias of supervised instruction-based image editing. The authors propose a method which does not require paired editing images for training via cycle edit consistency. Multiple types of experiments are conducted to demonstrate the effectiveness of the proposed method. The problem setting and the motivation for removing the triplet are recognized by all reviewers. However, the main cycle consistency idea is similar to CycleGAN and is widely seen, it is not very innovative at the technical level.

Besides, the superiority of the proposed method in the visual quality of generated images is also concerned. The proposed method does not have clear advantages over the compared methods. Although the authors conduct a user study, the setup and results of the study are not very convincing. Firstly, it is not clear why two (instead of one) best results are selected. Secondly, the study has 30 questions, which means only one test sample is edited across 15 image-edit instructions or different test sample are separately edited across only partial image-edit instructions. The sample size is too small. Thirdly, based on the small number of test samples, the similar results (e.g., 22% vs. 20% and 20% vs. 19%) cannot sufficiently demonstrate the clear superiority of the proposed method. Besides, the user study does not report the statistical significance, the validity of the statistical results is uncertain. 

In addition, the current user study does not consider the worst case, it cannot reflect whether the image generated by the proposed method have significantly worse quality in some cases. A better approach would be to allow the users to rank all the images generated by the six methods in each question, and compare the six methods after counting the average ranking scores of each method. Based on the above considerations, I think the current manuscript does not match the ICLR’s requirement and I do not recommend to accept this manuscript.