Multi-Reward as Condition for Instruction-based Image Editing

Authors: Xin Gu, Ming Li, Libo Zhang, Fan Chen, Longyin Wen, Tiejian Luo, Sijie Zhu
Primary area: generative models
Keywords: Instruction-based Image Editing

Abstract:
High-quality training triplets (instruction, original image, edited image) are essential for instruction-based image editing. Predominant training datasets (e.g., InsPix2Pix) are created using text-to-image generative models (e.g., Stable Diffusion, DALL-E) which are not trained for image editing. Accordingly, these datasets suffer from inaccurate instruction following, poor detail preserving, and generation artifacts. In this paper, we propose to address the training data quality issue with multi-perspective reward data instead of refining the ground-truth image quality. 1) we first design a quantitative metric system based on best-in-class LVLM (Large Vision Language Model), i.e., GPT-4o in our case, to evaluate the generation quality from 3 perspectives, namely, instruction following, detail preserving, and generation quality. For each perspective, we collected quantitative score in $0\sim 5$ and text descriptive feedback on the specific failure points in ground-truth edited images, resulting in a high-quality editing reward dataset, i.e., RewardEdit20K. 2) We further proposed a novel training framework to seamlessly integrate the metric output, regarded as multi-reward, into editing models to learn from the imperfect training triplets. During training, the reward scores and text descriptions are encoded as embeddings and fed into both the latent space and the U-Net of the editing models as auxiliary conditions. During inference, we set these additional conditions to the highest score with no text description for failure points, to aim at the best generation outcome. 3) We also build a challenging evaluation benchmark with real-world images/photos and diverse editing instructions, named as Real-Edit. Experiments indicate that our multi-reward conditioned model outperforms its no-reward counterpart on two popular editing pipelines, i.e., InsPix2Pix and SmartEdit. Code is released at https://github.com/bytedance/Multi-Reward-Editing.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730562529138

thread_type:
official_review

<thread_object>
creator: reviewer_sj5c

<content>
summary:
This paper introduces a dataset and benchmark for assessing image editing performance across multiple dimensions. The authors also utilize these multi-dimensional scores as rewards to enhance the effectiveness of image editing models.

strengths:
1. The paper underscores a valuable methodology for evaluating image editing datasets, which is highly beneficial in practical applications.
2. It proposes the use of evaluation scores to boost the performance of image editing models, which is novel in image editing.

weaknesses:
1. Although the use of GPT-4o for evaluation is efficient in practice, it lacks guaranteed consistency. The evaluation results could be influenced by underlying changes in GPT-4o, rendering the results unreliable. An alternative solution to this issue could be to train an independent evaluation model.

2. The RewardEdit-20K dataset continues to utilize images generated by InstructPix2Pix's model. For some challenging editing samples, scores cannot exceed 2 points. Even if the reward model can enhance performance in other samples, can it generate better results for these difficult cases generated by InsPix2Pix?

3. There is a lack of detailed analysis on the individual impact of each perspective reward on the improvement of editing models. Including such insights in an ablation study could strengthen the paper's contribution.

4. The practice of incorporating additional "reward scores" to enhance image generation quality is already established within the stable diffusion community (see https://civitai.com/articles/4248/what-is-score9-and-how-to-use-it-in-pony-diffusion). A more thorough discussion linking the proposed multi-reward framework to existing methodologies would enrich the manuscript's contribution.

questions:
1. Could you elaborate on the dimensionality of $c_R$ and $Linear(c_R)$, and explain how lines 281 and 287 can be implemented, given that both use the same notation of $Linear(c_R)$?
2. In line 369, is there a specific reason for training at a resolution of 256? Wouldn't training and inference at 512 yield better results?
3. Are the trained modules shared or trained separately between InstructPix2Pix and smartEdit?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730659877114

thread_type:
official_review

<thread_object>
creator: reviewer_etqx

<content>
summary:
This paper identifies significant issues related to data quality in the current Image Edit dataset. It proposes a pipeline for data cleaning and scoring using MLLM and introduces a cleaned dataset along with a training framework designed for multi-reward scenarios.

strengths:
- This paper discusses the issues present in the current dataset and utilizes MLLM for data cleaning. Through experiments and comparisons, it demonstrates the effectiveness of the cleaned dataset and the new training methods.
- Good writing and detailed experiments make this paper compelling.

weaknesses:
- W.1: The paper lacks a comparison with RL in T2I methods like DPO-Diffusion[1]. If I understand correctly, I believe that the Multi-Reward Framework is conceptually similar to methods like DPO-Diffusion. Therefore, I think it is reasonable and necessary to articulate the comparisons and distinctions between these approaches, especially the method difference.
- W.2: The article lacks some novelty. Of course, high-quality and abundant data can effectively enhance model performance, so I am uncertain about the extent to which the proposed Multi-Reward Framework improves upon traditional methods. For example, in Table 1, additional editing data (0.02M) was used for training. I believe it is fair to compare it with the exact same data using the same baseline method; otherwise, it is difficult to convince me whether the improvement comes from the high-quality data or the architecture. Perhaps experiments on the unprocessed REWARDEDIT-20K data could be added for comparison.

[1] Wallace, B., Dang, M., Rafailov, R., Zhou, L., Lou, A., Purushwalkam, S., Ermon, S., Xiong, C., Joty, S., & Naik, N. (2023). *Diffusion Model Alignment Using Direct Preference Optimization*. arXiv preprint arXiv:2311.12908.

questions:
See above, especially W.1
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703585084

thread_type:
official_review

<thread_object>
creator: reviewer_x16e

<content>
summary:
This paper aims to correct the noise supervision in instruction-based image editing models by using multi-view reward data as an additional condition. To achieve this, the authors collected a dataset named RewardEdit-20K, which contains 20,000 instances of multi-view reward data.

strengths:
1. This paper intorduce  a multi-view reward mechanism, instead of directly improving the quality of ground-truth images, the authors utilized GPT-4o to evaluate the training data from three key perspectives: instruction adherence, detail preservation, and generation quality.

2.The RewardEdit-20K dataset and the Real-Edit evaluation benchmark.

weaknesses:
1. The multi-view reward mechanism used in this paper relies entirely on GPT-4o’s evaluation. Although GPT-4o demonstrates strong capabilities in understanding and generating natural language, it may not fully capture the subtle nuances of human perception regarding image editing quality.

2. The cost for this reward is expensive as it using GPT4-o .

questions:
1. The human evaluations showed slightly lower scores than those generated by GPT-4o, though both showed consistency in ranking. Could the authors provide insights into why this discrepancy exists?

2.Given the reliance on the RewardEdit-20K dataset, a more detailed release plan for this dataset (and possibly pretrained models) could be helpful.

3. While the approach shows significant improvements, an analysis on instances where the multi-reward mechanism fails or provides subpar results would be beneficial. Understanding the limitations could offer insights for future iterations or refinements of the method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704904557

thread_type:
official_review

<thread_object>
creator: reviewer_rgo1

<content>
summary:
The authors introduce a high-quality editing reward dataset to address the problem of editing effectiveness due to training data quality issues with instruction-based image editing methods. This dataset contains assessment scores and reward texts, which are introduced as additional conditions to the instruction-based editing model to improve the model's ability. The authors also introduce an evaluation set to assess the quality of the instruction-based image editing model from multiple dimensions. Qualitatively and quantitatively, it is demonstrated that the present method effectively improves the quality of the instructional editing model.

strengths:
1.	A novel reward-based instruction editing framework that introduces evaluation scores for VLLM as well as rewarding textual feedback to improve the capability of instruction editing models. 
2.	The creation of Real-Edit provides a standardized approach to evaluate instructional editing methods in different scenarios.
3.	The method shows superior performance in both quantitative metrics and qualitative results, indicating robust editing capabilities.

weaknesses:
1. Both the quantitative assessment in Table 1 and the training strategy are based on the same scoring strategy for the GPT-4o, making the evaluation of the results overly dependent on the a priori of the VLLM and making it difficult to objectively validate the strengths of this method. One solution to this dilemma is to use other assessment metrics (e.g., CLIP scores) on the three dimensions to be adopted for comparison with other methods.
2. While experiments have shown that the introduction of additional reward text can improve image editing, unfortunately, there is no analysis of why introducing negative text in this way could help guide the diffusion process towards more effective editing.
3. Are there limitations to the introduction of the dataset REWARDEDIT-20K. Different instructional editing models may have been obtained by tuning on different training sets [1][2]. While this paper achieved an advantage on Ins-Pix2Pix and its improved version SmartEdit, is it still desirable to train other models using REWARDEDIT-20K obtained from the Ins-Pix2Pix training set?
[1] Instructdiffusion: A generalist modeling interface for vision tasks, CVPR 2024
[2] SEED-Data-Edit Technical Report: A Hybrid Dataset for Instructional Image Editing, arXiv 2024

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732517354728

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Rebuttal

comment:
We appreciate all the reviewers for their thoughtful and careful feedback.  
In this paper, we propose a new *comprehensive solution* to address the limitations in existing image editing, including new training data, network architecture, evaluation benchmarks, and evaluation metrics. Extensive experiments demonstrate that our proposed method can be combined with existing editing models, resulting in significant performance improvements and achieving state-of-the-art results in both GPT-4o and human evaluations.

As suggested by the reviewers, we have thoroughly revised our manuscript and address each of the issues raised in the reviews:

- Reviewer `RGo1`: We have added results on other evaluation metrics (*Q1*), further analyzed the reasons for improvements brought by negative text (*Q2*) and discussed the generalizability of the RewardEdit-20K dataset (*Q3*).

- Reviewer `X16E`: We compared and discussed the GPT-4o annotations and human annotations (*Q1*), further analyzed the costs of GPT-4o (*Q2*), the score discrepancies between GPT-4o and human evaluations (*Q3*), provided a detailed release plan (*Q4*), and added failure case analysis (*Q5*).
  
- Reviewer `eTqX`: We thoroughly discussed the connections and differences between our approach and methods like DPO-diffusion (*Q1*), and added ablation experiments to further demonstrate the effectiveness of our method (*Q2*).

- Reviewer `sJ5c`: We discussed the stability of GPT-4o (*Q1*), validated our method's improvements on challenging editing samples (*Q2*), added ablation experiments and analysis for each reward perspective (*Q3*),  compared it with existing reward-based methods(*Q4*), clarified the dimensionality of the linear layer (*Q5*), explained why we train at a resolution of 256 (*Q6*), and clarified that the MRC module is trained separately for different methods (*Q7*).

We once again express our heartfelt gratitude to all the reviewers for their valuable feedback, and we hope that our responses satisfactorily address all concerns. Please feel free to let us know if you have any remaining concerns and we are happy to address them!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper aims to use multi-view reward data as an additional condition to address the problem of editing effectiveness caused by the issues of training data quality. To achieve this, the authors introduce a high-quality editing reward dataset and propose a benchmark to evaluate the quality of the instruction-based image editing model from multiple dimensions. Qualitative and quantitative evaluations are presented and demonstrate the effectiveness of the proposed method. All reviewers give positive rating scores. Based on the above considerations, I recommend to accept this manuscript.