MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?

Authors: Zhaorun Chen, Yichao Du, Zichen Wen, Yiyang Zhou, Chenhang Cui, Zhenzhen Weng, Chaoqi Wang, Zhengwei Tong, Leria HUANG, Canyu Chen, Haoqin Tu, Qinghao Ye, Zhihong Zhu, Yuqing Zhang, Jiawei Zhou, Zhuokai Zhao, Rafael Rafailov, Chelsea Finn, Huaxiu Yao
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Multimodal Reward Models, Foundation Models Alignment, Reinforcement Learning from Human Feedback

Abstract:
While text-to-image models like DALLE-3 and Stable Diffusion are rapidly proliferating, they often encounter challenges such as hallucination, bias, and the production of unsafe, low-quality output. To effectively address these issues, it is crucial to align these models with desired behaviors based on feedback from a multimodal judge. Despite their significance, current multimodal judges frequently undergo inadequate evaluation of their capabilities and limitations, potentially leading to misalignment and unsafe fine-tuning outcomes. To address this issue, we introduce MJ-Bench, a novel benchmark which incorporates a comprehensive preference dataset to evaluate multimodal judges in providing feedback for image generation models across four key perspectives: alignment, safety, image quality, and bias. Specifically, we evaluate a large variety of multimodal judges including smaller-sized CLIP-based scoring models, open-source VLMs (e.g. LLaVA family), and close-source VLMs (e.g. GPT-4o, Claude 3) on each decomposed subcategory of our preference dataset. Experiments reveal that close-source VLMs generally provide better feedback, with GPT-4o outperforming other judges in average. Compared with open-source VLMs, smaller-sized scoring models can provide better feedback regarding text-image alignment and image quality, while VLMs provide more accurate feedback regarding safety and generation bias due to their stronger reasoning capabilities. Further studies in feedback scale reveal that VLM judges can generally provide more accurate and stable feedback in natural language (Likert-scale) than numerical scales. Notably, human evaluations on end-to-end fine-tuned models using separate feedback from these multimodal judges provide similar conclusions, further confirming the effectiveness of MJ-Bench.

Decision: reject
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730123408862

thread_type:
official_review

<thread_object>
creator: reviewer_rmzq

<content>
summary:
The paper organizes a new preference dataset called MJ-Bench, which captures four different aspects of text-to-image generation: text-image alignment, safety, image quality, and bias. 

The authors propose two methods to obtain feedback from multi-modal judges: single-input and multi-input judge. The paper also conducts detailed analysis, such as the consistency of the preference of the judges w.r.t different image modes.

strengths:
1. The paper tackles a crucial issue in evaluating text-to-image models. 
2. The paper includes detailed ablation studies with some insightful findings. For instance,  the analysis of the consistency of the preference of the judges w.r.t different image modes. 
3. The proposed dataset offers many formats (e.g., ranking and voting), which can enable a wider variety of preference modeling.

weaknesses:
1. The paper's novelty seems limited, as the whole pipeline has been proposed in the previous method. Besides, The selection of these four aspects lacks in-depth analysis.
2. The paper employs human evaluators for experimental evaluation multiple times, yet it fails to report the number of human evaluators involved. Since human evaluators may introduce bias, it is recommended to report this metric. If the number is small, it is advised to increase the scale of human evaluators.
3. The scale of the dataset was not compared with other existing datasets. As a result, the application scope of the dataset may be limited.
4. Since this work is a benchmark study, the quality of the samples within the benchmark was not evaluated. It is recommended to supplement the study with an experiment to assess the quality of the samples in the dataset.
5. The dataset only involved feedback from six judges to train a model (as indicated on page 7). It is suggested to supplement the study with an experiment where feedback is directly constructed from the benchmark data to train a model, in order to observe the outcomes.

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730601333026

thread_type:
official_review

<thread_object>
creator: reviewer_eath

<content>
summary:
This paper proposes a benchmark for evaluating LMMs about the judgement capability for T2I generation. The proposed benchmark set contains a few thousands (win, lose, prompt) triplets covering T2I evaluation aspects: alignment, safety, quality, and bias. To show that the proposed benchmark set can provide a fair platform to compare LMMs, this paper uses LMMs as the reward model and use DPO/DDPO to finetune SD1.5. The results show that the model finetuned with a better LMM (evaluated with the proposed benchmark) is preferred by human.

strengths:
- Evaluating and benchmarking the capability of LMMs is an important research direction of the community nowadays. This paper proposes a comparably comprehensive benchmark set and provides the comparison of a large amount of LMMs, which is valuable to the community.
- I like the design of the benchmark questions based on (win, lose, prompt) triplets, which, in my opinion, can make the human annotation easier compared with direct quality rating.

weaknesses:
- In terms of the presentation, I am unsure whether claiming evaluating "reward models" is a better idea than claiming evaluating "LMMs". Close-sourced LMMs such as GPT-4o/4v/Claude are not widely adopted T2I reward models due to their high cost and low throughput. In addition, Table 10 summarizes the number of evaluation questions for each category of the benchmark, which is of the pivotal importance and should be put in the main paper.
- In Table 10, how the number of evaluation questions for each category is determined? I found that most categories, scenarios, and subsets have random numbers of evaluation questions. If so, does this evaluation benchmark introduce bias by itself? For example, the questions for object (250) is more than 4 times of counting questions (55). Since all LMMs are evaluated by the averaged metrics, does the proposed benchmark biased towards LMMs that are better at object questions?
- Table 2 and 3 show the results of SD1.5 finetuned with DPO and DDPO, which, according to my understanding, is the "evaluation" to the proposed benchmark. However, I found that the LMM that work best as the reward (Table 2) is not aligned with the LMM having the highest evaluation scores based on the benchmark (Table 1). Does it mean the proposed benchmark is not good to evaluate T2I reward models? In addition, in Table 3, GPT-4o and GPT-4v achieve the best results with DPO and DDPO, respectively. I think this result suggest that the evaluation of reward model should be performed for certain RLHF methods. I suggest the authors to provide more discussion about this.

questions:
- I found that very limited details about how to use GPT-4o/4v/Gemini/Claude as the reward model for DPO/DDPO is provided in the paper. Do you use online or offline RL? If online, how to make use of these APIs and obtain the reward for DPO/DDPO fast? What if the API calls fail? If offline, what is the dataset used to finetune the model with DPO/DDPO?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716387000

thread_type:
official_review

<thread_object>
creator: reviewer_3dfz

<content>
summary:
* The paper introduces MJ-BENCH, a benchmark to improve feedback for text-to-image models on alignment, safety, image quality, and bias. 
* Some interesting points are found: e.g., Closed-source VLMs like GPT-4o outperform others overall, while smaller models excel in alignment and quality, and VLMs perform better on safety and bias. Human evaluations confirm MJ-BENCH’s effectiveness for model fine-tuning.

strengths:
* The writing and presentation of the entire paper are good.
* A comprehensive benchmark is provided, which can further advance research in the related field.
* It further promotes research on RLAIF (Reinforcement Learning from AI Feedbacks) and provides solid evidence for its effectiveness.

weaknesses:
* Please provide a comprehensive discussion that covers the related work. For example, some studies have also made efforts in aligning text-to-image models with feedback from MLLMs, such as VisionPrefe r[1], which also discusses the varying effectiveness of different MLLMs as annotators for alignment data in text-to-image tasks.

* It would be beneficial to discuss further the situations in which human judges and MLLM judges disagree, as this could provide valuable insights for future work.

* Besides, to better demonstrate the effectiveness of MJ-BENCH,  the authors are recommended to present some visualization cases of MJ-BENCH to help to offer a clearer and more comprehensive understanding of how well the dataset.

**Reference**
[1] Wu X, Huang S, Wei F. Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generation[J]. 2024.

questions:
Please see in Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730724398764

thread_type:
official_review

<thread_object>
creator: reviewer_qiub

<content>
summary:
This paper introduces MJ-BENCH, a novel benchmark that includes a comprehensive preference dataset to evaluate multimodal judges in providing feedback for image generation models across four key perspectives: alignment, safety, image quality, and bias. Detailed experimental analyses were conducted on CLIP-based scoring models, open-source VLMs, and close-source VLMs.

strengths:
The paper is well-written and logically coherent. The proposed benchmark is comprehensive, covering aspects such as alignment, safety, image quality, and bias.

weaknesses:
I would consider increasing my score if these concerns are addressed or resolved. 

**1. Comprehensiveness and fairness of the evaluation**: The paper evaluates models like CLIP, LLaVA and GPT-4, but lacks some popular alternative alignment evaluation models such as the Decompose method ([Davidsonian Scene Graph](https://arxiv.org/pdf/2310.18235)) and the Answer Probability Method ([VQAScore](https://arxiv.org/pdf/2404.01291)). Additionally, the paper claims that the alignment dataset was collected from Pick-a-pic, HPDv2, and ImageRewardDB, so evaluating PickScore-v1, HPS-v2.1, and ImageReward in the experiments is unfair because these models have already been trained on similar data or dataset formats.

questions:
1. Could you please explain how quality control is conducted in the construction of data across different evaluation dimensions? For example, how is a human verification process conducted, and what is the proportion of data that gets filtered?

2. Given the criticisms of CLIP-based methods as 'bag-of-words' and the lack of interpretability and reproducibility when directly asking VLMs to output scores, evaluating and comparing alternative evaluation methods is crucial. It's important to include popular alignment approaches that avoid these weaknesses, such as decomposition methods from [T2i-compbench](https://proceedings.neurips.cc/paper_files/paper/2023/file/f8ad010cdd9143dbb0e9308c093aff24-Paper-Datasets_and_Benchmarks.pdf)  and [Davidsonian Scene Graph](https://arxiv.org/pdf/2310.18235), as well as answer probability methods like [VQAScore](https://arxiv.org/pdf/2404.01291).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732418686322

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Paper Revision

comment:
We sincerely thank all reviewers for their recognition of the critical contributions of our work in advancing research on RLAIF for image generation, as well as their valuable feedback and efforts to help us improve the paper. Based on the reviewers' comments, we have made significant improvements to the paper (a revised PDF has been uploaded), with the main changes highlighted in blue text. Below, we summarize these changes:

1. We evaluated three additional models (two decomposition-based models: DSG and T2I-CompBench, and one probability-based model: VQAScore) as well as four of the most recent MLLMs on MJ-Bench. (Reviewer qiUB)
2. We conducted an additional experiment where we curated a clean test set for the alignment perspective and benchmarked models on this set to avoid data contamination and ensure fairness in our evaluation. (Reviewer qiUB)
3. We provided more comprehensive details and statistics about our data curation and quality control procedures. (Reviewers qiUB, eath, Rmzq)
4. We quantitatively demonstrated the high quality of the test samples in MJ-Bench by fine-tuning a text-to-image model solely using our preference data and comparing it with other models. (Reviewers eath, Rmzq)
5. We expanded the discussion on related work and highlighted the unique contributions of MJ-Bench. (Reviewer 3DFZ)
6. We trained a MoE-based reward model using a split from MJ-Bench and showed its superior performance in providing feedback for text-to-image generation. (Reviewers 3DFZ, Rmzq)
7. We conducted additional human evaluations using the standard win rate metric, comparing more models in an end-to-end manner to mitigate potential bias in human evaluations. (Reviewer eath)

Thank you once again for your thoughtful suggestions and efforts to help us improve this work. We hope our responses and revisions address your concerns, and we look forward to further engaging discussions during the follow-up period.

Best regards,  

Submission #9037 Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary of claims and findings: 

The paper introduces a benchmark named MJ-BENCH to evaluate multimodal judges for text-to-image generation models across four critical dimensions: alignment, safety, image quality, and bias. MJ-BENCH includes a comprehensive preference dataset comprising thousands of (win, lose, prompt) triplets. The authors propose two feedback mechanisms for multimodal judges: single-input and multi-input judges, and conduct detailed analyses, such as evaluating the consistency of judges' preferences across different image modes. Several insights are highlighted, including performance comparisons between closed-source and open-source VLMs under various metrics, model behaviors after fine-tuning, and validation through human evaluations. Overall, MJ-BENCH provides a comprehensive and well-rounded approach to evaluating and improving feedback mechanisms for text-to-image generation models, representing a valuable contribution to the community. 

Pros: 

The reviewers recognized MJ-BENCH as a comprehensive benchmark addressing critical challenges in evaluating text-to-image models. They highlighted its detailed ablation studies, which offer valuable insights, and its dataset, which supports various formats such as ranking and voting, enabling diverse applications in preference modeling. The benchmark's design and methodology were commended for advancing research in evaluating LMMs and fostering progress in Reinforcement Learning from AI Feedback (RLAIF).

Cons: 

The original submission lacks several important aspects, with reviewers raising concerns about the novelty, comprehensiveness, fairness, and presentation of the proposed MJ-BENCH benchmark:
- Novelty: The overall pipeline lacks significant novelty, as it is similar to previous methods. 

- Benchmark: The selection of evaluation dimensions (alignment, safety, image quality, and bias) lacks in-depth analysis. The scale and quality of the dataset were not compared to existing benchmarks, limiting its application scope. The benchmark's quality was not independently evaluated through experiments. Questions were raised about quality control in constructing data across evaluation dimensions, including human verification processes and data filtering proportions. Concerns were also raised about potential biases introduced by uneven question distributions across categories. The paper did not provide visualizations of MJ-BENCH cases to clarify the dataset's structure and effectiveness.
- Evaluation: The benchmark does not include several important alternative alignment evaluation models, such as the Decompose method (Davidsonian Scene Graph) and the Answer Probability Method (VQAScore). Evaluating PickScore-v1, HPS-v2.1, and ImageReward is deemed unfair, as these models were trained on datasets similar to those used for MJ-BENCH. The situations in which human judges and multimodal large language model (MLLM) judges disagree are not explored, which could provide valuable insights for future work.
- The paper lacks a comprehensive discussion of related works, such as VisionPrefer, which addresses similar challenges in aligning text-to-image models.
- Several important details were unclear or missing in the original submission. Some observations were also inconsistent (e.g., discrepancies in Tables 3 and 1).

Rebuttal: 

During the rebuttal phase, the authors provided extensive clarifications, explanations, and additional experimental results, including the evaluation of three additional models (DSG, T2I-CompBench, and VQAScore) and four recent MLLMs, conducting an experiment with a clean test set to ensure fairness, performing additional human evaluations, training an MoE-based reward model, expanding the discussion on related work, and offering comprehensive details on data curation and quality control.

Discussion and Recommendation:

The reviewers rated this paper as borderline. While the proposed benchmark has potential and could make an impact on the community, the original submission was weak in several key areas, as outlined above. Important aspects of benchmark construction and evaluation were missing. Although the authors' rebuttal provided extensive clarifications, explanations, and additional experimental results, it also led to substantial changes in the findings and claims presented in the original submission. For instance, the original paper did not consider several important alternative alignment evaluation models, which subsequently outperformed the models initially evaluated. These issues, along with concerns about the positioning of related work, novelty, and the incorporation of rebuttal content, along with the reorganization of the main paper and appendix, led the ACs to conclude that the paper requires major revisions and another review cycle. The ACs believe that with revision, the paper's impact can be greatly enhanced, and ACs encouraged the authors to continue this line of research for future submission.