MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for
  Text-to-Image Generation?
Authors: name: Zhaorun Chen, name: Yichao Du, name: Zichen Wen, name: Yiyang Zhou, name: Chenhang Cui, name: Zhenzhen Weng, name: Haoqin Tu, name: Chaoqi Wang, name: Zhengwei Tong, name: Qinglan Huang, name: Canyu Chen, name: Qinghao Ye, name: Zhihong Zhu, name: Yuqing Zhang, name: Jiawei Zhou, name: Zhuokai Zhao, name: Rafael Rafailov, name: Chelsea Finn, name: Huaxiu Yao

While text-to-image models like DALLE-3 and Stable Diffusion are rapidly
proliferating, they often encounter challenges such as hallucination, bias, and
the production of unsafe, low-quality output. To effectively address these
issues, it is crucial to align these models with desired behaviors based on
feedback from a multimodal judge. Despite their significance, current
multimodal judges frequently undergo inadequate evaluation of their
capabilities and limitations, potentially leading to misalignment and unsafe
fine-tuning outcomes. To address this issue, we introduce MJ-Bench, a novel
benchmark which incorporates a comprehensive preference dataset to evaluate
multimodal judges in providing feedback for image generation models across four
key perspectives: alignment, safety, image quality, and bias. Specifically, we
evaluate a large variety of multimodal judges including smaller-sized
CLIP-based scoring models, open-source VLMs (e.g. LLaVA family), and
close-source VLMs (e.g. GPT-4o, Claude 3) on each decomposed subcategory of our
preference dataset. Experiments reveal that close-source VLMs generally provide
better feedback, with GPT-4o outperforming other judges in average. Compared
with open-source VLMs, smaller-sized scoring models can provide better feedback
regarding text-image alignment and image quality, while VLMs provide more
accurate feedback regarding safety and generation bias due to their stronger
reasoning capabilities. Further studies in feedback scale reveal that VLM
judges can generally provide more accurate and stable feedback in natural
language (Likert-scale) than numerical scales. Notably, human evaluations on
end-to-end fine-tuned models using separate feedback from these multimodal
judges provide similar conclusions, further confirming the effectiveness of
MJ-Bench. All data, code, models are available at
https://huggingface.co/MJ-Bench.