LLaVA-Critic: Learning to Evaluate Multimodal Models
Authors: name: Tianyi Xiong; name: Xiyao Wang; name: Dong Guo; name: Qinghao Ye; name: Haoqi Fan; name: Quanquan Gu; name: Heng Huang; name: Chunyuan Li
Published: 2024-10-03 17:36:33

Abstract: We introduce LLaVA-Critic, the first open-source large multimodal model (LMM)
designed as a generalist evaluator to assess performance across a wide range of
multimodal tasks. LLaVA-Critic is trained using a high-quality critic
instruction-following dataset that incorporates diverse evaluation criteria and
scenarios. Our experiments demonstrate the model's effectiveness in two key
areas: (1) LMM-as-a-Judge, where LLaVA-Critic provides reliable evaluation
scores, performing on par with or surpassing GPT models on multiple evaluation
benchmarks; and (2) Preference Learning, where it generates reward signals for
preference learning, enhancing model alignment capabilities. This work
underscores the potential of open-source LMMs in self-critique and evaluation,
setting the stage for future research into scalable, superhuman alignment
feedback mechanisms for LMMs.