LLaVA-Critic: Learning to Evaluate Multimodal Models
Authors: name: Tianyi Xiong, name: Xiyao Wang, name: Dong Guo, name: Qinghao Ye, name: Haoqi Fan, name: Quanquan Gu, name: Heng Huang, name: Chunyuan Li

We introduce LLaVA-Critic, the first open-source large multimodal model (LMM)
designed as a generalist evaluator to assess performance across a wide range of
multimodal tasks. LLaVA-Critic is trained using a high-quality critic
instruction-following dataset that incorporates diverse evaluation criteria and
scenarios. Our experiments demonstrate the model's effectiveness in two key
areas: (1) LMM-as-a-Judge, where LLaVA-Critic provides reliable evaluation
scores, performing on par with or surpassing GPT models on multiple evaluation
benchmarks; and (2) Preference Learning, where it generates reward signals for
preference learning, enhancing model alignment capabilities. This work
underscores the potential of open-source LMMs in self-critique and evaluation,
setting the stage for future research into scalable, superhuman alignment
feedback mechanisms for LMMs.