Title: Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluators
Venue: NeurIPS 2025
Accepted: yes
Keywords: mllm-as-a-judge

automatic evaluation

multimodal reasoning

multimodal large language models
Abstract: Human-generated reward signals are critical for aligning generative models with human preferences, guiding both training and inference-time evaluations. While large language models (LLMs) employed as proxy evaluators, i.e., LLM-as-a-Judge, significantly reduce the costs associated with manual annotations, they typically require extensive modality-specific training data and fail to generalize well across diverse multimodal tasks. In this paper, we propose **Flex-Judge**, a reasoning-guided multimodal judge model that leverages minimal textual reasoning data to robustly generalize across multiple modalities and evaluation formats. Our core intuition is that structured textual reasoning explanations inherently encode generalizable decision-making patterns, enabling an effective transfer to multimodal judgments, e.g., with images or videos. Empirical results demonstrate that Flex-Judge, despite being trained on significantly fewer text data, achieves competitive or superior performance compared to state-of-the-art commercial APIs and extensively trained multimodal evaluators. Notably, Flex-Judge presents broad impact in modalities like molecule, where comprehensive evaluation benchmarks are scarce, underscoring its practical value in resource-constrained domains. Our framework highlights reasoning-based text supervision as a powerful, cost-effective alternative to traditional annotation-intensive approaches, substantially advancing scalable multimodal model-as-a-judge.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces KIRBY-Judge, a reasoning-guided multimodal evaluator that uses minimal textual reasoning data to asses model outputs across various modalities like text, images, and videos. The key insight is that structured textual explanations capture generalizable decision-making patterns, which enables effective transfer to multimodal evaluation without extensive modality-specific training. Empirical results show that KIRBY-Judge achieves performance competitive with or better than existing methods, even in low-resource domains.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The method is solid.  The approach is modality-agnostic, and once trained, the KIRBY-Judge can be applied to diverse evaluation tasks (vision-language tasks, audio quality scoring, molecular structure, etc.)\n- Overall, good experiments and execution.  Evaluation is conducted across diverse tasks, such as Image Understanding, Image Editing, Video Generation, Audio Understanding, and Molecule Evaluator.\n\nWeakness\n- It would be better if experimental results (Table 1, 2, 3) include stronger baselines like latest GPT4-o, o3 and Gemini 2.5 Pro.\n- Misses some related work. For example, there are various multimodal judge or reward benchmark works worth discussing or evaluating on: video LM as judge (https://arxiv.org/abs/2503.05977) and Multimodal RewardBench (https://arxiv.org/abs/2502.14191), in addition to MLLM-as-a-Judge and VL-RewardBench, etc. discussed in the paper.\n\nQUESTIONS:\n- Could the authors elaborate more on the intuition of generalizing from textual reasoning data to multimodal judge capability?\n- How exactly was the seed training data curated?  Could the authors provide some examples of their data?  Also, the data is mainly synthetic - could the authors elaborate on how to ensure the synthetic data is high quality and not collapsed?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new multimodal judge model, Kirby-Judge, designed to evaluate the performance of large multimodal models across a broad range of tasks. The proposed approach demonstrates strong robustness in handling multiple modalities, even when trained with minimal textual reasoning data. Experimental results show that Kirby-Judge outperforms state-of-the-art judge models that rely heavily on modality-specific training data, particularly in tasks such as image understanding, image editing, video generation and audio understanding. Additionally, the authors present an effective case study in which their approach is successfully adapted to evaluate multimodal models for molecular modeling.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The paper presents Kirby-Judge, a novel multimodal judge model designed to evaluate large multimodal models across a diverse range of tasks, which addresses a critical need in the field.\n- The proposed approach is robust to different input modalities and requires only minimal textual reasoning data for training, making it broadly applicable and data-efficient.\n- Experimental evaluations demonstrate that Kirby-Judge, in general, outperforms existing judge models trained with extensive modality-specific data, across a diverse  set of tasks.\n- The method significantly reduces the reliance on large amounts of specialized training data, enhancing its practicality and scalability in real-world applications.\n- The paper includes a well-executed case study showing the adaptability of Kirby-Judge to the molecular modeling domain, further illustrating the model’s flexibility and broader applicability.\n\nWeaknesses\n\n- The image understanding capabilities of Kirby-Judge are evaluated using the MLLM-as-a-Judge and VL-RewardBench benchmarks. However, more recent and comprehensive benchmarks such as MM-RewardBench (Yasunaga et al., 2025), TaskAnything, and JudgeAnything (Pu et al., 2025) are not considered in the analysis.\n    - Yasunaga et al., Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models, arXiv preprint arXiv:2502.14191, 2025\n    - Pu et al., Judge Anything: MLLM as a Judge Across Any Modality, arXiv preprint arXiv:2503.17489, 2025\n- The video understanding capabilities of Kirby-Judge are not evaluated. For this, the authors could consider benchmarks like JudgeAnything (Pu et al., 2025), which support multi-modality assessment, including video.\n- The evaluation of video generation and image editing capabilities is limited to the GenAI-Bench benchmark. A more comprehensive evaluation could include the recently proposed Q-Eval-100K (Zhang et al., 2025) and Video-Bench (Han et al., 2025).\n    - Zhang et al., Q-Eval-100K: Evaluating Visual Quality and Alignment Level for Text-to-Vision Content, CVPR 2025\n    - Han et al., Video-Bench: Human-Aligned Video Generation Benchmark, CVPR 2025\n- Several recently proposed MLLM-as-a-Judge models such as InternLM-XComposer2.5-Reward (Zang et al., 2025), CAREVL (Dai et al., 2025), and JudgeAnything (Pu et al., 2025) are not included in the experimental comparison, which limits the assessment of Kirby-Judge's competitiveness.\n    - Zang et al., Internlm-xcomposer2.5-reward: A simple yet effective multi-modal reward model, arXiv preprint arXiv:2501.12368, 2025\n    - Dai et al., From captions to rewards (CAREVL): Leveraging large language model experts for enhanced reward modeling in large vision-language models, arXiv preprint arXiv:2503.06260, 2025\n- The paper demonstrates that classical inference-time scaling techniques, such as majority voting over multiple reasoning paths, can further improve Kirby-Judge's performance. However, it does not explore whether Kirby-Judge itself can be used as a tool for inference-time scaling, e.g., selecting the best response from a set of diverse outputs generated by an MLLM. Evaluating Kirby-Judge in this context would provide valuable insight into its practical utility as a response selector.\n\nQUESTIONS:\n- Could the authors comment on how their model would compare against the recently proposed MLLM-as-a-Judge models such as InternLM-XComposer2.5-Reward, CAREVL, and JudgeAnything, or provide preliminary results if available?\n- Have the authors considered evaluating Kirby-Judge as a response selector as part of an inference-time scaling strategy?\n- Could video understanding assessments be carried out using the JudgeAnything benchmark?\n\nLIMITATIONS:\nThe authors have adequately discussed the limitations of their approach.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes Kirby-Judge, a multimodal judge model for automatic evaluation of image, videos, audio, and even molecules. The model is fine-tuned from Qwen2.5-VL/Omni-7B on a small scale (1K) of text-only data for LLM-as-judge data with high-quality reasoning steps. The hypothesis is that by learning from structured, reasoning-rich text data, the model can generalize to other modalities that the model originally can understand. Despite the minimal training cost, experiments show that Kirby-Judge can produce effective judging results aligned with human preferences.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The training (fine-tuning) cost is very affordable.\n\n2. The idea of transferring knowledge from text-only data to other modalities for LLM-as-judge is interesting.\n\n3. The comprehensive experiment results show promising performance compared with the best open-source and closed-source models.\n\n## Weaknesses\n\n1. In the main Tables 1 and 2, the base models' (Qwen2.5-VL-7B, Qwen2.5-Omni-7B) performance should be included to clearly understand the relative improvement brought by Kirby-Judge.\n\n2. In Figure 6, the standard deviation of multiple runs seems large. How does it compare with baseline models? Some more analysis on the model prediction stability/robustness could be helpful.\n\n3. As acknowledged in the supplementary material, the success of Kirby-Judge relies on the capabilities of the underlying multimodal LLM. If the base model does not have strong reasoning capabilities already, it would be hard to develop a judge model with limited fine-tuning data. In other words, the performance of Kirby-Judge may be heavily dependent on the base models (Qwen2.5-VL/Omni-7B), especially their base reasoning capabilities. Considering reasoning LLMs are being actively developed in the open-source community, this limitation is rather minor, but worth some investigation (e.g., training weaker models with the same data as Kirby-Judge).\n\nQUESTIONS:\nPlease check the Weaknesses described above.\n\nLIMITATIONS:\nThe authors have discussed the limitations in the supplementary material. Another minor limitation is that this work does not show whether this method can further scaled to larger models (e.g., 32B).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents the pipeline and the model for serving as a task evaluator that is trained using only textual reasoning data that can generalize its judgment capabilities to various modalities (vision, audio, video, even molecular inputs) without additional modality-specific training. The core idea of the proposed approach is that structured explanatory rationales (specifically generated for the responses evaluation and comparison) capture general decision-making patterns that can transfer across modalities. In the experiments, the authors fine-tune a base multimodal language model (specifically, Qwen-2.5 VL/Omni, 7B) on a small curated corpus of high-quality text-only evaluation examples with rich reasoning. After this training, the KIRBY-Judge (the fine-tuned model) can judge any input, i.e. perform zero-shot evaluation on image-text tasks, audio transcripts, video descriptions, and biological data, by leveraging the base model’s multimodal inputs. Empirically, KIRBY-Judge achieves competitive or superior performance to state-of-the-art evaluators despite using orders of magnitude less training data.\n\nSTRENGTHS AND WEAKNESSES:\nThe main strengths of the paper are as follows:\n\n1. The paper analyzes and empirically validates an important idea: that it is possible to effectively train models using small-scale datasets composed solely of high-quality, text-only reasoning examples. The main results suggest that when collecting modality-specific annotations is challenging, such datasets can still yield performance comparable to that of models trained on large-scale, modality-specific data.\n2. The specific molecular case study strengthens the contribution by demonstrating the method’s applicability to specific domain data. This shows that the proposed approach can generalize beyond conventional multimodal settings.\n3. The paper provides extensive experimental evaluation across a diverse set of benchmarks and modalities, which reinforces the robustness and reliability of the proposed method.\n\nThe main weaknesses of the paper are as follows:\n\n1. Despite the thorough comparisons with existing models, it would be valuable to include additional ablation studies to better isolate the effects of the proposed fine-tuning approach. First, it would be useful to evaluate the performance of the base models (e.g., Qwen-VL and Qwen-Omni) without any fine-tuning on the text-only dataset. This could clarify whether the instruction-following capabilities of the base models alone are sufficient for judgment tasks, or if the fine-tuning step is indeed necessary.\nSecond, assessing the performance gap on standard vision-language benchmarks before and after text-only fine-tuning would help determine whether this fine-tuning affects the model’s inherent multimodal understanding capabilities (either positively or negatively).\n2. An additional ablation study examining the effect of dataset size would also be valuable. Specifically, it would be helpful to understand how much text-only data is actually needed to achieve strong generalization performance without sacrificing the multimodal abilities of the initial checkpoint.\n\nQUESTIONS:\nI have the following questions for the authors:\n1. Did you evaluate the performance of the specifically prompted, but instruct-tuned multimodal models on your judgement tasks?\n2. Did you experiment with the size of the dataset for finetuning, can you assess the number of textual data necessary for training without sacrificing the multimodal abilities of the initial model?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-04"}