Title: Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: reward model

reinforcement leaning

chain-of-though reasoning
Abstract: Recent advances in multimodal Reward Models (RMs) have shown significant promise in delivering reward signals to align vision models with human preferences. However, current RMs are generally restricted to providing direct responses or engaging in shallow reasoning processes with limited depth, often leading to inaccurate reward signals. We posit that incorporating explicit long chains of thought (CoT) into the reward reasoning process can significantly strengthen their reliability and robustness. Furthermore, we believe that once RMs internalize CoT reasoning, their direct response accuracy can also be improved through implicit reasoning capabilities. To this end, this paper proposes UnifiedReward-Think, the first unified multimodal CoT-based reward model, capable of multi-dimensional, step-by-step long-chain reasoning for both visual understanding and generation reward tasks. Specifically, we adopt an exploration-driven reinforcement fine-tuning approach to elicit and incentivize the model's latent complex reasoning ability: (1) We first use a small amount of image generation preference data to distill the reasoning process of GPT-4o, which is then used for the model's cold start to learn the format and structure of CoT reasoning. (2) Subsequently, by leveraging the model's prior knowledge and generalization capabilities, we prepare large-scale unified multimodal preference data to elicit the model's reasoning process across various vision tasks. During this phase, correct reasoning outputs are retained for rejection sampling to refine the model (3) while incorrect predicted samples are finally used for Group Relative Policy Optimization (GRPO) based reinforcement fine-tuning, enabling the model to explore diverse reasoning paths and optimize for correct and robust solutions. Extensive experiments confirm that incorporating long CoT reasoning significantly enhances the accuracy of reward signals. Notably, after mastering CoT reasoning, the model exhibits implicit reasoning capabilities, allowing it to surpass existing baselines even without explicit reasoning traces.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces UNIFIEDREWARD-THINK, a unified multimodal reward model that uses explicit long Chain-of-Thought (CoT) traces for both vision-generation and vision-understanding tasks. It proposes a three-stage training pipeline:\n1. Cold-start distillation of GPT-4o CoT on a small image-generation preference set.\n2. Rejection sampling on large, diverse preference data to keep only correct reasoning traces.\n3. GRPO (Group-Relative Policy Optimisation) reinforcement fine-tuning with format and accuracy rewards.\n\nExperiments on GenAI-Bench, VideoGen-Reward, and VLReward-Bench show gains over the UnifiedReward baseline (e.g., +4.8 macro-accuracy for image understanding, +5.1 diff-accuracy for video generation). Ablations indicate that each stage adds to performance and robustness.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- Novel use of explicit CoT traces as an internal reward-shaping signal. Instead of only using CoT to explain a final prediction, the model learns intermediate reasoning steps and uses them to guide policy optimization. This bridges a gap between explainable reasoning and direct reward learning in multimodal settings.\n- Comprehensive benchmark suite. Evaluations span five distinct tasks—image CoT verification, video frame ordering, caption coherence, etc.—showing gains in both understanding and generation. Tasks cover different modalities and difficulty levels, indicating robustness beyond a single niche.\n- Demonstrated zero-shot generalization on held-out image types and longer reasoning chains. Figure 4 shows that UNIFIEDREWARD-THINK maintains high accuracy when chain lengths double, which suggests the model scales to more complex reasoning without retraining.\n- Clear qualitative examples of reasoning traces reveal how the model avoids common pitfalls (e.g., hallucinating unsupported details). These examples provide actionable insight into where CoT improves final scores, rather than opaque “black-box” shifts.\n\n### Weaknesses\n- Heavy reliance on GPT-4o quality. The entire pipeline hinges on distilling CoT traces from GPT-4o; if the base model produces systematic biases or errors, those propagate into the reward model. There is no analysis of GPT-4o’s failure modes or mitigation when its reasoning is flawed.\n- Human-annotation process is under-specified. The paper omits details on annotator demographics, pay, and inter-rater agreement when constructing rejection-sampling data. Without knowing how diverse or reliable the preference data are, it is difficult to assess fairness and reproducibility.\n- Computational cost and scalability are unclear. GRPO fine-tuning requires maintaining CoT trace consistency; the paper does not report GPU-hours or memory overhead relative to standard reward-models. For practitioners, it is unclear if performance gains justify added cost.\n- Limited analysis of downstream biases. Reward models can amplify biases present in training data—yet the paper does not examine whether UNIFIEDREWARD-THINK introduces or exacerbates biases along gender, race, or cultural lines in image-captioning tasks. A brief audit on a fairness benchmark would strengthen trust.\n\nQUESTIONS:\n1. Have you measured the frequency of incorrect or misleading explanations in distilled CoT data? How do such errors impact downstream reward optimization?\n2. Please detail the composition, pay structure, and inter-annotator agreement of human-preference datasets used during rejection sampling. This is critical for evaluating dataset quality and reproducibility.\n3. What are the GPU-hours, memory usage, and wall-clock time for GRPO fine-tuning compared to a standard SFT reward model on the largest benchmark?\n4. Have you evaluated UNIFIEDREWARD-THINK’s outputs on any fairness or bias benchmarks (e.g., CelebA-Bias)? If so, what mitigation strategies are needed when CoT traces encode societal stereotypes?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-01"}

comments: SUMMARY:
This paper proposes UNIFIEDREWARD-THINK, a multimodal reward model that introduces long CoT reasoning to improve the accuracy and robustness of reward signals. The approach consists of three stages: initializing the model with GPT distilled data (cold start), applying rejection sampling on large-scale preference data, and using GRPO reinforcement learning to optimize reasoning capabilities. Experiments show that the method outperforms existing approaches on both visual understanding and generation tasks, and exhibits strong implicit reasoning ability even without explicitly outputting reasoning traces.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The method is simple and easy to understand.

Weaknesses:
1. There are many R1-like papers nowadays that collect similar types of data for cold start and train with GRPO; this work lacks novelty.

2. The cold start and GRPO stages are similar to those used in most xx-R1 papers.

QUESTIONS:
1. How do you ensure the quality of the distilled cold-start data? For example, a long CoT reward dataset should guarantee both the correctness of the intermediate reasoning steps and the final scoring.

2. I'm curious why this model outperforms GPT-4o. What would happen if GPT-4o were combined with a similar long CoT reasoning approach?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces UnifiedReward-Think, a unified multimodal reward model that enhances vision-language model alignment with human preferences by incorporating explicit Chain-of-Thought (CoT) reasoning. The key idea is to improve the reliability and robustness of reward signals in both visual understanding and generation tasks through multi-step, structured reasoning.\n\nThe proposed training framework involves three stages:\n(1) Cold Start: Distill CoT reasoning traces from GPT-4o using a small image generation dataset to initialize the model with structured reasoning patterns.\n(2) Rejection Sampling: Use large-scale unified preference data to fine-tune the model on correctly reasoned examples, improving generalization across tasks.\n(3) GRPO-based Reinforcement Fine-Tuning: Employ Group Relative Policy Optimization on incorrectly reasoned samples, with verifiable format and accuracy rewards, to encourage exploration and correction of reasoning trajectories.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper studies a timely topic, and the research problem is valuable. Aligning vision models with human preferences is an important and fundamental problem in multi-modality learning.\n2. The paper writing is in general clear and easy to follow. The paper's structure is very clear. The examples provide in the paper are useful in helping understanding the proposed method.\n\nWeaknesses\n1. In Section 3.3, it is not clear how to know the correctness of the model-generated answers in rejection sampling, especially it considers both vision understanding and generation tasks.\n2. In Table 2, the performance improvement of the proposed method seems to be marginal.\n3. I have concerns regarding the novelty of the proposed method, since the combination of warm up + rejection sampling seems to be very standard in solving many problems.\n4. The paper could be more clear about what the \"reward\" is intended to be used for. It seems that the reward will be used for multiple tasks. If this is the case, then the paper should emphasize why the learning can be generalized into different tasks.\n\nQUESTIONS:\n1. How to guanrantee the generalizability of the resultant reward model?\n2. In Section 3.3, how to know the correctness of the model-generated answers in rejection sampling?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces UNIFIEDREWARD-THINK, a novel multimodal reward model designed to enhance the reliability and accuracy of reward signals for both visual understanding and generation tasks. The authors argue that current reward models are limited by their shallow reasoning processes and propose to integrate explicit long CoT reasoning to address this issue. The core of their methodology is a three-stage training pipeline: a \"Cold Start\" stage to learn the CoT format from distilled GPT-4o data, a \"Rejection Sampling\" stage to generalize CoT reasoning across various vision tasks, and a final \"GRPO\" stage to fine-tune the model on challenging cases using verifiable rewards. The authors present extensive quantitative and qualitative results demonstrating that their approach not to only improves the accuracy of reward signals with explicit CoT reasoning but also enhances the model's implicit reasoning capabilities, leading to superior performance even without the CoT.\n\nSTRENGTHS AND WEAKNESSES:\nPros\n\n1. The paper addresses a timely and significant problem in the field of large language models: the need for more reliable and interpretable reward models. The core idea of leveraging CoT reasoning to improve multimodal reward modeling is innovative.\n\n2. The proposed three-stage training pipeline is logical and systematically addresses the challenges of incorporating CoT reasoning into reward models. The use of distillation for the cold start, followed by rejection sampling and reinforcement learning, is a logical and effective approach to progressively build and refine the model's reasoning abilities. And the ablation studies demonstrate the contribution of each stage to the final performance of the model.\n\n3. The experimental results are thorough and provide strong evidence for the effectiveness of the proposed method. The authors evaluate their model on a wide range of benchmarks for both image and video understanding and generation.\n\nCons:\n\n1. While the reported performance in this paper is generally strong, the work lacks clear technical novelty. The proposed method consists of three components: supervised fine-tuning (SFT), rejection sampling, and GRPO, all of which are well-established post-training techniques. When properly applied, these methods are indeed effective. However, this raises the question: What is the actual technical contribution of the paper?\n\n2. The use of format and accuracy rewards feels somewhat superficial. Although the paper claims to promote longer chain-of-thought (CoT) reasoning, it lacks a rigorous evaluation of CoT quality. There is no explicit reward signal for distinguishing between high-quality and faulty CoTs, nor is there any post-hoc assessment of CoT correctness in the experiments. If a CoT contains factual errors or hallucinations, can it still be considered \"good\"? A high final accuracy alone does not necessarily indicate that the CoT itself is of high quality.\n\n3. The results reported in Table 2 are somewhat puzzling. In several cases, the base model outperforms the trained model on all three Tau metrics, which undermines the claimed effectiveness of the proposed training approach.\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-06"}