Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning
Authors: Yibin Wang
Zhimin Li
Yuhang Zang
Chunyu Wang
Qinglin Lu
Cheng Jin
Jiaqi Wang
Venue: NeurIPS 2025 poster ()
Keywords: reward model
reinforcement leaning
chain-of-though reasoning
Primary area: applications
TL;DR: This paper the first unified multimodal CoT-based reward model, capable of multi-dimensional, step-by-step long-chain reasoning for both visual understanding and generation reward tasks.

Abstract: Recent advances in multimodal Reward Models (RMs) have shown significant promise in delivering reward signals to align vision models with human preferences. However, current RMs are generally restricted to providing direct responses or engaging in shallow reasoning processes with limited depth, often leading to inaccurate reward signals. We posit that incorporating explicit long chains of thought (CoT) into the reward reasoning process can significantly strengthen their reliability and robustness. Furthermore, we believe that once RMs internalize CoT reasoning, their direct response accuracy can also be improved through implicit reasoning capabilities. To this end, this paper proposes UnifiedReward-Think, the first unified multimodal CoT-based reward model, capable of multi-dimensional, step-by-step long-chain reasoning for both visual understanding and generation reward tasks. Specifically, we adopt an exploration-driven reinforcement fine-tuning approach to elicit and incentivize the model's latent complex reasoning ability: (1) We first use a small amount of image generation preference data to distill the reasoning process of GPT-4o, which is then used for the model's cold start to learn the format and structure of CoT reasoning. (2) Subsequently, by leveraging the model's prior knowledge and generalization capabilities, we prepare large-scale unified multimodal preference data to elicit the model's reasoning process across various vision tasks. During this phase, correct reasoning outputs are retained for rejection sampling to refine the model (3) while incorrect predicted samples are finally used for Group Relative Policy Optimization (GRPO) based reinforcement fine-tuning, enabling the model to explore diverse reasoning paths and optimize for correct and robust solutions. Extensive experiments confirm that incorporating long CoT reasoning significantly enhances the accuracy of reward signals. Notably, after mastering CoT reasoning, the model exhibits implicit reasoning capabilities, allowing it to surpass existing baselines even without explicit reasoning traces.

Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuniin
Yibin Wang Zhimin Lit; Yuhang Zang": Chunyu Wang', Qinglin Lu Cheng Jin Jin Jiaqi Wang
College of Computer Science and Artificial Intelligence, Fudan University,
Shanghai Innovation Institute "Shanghai AI Lab, "Hunyuan, Tencent
https://codegoatt24
(a) Qualitative Case Study (Image Generation)
Which of the following images better?
Caption: [A wearing brown cap looking sitting his computer with black and brown dog resting next to him on the couch.]
Image I is better
Uni Riddewwoddd
<thinks
Image
Image 2:
Sementic consistency:
Image (7/10): The man wearing a cap and is sitting with dog. but the dog not black and brown described.
The setting
the computer not clearty visible. wearing
Image (9/10):
cap
clearly using computer. The dog is black and brown, matching the description. The setting consistent with the caption. Aesthetics
(b) Quantitative Comparison Results
Image has worm, inviting polette and good composition, with cozy atmosphere. Image (9/10): The Image is well-composed with good and the creating realistic
Image
UnifeedReward +think (Ours)
lighting and focus
appealing 3. Authenticity.
Image (6/10): The Image has more artistic. ilustrattie which
realism.
Image (9/10): Image appears realistic detailed, with attention to the textures the cap, dog fur, and computer.
VLRemidd
GerAl_Inngge Insige
GenAl Vide
Videoten
SharesPTydde
Total Image 74846m21
Image 94949x27
</thinko
Uni Undeeedwwrdd
Colld Start
RRaj. Sampling
GRPO
Snnwer-Image better:/answer
Figure 1: Overview of Comparison Results. (a) Our method enables multi multi-diment dimensional long CoT reasoning to improve reward signal accuracy. (b) Extensive quantitative results demonstrate our superiority in both vision understanding and generation reward tasks.
Abstract
Recent advances in multimodal Reward Models (RMs) have shown significant promise in delivering reward signals align vision models with human preferences. However, current RMs are generally restricted to providing direct responses or engaging shallow reasoning processes with limited depth, often leading inaccurate reward signals. We posit that incorporating explicit long chains of thought (CoT) into the reward reasoning process can significantly strengthen their reliability and robustness. Furthermore, we believe that once RMs internalize CoT reasoning, their direct response accuracy can also improved through implicit reasoning capabilities. To this end, this paper proposes UNIFTEDR EWARD- THINK, the first unified multimodal CoT--based reward model, capable of multi--iimenionnal, step-by-step long-chain reasoning for both visual understanding and generation reward tasks. Specifically, we adopt an exploration-driver reinforcement fine- tuning approach to elicit and incentivize the model's 's latent complex reasoning
'Equal contribution. 'Project lead, 'Coreesponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

ability: (1) We first use a small amount of image generation preference data to distill the reasoning process of GPT-4o, which then used for the model's 's cold start to learn the format and structure of CoT reasoning. (2) Subsequently, by leveraging the model's s prior knowledge and generalization capabilities, we prepare large-scale unified multimodal preference data to elicit the model's reasoning process across various vision tasks. During this phase, correct reasoning outputs are retained for rejection sampling to refine the model (3) while incorrect predicted samples are finally used for Group Relative Policy Optimization (GRPO) based reinforcement fine tueeetunngg, enabling the model to explore diverse reasoning paths and optimize for correct and robust solutions. Extensive experiments confirm that incorporating long CoT reasoning significantly enhances the accuracy of reward signals. Notably, after mastering CoT' reasoning, the model exhibits implicit reasoning capabilities, allowing it to surpass existing baselines even without explicit reasoning traces.
Introduction
In recent years, multimodal reward models (RMs) WWan et al. 2024 2025d Zang et al. 2025 Xiong et al. 2024 He et al., 2024 Xu et al.
2024 Lu et al. 2025a Li et al. 2025 Wang et al.
2025c have excelled at aligning vision model outputs with human preferences, providing crucial
reward signals to guide model training [Wang al. 2024 2025d Liu et al., 2225a Ouyang et al.
2022 Rafailov ct al. 2023 Schulman et al. [017]] and inference Gulcchre et al. [2033] Snell et al.
2024
Traditional reward models are typically trained on large-scale mlmman-annotaie
preference
data through supervised fine-tuning (SFT). At test time, most methods [He et al. 2024 Liu et al.
2025a Xu et al. 2024] directly assign scores or provide pairwise ranking for vision model outputs, relying on the Lndgl and intuitions acquired from the training data. While effective, these methods tend to lack interpretability, which makes it difficult for users to understand the underlying reasoning process behind the assigned scores or rankings. To this end, recent studies (Wang et al.
2025d Xiong et al. 2024 Wang et al. 2024 Li al. 2025 leverage the generative capabilities of Visual-Language Models (VLMs), enabling RMs to provide concise justifications alongside the assigned reward signals. Despite their success, their reasoning processes often lack rigorous logical structure and the capacity for multi multi-dimensional deep analysis, which may result in inaccurate reward signals in complex scenarios or misguided conclusions arising from flawed reasoning processes.
In light of these issues, we posit two key hypotheses: (1) Incorporating explicit long Chains-of- Thought (CoT) into the reward reasoning process is essential for significantly enhancing RM's reliability and robustness; (2) Once the model internalizes this ability, the accuracy of its directly provided reward signals, even without CoT reasoning traces, can also be improved by leveraging its implicit logical thinking capabilities. However, equipping RMs with CoT reasoning using traditional training schemes like SFT poses a highly challenge due to the scarcity of large-scale CoT reward data, as manual annotation requires substantial human resources and time.
In this work, argue that this challenge can be effectively addressed, VLMs inherently possess prior knowledge of complex reasoning; what is needed is an effective strategy to elicit and incentivize this capability (Guo et al.) (025). Therefore, this paper proposes UNIFIED REwARD- THINK, the first unified multimodal CoT--based reward model, capable of performing multi-dimensional step-by-step long-chain reasoning across both visual understanding and generation reward tasks. The core idea is to activate the model's latent long-chain reasoning capabilities through limited CoT reward data and to progressively reinforce and refine this capability through exploration-driver reinforcement fine-tuning that optimizes for accurate and robust reasoning patterns. Specifically, our training pipeline, as shown in Fig. consists of three stages: (1) Cold Start. We first use a small amount of labeled image generation preference data to distill the reasoning process from GPT-4o [Hurst et al. 2024), filtering for reasoning traces whose final answers align with the ground-truth (GT) labels. These samples serve to cold-start the model training, enabling it to learn the structure and format of long CoT reasoning. (2) Rejection Sampling. Next, we prepare large-scale unified multimodal labeled preference data to incentivize the model's CoT reasoning outputs across various visual reward tasks. Reasoning trajectories whose final answers match the GT label are retained for rejection sampling, reinforcing the distribution of correct reasoning patterns. (3) Group Relative Policy Optimization (GRPO). Finally, incorrectly reasoned samples are leveraged for GRPO-based reinforcement fine-tuning, enabling the model to explore diverse reasoning paths and optimize toward desirable outcomes defined by our

---

verified rewards (i.e. format and accuracy rewards). Unlike SFT, which merely imitates predefined answers, GRPO promotes trial-and-error learning by evaluating and refining the model's reasoning outputs based on verified rewards, thus encouraging deeper reasoning discovery rather than passive memorization, Notably, we also propose a multidimensional CoT reward scoring strategy to address a widely observed issue, Le., inconsistency between reasoning and final decisions, where reasoning may appear implausible despite a correct prediction. Specifically, as shown in Fig. | by scoring each candidate across various task-relevant dimensions and aggregating the scores from each dimension into a final decision, our approach enforces alignment between reasoning and outcomes, enhancing both the accuracy and reliabilitty of the reward signal. This design allows to rely solely on the final answer when filtering data during cold-start and rejection sampling, and applying reward supervision in GRPO, while implicitly ensuring the correctness of the reasoning process.
Extensive experiments demonstrate that incorporating long CoT reasoning significantly improves the accuracy and reliability of reward signals. Remarkably, experimental results also prove that once the model internalizes the CoT reasoning ability, it also exhibits strong implicit reasoning capabil- ities: even when providing direct reward outputs without explicit reasoning traces, it consistently outperforms existing baselines across all vision reward tasks.
We propose the first unified multimodal CoT reward model, UNIFIEDREWARD: THINK, capable of multi- dineii-dmmessonnll, step-by-step long-chain reasoning across both visual under- standing and generation tasks; (2) We demonstrate that explicit long CoT reasoning substantially enhances reward model reliability, and once mastered, also strengthens implicit reasoning, enabling more accurate reward signals even without explicit reasoning traces; (3) Extensive experiments validate the superiority of our method compared with existing baselines across all vision reward tasks. We hope our work can unlock reward models' reasoning potential to enhance interpretabillty. generalization, and alignment, enabling more trustworthy and human-alignes reward signals for multimodal generation and understanding.
Contributions: (1)
2 Related work
2.1 Multimodal reward models
Multimodal reward models have become increasingly important for aligning vision understanding and generation models with human preferences. A dominant approach is to fine-tune visual-language models (VLMs) [L et al. 2024a, Bai et al. 20256), exploiting their powerful multimodal alignment capabilities to learn human judgment- based reward functions. Earlier studies have explored reward
models on visual generation [Wang et al. 2024 Liu et al. [025a He et al., 2024] and understanding
(Zang et al. 2025 Xiong et al. 20024] tasks. For instance, [Wang et al. [2024] collects human feedback
and constructs human-rated video datasets to train the reward model. LiFT-Critic, which measures
how well the generated videos align with human expectations. [Zang et al. 2025) develops an
effective pipeline for constructing multimodal preference datasets and leverages existing high-quality data to train the reward model, Rewwar, enabling accurate evaluation of visual understanding outputs. However, these reward models are susk-specific. limiting their adaptability across diverse visual understanding and generation tasks. Therefore, [Wang et al. (025d] introduces UnifiedReward, unified reward model capable of assessing both image and video generation and understanding tasks, demonstrating that joint learning across diverse visual tasks can yield substantial mutual benefits.
Despite their effectiveness, these reward models are largely limited to providing direct responses [Liu et al. 2025al Xu et al./ 2024 He et al. 2024 Xu et al. (023) or engaging in shallow reasoning with limited depth [Wang et al. 2024 2025d Xiong et al. 2024] often resulting in inaccurate or unreliable reward signals in complex scenarios or misguided conclusions arising from flawed reasoning processes. To this end, we propose UNIFIEDREWAL SEREEEDREAARRTTHEAE -HHNK the first unified multimodal long CoT-ba reward model, enabling multi-diimesiionll long-chain reasoning for both visual understanding and generation tasks. The core idea is to use reinforcement learning to activate and enhance VLMs' latent reasoning capabilities, which will be discussed in the following section.
2.2 Reinforcement learning
Recently, reinforcement learning (RI. techniques have been extensively used to enhance the reasoning capabilities of Large-Language Models (LLMs), enabling them to effectively solve complex problems
3

---

Stage 1. Cold Start: Learning CoT Reward Format
Cap tion
<think> Semantic consistency:
Image 1 (9/10):
Image 2 (7/10)
Cold Start
Aesthetics:
Distillation
Total Score: Image ... .nImamm
</think> conswer> canae is better</answer>
Images
Stage 2. Rejection Sampling: Unified CoT Reward Generalization Fine-tuning
Images
Videos
Rejection Sampling
/Vi.
CoT Inference
Understanding
Answers
Generation
Stage 3
Unified Preference Training Data
Stage 3. GRPO: Unified CoT Reward Reinforcement Fine-tuning
Group Relative Policy Optimization +
Format Reward <thinko danswer....c/nswer
Incorrectly Predicted Training Data
Reinforcement Fine-tuning
Accuracy Reward ring /id.. X betfer
CoT Reward Completions
Ground Truth
Figure 2: Method Overview. (1) Cold Start: We first distill GPT-4o's reasoning process on a small amount of labeled image generation preference data to initialize the model's CoT reasoning format; (2) Rejection Sampling: Then, we leverage the model's generalization capabilities on large-scate scale unified multimodal preference data to elicit its CoT reasoning process across various vision tasks. Correctly predicted samples whose final answers match the ground-truth are retained for rejection sampling to further refine the model. (3) GRPO: Finally, incorrectly predicted samples are utilized for GRPO-based reinforcement Ane-tuning to further enhance step-by-step reasoning capabilities.
Jaech et al. 2024 Luong et al. 2024 Shao et al.
2024 Yang et al. 2024 Ying et al., 2024 Hui et al. 2024a c Wang et al. 2025a
bj. significant advancement in this helld is Deepseek-R -ero [uo et al. 2025 which introduced a novel approach for developing robust reasoning capabilities using Group Relative Policy Optimization (GRPO). By leveraging rule-based rewards, it enhances reasoning without the need for supervised fine-tuning (SFT), For VLMs, RL has mainly been applied to tasks such as mitigating hallucinations (Sun et al. 2023
2024| Jiao et al. 2024 Zhang et al.
Yu et al. 2024a], aligning models with human preferences (Yu et al. 2024b Zhou et al. 2024
improving visual perception [Liu et al., 2025b). and quality assessment Li et al. 2025
However, the application of RL in multimodal reward models is still under exploration. To address this gap, our work introduces a reinforcement fine-tuning strategy that leverages verifiable rewards with GRPO-based RL to integrate long CoT reasoning, enhancing the accuracy of reward signals by enabling multi-dimensional and step-by-step reasoning processes.
Method
3.1 Overview
This work aims to incorporate long Chain-of-Thought (CoT) reasoning into the reward model's decision-making process to enhance the reliability and robustness of reward signals. However, achieving this with traditional training methods like Supervised Fine-Tuning (SFT) remains highly challenging due to the scarcity of large-scale CoT-ba reward data. This work posits that Visual-

---

A dynamic shot of a robot dog munewvering through complex maze, sencors scarning the enviromment The canera cap tures the dog's &s precise movements, the
advaneed tech technology embedded in its design and the challenges of maze.
Video Generation
Samantic consietancy. Video (7/10 10): The video thows robot maze-like emviromentt but the very dynamic, the
complex. dog's
scanting environment very
(9/10): This video festures rabot moneuvering through complex The degls intricucies
depicled, aligring closely with the caption.
Tamporal coherance
(8/10): The video has smooth transitions and logical flow the dog's movements, making it essy
(9/ LO): video maintains a high of tempural coherence with the doy's movements being and mace's structure being clearly depicted.
Authenticitty
Video (K/ L0): The rabot dog design realistic, with detailed machanical fentures. Howeerr the mare could be more complex.
Video (9/10): The rabol dog and maze are highly defailed and realistic, with the maze's complexity
adding to the authenticlly.
Total score-Video 1: 7eBelkt23/niidd 2: 9.9w9.27
thinks
<anpwer-Viddo 2 betterc/anower>
Figure 3: Qualitative Results of Video Generation CoT Reward Reasoning. Given a pair of videos and the corresponding caption, our model performs quality assessment across semantic consistency, temporal coherence, and authenticity through CoT reasoning.
Language Models (VLMs) inherently possess prior knowledge of complex reasoning; the key challenge lies in devising an effective strategy to elicit and incentivize this capability. Therefore, we take the first step to propose the unified multimodal CoT-based reward model, UNIFIEDREWARD- THINK, adopting exploration-driven reinforcement fine-tuning to activate and refine the VLM's multi-dimensional and sep-by-step long chain reasoning across various vision reward tasks.
Specifically, as shown in Fig. 2 our pipeline includes three key stages: (1) Cold Start: use a small amount of distilled CoT reward data to initialize the reward model with the format and structure of multi-step reasoning (Sec. (.2); (2) Rejection Sampling: utilize large-scale unified preference data to elicit the model's s generalizable CoT reasoning across diverse vision tasks; correctly reasoned samples are retained for rejection sampling to reinforce accurate reasoning patterns (Sec. 3.3); (3) Group Relative Policy Optimization (GRPO): leverage incorrectly reasoned samples for GRPO-based reinforcement fine-tuning to further improve the model's CoT reasoning capabilities (Sec. 3.4). To
further ensure consistency between the reasoning process and final decisions, we design an effective multidimensional CoT reward score strategy (Sec. 3.5
3.2 Cold start: learning CoT reward format
In this work, we hypothesize that VLMs inherently possess the potential for complex, long-chain reasoning. However, due to the absence of exposure to reward modeling during pre-training, they often lack suitable format to articulate such reasoning in this context, which leads to inconsistent or shallow outputs. To address this, we initiate the training with a cold start phase, where a small amount of high-quality CoT reward data distilled from GPT-44 [Furst et al.) 2024 is used to explicitly teach the model the desired reasoning format and structure. Specifically, we begin by preparing a small set of image generation preference data, each consisting of an image pair, a prompt (i.e., instruction and image caption), and the ground-truth (GT) label. These samples are then fed into GPT-4o to generate the detailed long-chain reasoning process and the corresponding final answer. Among the generated samples, we retain only those reasoning trajectories whose final answers align with the GT label, denoted (x.y) where the original input, and .....2...... .r the distilled output, which are subsequently used to cold-start the training. The objective function is defined as:
T
Locodd sturt (0) =
Σ
log (y, x))
where 0 represents the parameters of the reward model. This stage serves to initialize the model's ability to follow structured CoT reasoning format.
3.3 Rejection sampling: unified CoT reward generalization fine-tuning
After learning the CoT reasoning format through cold-start training, we further elicit and expand the model's 's reasoning capabilities to vision understanding and generation tasks. This is inspired by
5

---

prior study Waan et al. 2025d], which highlights the benefits of multi-task joint training: reward reasoning abilities learned in one task can generalize well and effectively enhance performance in other vision tasks. Therefore, we apply rejection sampling to reinforce the learning of correct reasoning patterns to improve both reliability and accuracy in vision understanding and generation tasks. Specifically, we first prompt the model to perform generalization CoT reasoning large-scale unified preference datasets, leveraging its prior knowledge across various tasks (e.g., evaluating video temporal consistency and woooo--uuuttton-nsseee accuracy). In vision generation tasks, each input sample consists of a textual prompt (instruction and caption) and a corresponding image/video pair, whereas in vision understanding tasks, it comprises a textual prompt (instruction and query) along with an image/video. Then, we retain only the reasoning samples whose trajectories lead to a final answer consistent with the ground-trune -ruth label, while discarding incorrect reasoning. The training objective for this stage is similar to Eqn. | but it utilizes the filtered reasoning data of diverse vision tasks (x', y') obtained through rejection sampling.
This process concentrates the training distribution around accurate reasoning patterns and enhances the model's generalization ability actoss diverse visual domains.
3.4 GRPO: unified CoT reward reinforcement fine-tuning
In the rejection sampling stage, subset of challenging data featuring intricate reasoning patterns is filtered, which the model has yet to fully comprehend and master. To ensure the model fully learns the underlying knowledge in the training dataset and further enhances its reasoning ability, we introduce GRPO-based reinforcement fine-tuning [Guo et al. 2023]. Specifically, in GRPO, the policy model To generates multiple candidate responses for a given imput, which are evaluated using predefined verifiable reward functions, providing corresponding reward signals. These signals guide policy updates, encouraging alignment with high-quality reasoning while constraining deviations from the reference model Tee. This approach enables the model to explore diverse reasoning processes, guiding toward the correct reasoning trajectory and improving ability to handle complex scenarios. In the following, we will first introduce the design of verifiable rewards in our work, followed by detailed description of the GRPO training process.
3.4.1 Verifiable reward
In GRPO, verifiable rewards are essential for guiding the model's learning by offering rule-based feedback. In this work, we employ two types of verifiable rewards i.e., format reward and accuracy reward, to ensure both the quality and accuracy of the model's responses:
Format reward ensures that the generated response follows specific reasoning structure, which is critical for maintaining clarity and consistency in the reasoning process. In this case, the response is expected to contain two key tags: (think) and (answer). These tags are used to delineate the model's reasoning process and the final answer, respectively. If both tags are present and properly formatted in the response, the reward Ryme is set 1. Otherwise, the reward R fot is O. This mechanism helps reinforce the importance of structuring the response. Although most VLMMs already exhibit strong instruction- -orllciion-oolowwinng capabilities, the format reward can still serve as safety check to filter out occasional formatting errors, which, though minor, may negatively affect model performance.
Accuracy reward evaluates whether the final answer output o (i.e., Image/Video//nswen Answer X is better), enclosed within the (answer) tag, exactly matches the ground truth. This reward serves as a reliable signal to ensure that the model produces correct answers. If matches the ground truth precisely, the reward is set to 1; otherwise, it is O. Formally, the accuracy reward is defined as:
Race {
if = ground truth otherwise.
This is crucial for reinforcing accurate reasoning and encouraging the model to generate correct responses. Finally, the overall verifiable reward R formulated as:
R = Rm + Race-
By incorporating both the format and accuracy rewards into the GRPO training, we provide the model with explicit feedback that encourages it to generate responses that are both sell-structured and factually accurate.

---

Image Understanding
Can you elaborate on the details of the image?
Answer I: The image shows male ternis player in mid-action, executing serve. He is cressed teal athletic shirt with short sieeves, suitable for sports activity. The player has dark the
curly hair and looking upwards, concentrating left hand. His right arm drown back, holding racket preparation swing ot the ball. The background Features green barrier with the word \"MASTERS\' written across in white lettering. suggesting this professional level tennis tournament.
ball tossing
with
MAST S S
Answer The image capture saptures young black shorts, and in the process serving a tennis ball. The ball in mid-air, and man fully engaged in the game, There are several tenns rackets seattered around the court, indicating that the game is progress. Additionally, there are several chairs placed around the court, suggesting that spectators watching the game....
playing tenris court.
wearing blue
cthinks
Semantic accuracy.
Answer 1 (9/10): Accurately describes the player's attine, action, and background details such as the \"MASTERSY" sign. description of the player's focus and posture is precise.
Answer 2 (s/10): Incorrectly mentions several tennis rackets and chairs around the court, which are not visible in the image.
The shirt color described inaccurately blue instead of teal.
Correctness:
Answer 1 19/10): Logically consistent with the image content, correctly identifying the player's action and attire.
Answer 2 (5/10): Contains factual inaccuracies regarding the presence of multiple rackets and chairs, which are not depicted in
the
Clarity:
Answer 1 (8/10): Clearly and Fluently expressed, providing vivid depiction of the scene.
Answer 2 (7/10): Generally clear but includes incorrect details that could confuse the reader.
Total score: Answer 39+8-26//nnsww 2: 645+7e18:/
Answer betterd/ answers
Video Understanding
What emotions are conveyed the video, and how do visual details support your concept?
Answer 1: The video conveys Sense of happiness and freedom. The boy's big smile and relaxed posture show is enjoying himself. The bright sunlight and the clean, quiet neighborhood penceful
highlighting the carefrea Answer 2: The video captures strong theme youthful freedom and Innocence. The boy's genuine smile, casual clathing, and confident grip on the handlebars suggest both comfort and excitement. The worm sunlight enhances the feeling of joy and security, while the neatly
of childhood.
fences and wide sidewalks- amphasizes safe environment nurtures explonation and independence. These visual elements collectively celebrate simple, unburdened pleasures of childhood.
maintained suburban setting-with
cthinks
Samantic accurecy:
(B/10) The captures emations happiness and freedom well, supported bay's smile and peaceful visual details.
suburban setting, However, it could delve deeper
Answer (9/10) This answer effectively corweys the themes of youthful freedom and innocence, supported by the boy's expression
and the suburban environment. If provides a more comprehansive interpretation.
Correctness:
Answer (7/10)- interpretation correct somewhat simplistic. focuses moinly on the boy's emotions without much depth.
Answer (9/10) The onser is factually correct and provides a masnced understanding of the scene, including the setting and the boy's demeanor.
3. Clarity:
clear easy to understand, with straightforword explanation emotions and setting. Answer (9/10) The onswer very clear and well teltlaatdd, with detailed observations that enhance the interpretation.
(B/10) The
Total score: Answer 1: 847+8:23/vAnswer 2: 949+9:27c/ 1099:277//hii thinks
Sntower-Answer 2 better /answers
Figure 4: Qualitative Cases of Image and Video Understanding CoT Reward Reasoning. Given an image video, a query, and pair of candidate answers, our model performs quality assessment across semanntc accuracy, factual correctness, and clarity through CoT reasoning.
3.4.2 Reinforcement fine-tuning
Given an input x, GRPO first samples N distinct responses (o(1), _02) ...., N) from the previous policy model Tew.. Each response is evaluated using our verifiable rewards, i.e. format and accuracy rewards, resulting in corresponding reward scores {R(() ((()) ..... H( Then, GRPO normalizes these scores and quantifies the relative quality of each response by computing the advantage of each response using the standardized reward:
R(O memna (R()...,
( R(N)))
A
std ..... R(N)))
where 1 quantifies the relative quality of the i-th response in comparison to other candidates within the same sampled group.
Then, GRPO estimates magnitude of the policy update by computing the likelihood ratio of each response under the new policy Te... relative to the old policy Te defined as:
Te (0( x)
(6]
THau (ol1) xx)
7

---

To stabilize training and avoid overly aggressive updates, the ratio is clipped to a bounded interval [1 6, + 61. Moreover, to ensure that the updated policy does not diverge significandly from fixed reference model Tee, KI. divergence penalty term is introduced with weighting factor B3. The final optimization objective is defined as:
Lgrpo = Ex 76
min
clip(r (0) (rip)),,.ll) 8, 1)) - 3 DKL OKK((Te (Tew |l meet]
where X' denotes the set of training sample input and ( || denotes the KL. divergence.
By integrating normalized reward advantages, clipped importance sampling, and reference- regularization, GRPO enables stable and effective policy optimization, driving our model toward generating higher-quality, verifiably correct CoT reasoning paths.
3.5 Multidimensional CoT reward scoring strategy
In the cold-start and rejection sampling stages, we filter data by retaining only those samples whose final answers align with the ground truth, and in the GRPO stage, supervision is likewise applied solely to the final answers. However, both GPT-4o and existing reward models often suffer from inconsistencies between reasoning traces and final decisions, where reasoning may appear implausible even if the prediction is correct, leading to unreliable supervision. To address this issue, we propose a multidimensional CoT reward scoring strategy that tightly couples the reasoning process with the final score. Specifically, the model evaluates each candidate (image, video, or textual response) along multiple task-relevant dimensions, aggregates the dimension- wise scores into a final decision, and thereby enforces consistency between reasoning and outcomes (Figs. | 3 4. By grounding the final prediction in multidimensional and interpretable reasoning processes, our design implicitly ensures reasoning correctness even when data filtering or supervision relies solely on the final answer, as in all training stages, thereby enhancing both the accuracy and reliability of our reward model.
4 Experiments
4.1 Experimental setup
Datasets. For Image Generation, we utilize HPD (25.6K) (Christodoulou and Hachlmann-JJrgenssee
2024]. OIP (7.4K)) EvalMuse (3K) [Han et al. 2024]. all preprocessed by [Wang et al. [025d] as well as OpenAI-40 12i 1 human pateran_nreeeeceeee (6.7K) collected by Rapidat For Video Generation, we employ VideoDPO (10K) al, and Text2Video- Preferences (5.7K), also collected by Rapidata. For Image Understanding. we sample 30K data from LLa 2024). For Video Understanding, we adopt ShareGPTVideo-DPP (17K) (Zhang et al.) 2024bb. In the cold-start stage, we distill image generation CoT reward reasoning samples from GPT-4o, constructing our CeaeeeGGen-----wwwreeeee generation datasets, with the remaining data reserved for the subsequent training stages.
[Xiong et al.,
The input data are randomly sampled from the image
Reward model. We adopt enifiedRewarrd (Wang et al. 2025d) as our base model, which is capable of assessing both image/video generation and understanding. We leverage its strong performance and extensive prior knowledge in visual perception and generation, and further activate its latent capacity for long-chain CoT reasoning-
Training details. For both the cold-start and rejection sampling stages, training is performed with a batch size of 16 gradient accumulation steps, a learning rate of 2.5 10-6, and warm-up ratio of 0.3, using 8 NVIDIA H100 (80GB) GPUs For GRPO, training is conducted with a batch size of 1, a single gradient accumulation step, a learning rate of 1 10-6, and a KL penalty coefficient of 3 = 0.04. The number of generated responses N is set to 8, using 64 NVIDIA H20 (97GB) GPUs.
Evaluations. We evaluate image and video understanding reward assessment VLRewardBench [L al. 2024b) and ShareGPTVidde (Zhang et al. 2024b), using 5K test samples, respectively. For generation evaluation, adopt GenAl-Bench Jiang et al. 2024]. which covers both image and video reward benchmarks. Additionally, we utilize RewarrdBeec [Liu et al.) 2025a to further assess video generation.
http:///www...tooooooo/////oeeee oeteeeeeeeeeeei.ioeeoeoeeeeeeeee -hinarized
8

---

Table 1: Image Understanding Assessment Comparison. We evaluate baselines across different understanding aspects on VLRewardBench.
Overall Accuracy
Macro Accuracy
Models
General
Hallu.
Reason.
Gemini-1.5-HB
50.8
72.5
64.2
67.2
62.5
GPT-4o
49.1
67.6
70.5
65.8
62.4
I.aVA-Critic
47.4
38.5
53.8
46.9
46.6
UnifiedReward
76.5
58.1
65.1
67.5
66.6
Ours (w/ CoT)
77.9
70.5
65.4
73.1
71.3
Ours
78.1
72.7
66.0
73.8
72.3
Table Image and Video Generation Assessment Comparison. "tau" indicates that accuracy is calculated with ties, and "diff" excludes tied pairs when calculating accuracy.
Image Generation
Video Generation
Method
Method
--nnchhench diff
GenAI-Bende
tau
tau
diff
tau
diff
PickScore
53.2
67.2
VideoScore
46.2
70.6
42.1
49.9
HPSv2
51.6
68.4
LiFT
41.2
60.1
40.6
58.3
IewgeReward
47.8
65.0
VisionReward
52.1
73.1
57.4
68.2
VisionReward
46.8
66.4
VideoReward
50.2
73.3
60.1
73.9
UnifiedReward
54.8
70.9
UnwriedReward
60.7
77.2
66.6
79.3
Ours (w/o CoT)
54.1
71.9
Ours (w/o CoT)
57.8
81.6
64.9
Ours
72.5
Ours
82.3
80.5
4.2
Comparison results
The quantitative results, presented in Tabs. and Fig. [ demonstrate the superiority of our model across visual generation and understanding tasks. image and video generation tasks, our model does not account for tie scenarios during evaluation, since such training data. Nevertheless, it consistently outperforms existing methods across all other evaluations. Notably, compared the base model UnifiedReward, incorporating reasoning yields substantial performance gains across all tasks. In particular, our model achieves significant improvements image understanding reward tasks. This is intuitive, as CoT reasoning in vision tasks fundamentally centers on a deeper understanding of the visual content. Additionally, we explore whether our model, after learning long CoT reasoning, can improve the accuracy of direct reward signals, i.e., without explicit CoT reasoning. Experimental results show that the model still outperforms existing methods by leveraging implicit logical reasoning, with only a slight drop in performance compared to explicit CoT reasoning. These results strongly validate the effectiveness and superiority of our approach. Our qualitative results are provided in Figs. and
not included in the
dimensinaan and multi-step
4.3 Ablation studies
Ablation of each training stage. We conduct ablation studies to assess the effectiveness of each training stage. As shown in Tabs. 3 and after the cold start phase, though the model learns the CoT reasoning format, it still struggles with accurate reward prediction. Notably, introducing rejection sampling leads to clear improvements by retaining correctly reasoned samples for supervised fine-tuning, thereby reinforcing desirable reasoning patterns. Further, the GRPO stage yields the most substantial gains, as focuses on previously mispredicted cases, allowing the model to explore multiple reasoning paths and converge on more accurate solutions. These results highlight the complementary roles of each stage and demonstrate how our staged training strategy progressively enhances CoT-based reward modeling.
Ablation of GRPO without CoT reasoning. To further validate necessity learning the CoT reasoning process, we evaluate a GRPO variant that removes CoT and directly optimizes reward predictions based on final answers. shown Tabs. and although this approach yields slight improvements over the baseline, the gains are significantly limited. This suggests that learning

---

Table 3: Ablation Results of Image Understanding Assessment. We conduct ablation experiments under different settings and evaluate them across multiple aspects on VLRewardBench.
Overall Accuracy
Macro
Models
General
Hallu.
Reason.
Accuracy
WnifiedReward
76.5
58.1
65.1
67.5
66.6
+GRPO (w/o CoT)
77.8
59.1
65.5
69.0
67.4
UnwfiedReward
76.5
58.1
65.1
67.5
66.6
+cold start
76.0
56.7
65.2
66.9
66.0
+ejeecioon sampling
77.6
64.9
65.4
72.1
69.3
+GRPO(Ours)
78.1
72.7
66.0
73.8
72.3
Table 4: Ablation Results of Image and Video Generation Assessment. "tau" indicates that accuracy is calculated with ties, and "diff" excludes tied pairs when calculating accuracy.
Image Generation
Video Generation
Method
Method
GenAI-Bench diff
GenAI-Bench
VideoGen-Reward
tau
tau
diff
tau
diff
Unif
54.8
70.9
UnifiedReward
60.7
77.2
66.6
79.3
+GRPO (w/o CoT)
54.1
71.3
+GRPO (w/o Co'T)
56.7
78.4
64.8
79.5
UnifiedReward
54.8
70.9
UnifiedReward
60.7
77.2
66.6
79.3
+cold start
68.6
+cold start
75.3
76.8
+rejection sampling
72.0
+rejection sampling
78.9
79.7
+GRPO(Ours)
72.5
+GRPO(Ours)
82.3
80.5
from final answers alone fails to teach the model the underlying reasoning process. In contrast, CoT-based GRPO guides the model to explore multiple reasoning trajectories and gradually converge toward the correct path, leading to deeper understanding and more robust generalization. These results show that the effectiveness of our GRPO-baser reinforcement fine-tuning stems from explicitly modeling the reasoning process, rather than simply reinforcing the final answer.
Limitations and future works
While our method introduces form CoT reasoning improve reward modeling, this inevitably increases inference time during reasoning. However, we show that once the model has mastered CoT reasoning, it can leverage implicit reasoning enhance answer accuracy generating CoT traces. This suggests strong internalization of the reasoning process. Besides, although our reinforcement fine-tuning strategy successfully activates the model's latent long CoT reasoning ability using only small amount of high-quality data, the prior study (Yue et al. 2025 has shown that reinforcement learning cannot fundamentally extend model's capability: it can only amplify the potential already acquired during supervised fine-tuning (SFT). Therefore, to further push the boundaries of CoT-based reward reasoning, scaling up high-quality CoT supervision still remains a promising direction.
without explicitly
Conclusion
We propose UNIFIEDEWARRDTHHING the first unified multimodal CoT reward model capable of multi-dimensional. step-by-step reliable reward reasoning for both visual understanding and generation tasks. Specifically, we adopt an exploration-driven reinforcement fine-tuning approach to elicit and incentivize the model's latent complex reasoning ability, including three key stages: cold start, rejection sampling, and GRPO-based reinforcement fine-tuning. Our experiments demonstrate that CoT reasoning not only improves the accuracy and robustness of reward signals but also equips model with strong implicit reasoning capabilities, enabling superior performance even without explicit CoT outputs. We hope our work can unlock reward models' reasoning potential to enhance interpretability and generalization, enabling more trustworthy and human-aligned reward signals for multimodal understanding and generation.
10

---

7 Acknowledgments
This work was supported by the National Natural Science Foundation of China (Grant No. 62176064), AI for Science Foundation of Fudan University (FudanX24A1023), and Shanghai Innovation Institute.
References
Bai, K. Chen, X. Liu, J. Wang. W. Ge, S. Song, K. Dang, P. Wang. S. Wang, J. Tang, et al. Qwen2. 5-vl technical report. arXiv preprint XXX2:2502.11323 2025a.
S. Bai, K. Chen, X. Liu, J. Wang. W. Ge, S. Song, K. Dang. P. Wang, S. Wang, J. Tang, al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025b.
D. Christodoulou and M. Kuhlmann- Jorgnnnsnn.. Finding the subjective truth: Collecting 2 million votes for comprehensive gen-ai model evaluation. arXiv preprint arXiv:2409.1 2024.
C. Gulcehre, T. L. Paine, S. Srinivasan, K. Konyushkova, L. Weerts, A. Sharma, A. Siddhant, A. Ahern, M. Wang, C. Gu, et al. Reinforced self-training (rest) for language modeling. arXiv 2023.
preprint ar arXiv:2308.0890
D. Guo, D. Yang. H. Zhang, Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, al. Deepseek-r1: Incentivizing reasoning capability in iims via reinforcement learning. arXiv preprint arXiv:2501. /29488 2025.
S. Han, H. Fan, Fu, L. Li, T. J. Cui, Y. Wang, Y. Tai, J. Sun, C. Guo, et al. Evalmuse-40k: A reliable and fine-grained benchmark with comprehensive human annotations for text-to-image generation model evaluation. arXiv preprint arXiv:2412. 18/50, 2024.
X. He, D. Jiang, G. Zhang, M. Ku, A. Soni, S. Siu, H. Chen, A. Chandra, Z. Jiang, A. Anulraj, K. Wang, Q D. Do, Y. Ni, B. Lyu, Y. Narsupalli, R. Fan, Z. Lyu, Y. Lin, and W. Chen. Videoscore: Building automatic metries to simulate fine-grained human feedback for video generation. arXiv preprint arXiv:2406.1522 15252, 2024.
B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12/1 12186, 2024.
A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda,
A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276. 2024.
A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, al. Openai ol system card. arXiv preprint arXiv:2412.1672 2024.
D. Jiang, M. Ku, T. Li, Y. Ni, S. Sun, R. Fan, and W. Chen. Genai arena: An open evaluation platform for generative models. arXiv preprint arXiv:2406. 044424485,4 2024.
F. Jiao, G. Guo, X. Zhang, N. F. Chen, S. Joty, and F. Wei. Preference optimization for reasoning with pseudo feedback. arXiv preprint arXiv:2411.16345 2024.
Y. Kirstain, A. Polyak, U. Singer, S. Matiana, Penna, and O. Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. NeurlPS, (6:36652-36665 2023.
B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang. Y. Li, Z. Liu, and C. Li. Llava- onevision: Easy visual task transfer. arXiv preprint 2tti::20080332266 2024a.
L. Li, Y. Wei, Z. Xie, X. Yang, Song, P. Wang, C. An, T. Liu, S. Li, Y. Lin, et Vlreward- bench: A challenging benchmark for vision-language generative reward models. arXiv preprint arXiv:2411. 17451, 2024b.
W. Li, X. Zhang, S. Zhao, Y. Zhang, Li, Zhang, and J. Zhang. Q-insight: Understanding image quality via visual reinforcement learning. arXiv preprint N...i:25533.2667. 2025.
J. Liu, G. Liu, J. Liang, Z. Yuan, X. Liu, M. Zheng, X. Wu, Q Wang, W. Qin, M. Xia, X. Wang, X. Liu, Yang, P. Wan, D. Zhang, Gai, Yang, and W. Ouyang. Improving video generation with human feedback. arXiv preprint 2511.32501.13918 2025a.

---

R. Liu, H. Wu, Z. Ziqiang, C. Wei, Y. He, R. Pi, and Q. Chen. Videodpo: Omni-preference alignment for video diffusion generation. arXiv preprint arXiv:2412.14167, 2024.
Z. Liu, Z. Sun, Y. Zang, X. Dong, Y. Cao, H. Duan, D. Lin, and J. Wang. Visual-rft: Visual reinforcement finettuning arXiv preprint arXiv:2503001785, 2025b.
T. Q. Luong, X. Zhang, Z. Jie, P. Sun, X. Jin, and H. Li. Reft: Reasoning with reinforced fine-tuning, arXiv preprint arXiv:2401 0000000077,73 3, 2024.
L. Ouyang, Wu, X. Jiang. D. Almeida, C. Wainwright, Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. NewrIPS, 35: 17730-27744, 2022.
R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly reward model. NeurIPS, 36:53728-53311 2023.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: arXiv: 170.066347, 2017.
Z. Shao, Wang. Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Li, Y. Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03322 2024.
C. Snell, J. Lee, K. Xu, and A. Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314 2024.
Z. Sun, S. Shen, S. Cao, H. Liu, C. Li, Y. Shen, C. Gan, L.-Y. Gui, Y.-.. Wang, Y. Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint Xiv:2309.115255 2023.
S. Wang, Z. Guo, C. Lu, and J. Yan. Fractional langevin dynamics for combinatorial optimization via polynomial-time time escape. In Advances Neural Information Processing Systems, 2025a.
S. Wang, H. Xu, Y. Zhang, J. Lin, C. Lu, X. Wang. and W. Li. Where paths collide: comprehensive survey of classic and learning- based multi-agent pathfinding. arXiv preprint arXiv:2505. 19219, 2025b.
Y. Wang. Z. Tan, J. Wang. X. Yang, C. Jin, and H. Li. Lift: Leveraging human feedback for text-to-video model alignment. arXiv preprint arXiv:2412.04814 2024.
Y. Wang, Z. Li, Y. Zang, Y. Zhou, J. Bu, C. Wang, Q. Lu, C. Jin, and J. Wang. Pref-grpo: Pairwise preference howard-based grpo for stable text-to-image reinforcement learning. arXiv preprint orXiv:2508.2075) 2025c.
Y. Wang, Y. Zang, H. Li, C. Jin, and W. Jiaqi. Unified reward model for multimodal understanding and generation. arXiv preprint MxXiv:5503.05236 2025d.
Xiong, X. Wang, Guo, Q. Ye, H. Fan, Q. Gu, H. Huang, and C. Li. Llava-critic: Learning to evaluate multimodal models. arXiv preprint 1rX.v:2410.02772, 2024.
J. Xu, X. Liu, Y. Wu, Y. Tong, Q. Li, M. Ding, J. Tang, and Y. Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. NeurIPS, 36:15903-15935, 2023.
Xu, Y. Huang, J. Cheng, Yang, Xu, Y. Wang, W. Duan, Yang, Q. Jin, S. Li, J. Teng, Z. Yang, W. Zheng, X. Liu, M. Ding, X. Zhang, X. Gu, S. Huang, M. Huang, J. Tang, and Y. Dong. Visionreward: Fine-grad graingd muulti-dimensionll human preference learning for image and video generation. arXiv preprint arXiv:2412.2100 2024.
A. Yang, B. Zhang, B. Hui, B. Gao, B. Yu, C. Li, D. Liu, J. Tu, J. Zhou, J. Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via iepfoimpro arXiv preprint arXiv:2409. 12122, 2024.
H. Ying. S. Zhang, L. Li, Z. Zhou, Y. Shao, Z. Fei, Y. Ma, J. Hong, K. Liu, Z. Wang, et al. Internlm-math: Open math large language models toward verifiable reasoning. arXiv preprint 202.032402.06333, 2024.
12

---

T. Yu, Y. Yao, H. Zhang, T. He, Y. Han, G. Cui, J. Hu, Z. Liu, H.-T. Zheng, M. Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In CVPR, pages 13807-13816, 2024a.
T. Yu, H. Zhang, Y. Yao, Y. Dang, D. Chen, X. Lu, G. Cui, T. He, Z. Liu, T.-S. Chua, et al. Rlaif-v: Aligning milms through 1125 2024b.
ai feedback for super gpt-4v trustworthiness. arXiv preprint
Y. Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang, S. Song, and G. Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint arXiv:2504. 13837, 2025.
Y. Zang, X. Dong, Zhang, Y. Cao, Z. Liu, S. Ding, S. Wu, Ma, H. Duan, W. Zhang, Chen, A simple yet effective multi-modal reward
D. Lin, and J. Wang. 2eopllm-xxooooe225-rree.. model. arXiv preprint 2231:2550..2368, 2025.
K. Zhang, Li, Dong, Xu, Zhang, J. Su, Liu, and Z. Jin. Codedpo: Aligning code models with self generated and verified source code. arXiv preprint arXiv:241000560,, 2024a.
R. Zhang, L. Gui, Z. Sun. Y. Feng, K. Xu, Y. Zhang, D. Fu, C. Li, A. Hauptmann, Y. Bisk, et al. Direct preference optimization of video large multimodal models from language model reward. arXiv preprint arXiv:2404.0/255 2024b.
Y. Zhang, S. Wu, Y. Yang. J. Shu, J. Xiao, C. Kong, and J. Sang. ol-coder: ol replication for coding. arXiv preprint 2rXiv:2412.00155 2024c.
Y. Zhou, C. Cui, R. Rafailov, C. Finn, and H. Yao. Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint 1111 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction have clearly stated the paper's s contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work discussed in the Supplementary Materials.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
This paper does not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: This paper fully disclose all the information needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: We have released all the code in the supplementary and prepared a detailed file "readme. md" for reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips. //nips.cc/ppl wwwppbblccgguiicuceccccceoeooopp /eegguddsss/Ceebbbinninn
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: This paper specify all the training and test details in Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: This paper provides sufficient information the computer resources in Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't ' make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:p/://.aaepp....ccpppp/lclla
Answer: [Yes]
Justification: The research conducted in the paper conforms, every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper has discussed both potential positive societal impacts and negative societal impacts of the work performed the Supplementary Materials.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators or original owners assets are properly credited and the license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
20

---

Table 5: Ablation of Rejection Sampling Training Stage. We conduct an ablation study to assess the effect of the rejection sampling stage on different benchmarks.
Method
VLReward
GenAI-Image
GenAI-Vides
VideoGen
Baseline
67.5
70.9
77.2
79.3
UnfftedReward-Thing
73.8
72.5
82.3
80.5
w/o Rejection Sampling
70.4
71.6
79.2
79.8
A Further methodological insights
Cold start only with image generation preference data: why it works. Our experiments demon- strate that using a small amount of high-quality image generation CoT reward reasoning data, instead of distilling data for every task, is sufficient for the model to learn the CoT reasoning format and structure across all visual reward tasks. The underlying reason lies in the fact that video tasks can be seen as multi -mmge understanding problems. Video frames, like images, also require recognizing objects, spatial relationships, and context. Once the model masters CoT reasoning for images, can naturally extend this reasoning to videos by leveraging its inherent prior knowledge of temporal dynamics and sequential visual understanding. Therefore, by learning CoT reasoning from images, the model can seamlessly generalize to both static and dynamic visual tasks, eliminating the need for separate distillation for each task.
Rejection sampling for unified reward generalization fine-tuning: why we need it. After the cold start, the model has already internalized the format and structure of CoT reasoning. With prior knowledge across tasks, can generate accurate CoT-basee reward analyses for many simple scenarios. However, directly applying GRPO to the entire training set would be inefficient and computationally costly. Besides, GRPO offers limited gains on samples that the model has already mastered. Therefore, performs well on. This not only reduces training cost but also reinforces the distribution of correct reasoning patterns by prioritizing high-confidence outputs. More challenging or ambiguous samples are passed to the GRPO stage, where the model explores diverse reasoning trajectories and gradually learns to prefer more accurate solutions. Ablation results of this training stage are provided in Tab, [
first apply rejection sampling to filter out cases that the model already
Why we trust the CoT reasoning when the final answer is correct during GRPO. In MLLMs, common failure mode during CoT reasoning is the disconnect between the reasoning steps and the final answer: models may produce plausible reasoning chains but ultimately rely on shortcuts intuition to generate the conclusion. To mitigate this, explicitly structure the CoT process by having the model score each image across multiple dimensions and then aggregate these scores to derive the final decision. This enforces step-by-step alignment between intermediate reasoning and outcome, ensuring that a correct final answer emerges from a coherent and interpretable reasoning process. As a result, in GRPO, verifying the correctness of the final answer implicitly validates the reasoning trajectory, offering a principled yet efficient way to supervise complex CoT generation.
Why learning can be generalized to different tasks. In this work, unified multimodal CoT reward model is designed to support both vision generation and understanding tasks across both images and videos. Although the downstream tasks may differ, the input-output structure to the reward model is unified: in all cases, the model receives a combination of visual content (image or video) and caption textual response, and outputs reward judgment signal indicating the relative quality or preference between candidate outputs. Because of this consistent structure, the reward model learns to evaluate visual-text pairs in a task-agnostic manner, allowing to generalize across different modalities and task types. This design enables us to train the reward model jointly on diverse datasets and apply it effectively across vision reward tasks.
B Robustness on different baselines
To further demonstrate the robustness of our method across different base models, we additionally train UNIFFEDREWARD-THNN on Qwen2.5-VL-7b [Bai et al. 2025a As shown in Tab. 6 lever- aging the stronger capability of the base model, the Qwen2.5-V based model achieves consistent improvements.
21

---

Table 6: Performance Comparison on Different Backhones. We compare the performance of UNIFIEDREWARD- THINK trained on LLaVA-OneVision and Qwen2.5-VL.
Gennl-Benchh
VLRewardBench
Overall Accuracy
Macro Accuracy
UnffeddReaaddTTiink
Image
Vidco
General
Hallo.
Reason.
LLaVA OneVision 7B Qwen2.5.VI.n77
72.5
82.3
78.1
72.7
66.0
73.8
72.3
Table Exploring CoT Distillation from Different Models. We compare the impact of distilling CoT samples from GPT-4o versus Qwen2.5-VL.
Method
Reward
GenAT-Image
GenAT-
VideoGen
Baseline
67.5
70.9
77.2
79.3
Ours (distilliigg GPT-4o)
73.8
72.5
82.3
80.5
Ours (distilling Qwcn2.5-V1.-7hh)
73.2
73.6
81.7
80.2
C Role of distilled CoT samples on cold-start stage
In our pipeline, the cold-start CoT data serves primarily to teach the model the structure and format of CoT--style reasoning, rather than to enhance reward decision quality. The generalization and strengthening of the model's reward discrimination capability are achieved through the subsequent stages of rejection sampling and GRPO, which operate on large-scale unified multimodal vision preference data and iteratively reinforce correct reasoning trajectories. Our experimental results in Tab. 7 also show that even when using distilled CoT data from weaker models (Dwen2.5VL-77. instead GPT-4o for cold-start, the model can still achieve comparable performance after subsequent reinforcement through rejection sampling and GRPO.
D Details of our cold-start dataset: ImageGen-Cof-Rewwarde
In this work, we distill a small set CoT reward reasoning samples for image generation from GPT- 4o (Hurst et al. 2024), which are used to cold-start our model and teach it the format and structure of CoT reasoning. Specifically, we randomly select 10K samples from HPD, EvalMuse, and OIP datasets and input them into GPT-4o for distillation. To ensure consistency between the long-char mang-hhiin reasoning process and final decision, we carefully designed the CoT reasoning format: the model is prompted to independently score each image along multiple evaluation dimensions, and the final judgment is made based on the aggregated scores across all dimensions. This structured approach mitigates inconsistencies between intermediate reasoning and final conclusions. An example prompt template shown in Fig. 5) We retain only those CoT outputs whose final conclusion matches the ground truth preference label. After filtering, obtain total of 5K high-quality CoT reward reasoning samples for image generation, constructing our cold-start dataset: ImageGen-CoT-Rewarrd-
E
More experimental details
E.1
Base model.
This work adopts UnifiedReward (Wang al. 2025d as our base architecture, which unifies image/video generation and understanding reward tasks, demonstrating the mutual benefits of multi- task learning and achieving strong performance across various vision reward benchmarks. However, its reasoning is limited to direct responses or shallow thinking, lacking the capability for long and structured CoT reasoning, which may lead to lower accuracy and weaker interpretability scenarios. Inspired by this work, we build upon UnifiedRewart and further activate its latent long CoT reasoning capabilities across different vision reward tasks, aiming enhance the accuracy and robustness of the reward signals.
complex
22

---

E.2 Reward model baselines.
We compare our method against a series of strong reward model baselines across both image and video domains, covering generation and understanding tasks.
PickScore [Krstain et al., 2023] is a text-to-image preference model that integrates CLIP-based vision-language features with reward modeling strategy inspired by InstructGPT. It is trained on the Pick-a-Pic dataset to align image generation outputs with human preferences.
HPSv2 Christodoulou and Kuhlmann-Jorgen Jorgensen 2024] builds upon CLIP and is fine-tuned using the HPD_ ___2 [Christodoulou and Kuhlmann- JJrgenssee 2024) to predict human preferences over generated images. It demonstrates strong performance in pairwise ranking tasks and serves
representative image generation reward baseline.
ImageReward [Xu et al. 2023) is trained on a large-scale preference dataset containing 137k human expert comparisons through both rating and ranking. It is specifically designed capture subtle aspects of human preferences in texl-to-image generation quality.
LLa VA-Criiii [Xiong al. 2024] extends large language models for evaluating image understanding through a critic-style framework. It is trained on high-quality diverse criteria such accuracy, relevance, and hallucination, supporting both pointwise and pairwise evaluation.
followiigg data covering
VisionReward Xu et al. for both image and video domains. It trains separate reward models tailored human preferences collected through carefully curated datasets, offering strong baselines for visual content assessment.
2024
introduces fine-grained, multi-dimensional evaluation framework
VideoScore [He al., 2024 focuses on assessing video generation quality. is trained on the scores over 37.6K videos, each evaluated across multiple aspects including fidelity, consistency, and alignment.
VideoFeedback dataset comprising
LiFT-Critic [Wang et al., 2024] is a reward model developed under the LiFT framework, which aligns text-to-video models using human feedback. Trained on LiFT-HRA- dataset containing 10K human-labeled samples with both scores and rationales- captures detailed human evaluation signals across multiple dimensions.
VideoReward Liu et al. 2025al offers a multi-dimensional assessment for video generation tasks. trained on large-scale I82K dataset of human-labeled comparisons collected from outputs of 12 video generation models, providing strong performance on complex video benchmarks.
Rewad Wang et al. 2025d serves as our base architecture. It leverages multi-task learning across diverse image and video generation and understanding datasets. By unifying multimodal reward tasks into single framework, UnifiedReward demonstrates mutual enhancement effects and establishes a solid baseline for holistic visual reward modeling.
Our UnifiedReward-Thiig extends UnifiedReward by integrating explicit long CoT reasoning across both visual understanding and generation tasks. Through three-stage training includin-n cold start to learning CoT reward format, rejection sampling for unified CoT reward generalization fine-tuning, and GRPO for unified CoT reward reinforcement fine-tuning, the model achieves stronger accuracy and interpretability in reward assessment. It also generalizes well without explicit reasoning, leveraging implicit CoT capabilities for robust performance.
E.3
Evaluation benchmarks
BeeewardBench [i et al. 2024b serves diverse benchmark for evaluating image understand- ing capabilities, featuring 1 1,250 carefully curated samples across general vision-language queries, hallucination detection, and complex reasoning. To ensure robust evaluation, response orders randomly shuffled during testing.
ShareGPTVideo [Zhang et al. 2024b) provides large-scale video-caption pairs and human preference data, covering various aspects of video understanding such as temporal reasoning, spatial relations, and factual grounding. We 3K for evaluation in reward modeling experiments.
23

---

GenAI-Bench (Diagg et al. 2024 is multimodal generation benchmark designed to assess how well models align with human preferences across image and video generation tasks. We adopt its image and video generation subsets for evaluating generative reward performance.
[Liu et al.) 2025a offers large-scale benchmark tailored for evaluating video reward models, consisting of 26.5k video pairs labeled by humans. Each pair is ranked according to multiple criteria, and we use the Overall Quality scores benchmark the performance of the model.
GeleoGen-RewardBeee
F Regarding performance on "Tau" metric
In both GenAI-Bence and VideoGen benchmarks shown in Tab. 2 the "Tau" metric is specifically designed to evaluate model behavior in fie cases, where two images or videos are equally preferred. In this work, our approach is explicitly focused on enhancing the model's discriminative ability, that is, its capacity to distinguish between better and worse samples. While existing reward models are generally effective at identifying outputs with large quality differences, they often struggle to make fine-graines distinctions between two high-quality candidates. To address this gap, introduce a multidimensional CoT reasoning process, aimed at improving the model's s sensitivity to subtle differences in quality. As result, intentionally exclude tie cases from our training data, and our method is not optimized for these scenarios. This design choice deliberate and aligns with practical reinforcement learning, where reward models are expected make clear and decisive judgments between competing outputs in order to effectively guide policy optimization. In contrast, baseline models were specially trained datasets that include such tie scenarios, enabling them to perform well in such situations. Although baseline models may achieve higher "Tau" scores, our method, both with and without CoT reasoning, consistently demonstrates superior performance in "Diff." metrics where a quality difference exists between the two candidates. We believe these results convincingly demonstrate the effectiveness of our proposed training approach.
objectives in vision-basee
G Prompting templates and more qualitative cases
We provide more qualitative cases across diverse vision tasks with prompting templates in Figs. and
H
Societal impacts
Our work introduces unified multimodal CoT reward model capable of high- interpretable as- sessment across diverse multimodal tasks. This advancement can significantly enhance the alignment of generative models with human preferences in real-world applications such Al-assisted content creation, and education. By improving both the accuracy and interpretability of reward signals, our method contributes more transparent and controllable AI behaviors, potentially increasing public trust in generative technologies. However, as reward models become more capable and general, they may also be misused to reinforce harmful biases in generation models, especially if the training data or preference annotations reflect subjective or skewed human values. We encourage future work to further examine the ethical implications of large-scale reward modeling and to include fairness-awan training strategies.
I Ethical statement
In this work, we affirm our commitment to ethical research practices and responsible innovation. To the best of our knowledge, this study does not involve any data, methodologies, or applications that raise ethical concerns. All experiments and analyses were conducted in compliance with established ethical guidelines, ensuring the integrity and transparency of our research process.
24

---

Image Generation
Given caption and images generaled based oap saption, please analyze detail two provided images. Evaluste them on vanous dimensions such as semontic consistency (how closely the image content aligns with the cap fion), aesthetics (compesition, color usege, 4rtistic expression), authenticlly (reslism and atiuntion to datail), and any other factors you doam relevant. for each evolustion dimension, provide a scone belween 1-10 for buth images (s.g. Image I: 8/10, Image 2: 6/10) provide concise ralionale for the summing dimension
photo of boy sating romen in reastaurant
Calculate the total
chan-of-thhught tag. output exactly ail detalled ressoning cthinko and c/thinio total
for each
tags. detall your reasoning steps,
following strings: Vimage betherl Vimage allowed the sanower> section.
Exomple output Format:
<thinko
1. Samanhic consistency: Image 1 (9/10) - ... Image (7/10)
Aestnefics: Image (8/10) ui Image (8/10)
Authentictyy Image 1 (B/L0)
Image (5/10)
[Adsi fional dimersions if any): Image 2 (8/10) Image (6/10)
Total
Image 9+8+8+6n31
Image
c/thinko
conswer- Imoge 1 better:/answen
Note: In the example above, scores and the final answer are placeholders meant only to demonstrate the format. Your actual evaluction should be based on the quality of two given images.
Your tesk provided ai follows:
Text Caption: [(promptt]
1. Semantic consistency:
Image (9/10): The image shows boy eating ramen in restaurant setting, which aligns well with the cap caption. The presence of bowl of ramen and the restaurant environment are clear.
Image (8/10): This image also depicts eating ramen restaurant. However, the focus on the boy's face more Intense, flightly reducing the emphosis on restaurant setting.
2. Aesthatics:
on the boy -Image (7/10): image good composition, but the focus on boy's face might distract from
Image (8/10): composition is balanced, warm lighting that enhances dining atmosphere.
and the ramen is clear and visually appealing.
aesthetic.
The lighting is slightly harsher, affecting the ambiance.
3. Authenticity:
Image (9/10): The details in the image, such as the texture of the ramen and the restaurant background, are realistic and corwincing.
Image (7/10): While image realistic, the focus the boy's face and the lighting make it slightly less authentic terms of capturing a typical restaurant dining experiance.
Total scors:
Image 1: 94849-26
Image 8+747=22
</thinko
Innwerr-Image
better:/answer>
Figure 5: More Qualitative Results of Image Generation CoT Reward Reasoning, Given pair of images and the corresponding caption, our model performs quality assessment across semantic consistency, aesthetics, and authenticity through CoT reasoning.
25

---

Video Generation
detail geteraled dimenions semontic consistency closely across detail), any other factors you deam relevant. For
caplion, please
The camera remains still, woman with long blonde
aligns caption), temporal coherence (smoothness and logical frames), suthanticity (realism eath evaluation dimension, provide score between 1-10 for both videos (e.g. Video 8/10. Video 5/LD) and provide concise rationale for the score. Calculate the total score for each video by sunming all dimension scores. Use taain-of-hhuugh process to detail your reasoning steps, and enclose all your detailed ressoning within cthinks and o/ c/tinnss fage. Then, in the conswers tag. oxtput exactlly one of the following strings: 'Video is better' or Video better' based the total scores. No additional text is allowed the canswer> section.
wearing a green blouse turns on the faucet with her hands, water faucet, the background is modern kitchen, bright natural lighting.
out from the
Exomple output format:
chinnoo
... Semantic lency: (9/10)
Temporal
[Additional dimensions any): (8/10) .... (6/10)
Total score:
Video 1: 9-847=6-30
Video 2: 74645x8-26
</ c/hhinko
canswer-Videe better// onewers
Note: In the example above, scores and the Final ansiwer are placehalders meant only to demonstrate the format. Your actual evalua fion should be based the quality of two given videos.
O
Text figtion: ((prompti)
Your tosk is provided followe:
1. Semantic consistency:
shows woman a green blouse, water flow is not visible. The setting is a modern kitchen, aligning with the caption.
(5/10):
dark,
on,
long blonde turned on, and water flows out, consistent with the description.
(9/10):
features
green matching the caption. The faucet
2. Temporal coherence:
Video (4/10):
actions are smooth, the turning on clearly depicted. scene
coherent flow.
(9/10):
shows clear, action turning faucet
Flowingg maintaining logical
sequence.
Authenticity:
Video (6/10): The setting and attire are realistic, but the lack of visible water flow redures authanticity.
VVideo (9/10): The video is realistic, with natural lighting and believable scene of water Flowingg from the faucet.
Total score:
Video 1: 5e4+6-15
9+9+9=27
</thinks
conswer-Video better:/answer
Figure 6: More Qualitative Results of Video Generation CoT Reward Reasoning. Given pair of videos and the corresponding caption, our model performs quality assessment across semantic consistency, temporal coherence, and authenticity through CoT reasoning.
26

---

Image Understanding
Describe this image in detail.
Answer I: The image Features table filled with several cakes, including a birthday cake and fateeoous cupcokes. The cupcakes are decorated to resemble verious movie characters, making them visually appealing and fun. There unique presentation. to the festive stmosphere.InlaTin in the background. Overall, the image showcases delightful assortment of creatively deconated cakes cuprokes, orranged for special birthday celebration. table
cupcakes, cut half Twinkles, placed like a popular cartoon character, adding dining table the center of the room, with refrigerator visible
creating
birthday
fashioned
and confetti, creating festive atmosphare for the special ocossion. Among the assor tment of creatively decorated cakes cupcokes are some unique treats, including mershmallows, candies, and cups. The movie character cupcakes are particularly eye-catching, with their intricate designs and attention to detail. However, the standout centerpiece the birthday cake fashianed after popular cartoon character, background, refrigerator can be seen, perhaps holding additional treats -gelebration.
colorful balloons, party
delight the guest
honor.
Given a question a reference Image, please analyze detail the two provided answers (Answer Answer 2).
Evalucte them based on the following three core dimensions:
1. Semantic acouracy: How well the ontwer reflects the visusi content of the image
tornetness: Whether
logically and factually
3. Clarity: Whether answer clearly and fluently expressed
You may also consider additional dimensions if you find them relevant (e.g, reasoning ability, attention to detail, multimodal grounding. etc.).
for each dimension, provide score from to 10 for both arswers, and briefly exploin reasoning, Then, compute the total score for within cthinks
explicifly adding the and </thinks tays. Then, the conswers tag, output exactly one of the following: 'Answer better' or 'nnswr better'. other
dimensions and showing the calculation. Enclose your full
text allowed in the conswer> section. Exomple format
<thinks
Semantic accuracy: Answer (9/10) .... Answer (7/10)
2. Correctness: Answer 1 (8/10) .... Answer 2 (7/10)
3. Clarity: Answer L (9/10) .. Answer 2 (8/10)
(Additional dimensions if ony Anser (6/10)-
Answer (7/10) score:\aAnswer 9+889++6322yAmmeer 7474847:29
</ </tnink>
canswer-Answer 1 is setter</answer>
Note: In the example above, scores and the finel answer are placeholders meant only to demonstrate the format. Your actual evaluation should be based the quality of two given onswers.
Your task is provided follows:
Question:
[[Query[lnkrsswrr 1: [RR]])VAnnwer [[R2]])
<think>
1. Semantic accuracy:
Answer (8/10) Correctly describes presence cupcokes decorated characters resembling Minions and birthday cake. The mention of "out half Twinkies" accurate but there no visible refrigerator the background.
Answer 2 (6/10) Describes festive scene with balloons, party hats, and confetti, of which visible the image. Correctly identifies the character cupcakes and a birthday cake but inaccurately describes other elements.
2. Correctness:
Answer (7/10) - Generally correct in describing the cupcakes and cake, but the mention of a refrigerator and the broader setting is incorrect.
Answer 2 (5/10) Incorrectly adds elements like balloons and party hats that are not present, offecting overall correctness.
3. Clarity:
Answer (9/10) Clear and concise, effectively communicates the main elements of the image, despite minor
inaccuracies.
Answer 2 (8/10)- Generally clear but includes inaccuracies that may confuse the reader regarding the actual
content of the image.
Total score:
Answer 1: 8+749-24
Answer 2: 6+5+8=19
</ /hinks
<answer-Answer better</answer>
Figure 7: More Qualitative Cases of Image Understanding CoT Reward Reasoning. Given an image, a query, and pair of candidate answers, our model performs quality assessment across semantic accuracy, factual correctness, and clarity through CoT reasoning.
27

---

Video Understanding
Why are fireworks an important part of the lion dance performance the video?
performance. They ermance festive stmosphere, making the event more exciting and eye - catching for the large crowd gathered. The bright bursts of fineworks also symbolize good luck and prosperity, which aligns with the cultural dignificance of lion dances in many celebrations.
Anwer Fireworks spectocular visual element lion
Answer Fireworks play multi fareted role the dance performance. Visually, they lion''s movements,
create dazzling backdrap that heightans the drama and grandeur
captivating the audience''s attention. Culturally, fireworts are deeply rooted in traditions of warding off evil spirits and welcoming good fortune, just os the lion dence itself is belleved to bring luck and Their combination intensifiet fies symbolic more profound memorable cultural experience
present. of the performance, anser more depth, covering both the visual and cultural aspects in greater detall and shawing better understanding of the cutturel, significnnce and the synergy between. the two elements.
drive bad
Given question reference video, please evaluate the provided answers (Answer
Anzwer 2). Judge them based
following key dimersions:
I. Semantic accuracy: Does the answer align with the visusl and temporal content in the video?
2. Conrectness: Is the answer factually and logically correct?
Clarity:
expressed fluently, clearly, coherently?
You encouroged consider any additional dimensions relevant (e.g. temporal reasoning, cousal understanding, detail, emotional perception, etc.).
For each dimension, assign score from 1 to 10 for both answers and explain briefly. Then, compute and explicitly show the total score addition of dimension better" Example Format:
full reasoning in ethinko togs. In the tag. output exactly of the following: additional commentory allowed conswer> section.
<thinks
Samontic accurecy: Arswer (8/10) ... Answer (9/10)
Correctness:
(7/10)
(6/10)
Clarity: Answer 1 (9/10) ... Answer (8/10)-
[Additional dimensions If any]: Answer (7/10)
.... Answer 2 (6/10)
Total AArsewrreeerr 1: 3:7s947:31MAnsweer 2: 946+8+6:29
</thinks
canswer-Anower better /anwerr
Your task is provided as follows:
Question: (Query/inArswer I: [R]]Annnser 2: (R2fln
cthinks
Samentic accuracy:
Answer 1 (8/10): The answer accurately describes the visual and cultural significance of fireworks in the lion dance performance. It mentions the enhancement of the festive atmosphere and the symbolic meaning of good luck and prosperity.
Answer 2 (9/10): This answer provides a more detailed explanation, covering both the visual impact and cultu significance. elaborates on symbolic power of fireworks relation wording off evil spirits welcoming good fortune.
2. Correctness:
Answer 1 (9/10): The explanation factually correct, aligning well with the cultural context of lion dances and
the role of fireworks.
Answer (10/10): answer also factually correct and provides a comprehensive understanding cultural and symbolic aspects of fireworks in the context of the lion dance.
3. Clarity:
Answer 1 (8/10): The answer is clear and well-structured, effectively conveying the importance of fireworks the performance.
Answer (9/10): The answer very clear and detailed, offering a thorough explanation enhances
understanding-
Total score:
Answer 1: 8+9+8=25
Answer 2: 9+10+9=28
c/ thinks
canswer-Answer betterc/answer>
Figure 8: More Qualitative Cases of Video Understanding CoT Reward Reasoning. Given a video, a query, and a pair of candidate answers, our model performs quality assessment across semantic accuracy, factual correctness, and clarity through CoT reasoning.
28


Reviews:
[{"review_id": "t1jnjIWhUQ", "reviewer": "Reviewer_KVGP", "summary": "This paper introduces UNIFIEDREWARD-THINK, a novel multimodal reward model designed to enhance the reliability and accuracy of reward signals for both visual understanding and generation tasks. The authors argue that current reward models are limited by their shallow reasoning processes and propose to integrate explicit long CoT reasoning to address this issue. The core of their methodology is a three-stage training pipeline: a \"Cold Start\" stage to learn the CoT format from distilled GPT-4o data, a \"Rejection Sampling\" stage to generalize CoT reasoning across various vision tasks, and a final \"GRPO\" stage to fine-tune the model on challenging cases using verifiable rewards. The authors present extensive quantitative and qualitative results demonstrating that their approach not to only improves the accuracy of reward signals with explicit CoT reasoning but also enhances the model's implicit reasoning capabilities, leading to superior performance even without the CoT.", "questions": "See weakness", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 1, "strengths_and_weaknesses": "Pros\n\n1. The paper addresses a timely and significant problem in the field of large language models: the need for more reliable and interpretable reward models. The core idea of leveraging CoT reasoning to improve multimodal reward modeling is innovative.\n\n2. The proposed three-stage training pipeline is logical and systematically addresses the challenges of incorporating CoT reasoning into reward models. The use of distillation for the cold start, followed by rejection sampling and reinforcement learning, is a logical and effective approach to progressively build and refine the model's reasoning abilities. And the ablation studies demonstrate the contribution of each stage to the final performance of the model.\n\n3. The experimental results are thorough and provide strong evidence for the effectiveness of the proposed method. The authors evaluate their model on a wide range of benchmarks for both image and video understanding and generation.\n\nCons:\n\n1. While the reported performance in this paper is generally strong, the work lacks clear technical novelty. The proposed method consists of three components: supervised fine-tuning (SFT), rejection sampling, and GRPO, all of which are well-established post-training techniques. When properly applied, these methods are indeed effective. However, this raises the question: What is the actual technical contribution of the paper?\n\n2. The use of format and accuracy rewards feels somewhat superficial. Although the paper claims to promote longer chain-of-thought (CoT) reasoning, it lacks a rigorous evaluation of CoT quality. There is no explicit reward signal for distinguishing between high-quality and faulty CoTs, nor is there any post-hoc assessment of CoT correctness in the experiments. If a CoT contains factual errors or hallucinations, can it still be considered \"good\"? A high final accuracy alone does not necessarily indicate that the CoT itself is of high quality.\n\n3. The results reported in Table 2 are somewhat puzzling. In several cases, the base model outperforms the trained model on all three Tau metrics, which undermines the claimed effectiveness of the proposed training approach.", "was_revised": true, "final_justification": "I have read the new rebuttal. I think parts of my concerns have been addressed. Therefore, I raised my score.", "rebuttal": "Thank you for your valuable feedback. We respectfully address your concerns as follows:\n## 1. Response to Concern 1: Regarding novelty\n1. Our core innovation lies not in proposing a new technique, but in **strategically addressing a fundamental and underexplored challenge in multimodal reward model-based RL**: the unreliability of reward signals caused by direct response or shallow reasoning in existing vision reward models.\n\n2. To the best of our knowledge, **this is the first work to introduce a unified multimodal CoT reward model**, capable of performing long CoT reward reasoning across both image and video generation and understanding tasks.\n\n3. To address the challenge, we posit that incorporating explicit, multi-step CoT reasoning into the vision reward modeling process significantly improves the reliability and robustness of the reward signals. **However, this is not a trivial application of existing techniques**. Several unique challenges may arise in this context:\n\n   1. **Lack of multidimensional reward evaluation**: Resultant models may tend to provide oversimplified or single-facet CoT reward reasoning, which is inadequate for complex visual reward tasks. For instance, in image generation, a comprehensive evaluation should consider not only semantic consistency, but also dimensions such as authenticity and aesthetics. Neglecting any of these aspects can lead to unreliable or biased reward signals.\n   2. **Inconsistency between reasoning and final decision**: A widely observed issue in CoT-based models, where the reasoning appears implausible but the final prediction is correct. This mismatch directly undermines the reliability of the reward model, as it leads to inaccurate supervision signals during reinforcement learning.\n\n4. To address these issues, **we introduce several key innovations in CoT reward scoring strategy design**:\n\n   1. **Multidimensional Evaluation Thinking**:  \n     We design the model to perform multidimensional evaluations for each candidate (image, video, or textual response), based on the specific requirement of vision reward tasks. This enables the model to reason over diverse and task-relevant aspects, thereby enhancing the accuracy and reliability of the reward signal.\n\n   2. **Reliable Reasoning and Decision Alignment**:  \n     The model is designed to explicitly assign scores to each evaluation dimension and then compute a final aggregated score across all dimensions. The candidate with the highest total score is selected as the preferred choice. This design enforces alignment between the intermediate reasoning process and the final decision, thereby mitigating a common issue in CoT-based models, i.e., the misalignment between reasoning and final prediction.\n\n5. Based on these reward designs, we **strategically adapt and integrate post-training techniques** to elicit and reinforce CoT reasoning:\n\n   1. **Cold Start**: We first construct a distilled dataset of 5k CoT reward reasoning samples for image generation, serving as a cold start to help the model learn CoT reasoning structures.\n   2. **Rejection Sampling**: We then extend this capability to all vision tasks (image/video understanding and generation), by applying our model to large-scale unified multimodal reward datasets that contain only preference annotations (image/video/response X is better): we prompt our model to generate explicit CoT reasoning for each data. If the generated reasoning results in final prediction that aligns with the ground-truth preference, we apply rejection sampling to reinforce these correct reasoning patterns.\n   3. **GRPO**: Conversely, when the reasoning leads to incorrect predictions, we apply GRPO, which encourages the model to explore diverse reasoning paths and optimize toward outputs that better match the verified preferences.\n\nAs shown in Tables 3 and 4, applying GRPO directly to the baseline without our CoT reward scoring strategy and the proposed three-stage training pipeline yields only limited performance gains. In contrast, our full method achieves substantially better results.\n\n6. **Empirically**, our model shows substantial improvements across multiple vision reward benchmarks. Interestingly, we observe that once the model acquires long CoT reasoning ability, it can still make more accurate reward judgments even without explicitly outputting reasoning traces, suggesting **it has internalized implicit logical reasoning mechanisms**.\n\n7. We believe our work leads to meaningful and impactful improvements in enhancing the reliability of reward signals in multimodal reward models, and presents a valuable contribution to the community that should not be overlooked.\n\n---\n\n## 2. Response to Concern 2: Regarding CoT quality during GRPO\n\n1. We agree that evaluating the quality of CoT reasoning is a critical aspect of ensuring the reliability of reward modeling. Indeed, **rigorously assessing reasoning quality remains a broadly acknowledged challenge across CoT-based models**. To date, there is no widely adopted and effective solution for systematically evaluating the correctness and robustness of intermediate reasoning steps during training, especially in complex multimodal tasks.\n\n2. In our work, we mitigate this concern by **designing a multidimensional analysis and scoring CoT process**. Specifically, the model is guided to assess each candidate (image/video/response) along multiple well-defined dimensions, and then aggregate these intermediate scores to produce the final preference decision. This step-by-step structure enforces alignment between intermediate reasoning and the final outcome, helping ensure that correct decisions are derived from reliable and interpretable reasoning trajectories.\n\n3. Thanks to this design, during GRPO training, **we treat the correctness of the final prediction as a proxy for the quality of the CoT reasoning path**. This allows us to supervise the CoT process implicitly but effectively, without requiring manual annotation of each reasoning step, which demands substantial human annotation effort.\n\n4. In summary, **our method implicitly** **favors reasoning chains that consistently lead to reward-aligned outcomes**, which also have been discussed in Appendix A.\n\n5. We fully acknowledge the importance of explicit post-hoc evaluation of CoT correctness, as you suggested. This is an excellent direction, and we plan to incorporate such evaluation in future iterations of our work.\n\n---\n\n## 3. Response to Concern 3: Regarding Tau metric\n\n1. We would like to first clarify the interpretation of the Tau metrics reported in Table 2. In both the GenAI-Bench and VideoGen benchmarks, **the Tau metric are specifically designed to evaluate model behavior in tie cases**, where two images or videos are equally preferred.\n\n2. In this work, our approach is explicitly focused on enhancing the model’s discriminative ability, that is, its capacity to distinguish between better and worse samples. While existing reward models are generally effective at identifying outputs with large quality differences, they often struggle to make fine-grained distinctions between two high-quality candidates. To address this gap, **we introduce a multidimensional CoT reasoning process, aimed at improving the model's sensitivity to subtle differences in quality**. As a result, we intentionally exclude tie cases from our training data, and our method is not optimized for these scenarios. This design choice is deliberate and aligns with practical objectives in vision-based reinforcement learning, where **reward models are expected to make clear and decisive judgments between competing outputs in order to effectively guide policy optimization**.\n\n3. In contrast, baseline models were specially trained on datasets that include such tie scenarios, enabling them to perform well in such situations.\n\n4. Although baseline models may achieve higher Tau scores, our method, both with and without CoT reasoning, **consistently demonstrates superior performance in **Diff.** metrics** where a quality difference exists between the two candidates.\n\n5. We believe these results convincingly demonstrate the effectiveness of our proposed training approach, which is also acknowledged by Reviewer HVvo."}, {"review_id": "CRtlWDzNqt", "reviewer": "Reviewer_m2dN", "summary": "This paper introduces UnifiedReward-Think, a unified multimodal reward model that enhances vision-language model alignment with human preferences by incorporating explicit Chain-of-Thought (CoT) reasoning. The key idea is to improve the reliability and robustness of reward signals in both visual understanding and generation tasks through multi-step, structured reasoning.\n\nThe proposed training framework involves three stages:\n(1) Cold Start: Distill CoT reasoning traces from GPT-4o using a small image generation dataset to initialize the model with structured reasoning patterns.\n(2) Rejection Sampling: Use large-scale unified preference data to fine-tune the model on correctly reasoned examples, improving generalization across tasks.\n(3) GRPO-based Reinforcement Fine-Tuning: Employ Group Relative Policy Optimization on incorrectly reasoned samples, with verifiable format and accuracy rewards, to encourage exploration and correction of reasoning trajectories.", "questions": "1. How to guanrantee the generalizability of the resultant reward model?\n2. In Section 3.3, how to know the correctness of the model-generated answers in rejection sampling?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The paper studies a timely topic, and the research problem is valuable. Aligning vision models with human preferences is an important and fundamental problem in multi-modality learning.\n2. The paper writing is in general clear and easy to follow. The paper's structure is very clear. The examples provide in the paper are useful in helping understanding the proposed method.\n\nWeaknesses\n1. In Section 3.3, it is not clear how to know the correctness of the model-generated answers in rejection sampling, especially it considers both vision understanding and generation tasks.\n2. In Table 2, the performance improvement of the proposed method seems to be marginal.\n3. I have concerns regarding the novelty of the proposed method, since the combination of warm up + rejection sampling seems to be very standard in solving many problems.\n4. The paper could be more clear about what the \"reward\" is intended to be used for. It seems that the reward will be used for multiple tasks. If this is the case, then the paper should emphasize why the learning can be generalized into different tasks.", "was_revised": true, "final_justification": "Most of my concerns have been addressed, including some of the technical details, the novelty, and experimental results. I will improve my score.", "rebuttal": "We appreciate the reviewer’s valuable feedback. We respectfully address your concerns as follows:\n## 1. Response to Weakness 1 & Question 2: Regarding correctness of model-generated answers in rejection sampling\n  In our work, we ensure the correctness and reliability of CoT reward samples in rejection sampling through two effective strategies:\n1. **Multidimensional CoT Scoring Mechanism**: \n    We design a multidimensional CoT analysis and scoring process to structure reasoning process. Specifically, the model is guided to **assess each candidate (image/video/response) along multiple well-defined dimensions, and then aggregate these intermediate scores to produce the final preference decision**. This step-by-step scoring mechanism enforces alignment between reasoning and outcome, helping ensure that the final decision is grounded in interpretable and logically coherent reasoning.\n\n1. **Human Preference-Aligned Ground Truth Supervision**:\n    1. For **generation** tasks, we conduct sampling on datasets such as HPD and VideoDPO, where each sample consists of **a prompt, a pair of candidate images or videos, and human-labeled ground-truth (GT) preferences** indicating which candidate is better. During sampling, the model is prompted to perform explicit CoT reasoning to compare the two candidates. If the **final prediction of the model matches the GT label**, we consider the generated CoT reasoning and decision to be correct and apply rejection sampling to reinforce these correct reasoning patterns.\n\n    2. For **understanding** tasks, we apply a similar approach on datasets like LLaVA-Critic and ShareGPTVideo-DPO, where each sample includes **a prompt and two candidate responses, along with labeled GT preferences**. Again, if the **model’s CoT-guided final prediction aligns with the GT**, we regard the reasoning as correct and apply rejection sampling.\n\n---\n## 2. Response to Weakness 2: Regarding the performance improvement in Table 2\n\n1. We would like to point out that while there is still room for improvement, our model has already achieved significant gains across most vision reward tasks, also acknowledged by Reviewer HVvo.. Specifically:\n\n   - **Image Generation Reward Task (GenAI-Bench):**  \n     Our model achieves a **+1.6% gain with CoT** and **+1.0% gain without CoT** over the best baseline.\n\n   - **Video Generation Reward Task (GenAI-Bench):**  \n     Our model outperforms the baseline by a significant margin of **+6.1% with CoT** and **+5.4% without CoT**, demonstrating its effectiveness in handling the complexity and temporal reasoning required for video evaluation.\n\n   - **Multimodal Understanding Task (VLRewardBench):**  \n     On this challenging and widely adopted understanding benchmark, our model achieves a **+6.3% improvement with CoT** and **+5.6% without CoT** in Overall Accuracy, showing robust generalization beyond generation tasks.\n\n---\n## 3. Response to Weakness 3: Regarding the novelty\n\n1. Our core innovation lies not in proposing a new technique, but in **strategically addressing a fundamental and underexplored challenge in multimodal reward model-based RL**: the unreliability of reward signals caused by direct response or shallow reasoning in existing vision reward models.\n\n2. To the best of our knowledge, **this is the first work to introduce a unified multimodal CoT reward model**, capable of performing long CoT reward reasoning across both image and video generation and understanding tasks.\n\n3. To address the challenge, we posit that incorporating explicit, multi-step CoT reasoning into the vision reward modeling process significantly improves the reliability and robustness of the reward signals. **However, this is not a trivial application of existing techniques**. Several unique challenges may arise in this context:\n\n   1. **Lack of multidimensional reward evaluation**: Resultant models may tend to provide oversimplified or single-facet CoT reward reasoning, which is inadequate for complex visual reward tasks. For instance, in image generation, a comprehensive evaluation should consider not only semantic consistency, but also dimensions such as authenticity and aesthetics. Neglecting any of these aspects can lead to unreliable or biased reward signals.\n   2. **Inconsistency between reasoning and final decision**: A widely observed issue in CoT-based models, where the reasoning appears implausible but the final prediction is correct. This mismatch directly undermines the reliability of the reward model, as it leads to inaccurate supervision signals during reinforcement learning.\n\n4. To address these issues, **we introduce several key innovations in CoT reward scoring strategy design**:\n\n   1. **Multidimensional Evaluation Thinking**:  \n     We design the model to perform multidimensional evaluations for each candidate (image, video, or textual response), based on the specific requirement of vision reward tasks. This enables the model to reason over diverse and task-relevant aspects, thereby enhancing the accuracy and reliability of the reward signal.\n\n   2. **Reliable Reasoning and Decision Alignment**:  \n     The model is designed to explicitly assign scores to each evaluation dimension and then compute a final aggregated score across all dimensions. The candidate with the highest total score is selected as the preferred choice. This design enforces alignment between the intermediate reasoning process and the final decision, thereby mitigating a common issue in CoT-based models, i.e., the misalignment between reasoning and final prediction.\n\n5. Based on these reward designs, we **strategically adapt and integrate post-training techniques** to elicit and reinforce CoT reasoning:\n\n   1. **Cold Start**: We first construct a distilled dataset of 5k CoT reward reasoning samples for image generation, serving as a cold start to help the model learn CoT reasoning structures.\n   2. **Rejection Sampling**: We then extend this capability to all vision tasks (image/video understanding and generation), by applying our model to large-scale unified multimodal reward datasets that contain only preference annotations (image/video/response X is better): we prompt our model to generate explicit CoT reasoning for each data. If the generated reasoning results in final prediction that aligns with the ground-truth preference, we apply rejection sampling to reinforce these correct reasoning patterns.\n   3. **GRPO**: Conversely, when the reasoning leads to incorrect predictions, we apply GRPO, which encourages the model to explore diverse reasoning paths and optimize toward outputs that better match the verified preferences.\n\nAs shown in Tabs. 3 and 4, applying GRPO directly to the baseline without our CoT reward scoring strategy and the proposed three-stage training pipeline yields only limited performance gains. In contrast, our full method achieves substantially better results.\n\n6. **Empirically**, our model shows substantial improvements across multiple vision reward benchmarks. Interestingly, we observe that once the model acquires long CoT reasoning ability, it can still make more accurate reward judgments even without explicitly outputting reasoning traces, suggesting **it has internalized implicit logical reasoning mechanisms**.\n\n7. We believe our work leads to meaningful and impactful improvements in multimodal reward models and presents a valuable contribution to the community that should not be overlooked.\n\n---\n\n## 4. Response to Weakness 4 & Question 1: Regarding why the learning can be generalized to different tasks\n\n1. In this work, our unified multimodal CoT reward model is designed to support both vision generation and understanding tasks, across both images and videos. **Although the downstream tasks may differ, the input-output structure to the reward model is unified**: in all cases, the model receives a combination of visual content (image or video) and caption or textual response, and outputs a reward judgment signal indicating the relative quality or preference between candidate outputs.  \n\n   Because of this consistent structure, **the reward model learns to evaluate visual-text pairs in a task-agnostic manner**, allowing it to generalize across different modalities and task types. This design enables us to train the reward model jointly on diverse datasets and apply it effectively across vision reward tasks.\n\n2. However, equipping reward models with CoT reasoning **using traditional training schemes like SFT poses a highly challenging problem** due to the scarcity of large-scale multimodal CoT reward data.\n\n3. We posit that **VLMs already possess latent capabilities for multi-step reasoning**. The key is designing a scalable and effective approach to elicit and reinforce such reasoning behavior across different vision reward tasks.\n\n4. To this end, our pipeline strategically incorporates three stages:\n   1. **Cold Start: Learning CoT Reward Format**  \n   2. **Rejection Sampling: Unified CoT Reward Generalization Fine-tuning**  \n   3. **GRPO: Unified CoT Reward Reinforcement Fine-tuning**  \n\n5. Through this three-stage strategy of learning, consolidation, and reinforcement, our model successfully transfers the CoT reasoning ability learned **from a small number of image generation reward tasks to all visual reward tasks**."}, {"review_id": "zrxRck5CgW", "reviewer": "Reviewer_qbY9", "summary": "This paper proposes UNIFIEDREWARD-THINK, a multimodal reward model that introduces long CoT reasoning to improve the accuracy and robustness of reward signals. The approach consists of three stages: initializing the model with GPT distilled data (cold start), applying rejection sampling on large-scale preference data, and using GRPO reinforcement learning to optimize reasoning capabilities. Experiments show that the method outperforms existing approaches on both visual understanding and generation tasks, and exhibits strong implicit reasoning ability even without explicitly outputting reasoning traces.", "questions": "1. How do you ensure the quality of the distilled cold-start data? For example, a long CoT reward dataset should guarantee both the correctness of the intermediate reasoning steps and the final scoring.\n\n2. I'm curious why this model outperforms GPT-4o. What would happen if GPT-4o were combined with a similar long CoT reasoning approach?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The method is simple and easy to understand.\n\nWeaknesses:\n1. There are many R1-like papers nowadays that collect similar types of data for cold start and train with GRPO; this work lacks novelty.\n\n2. The cold start and GRPO stages are similar to those used in most xx-R1 papers.", "was_revised": true, "final_justification": "The author has addressed my earlier questions about cumulative errors in CoT and related issues.", "rebuttal": "Thank you for your feedback. We respectfully address your concerns as follows:\n\n## 1. Response to Weaknesses 1&2: Regarding novelty\n\n1. Our core innovation lies not in proposing a new technique, but in **strategically addressing a fundamental and underexplored challenge in multimodal reward model-based RL**: the unreliability of reward signals caused by direct response or shallow reasoning in existing vision reward models.\n\n2. To the best of our knowledge, **this is the first work to introduce a unified multimodal CoT reward model**, capable of performing long CoT reward reasoning across both image and video generation and understanding tasks.\n\n3. To address the challenge, we posit that incorporating explicit, multi-step CoT reasoning into the vision reward modeling process significantly improves the reliability and robustness of the reward signals. **However, this is not a trivial application of existing techniques**. Several unique challenges may arise in this context:\n\n   1. **Lack of multidimensional reward evaluation**: Resultant models may tend to provide oversimplified or single-facet CoT reward reasoning, which is inadequate for complex visual reward tasks. For instance, in image generation, a comprehensive evaluation should consider not only semantic consistency, but also dimensions such as authenticity and aesthetics. Neglecting any of these aspects can lead to unreliable or biased reward signals.\n   2. **Inconsistency between reasoning and final decision**: A widely observed issue in CoT-based models, where the reasoning appears implausible but the final prediction is correct. This mismatch directly undermines the reliability of the reward model, as it leads to inaccurate supervision signals during reinforcement learning.\n\n4. To address these issues, **we introduce several key innovations in CoT reward scoring strategy design**:\n\n   1. **Multidimensional Evaluation Thinking**:  \n     We design the model to perform multidimensional evaluations for each candidate (image, video, or textual response), based on the specific requirement of vision reward tasks. This enables the model to reason over diverse and task-relevant aspects, thereby enhancing the accuracy and reliability of the reward signal.\n\n   2. **Reliable Reasoning and Decision Alignment**:  \n     The model is designed to explicitly assign scores to each evaluation dimension and then compute a final aggregated score across all dimensions. The candidate with the highest total score is selected as the preferred choice. This design enforces alignment between the intermediate reasoning process and the final decision, thereby mitigating a common issue in CoT-based models, i.e., the misalignment between reasoning and final prediction.\n\n5. Based on these reward designs, we **strategically adapt and integrate post-training techniques** to elicit and reinforce CoT reasoning:\n\n   1. **Cold Start**: We first construct a distilled dataset of 5k CoT reward reasoning samples for image generation, serving as a cold start to help the model learn CoT reasoning structures.\n   2. **Rejection Sampling**: We then extend this capability to all vision tasks (image/video understanding and generation), by applying our model to large-scale unified multimodal reward datasets that contain only preference annotations (image/video/response X is better): we prompt our model to generate explicit CoT reasoning for each data. If the generated reasoning results in final prediction that aligns with the ground-truth preference, we apply rejection sampling to reinforce these correct reasoning patterns.\n   3. **GRPO**: Conversely, when the reasoning leads to incorrect predictions, we apply GRPO, which encourages the model to explore diverse reasoning paths and optimize toward outputs that better match the verified ground-truth preferences.\n\n5. It is important to highlight that **solely relying on cold start and GRPO is insufficient** for this task due to following challenges:\n\n   - **Scalability**: Directly applying GRPO over a large scale of preference-only samples after cold start would incur high computational costs.\n   - **Low rollout quality**: After cold start, the model’s performance is still unsatisfactory as evidenced in Tables 3 and 4, making it difficult to generate correct answers during RL rollout in hard samples. \n   - **Low RL benefit on known samples**: For cases the model can already handle well after cold-start, GRPO provides limited learning gains while incurring high computation.\n\n   Therefore, we introduce **rejection sampling** as an intermediate stage to:\n   - **Filter out easy samples** that the model already gets right, reducing redundancy in GRPO, by consolidating high-quality correct reasoning patterns through these samples.\n   - **Allocate more RL effort to hard or ambiguous cases** where exploration is needed through GRPO.\n\n   We additionally include ablation results as shown in the table below. The results demonstrate that removing rejection sampling leads to a consistent drop in performance. \n\n|                         | VLReward. | GenAI-Image | GenAI-Video | VideoGen. | ShareGPTV. |\n|-------------------------|-----------|-------------|-------------|-----------|------------|\n| baseline         | 67.5      | 70.9       | 77.2        | 79.3      | 77.3       |\n| UnifiedReward-Think    | **73.8**      | **72.5**       | **82.3**        | **80.5**      | **79.6**       |\n| **w/o rejection sampling**  | 70.4  | 71.6    | 79.2    | 79.8  | 78.5   |\n\n6. **Empirically**, our model shows substantial improvements across multiple vision reward benchmarks. Interestingly, we observe that once the model acquires long CoT reasoning ability, it can still make more accurate reward judgments even without explicitly outputting reasoning traces, suggesting **it has internalized implicit logical reasoning mechanisms**.\n\n7. We believe our work leads to meaningful and impactful improvements in multimodal reward models and presents a valuable contribution to the community that should not be overlooked.\n\n---\n\n## 2. Response to Question 1: Regarding correctness of cold-start data\n\n1. In our work, we ensure the correctness and reliability of the distilled CoT reward samples through two effective strategies:\n\n   - **Multidimensional Evaluation and Scoring Thinking**:  \n     We design a multidimensional analysis and scoring CoT process to structure reasoning. Specifically, **we prompt GPT-4o to evaluate each image along several well-defined dimensions, and then aggregate the intermediate scores to arrive at a final preference decision**. This step-by-step scoring mechanism enforces alignment between reasoning and outcome, helping ensure that the final decision is grounded in interpretable and logically coherent reasoning.\n   - **Human-Aligned Ground Truth Supervision**:  \n     We distill CoT reasoning samples from 10K human-preference-annotated image generation pairs, randomly sampled from four datasets: HPD, OIP, EvalMuse, and OpenAI-4o_t2i_human_preference. Each sample contains **a prompt, a pair of generated images, and a human-annotated ground-truth (GT) label** indicating the preferred image. We prompt GPT-4o to perform explicit CoT reasoning and make a comparative judgment. If **GPT-4o's final decision matches the GT label**, we retain the generated CoT reasoning and decision. This filtering process yields a final distilled dataset of approximately 5K high-quality CoT reward samples, which we use to cold-start our reward model training.\n\n   These measures jointly help ensure that the distilled reasoning traces are not only logically consistent but also aligned with human preference signals.\n2. Furthermore, in our pipeline, the cold-start CoT data serves **primarily to teach the model the structure and format of CoT-style reasoning**, rather than to enhance reward decision quality. The generalization and strengthening of the model’s reward discrimination capability are achieved through the subsequent stages of rejection sampling and GRPO, which operate on large-scale unified multimodal vision preference data and iteratively reinforce correct reasoning trajectories.\n3. Our experimental results in the table also show that even when using distilled CoT data from weaker models (**Qwen2.5VL-72b**) instead of GPT-4o for cold-start, the model can **still achieve comparable performance** after subsequent reinforcement through rejection sampling and GRPO.\n\n|                         | VLReward. | GenAI-Image | GenAI-Video | VideoGen. | ShareGPTV. |\n|-------------------------|-----------|-------------|-------------|-----------|------------|\n| baseline         | 67.5      | 70.9       | 77.2        | 79.3      | 77.3       |\n| UnifiedReward-Think (distilled **4o**)     | **73.8**      | 72.5       | **82.3**        | **80.5**      | 79.6       |\n| UnifiedReward-Think (distilled **Qwen2.5VL**)  | 73.2  | **73.6**    | 81.7    | 80.2  | **80.9**   |\n\n---\n## 3. Response to Question 2: Regarding GPT-4o's performance\n1. While GPT-4o is a powerful general-purpose multimodal model, **it is not specifically optimized for reward modeling**, especially in terms of aligning its reward reasoning process with ground-truth human preferences across diverse image and video tasks. In contrast, **achieving reliable reward decisions requires more than general reasoning capability**. In our work, this is accomplished through our carefully designed **multidimensional evaluation and scoring reward mechanism, combined with a strategically structured three-stage training pipeline**. Together, these components ensure the generalizability and reliability of our model across both image/video understanding and generation reward tasks.\n2. Since GPT-4o is a **closed-source model**, we are unable to directly apply our CoT-based reward modeling training pipeline to it. Nonetheless, **we believe that integrating our approach into GPT-4o would likely lead to significant improvements** in its performance on this tasks."}, {"review_id": "55IixYMOY5", "reviewer": "Reviewer_HVvo", "summary": "The paper introduces UNIFIEDREWARD-THINK, a unified multimodal reward model that uses explicit long Chain-of-Thought (CoT) traces for both vision-generation and vision-understanding tasks. It proposes a three-stage training pipeline:\n1. Cold-start distillation of GPT-4o CoT on a small image-generation preference set.\n2. Rejection sampling on large, diverse preference data to keep only correct reasoning traces.\n3. GRPO (Group-Relative Policy Optimisation) reinforcement fine-tuning with format and accuracy rewards.\n\nExperiments on GenAI-Bench, VideoGen-Reward, and VLReward-Bench show gains over the UnifiedReward baseline (e.g., +4.8 macro-accuracy for image understanding, +5.1 diff-accuracy for video generation). Ablations indicate that each stage adds to performance and robustness.", "questions": "1. Have you measured the frequency of incorrect or misleading explanations in distilled CoT data? How do such errors impact downstream reward optimization?\n2. Please detail the composition, pay structure, and inter-annotator agreement of human-preference datasets used during rejection sampling. This is critical for evaluating dataset quality and reproducibility.\n3. What are the GPU-hours, memory usage, and wall-clock time for GRPO fine-tuning compared to a standard SFT reward model on the largest benchmark?\n4. Have you evaluated UNIFIEDREWARD-THINK’s outputs on any fairness or bias benchmarks (e.g., CelebA-Bias)? If so, what mitigation strategies are needed when CoT traces encode societal stereotypes?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- Novel use of explicit CoT traces as an internal reward-shaping signal. Instead of only using CoT to explain a final prediction, the model learns intermediate reasoning steps and uses them to guide policy optimization. This bridges a gap between explainable reasoning and direct reward learning in multimodal settings.\n- Comprehensive benchmark suite. Evaluations span five distinct tasks—image CoT verification, video frame ordering, caption coherence, etc.—showing gains in both understanding and generation. Tasks cover different modalities and difficulty levels, indicating robustness beyond a single niche.\n- Demonstrated zero-shot generalization on held-out image types and longer reasoning chains. Figure 4 shows that UNIFIEDREWARD-THINK maintains high accuracy when chain lengths double, which suggests the model scales to more complex reasoning without retraining.\n- Clear qualitative examples of reasoning traces reveal how the model avoids common pitfalls (e.g., hallucinating unsupported details). These examples provide actionable insight into where CoT improves final scores, rather than opaque “black-box” shifts.\n\n### Weaknesses\n- Heavy reliance on GPT-4o quality. The entire pipeline hinges on distilling CoT traces from GPT-4o; if the base model produces systematic biases or errors, those propagate into the reward model. There is no analysis of GPT-4o’s failure modes or mitigation when its reasoning is flawed.\n- Human-annotation process is under-specified. The paper omits details on annotator demographics, pay, and inter-rater agreement when constructing rejection-sampling data. Without knowing how diverse or reliable the preference data are, it is difficult to assess fairness and reproducibility.\n- Computational cost and scalability are unclear. GRPO fine-tuning requires maintaining CoT trace consistency; the paper does not report GPU-hours or memory overhead relative to standard reward-models. For practitioners, it is unclear if performance gains justify added cost.\n- Limited analysis of downstream biases. Reward models can amplify biases present in training data—yet the paper does not examine whether UNIFIEDREWARD-THINK introduces or exacerbates biases along gender, race, or cultural lines in image-captioning tasks. A brief audit on a fairness benchmark would strengthen trust.", "was_revised": true, "final_justification": "I am maintaining my score of 4. The authors' detailed rebuttal has clarified several points from my initial review. However, a significant debate regarding the paper's core contribution remains, which makes this a difficult recommendation to make.\n\n### Addressed and Remaining Concerns\n\n1.  **Methodological Details:** The authors have provided helpful clarifications on their pipeline, including the data distillation process, the use of public datasets, and computational costs. My follow-up question regarding the reward function design was also addressed with new experimental data. This data helps to mitigate the concern about reward ambiguity, though it's worth noting that this important justification was absent from the original paper and the reward mechanism remains an indirect proxy for reasoning quality.\n\n2.  **Unresolved Weakness: Technical Novelty:** A significant and unresolved weakness, as pointed out by multiple reviewers, is the paper's limited technical novelty. The methodology is primarily a well-executed combination of existing techniques (SFT, Rejection Sampling, GRPO). For a top-tier venue like NeurIPS, the lack of a more fundamental algorithmic contribution is a considerable drawback.\n\n---\n### Weighting and Final Recommendation\n\nMy final recommendation balances the paper's clear empirical strengths against its debatable novelty.\n\n* **Reasons to Accept:** The paper tackles a novel and important problem, presenting a complete system that achieves strong state-of-the-art results across a comprehensive suite of multimodal tasks. The empirical evidence is robust.\n\n* **Reasons to Reject:** The core methodology lacks fundamental originality. The contribution is more of an engineering or system-building success rather than a leap in algorithmic innovation, which raises the bar for acceptance.\n\nUltimately, I am **leaning towards acceptance on the borderline**. I give more weight to the strong empirical evidence and the importance of the problem being solved.", "rebuttal": "Thank you for your valuable feedback. We respectfully address your concerns as follows:\n\n## 1. Response to Weakness 1 & Question 1: Regarding correctness of distilled cold-start data\n\n1. In our work, we ensure the correctness and reliability of the distilled CoT reward samples through two effective strategies:\n\n   1. **Multidimensional Evaluation and Scoring Thinking**:  \n     We design a multidimensional analysis and scoring CoT process to structure reasoning. Specifically, **we prompt GPT-4o to evaluate each image along several well-defined dimensions, and then aggregate the intermediate scores to arrive at a final preference decision**. This step-by-step scoring mechanism enforces alignment between reasoning and outcome, helping ensure that the final decision is grounded in interpretable and logically coherent reasoning.\n\n   2. **Human-Aligned Ground Truth Supervision**:\n     We distill CoT reasoning samples from 10K human-preference-annotated image generation pairs, randomly sampled from four datasets: HPD, OIP, EvalMuse, and OpenAI-4o_t2i_human_preference. Each sample contains **a prompt, a pair of generated images, and a human-annotated ground-truth (GT) label** indicating the preferred image. We prompt GPT-4o to perform explicit CoT reasoning and make a comparative judgment. If **GPT-4o's final decision matches the GT label**, we retain the generated CoT reasoning and decision. This filtering process yields a final distilled dataset of approximately 5K high-quality CoT reward samples, which we use to cold-start our reward model training.\n\n   These measures jointly help ensure that the distilled reasoning traces are not only logically consistent but also aligned with human preference signals.\n\n2. Furthermore, in our pipeline, the cold-start CoT data serves **primarily to teach the model the structure and format of CoT-style reasoning**, rather than to enhance reward decision quality. The generalization and strengthening of the model’s reward discrimination capability are achieved through the subsequent stages of rejection sampling and GRPO, which operate on large-scale unified multimodal vision preference data and iteratively reinforce correct reasoning trajectories.\n3. Our experimental results in the table also show that even when using distilled CoT data from weaker models (**Qwen2.5VL-72b**) instead of GPT-4o for cold-start, the model can **still achieve comparable performance** after subsequent reinforcement through rejection sampling and GRPO.\n\n|                         | VLReward. | GenAI-Image | GenAI-Video | VideoGen. | ShareGPTV. |\n|-------------------------|-----------|-------------|-------------|-----------|------------|\n| baseline         | 67.5      | 70.9       | 77.2        | 79.3      | 77.2       |\n| UnifiedReward-Think (distilled **4o**)     | **73.8**      | 72.5       | **82.3**        | **80.5**      | 79.6       |\n| UnifiedReward-Think (distilled **Qwen2.5VL**)  | 73.2  | **73.6**    | 81.7    | 80.2  | **80.9**   |\n\n\n---\n\n## 2. Response to Weakness 2 & Question 2: Regarding reliability of the model-generated answers in rejection sampling\n\nWe would like to clarify that our rejection sampling process **does not involve any additional human annotation**. Similar with cold-start data, we ensure the correctness and reliability of the rejection sampling data through two effective strategies . Specifically:\n\n1. **Multidimensional Evaluation and Scoring Thinking**  \n    To ensure the reliability of the CoT process during sampling, we design a **multidimensional analysis and scoring CoT process**. Specifically, the model is guided to assess each candidate (image/video/response) along multiple well-defined dimensions, and then aggregate these intermediate scores to produce the final preference decision. This step-by-step structure enforces alignment between intermediate reasoning and the final outcome, helping ensure that correct decisions are derived from reliable and interpretable reasoning trajectories.\n\n2. **Human-Aligned Ground Truth Supervision**:\n    1. **Generation** Tasks :We conduct sampling on datasets such as HPD and VideoDPO, where each sample consists of **a prompt, a pair of candidate images or videos, and human-labeled ground-truth (GT) preferences**. The model is prompted to perform explicit CoT reasoning to compare the two candidates. If **the final prediction matches the GT label**, we consider the generated CoT reasoning and decision to be correct and apply rejection sampling to reinforce these correct reasoning patterns.\n\n    2. **Understanding** Tasks: We apply a similar approach on datasets like LLaVA-Critic and ShareGPTVideo-DPO, where each sample includes **a prompt and two candidate responses, along with labeled GT preferences**. Again, if the model’s CoT-guided final prediction aligns with the GT, we regard the reasoning as correct and apply rejection sampling.\n---\n\n## 3. Response to Weakness 3 & Question 3: Regarding computational cost of GRPO fine-tuning and scalability\n\n### (1) GPU-hours and Memory Overhead of GRPO  \nDuring the GRPO fine-tuning phase, we trained on approximately 40K data points using ZeRO-3 distributed training across 64 H100 GPUs for 2 epochs, which took around **12 hours**. Each GPU utilized approximately **60GB of memory**.  \nThe inference speed is around **3 seconds for image reward tasks** and **7 seconds for video reward tasks**.\n\nIn comparison, the UnifiedReward baseline was trained on 236K samples using ZeRO-2 across the same number of GPUs for 3 epochs, requiring around **10 hours**, with **70GB memory** usage per GPU. The inference speed is around **1 second for image reward tasks** and **4 seconds for video reward tasks**.\n\nWhile GRPO introduces a slightly higher training overhead, this is justified by its ability to support long, multi-dimensional CoT reasoning, which leads to **consistent and significant performance improvements** across both image and video reward tasks.\n\n### (2) Scalability  \n1. Our framework is **designed to scale effectively**. In this work, we have curated and unified a wide range of open-source datasets covering both image and video generation/understanding tasks.  \n2. We believe that **the model continues to benefit as more data becomes available**. We are committed to continually integrating more newly released datasets to enhance the scalability and robustness of our approach.\n\n---\n\n## 4. Response to Weakness 4 & Question 4: Regarding analysis of downstream biases\n\nWe appreciate the reviewer’s thoughtful comment on fairness and bias. We would like to note the following:\n\n\n1. **Our method itself does not introduce new sources of societal bias**. Any potential biases in the CoT traces would likely originate from the open-source datasets used for training.\n\n2. In real-world tasks, we recommend incorporating **human verification or filtering mechanisms** when sensitive applications are involved. This can effectively mitigate unintended stereotypes that may appear in the data or model outputs.\n \n3. As a **research work**, we did not have the resources or capacity to conduct a dedicated fairness audit. Nonetheless, we acknowledge the importance of fairness evaluation and view it as a valuable direction for future exploration."}]

Author rebuttal:

Meta-review: This paper proposes UnifiedReward-Think, a multimodal reward model that integrates chain-of-thought (CoT) reasoning into preference modeling across both vision and language tasks. The main contributions are (i) a rejection sampling strategy for generating reliable CoT supervision signals, (ii) GRPO fine-tuning to optimize over reasoning paths, and (iii) a multimodal evaluation framework. Reviewers generally agreed that the work addresses a fundamental challenge in reward modeling—the unreliability of sparse outcome-based signals—by leveraging reasoning traces to improve alignment. Strengths noted include the non-trivial novelty of combining multimodal CoT with reinforcement fine-tuning, solid theoretical motivation, and empirical gains on diverse benchmarks. Weaknesses include limited experimental breadth (a few missing baselines, not fully tested for out-of-distribution generalization) and some exposition issues in the introduction, though the rebuttal added new results, clarified distinctions from prior work, and promised revisions. After the rebuttal, most reviewers increased their confidence that the contribution is significant and technically sound. Overall, the paper offers a well-justified advance at the intersection of multimodal alignment and RLHF. I would recommend acceptance.
Decision: Accept (poster)