R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning
Authors: YiFan Zhang
Xingyu Lu
Xiao Hu
Chaoyou Fu
Bin Wen
Tianke Zhang
Changyi Liu
Kaiyu Jiang
Kaibing Chen
Kaiyu Tang
Haojie Ding
Jiankang Chen
Fan Yang
Zhang Zhang
Tingting Gao
Di ZHANG
Guorui Zhou
Liang Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Large Language Model
Multimodal Reward Model
Stable Reinforcement Learning
Long-CoT Reasoning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: This paper introduces StableReinforce, a novel RL algorithm designed to stably and effectively train Multimodal Reward Models (MRMs), leading to the R1-Reward model which significantly outperforms previous SOTA models on key benchmarks.

Abstract: Multimodal Reward Models (MRMs) play a crucial role in enhancing the performance of Multimodal Large Language Models (MLLMs). While recent advancements have primarily focused on improving the model structure and training data of MRMs, there has been limited exploration into the effectiveness of long-term reasoning capabilities for reward modeling and how to activate these capabilities in MRMs. In this paper, we explore how Reinforcement Learning (RL) can be used to improve reward modeling. Specifically, we reformulate the reward modeling problem as a rule-based RL task. However, we observe that directly applying existing RL algorithms, such as Reinforce++, to reward modeling often leads to training instability or even collapse due to the inherent limitations of these algorithms. To address this issue, we propose the StableReinforce algorithm, which refines the training loss, advantage estimation strategy, and reward design of existing RL methods. These refinements result in more stable training dynamics and superior performance. To facilitate MRM training, we collect 200K preference data from diverse datasets. Our reward model, R1-Reward, trained using the StableReinforce algorithm on this dataset, significantly improves performance on multimodal reward modeling benchmarks. Compared to previous SOTA models, R1-Reward achieves a 8.4% improvement on the VL Reward-Bench and a 14.3% improvement on the Multimodal Reward Bench. Moreover, with more inference compute, R1-Reward's performance is further enhanced, highlighting the potential of RL algorithms in optimizing MRMs.

Published as a conference paper at ICLR 2026
R1-REWARD: TRAINING MULTIMODAL REWARD MODEL THROUGH STABLE REINFORCEMENT LEARNING
Yi-Fan Zhang, Zhang Zhang; Liang Wang Institute of Automation, Chinese Academy of Sciences (CASIA)
Xingyu Lu, Xiao Hu Tsinghua University (THU)
Chaoyou Fu" Nanjing University (NJU)
Bin Wen, Tianke Zhang, Changyi Liu, Kaiyu Jiang, Kaibing Chen, Kaiyu Tang, Haojie Ding, Jiankang Chen, Fan Yang, Tingting Gao, Di Zhang, Guorui Zhou
KuaiShou
ABSTRACT
Multimodal Reward Models (MRMs) play crucial role in enhancing the performance of Multimodal Large Language Models (MLLMs). While recent advancements have primarily focused on improving the model struc- ture and training data of MRMs, there has been limited exploration into the effectiveness of long-term reasoning capabilities for reward modeling and how activate these capabilities in MRMs. In this paper, we explore how Reinforcement Learning (RL) can be used to improve reward modeling. Specifically, we reformulate the reward modeling problem as a rule-based RL task. However, we observe that directly applying existing RL algorihmms, such as Reinforce++, to reward modeling often leads to training instability even collapse due to the inherent limitations of these algorithms. To address this issue, we propose the StableReinforce algorithm, which refines the training loss, advantage estimation strategy, and reward design of exist- ing RL methods. These refinements result in more stable training dynamics and superior performance. To facilitate MRM training, we collect 200K pref- erence data from diverse datasets. Our reward model, R1-Reward, trained using the StableReinforce algorithm on this dataset, significantly improves performance on multimodal reward modeling benchmarks. Compared to previous SOTA models, RI-Reward achieves a 8.4% improvement on the VL Reward-Bent Bench and 14.3% improvement on the Multimodal Reward Bench. Moreover, with more inference compute, R1-Reward's 's performance is further enhanced, highlighting the potential of RL frameworks in MRMs.
1 Introduction
High-quality Multimodal Reward Models (MRMs) Pu et al. (2025): Chen et al. (2024a)
Wang et al. (2025a) play crucial role in the development of Multimodal Large Language Models (MLLMs) Wang et al. (2024a); Deitke et al. (2024); Dai et al. ) Agrawal al.
(2024). In the training phase, from an algorithmic perspective, the MRM provides reward signals for RL RLSunn et al. (2023); Ouyang et al. (2022), directly influencing the stability and final outcomes of training. From data perspective, powerful MRM enables high-quality data filtering, improving data quality by removing noisy samples /hhag et al. (2025a) ) Lu et al. (2025). In the inference phase, the MRM facilitates test-time scaling strategies, such the best-of-N strategy, to select the optimal responses Wang et al. (2025a). In the evaluation
"Corresponding Author 'Project Leader
1

---

Published as a conference paper at ICLR 2026
Baselines
R1- Reward
BCPT-4o
0--25-Be
Bitt-Boooed
D'iningg
#Holingght
MM-RLHF Reward Bench
VL RewarddB-ecch
Multimodal Reward -Bench
822
R24
02.
60.6
70.
66.6
66-1
Figure 1: R1-Reward performance on multimodal reward benchmarks. Performance improves significantly when using a majority voting strategy (Voting@5/Il 115) over multiple inference samples.
10"
Method
Method
StableReinforce
Response Length
Reinffree
Policy Loss
10
Reinforce*
Reinforce**
10"
600
550
100
200
300
50
100
150
200
250
300
350
Step
Steps
(a) Policy Loss Convergence
(b) Response Length During Training
Figure 2: Detailed comparison between StableReinforce and Reinforce+++ (a) StableRe- inforce exhibits faster and more stable convergence of the policy loss during training. (b) StableReinforce continuously performs length compression, improving efficiency. Rein- force++ collapses around step 150, whereas StableReinforce remains stable, demonstrating its enhanced robustness. Additionally, after RL training with StableReinforce, the average response length is reduced by approximately 15% compared to base model, suggesting potential improvements reasoning token efficiency.
phase, a good MRM can serve as an evaluator to simplify the evaluation process, especially in open-ended scenarios Xiong et al. (2024).
Recently, reinforcement learning DeepSeek-A1
(2025) has gained widespread application in
(022a), achieving remarkable improvements Shen et al. (2025a), multimodal reasoning tasks| Huang et al. (2025) Meng et al. (2025), video understanding tasks Feng et al. (2025), and omni-tasks Zhao al. (2025). Compared to traditional post-training strategies such as supervised fine-tuning and direct preference optimization Rafailov et al. (2023). RI. offers better generalization Chu et al. (2025) and demonstrates the ability to induce long-term reasoning capabilities DeepSeek-Al capabllitessDeepSeeek.
the post-training process of MLLMs Yu et al.
in traditional vision tasks Liu et al. (2025a)
(2025). However, recent improvements in MRMs have primarily focused on data Xiong et al. (2024) and structural aspects Zhang et al. (2025a), with little discussion whether RL can be used to introduce long-term reasoning in order to improve multimodal reward modeling performance.
In this paper, we investigate whether RL algorithms can be applied to multimodal reward modeling tasks? Intuitively, the reward modeling problem can be transformed into rule- based RL task, where the input consists
given question and two answers. The target of the policy is to decide which answer is better. The reward during training can be obtained by comparing whether the model's judgment is consistent with the ground truth. Our goal is to enable the model to perform long-term reasoning and then provide the correct judgment. However, RL for reward modeling presents several unique challenges, and directly using traditional RL. methods can easily cause training to collapse:
2

---

Published as a conference paper at ICLR 2026
1. Limitation of PPO Schulman et al. (2017) and Related Algorithms Shao et al. (2024a). PPO and related algorithms rely on clipping the loss function to ensure training stability. However, we observe that when the advantage is negative and the current policy differs significantly from the reference policy, simple clipping fails to prevent instability, which may cause the training process to diverge even crash.
2. Instability of Advantage Normalization. We observe that in the later stages of training,
where the majority of rewards in a single batch are either 1 or with very low variance, the commonly used advantage normalization technique (subtracting the mean and dividing by the variance) in algorithms such as GRPO Shao et al. (2024a) and Reinforce+ Hu (2025) can lead to extremely large or small advantage values for some samples. This can cause significant instability during training.
Inconsistency Between Reasoning and Results. During training, we frequently observe inconsistencies between the model's reasoning process and its final output. The model may judge one answer as better during reasoning but ultimately output an opposite conclusion. This happens because rule-based RL only scores the result without supervising the reasoning process, leading the model to learn to generate correct answers without coherent reasoning.
To this end, we propose StableReinforce, which introduces several modifications to traditional RL methods. Specifically, we refine the clipping operation to mitigate numerical instability caused by large updates and introduce a robust advantage normalization technique that limits the impact of outliers. On the reward function design front, StableReinforce introduces a novel mechanism: the use of an MILM as a referee. This referee evaluates the consistency between the model's reasoning process and the result, ensuring that the reasoning aligns with the output. This consistency reward promotes more accurate and logically coherent menision-making.. During the training phase, directly training the MLLM using reinforcement learning yields suboptimal results. Therefore, a progressive difficulty training strategy is adopted. Initially, 200K preference data collected from publicly available datasets, and GPT-4o generates corresponding thinking processes, referred to as R1 R1-Reward-2000 to serve as cold-start data. Meanwhile, for each sample, the number of sampling attempts GPT-4o requires to infer a conclusion matching the ground truth is recorded, which is considered the difficulty level of that sample. In the reinforcement learning phase, samples where GPT-4o requires at least two sampling attempts to arrive at the correct answer, or fails to answer correctly even after three attempts, are selected as training data. These samples are then used to train the model with the StableReinforce algorithm.
As shown in Figure 2 the RL phase effectively performs token compression, and also resulting in noticeable performance improvement in our experiments. Ri-Reward performs excellently on common multimodal reward modeling benchmarks. As shown in Figure 1 R1- Reward outperforms the state-of-the-art (SOTA) on all the three benchmarks. Furthermore, R1-Reward exhibits strong inference time scalability. By sampling only five times and selecting the most frequent answer as the correct one, the accuracy of reward modeling improves substantially. On the MM-RLHF Reward Bench Zhang et al.) (2025a)), VL Reward- Bench Li et al. (2024a), and Multimodal Reward Bench Yasunaga et al. (2025) R1-Reward achieves improvements of 3.5%, 13.5%, and 14.6%, respectively, compared to SOTA. Finally, downstream validation shows consistent improvements across multiple benchmarks when using R1-Reward train smaller MLLMs, confirming the practical utility and transferability of our reward modeling approach.
2 Preliminary and Limitations
Standard Reward Models. Typically, a reward model starts with an LLM o, where the LLM head hy replaced with linear reward head 4, enabling the model output a scalar reward. These models are trained using numan-providee a preferred and less preferred response yw, yt, the reward model is optimized to assign
pairwise comparisons. Given a query
higher rewards to preferred responses: (Reward (0) Ex.y=.
log a (r(vnl [x) -r(u(xx)
where r(y]x) is the reward and a is the sigmoid function.
3

---

Published as a conference paper at ICLR 2026
Proximal Policy Optimization (PPO) Schulman et al. (2117) is a commonly used al- gorithm in RL that aims to optimize a policy directly while maintaining stable and ef- ficient learning. PPO belongs to the family of policy gradient methods, where the ob- jective is to improve the policy by maximizing the expected cumulative reward. Un- like traditional policy gradient methods, which can suffer from large updates and in- stabiliit, PPO introduces novel way to constrain policy updates, ensuring both effi- cient and stable learning. The objective function for PPO is defined as: (P))
4.)] where To(a)]s)) is the proba- bility of taking action a at state 8; under the current policy f; Teeu (a,u(a (s)) is the probability under the old policy with parameters Oola; A, is the advantage estimate at time t, which mea-
1 1 E, min xo(aele)) (ails))) A,, clip xo(exls))
(ails))
sures the relative desirability of the action taken; and e is a small hyperparameter (typically 0.1 S S 0.3) that controls how much the policy can change.
The first term in the minimum represents the standard objective, while the second term applies a clipping mechanism. The clip function restricts the ratio of the new policy to the old policy to stay within the interval [1 1 +c]. If the ratio exceeds this range, the objective is capped, preventing large updates that could destabilize the learning process.
Reinforce++ Hu (2025) Enhancements. +einforce+ incorporates several key optimiza- tions to enhance training stability and efficiency of PPO. One is the addition of token-level Kullback-Leibler (KL) divergence penalty between the RL model and the supervised fine- tuning (SFT) model distributions. Additionally, Reinforce++ introduces global batch-level reward normalization, clipping, and scaling for stability, as well as advantage normalization:
Anurmalized = A-#a Where μA and a are the mean and standard deviation of the advantage values. REINFÖRCE++ is shown to be more stable compared to GRPO Shao et al. (2024b)
and faster than PPO Xie al. (2025a): Cui et al. (2025).
2.1 DRAWBACKS OF TRADITIONAL PPO/REINFORCE++
During our training process, we observed two core issues in the Reinforce++ algorithms that can easily lead to model instability and poor performance, especially for reward model training.
Instability Caused by Training Losses. The typical PPO loss function is implemented
as follows, given the log probabilities log No (aelse), log T (oe]ss), and advantages. The pseudocode in for calculating the loss is shown in Algorithm | (lines 0-3). If the ratio
so(or)e)) differs significantly, two main issues arise. First, the expression (log_probs (a.18)
uod_log_pe log_probs). exp() can lead to numerical instability. When the difference in token proba- bilities is large, the exponential function may overflow, causing the model to crash. Even forch.miinsurrl,, could result in an excessively large loss due to the minimization objective. For ex- ample: let log_probs
the computation proceeds normally, if the advantage is negative,
[-0.1, -0.1, -011), old old_log-probs = [-10,- -0.2, -0.2, -), and advantages = 1-1.0.-1.00 0.5, 0.5), the resulting loss values might be: loss = [19930.4, 1.1, -.5,, 67.1]. Such large losses can make the optimization process highly unstable. Currently, many training methods remove the KI. divergence constraint Meng et al. (2025); Peng et al.] (2025), allowing each mini-batch to perform multiple parameter updates, thereby improving data usage efficiency Hu (2025); /Shhlman et al. (2017). The former accelerates model updates, while the latter further increases the discrepancy between log ne (at (st) and
log Ta,s (a.ls)). Consequently, in these cases, the ratio between these two values can diverge significantly, leading to instability.
Instability Caused by Advantage Normalization. In addition to the training loss, the data labels for the reward model are relatively simple, consisting of only two labels: 1 and 2, which makes them easy to learn. As a result, during training, there is a high probability that the majority of the batch will correctly predict the rewards. In extreme cases, such as a batch containing 255 rewards of and 1 reward of 0, this highly imbalanced distribution, when subjected to z-Normalization, can lead to significant numerical disparities. Particularly, the advantage corresponding to the 0 reward in this example would be normalized to -15.96. A large advantage value like this can cause instability.
4

---

Published as a conference paper at ICLR 2026
Algorithm Pseudocode of PPO Loss Function in PyTorch-like style.
probor log probabi iiiis of the current policy old pro_ log probabilities the previous polloy advantages; advantage estimates for the actions
opsilon: clipping paranetor for PPO objective
4 Our Pre-Clip strategy: clip the log difference to prevent large values dog_dife log_probs leg__egg loq_dit" forch.clamm (lag_ditf, maxenp. (163), logenp. similar to ratio torch, exp (log_diff)
strategy
0. ratio plog_probs - pad_lag_probb) .epp))eeppp cex(() compute the prabability ratio
1. surrl ratic advantages f first surrogate objective
2. surr2 ratio.clampa epsi 1 epai lon] advantages second surrogate with clipping
The final loss is the minimum of the two surrogates negative loss minimisation
-torch. taurrl,
3 R1-Reward
3.1 OUR TRAINING ALGORITHM: STABLEREINFORCE
To overcome the drawbacks and enhance the stability of reinforcement learning training, propose two strategies: pre-CLIP and advantage filter, which respectively remove unstable gradients and advantages that deviate excessively from the overall distribution. In terms of reward design, we introduce the consistency reward to ensure consistency between reasoning and the final answer.
Pre-CLIP. As shown in Algorithm I under the "Our Pre-Clip strategy", our core approach to clip large ratios before computing the exponential the log probability. The value of 1e3 is hyperparameter that we find works well and the method is relatively insensitive to hyperparameter variations. The main purpose of this step is to mitigate the impact of noisy data on the overall training process with prgooabbbbilty clamping:
(c))))) + exp clipp
log .T,
log 8min log 6max
T8 (arlst
where 8mi = 10-3 6max
10 control allowable probability ratio bounds. By clipping the ratio before applying the exponential function, we can prevent overflow issues due to excessively large differences in the ratios. Additionally, this clipping ensures that large log-probability differences are mitigated, particularly when the advantage is negative, thus maintaining training stability.
Advantage Filter. To prevent the influence outliers due to the extreme imbalance in the advantage distribution, we apply the 3-sigma rule. For the standardized advantage,
Astandardized A-PA, we retain only those advantages that fall within the range of [-3, 3 F사
This range corresponds to values within standard deviations from the mean in the original distribution, as the standardization process converts the data to z-scores (unitless measures in terms of standard deviations). In the extreme case from the previous subsection, this ensures that all samples with original rewards of 1 are selected, while extreme negative advantages are excluded.
{ A Astandardized if [Astandardized otherwise
-- AA
Astandardized
T +
The final StableReinforce objective function with clipping applied:
After applying Z-normalization in the original text, the distribution becomes standard normal distribution, meaning it has mean of and standard deviation of 1.

---

Published as a conference paper at ICLR 2026
Table 1: Prompt template for reward model training.
You are highly skilled and impartial evaluator tasked with comparing two responses generated by Large Multimudal Model for given question.
Start with thorough, side-by-side comparative analysis enclosed within <think> and </think> tags. A tie is not permitted; you must choose better option.
Conclude with a single numeric choice enclosed within <answer> and </answer> tags:
-ulppu "1" Response belter; Output "2" Response 2 is betler.
Input
[Question]: !question.; [Response (answerl) ); [Response 2] (answer2)
Output Format (strictly follow)
<think>Your detailed comparative
https//se//haaaoooosssssses///l /
Σ min TT((a)(1))
A,, clip The (a)(()))) Thas (s))
1, StableReinforce (0)
6, + A,
[
Toas (at)
We discuss the difference between StableReinforce and recent RL methods for LLMs in section B
3.2 REWARD FUNCTION AND TRAINING DATA
Inspired by DeepSeek-RI DeepSeek-AI (2025), we aim to directly use RL to guide the reward model in generating the best analysis content, in order to produce high-quality model output comparisons. As a result, the prompt format in Table transforms the reward modeling task into straightforward rule-based reinforcement learning problem. By defining the model's output format, we only need to define our reward functions to complete the training process:
Formatting Reward. The model's output must adhere to a specific format of <<hhin>>
</answer>', which encourages the model to rea- son before generating the final output. This ensures that the model reflects on the reasoning process before making its final decision, enhancing both the quality and interpretability of the generated content.
Result Reward. The model's generated final result must align with human pref- erences. This primarily involves ensuring that the model's output ranking labels consistent with those of human experts, enhancing the overall usefulness and credibility.
Inconsistency Between Reasoning and Results. However, simply following existing work DeepSeek-Al (2025); Yu et al. (2025b) in setting has led to unexpected results. During training, we observe discrepancies between the model's reasoning and its final an- swer. For example, the reasoning might conclude that response 2 is better but the model outputs answer 1, as seen in <think>... response is better</ <answer>> SaatttueriIIaassisiii /answerr> This inconsistency arises because, we provide no supervision for the reasoning process and only score based on the outcome. When a sample demonstrates poor reasoning but produces the correct answer, this pattern inadvertently reinforced, leading the model to believe that reasoning and the final answer are not necessarily linked. Consequently, the model may learn to generate correct answers without a coherent reasoning process. This could even result in the model treating the reasoning process irrelevant, or worse, outputting repeti- tive content or random noise. To address this issue, we introduce an additional component, Owwen25-vV--7--nnstruee
supervisor to verify whether the reasoning and the final result are consistent. This addition helps ensure that the reasoning process and output align well, introducing the following reward function:

---

Published as a conference paper at ICLR 2026
Table 2: Summary of datasets used for training, including the category (text or image), dataset name, the number of original samples, and the number of samples selected for final training.
Datasel RLAIFF et al.
Original Sample
Dataset
Original
*Smple
2024a
74,802
MM-RI. SMM-RIIF-II Long Zhang al.
2025a
41,163
41,163
VL-Feedback
80,258
MM-RL Shortt Zhang
46,281
POVID POVID Zhoir
17.184
100k
Mec Zhang et a.
20258
8,306
8,306
WildViston-Danie
22024a
10,383
MIMRLHFF-aaeey Zhang St
9,990
9,990
Consistency Reward. The model's final result must be consistent with its intermedi- ate reasoning process. This function ensures that the final answer is directly derived from the model's reasoning process, rather than being generated in isolation from the reasoning steps.
Integrating the consistency reward as a separate reward and combining it with the previous two reward functions can lead to situation where the model, despite selecting the wrong answer, may still receive high overall reward due the consistency component. This could result in the model overly prioritizing consistency. To mitigate this issue, the final reward is designed as follows:
Final Reward Result Reward (1 0.5 Consistency Reward) +0.5 + Formatting Reward.
This ensures that the consistency reward is only taken into account when the result is correct, thereby preventing the model from excessively favoring consistency in cases where the outcome is incorrect.
Dataset Construction. As shown in Table 2 we sample preference data from multiple existing datasets for training. To ensure data quality and diversity, sample all instances from the human-annotated dataset MM-RLHF, and an additional 100,000 samples from other multimodal preference datasets. The final dataset is termed R1-Reward-2001 combines these diverse instances to create a robust training foundation for our model. We then randomly shuffle the data to ensure balanced ratio of answers and (1:1), preventing the model from favoring specific answer. Each sample consists of a quadruple: (question, answer 1, answer 2, ground truth choice).
which
Saain--o--Touught Wei et al. (2022) Cold Start. Since MLLMs are not initially trained for reward modeling tasks, directly using MLLMss for reinforcement learning training yields poor and unstable results. Therefore, we first use GPT-4o as annotator for each sample in the R1-Reward-200k dataset, utilizing the prompts from Table to construct SFT data. The temperature is set to O, with a maximum of 3 attempts. We also record how many attempts GPT-4o needs to generate a final response that is judged to be correct (i.e., the same to the ground truth choice). This SFT training phase teaches the model the basic format and familiarizes it with the reward modeling task.
Long-CoT
RL Training Data. We use all samples from the SFT phase with at least 2 attempts, as well as samples where GPT-4o fails to produce the correct answer after three attempts. These samples exhibit smaller differences between answer and answer 2, making them more difficult.
4 Experiments
Implementation Details. Both SFT and RL experiments are conducted on 4xH800 (80G) GPUs. The SET phase trains for epoch and takes approximately 8 hours, while the RL phase trains for epochs and takes 12 hours. We use SivenViL-25-77-Innstrr
as the base model for training. During the SFT phase, the learning rate set to 1e-5, and the batch size is set to 256. We use the OpenRLHF Hu et al. (2024) framework for RL. The training batch size is set to 128, and the rollout batch size is set 256. The learning rate is set 1e-6, and the initial KL coefficient is set to 0.
Baseline Algorithm. At the algorithmic level, we primarily compare two entities: the reward model and MM-RL-HF-Reware /hhng et al. (2025a). For the former, we replace the language
7

---

Published as a conference paper at ICLR 2026
Table 3: VLReward Bench. Performance comparison of our reward model (R1-Reward) with existing open-source and private counterparts.
Models
*farrm
General
Hallucination
Reasoning
Overall Acc
Macro Acc
ProprieLacy
Claude-3.5 (2024-06-22)
43.40
55.00
62.30
55.30
53.60
GPT-4 (2024-08-06)
49.10
67.60
70.50
65.80
62.40
Gemini-1.5-Pro (2024-09-24)
72.50
64.20
67.20
62.50
Claude-3.--Sonnee
68.08
70.70
60.81
66.31
66.53
Open Suugue
V. et 2025
7B 16.55
8.93
22.11
16.48
16.53
SIME Zhang etal Auzea
7B
7:23
27.09
18.60
19.04
17.64
InternVI PPZCChin Tanr dila
44.50
LLaVA-Crtia Nione
20124
8B
54.60
38,30
59.10
41.20
44,00
Molmo
Deitkl
72B
33.91
42.30
54,90
44.10
43,70
QwenZ-vE
el 20248
72B
36 38.10
32,80
58.00
39.50
43.00
NVLM-D
205))
728
38,90
31.60
62,00
40.10
44.10
Llama-3
Minghad Tani
90B
Bee cdd Modies
MM-RLHH- Zhang
2015a
7B
45.04
50.45
57.55
50.15
51.01
MCC-2.-Reeware Zang dlal
7B
84.70
62.50
62.90
65,80
70.00
Ours
RI-Reward
7B
63.84
85.71
64.78
71.92
Viting815
7B
66.32
89.06
73.70
76.46
76.36
head of the base LLM with two-layer MLP that outputs float value the reward. Training is done using a binary classification loss. For the latter, in addition to the traditional binary classification loss, an additional critic loss is required. Specifically, the model first outputs an evaluation of the candidate, and then, based on the evaluation, the reward head provides the reward value.
Evaluation Benchmarks and Metrics. The multimodal benchmark includes VL-Reward Bench et al. (2024a)), Multimodal RewardBench Yasunaga et al. (2025), and MM-RLHF. Reward Bench Zhang al. (2025a). VL-Reward Bench uses two metrics: Overall Accuracy, which measures the percentage of model decisions aligning with human preferences, and Macro Average Accuracy, which calculates the average accuracy across task categories to address distribution imbalance. Multimodal RewardBench evaluates reward models in six areas: correctness, preference, knowledge, reasoning, safety, and visual question answering (VQA). MM-RLHii-Reward Bench has two metrics: 1. Acc, measuring correct identification of the preferred response, and 2. Acc+, which assesses the model's ability to correctly rank all response pairs, focusing challenging cases with small ranking differences or danstto-distinguish pairs.
4-1
MAIN RESULTS
We evaluate the performance of R1-Reward on three common multimodal reward model benchmarks. On the VLReward Bench (Table R1-Reward achieves the best overall perfor- mance, with an average accuracy of 71.92%. This represents a roughly 9.3% improvement in overall accuracy compared to the previous best open-source model, IXC-2.5-Reward. Notably, Rxw-ad trains on more than 1 million samples, while our training data consists of 200k samples, highlighting significant improvement in data efficiency. In comparison to other open-source models, R1-Reward demonstrates a larger margin of improvement. Among closed-source models, Gemini-1..-Pro performs the best, but R1-Reward outper- forms across all dimensions, further demonstrating its superiority.
On the Multimodal Reward Bench (Table R1-Reward achieves the best performance across all dimensions, with a 14.3% improvement over the previous state-of-the-art. noting that the Multimodal Reward Bench is derived from over ten existing benchmarks and reconstructed into a unified set, with minimal overlap with our training data. This further demonstrates R1-Reward's remarkable generalization ability across different datasets. The performance R1-Reward on the MM-RLHF-RRewadd Bench and Test-Time Scaling analysis is shown in Section. C44 and C.5
worth
8

---

Published as a conference paper at ICLR 2026
Table 4: Multimodal Reward Bench. Performance comparison of our reward model (R1- Reward) with existing open-source and proprietary counterparts.
Knowledge
Reasoning
Safety
VQA
Preference
Math
Coding
GPI-4
20.8
62.6
69.0
72.0
62.6
60.1
74.5
87.2
Gemini 1.5 Pro
71.9
63.5
62.7
66.3
68.9
56.5
94.5
$7.2
Clasde Sonnet
71.5
62.6
67.8
73.9
68.6
68.3
76.5
85.6
Clasde 37 Sonnet
71,9
36.8
637
75.1
76.3
71.3
72.0
56.8
Open SouECe Nodels
SIME Zhank stal
42.0
42.3
52.2
47.5
45.5
35.5
19.1
53.8
VITA-1
35.6
54.3
58.1
30.
11am-3. Soneee
Minghao Yang
37.8
63.8
5888
Molmo Declp
52.0
99.3
62.3
Qwen-2
36,4
02.5
7012
$8.99
9631
85.3
MM-RU HF-Reaar Thang
2005a
7B
67.1
61.7
62.
54.3
58.4
57.5
92.9
26.8
DXC Reward Zang
7B
66.6
607
6122
56.8
63.0
90.5
99.9
61.1
Ri-Reward
7B
80.2
22.5
24.0
24.9
83.1
27A
99.6
56.5
Voting015
7B
63.
78.
77.2
766
81.3
85.6
95.4
87.0
4.2 ABLATIONS AND ANALYSIS
R1-Reward Demonstrates High Data Efficiency. In Table we compare the performance of traditional reward model (using two-layer MLP the reward head) and MM-RLHF Reward (which first generates a critic and then generates the reward) trained on the same dataset. For MM-RLHF-RRward, the training data must include evaluation for each response. To achieve this, we use GPT-4o to generate corresponding evaluations for each sample, which may be slightly less accurate than the human annotations used in the original work. All the models' backbones are the same. As shown in the table, the traditional reward model, when trained with only 200K data samples, performs poorly, In most cases, MM-RLHF outperforms the traditional reward model. Its superior performance in the "hallucination" dimension likely due the generated critic. Comparing these two baselines, the reinforcement learning- based approach significantly enhances the reward modeling capabilities, even with the same amount of data. Moreover, our SFT approach shows advantages over both the traditional reward model and HIM-LLHF-Reward We believe this is primarily due to that we allow direct comparison of two responses during the scoring process, whereas existing methods score responses independently before comparing them.
Based on the appendix experiments, our additional studies provide comprehensive valida- tion of the R1-Reward model and StableReinforce algorithm across multiple dimensions. The appendix demonstrates that each component of StableReinforce (advantage filter, pre- clip, and consistency reward) is essential for training stability, with ablations showing that removing any component leads to decreased performance and potential training collapse. Hyperparameter analysis confirms the robustness of our design choices. The model shows strong generalization capabilities, achieving state-of-the performance the challenging MM-RLHF-Rewan Bench and demonstrating effective test-time scaling where performance consistently improves with increased sampling. Human evaluation validates that our ap- proach produces superior reasoning quality, with annotators preferring R1-Reward's reason- ing processes in 72.5% of cases. The method proves robust annotation quality, maintaining strong performance even when using weaker models like @ween.55VVL-BB instead of GPT- 4o for data construction. Finally, downstream validation shows consistent improvements across multiple benchmarks when using R1-Reward to train smaller MILMs, confirming the practical utility and transferability our reward modeling approach.
Conclusion
In this paper, we introduce R1-Reward, a MRM trained using the StableReinforce algo- rithm. We demonstrate that RL can be effectively applied to reward modeling, significantly enhancing its performance. Our approach addresses key challenges, including training in- stability, the advantage normalization limitation, and inconsistencies between reasoning and results. By incorporating techniques such as pre-clipping advantage filtering, consistency reward and progressive difficulty training strategy, StableReinforce stabilizes training

---

Published as a conference paper at ICLR 2026
and improves model performance. Experiments show that R1-Reward outperforms SOTA models on several multimodal reward model benchmarks, with significant improvements in accuracy and data efficiency. While R1-Reward demonstrates excellent test-time scaling capabilities and lays the foundation for future research on incorporating reinforcement learning into MRMs, we acknowledge the limitations of our current work. Specifically, only test a basic majority voting strategy for test-time scaling, and more advanced methods could potentially yield further performance improvements Liu et al. (2025b). Additionally, the performance of R1-Reward on the MM-RLHF Reward Bench shows only modest im- provements over previous SOTA models. Therefore, improving training strategies further enhance the foundational capabilities of reward models remains an important open problem.
LLM Usage Declaration
In this research, LLMs were used exclusively for grammar checking and to assist with the clarity of language. No LLM was involved in the ideation or content generation processes. The authors take full responsibility for all content presented in the paper, including any generated by the LLM. We have ensured that the use of LLMs complies with ethical standards and does not constitute any form of scientific misconduct or plagiarism.
Acknowledgement
This work was jointly supported by the National Key R&D Program of China (Grant No. 2022ZD0117901), the National Natural Science Foundation of China (Grant Nos. 62506158 158, 62441234, 62322607, 62276261, 62373355, 62306311, and 62236010), the Basic Research Pro- gram of Jiangsu (Grant No. BK20251183), and the Beijing Natural Science Foundation (Grant No. L252033).
References
Shu Pu, Yaochen Wang, Dongping Chen, Yuhang Chen, Guohao Wang, Qi Qin, Zhongyi Zhang, Zhiyuan Zhang, Zetong Zhou, Shuang Gong, et al. Judge anything: Mllm as judge across any modality. arXiv, 2025.
Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. Mllm-as-ajuudge: Assessing multimodal llm-as-a-judge with vingonnl benchmark. In Forty-first International Conference on Machine Learning, 2024a.
Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jinguo Zhu, Xiangyu Zhao, Yangzhou Liu, Yue Cao, Shenglong Ye, Xizhou Zhu, Lewei Lu, Haodong Duan, Yu Qiao, Jifeng Dai, and Wenhai Wang. Visualprm: An effective process reward model for multi- modal reasoning, 2025a.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan,Jina Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv, 2024a.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv, 2024.
Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nvlm: Open frontier- class multimodal 1lms. Xiv, 2024.
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b. at Xiv, 2024.
10

---

Published as a conference paper at ICLR 2026
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented rlhf. 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhiní Agarwal, Katarina Slama, Alex Ray, et al. Training language mod- els to follow instructions with human feedback. Advances in neural information processing systems, 2022.
Yi-Fan Zhang, Tao Yu, Haochen Tian, Chaoyou Fu, Peiyan Li, Jianshu Zeng, Wulin Xie, Yang Shi, Huanyu Zhang, Junkang Wu, et al. Mm-rlhf: The next step forward in multimodal Ilm alignment. arXiv, 2025a.
Jinda Lu, Junkang Wu, Jinghan Li, Xiaojun Jia, Shuo Wang, YiFan Zhang, Junfeng Fang, Xiang Wang, and Xiangnan He. Dama: Data- and model-aware alignment of multi-modal llms. arv, 2025.
Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. Llava-critic: Learning evaluate multimodal models. CVPR, 2024.
DeepSeek-Al. Deepseek-r1: Incentivizing reasoning capability in lims via reinforcement learning. arXiv, 2025.
Tao Yu, Chaoyou Fu, Junkang Wu, Jinda Lu, Kun Wang, Xingyu Lu, Yunhang Shen, Guibin Zhang, Dingjie Song, Yibo Yan, et al. Aligning multimodal IIm with human preference: A survey. Xru, 2025a.
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arv, 2025a.
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vim-rl: A stable and generalizable rl-style large vision-language model. Xiv, 2025a.
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arviv, 2025.
Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, et al. Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning. arv, 2025.
Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-rl: Reinforcing video reasoning in milms. arXiv, 2025.
Jiaxing Zhao, Xihan Wei, and Liefeng Bo. R1-omni: Explainable omni-multimodal emotion recognition with reinforcing learning. arXiv, 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systemss, 36:53728-533 53741, 2023.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model mest-training. Xiv, 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXio preprint ar arXi::1707.063477 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv, 2024a.
11

---

Published as a conference paper at ICLR 2026
Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arxio, 2025.
Lei Li, Yuancheng Wei, Zhihui Xie, Xuqing Yang, Yifan Song, Peiyi Wang, Chenxin An, Tianyu Liu, Sujian Li, Bill Yuchen Lin, Lingpeng Kong, and Qi Liu. Vlrewardbench: A challenging benchmark for vision-language generative reward models. arXiv, 2024a.
Michihiro Yasunaga, Luke Zettlemoyer, and Marjan Ghazvininejad. Multimodal reward- bench: Holistic evaluation of reward models for vision language models. arXie, 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ar arii 2024b.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rule-based reinforcement learning. Xiv, 2025a.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arxiv, 2025.
YingZhe Peng, Gongrui Zhang, Xin Geng, and Xu Yang. Lmm-rl. https://github.cr TideDra/Imo /mm r1. 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv, 2025b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thous though prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
Tianyu Yu, Haoye Zhang, Qiming Li, Qixin Xu, Yuan Yao, Da Chen, Xiaoman Lu, Ganqu Cui, Yunkai Dang, Taiwen He, Xiaocheng Feng, Jun Song, Bo Zheng, Zhiyuan Liu, Tat- Seng Chua, and Maosong Sun. Rlaif-v: Open-source ai feedback leads to super gpt-4v trustworthiness. CVPR, 2024a.
Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. 2023.
Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning. arviv, 2024.
Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, and Bill Yuchen Lin. Wildvision: Evaluating vision-language models in the wild with human preferences. arv, 2024a.
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and high-performance rihf framework. arXiv, 2024.
Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Yangze Li, Zuwei Long, Heting Gao, Ke Li, et al. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction. Xiv, 2025.
Jiawei Zhang, Tianyu Pang, Chao Du, Yi Ren, Bo Li, and Min Lin. Benchmarking large multimodal models against common corruptions. arviv, 2024a.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic lingus-iig tasks. Xiv, 2023.
12

---

Published as a conference paper at ICLR 2026
Xiaoyu Tan Minghao Yang, Chao Qu. Inf-orm-Ilama3-1-700 2024.
URL
hhttss:/buggginffceccc////t/ http://iiiii/I--------aaaaaaaa 1 (b
Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Ziyu Liu, Shengyuan Ding, Shenxi Wu, Yubo Ma, Haodong Duan, Wenwei Zhang, et al. Internlm-xcompoeed 5-reward: A simple yet effective multi-modal reward model. arXiv, 2025.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-s multimodal models. arXiv, 2025.
Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling. arXiv, 2025b.
Chaoyou Fu, Yi-Fan Zhang, Shukang Yin, Bo Li, Xinyu Fang, Sirui Zhao, Haodong Duan, Xing Sun, Ziwei Liu, Liang Wang, et al. Mme-survey: A comprehensive survey evaluation of multimodal llms. ar 2024a.
Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, et al. Mme-realworld: Could your multimodal Ilm challenge high-resolution real-world scenarios that are difficult for humans? ICLR, 2024b.
Feng Li, Hao Zhang, Yi-Fan Zhang, Shilong Liu, Jian Guo, Lionel M Ni, PengChuan Zhang, and Lei Zhang. Vision-language intelligence: Tasks, representation learning, and large models. ar arviv, 2022.
Huanyu Zhang, Chengzu Li, Wenshan Wu, Shaoguang Mao, Yifan Zhang, Haochen Tian, Ivan Vulié, Zhang Zhang, Liang Wang, Tieniu Tan, et al. Scaling and beyond: Advancing spatial reasoning in milms requires new recipes. ar preprint ar arXio:2504.1507, 2025b,
Biao Yang, Bin Wen, Boyang Ding, Changyi Liu, Chenglong Chu, Chengru Song, Chongling Rao, Chuan Yi, Da Li, Dunju Zang, et al. Kwai keye-vl 1.5 technical report. arXiv preprint arXio:2509.01563 2025a.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. Xiv, 2025.
OpenAl. Introducing openai o1-preview. 2024. URL http:::/ppeeaii.comm//////////// cow/inex//////////x//////x index/
Latrodduciagg--ppeaaa-------eeee
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv, 2024b.
Llama3 Team. The llama 3 herd of models. arXiv, 2024a.
Yunhang Shen, Chaoyou Fu, Shaoqi Dong, Xiong Wang, Yi-Fan Zhang, Peixian Chen, Meng- dan Zhang, Haoyu Cao, Ke Li, Xiawu Zheng, Yan Zhang, Yiyi Zhou, Ran He, Caifeng Shan, Rongrong Ji, and Xing Sun. Long-vita: Scaling large multi-modal models to 1 million tokens with leading short-context accuracy, 2025b.
Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, Liang Wang, and Rong Jin. Beyond Ilava-hd: Diving into high-resolution large multimodal models. arxiv, 2024c.
Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. Llava-mini: Efficient image and video large multimodal models with one vision token, 2025c. URL https: : //arxivv hrg//bbs/2501 03895
13

---

Published as a conference paper at ICLR 2026
Yi-Fan Zhang, Weichen Yu, Qingsong Wen, Xue Wang, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. Debiasing
D multimodal large language models. arXiv, 2024d.
Yadong Li, Jun Liu, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Guosheng Dong, Da Pan, et al. Baichuan-omni-1.5 technical report. arXio, 2025a.
Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Shaoqi Dong, Xiong Wang, Di Yin, Long Ma, et al. Vita: Towards open-source interactive omni multimodal llm. ar Xiv, 2024b.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv, 2024.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv, 2024b.
Wulin Xie, Yi-Fan Zhang, Chaoyou Fu, Yang Shi, Bingyan Nie, Hongkai Chen, Zhang Zhang, Liang Wang, and Tieniu Tan. Mme-unify: A comprehensive benchmark for unified multimodal understanding and generation models. arviv, 2025b.
Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vulić, and Furu Wei. Imagine while reasoning in space: Multimodal risualization-o-tthuuht. In The Forty-Second International Conference on Machine Learning, 2025b.
Huanyu Zhang, Wenshan Wu, Chengzu Li, Ning Shang, Yan Xia, Yangyu Huang, Yifan Zhang, Li Dong, Zhang Zhang, Liang Wang, et al. Latent sketchpad: Sketching visual thoughts to elicit multimodal reasoning in milms. arXiv preprint ar Xio:20500.24514 2025d.
Yan Yang, Haochen Tian, Yang Shi, Wulin Xie, Yi-Fan Zhang, Yuhao Dong, Yibo Hu, Liang Wang, Ran He, Caifeng Shan, et al. A survey of unified multimodal understanding and generation: Advances and challenges. Authorea Preprints, 2025b.
OpenAl. Thinking with images. htttp://oppnnii.ii http:////onaaaco/////eaooo / thiiking--iithh with--nngssst 2025.
Yi-Fan Zhang, Xingyu Lu, Shukang Yin, Chaoyou Fu, Wei Chen, Xiao Hu, Bin Wen, Kaiyu Jiang, Changyi Liu, Tianke Zhang, et al. Thyme: Think beyond images. arXiv preprint ar arXiv:2508 11630, 2025e.
Yi-Fan Zhang, Haihua Yang, Huanyu Zhang, Yang Shi, Zezhou Chen, Haochen Tian, Chaoyou Fu, Haotian Wang, Kai Wu, Bo Cui, et al. Basereward: A strong baseline for multimodal reward model. ar preprint arXiv:2509.1112
2025f.
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in lims. arXiv, 2024a.
Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, and Junge Zhang. Uncertainty-aware reward model: Teaching reward models to know what is unknown. arXiv, 2024.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models, 2024b.
Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan, Chao Zhang, et al. Self-generated critiques boost reward modeling for language models. ar Xiv, 2024b.
Nicolas Le Roux, Marc G Bellemare, Jonathan Lebensold, Arnaud Bergeron, Joshua Greaves, Alex Fréchette, Carolyne Pelletier, Eric Thibodeau-Laue Laufer, Sándor Toth, and Sam Work. Tapered off-policy reinforce: Stable and efficient reinforcement learning for 1lms. ar Xiv, 2025.
Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, et al. Minimax-01. Scaling foundation models with lightning attention. arXiv, 2025c.
14

---

Published as a conference paper at ICLR 2026
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl -lerooo-ikke training: A critical perspective. arXiv, 2025c.
Penghao Wu and Saining Xie. V?: Guided visual search as a core mechanism in multimodal llms. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 13084-13094, 2024.
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal arxiv: dataset for improving scientific comprehension of large vision- language models. arviv, 2024c.
Xiyao Wang, Zhengyuan Yang, Chao Feng, Hongjin Lu, Linjie Li, Chung-Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. Sota with less: Mcts-guided sample selection for data-efficient visual reasoning self-improvement. arXiv preprint ar arXic:2504.07934 2025b.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs /b0/2200.00300,, 2024c.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216-233, Springer, 2024b.
Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. Mme-realworld: Could your multimodal Ilm challenge high-resolution real-world scenarios that are difficult for humans? ICLR, 2025g.
Lin C Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arviv, 2024b.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In ICLR, 2024b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Xrv, 2023.
15

---

Published as a conference paper at ICLR 2026
A Related Work
MLLMs. Thanks to the success of language models, MLLMs have rapidly developed in
recent years, with their task handling capabilities and model performance advancing at
al. (2024b); Yu et al.) (2025a) ) Li et al. (2022); Zhang et al.) (2025b); ; Yang et al. (2025a). For example, traditional multi-modal large models pertorm well in handling complex high-resolution images and human dialogue Bai et al. (2025); OpenAL. (2024); Li al. (2024b); [eam (2024a); Wang et al. (2024a). A series of works focus on improving the context length Shen et al. (2025b), computational efficiency Zhang et al. (2024c 2025c), reducing hallucinations Lu et al. (2025); Zhang et al. (2024d), enhancing conversational abilities abilitiesxiiong et al. (2024). and aligning with preferences ZZann et al. (2025a). Omni-MLLMs are capable of simultaneously processing multiple modalities such as speech, video, images Li et al. (2025a); Zhao et al. (2025), and even interacting with users via voice Fu et al. (2024b) 2025 Unity-MLLMs can perform mixed-modal generation Xie let al. (2024)) leam (20246); Xie et al. (2025b);; Li et al. (2025b); Zhang et al. (2025d); Yang et al. (20256) for example, generating an image with auxiliary lines while understanding math problem, enhancing both generation and comprehension abilities. Recently, with the success of Open Al's O1 model and Deepseek's R1 model DeepSeek-Al (2025), the rule-based reinforcement learning approach has gained sigmificant attention in the multi-modal field. At the same time, "think with image" paradigm ApenAT (2025); Zhang et al. (2025e) has also attained much attention for visual reasoning. Various studies are devoted to enhancing the reasoning capabilities of multi-modal models. However, as far as we know, no work has yet explored whether the reinforcement learning paradigm can be transferred into reward modeling.
fast pace Fu et al. (2024a); ); 2
Reward Model Training. The reward models most relevant to this paper are pure text reward models and multi-modal reward models. There are generally three main approaches to reward modeling. The first approach is directly use a language model or multi-modal model as the reward model by designing precise prompts that allow them to output score or ranking Xiong et al., (2024); Zhang et al. (2025f). However, this method heavily ability and comprehension. The second approach involves connecting the latent representation of a language model to a reward head (typically an MLP linear layer), where the model directly outputs a score. During training, the reward modeling is converted into a binary classification task. This approach is computationally efficient, but it lacks interpretability Liu et al. (2024a); Zang et al. (2025); Minghao Yang (2024); Lou et al. (0024); WWng et al. (2024b). ) The final type of model simultaneously learns to evaluate the question-answe pair and creates an additional reward head to provide the score Yu et al.] (2024b);; Zhang et al. (2025a). This model strikes balance between interpretability and computational efficiency, but usually requires specific data formats or training strategies. This paper proposes training a reward model through reinforcement learning. The model first outputs an inference for a given question-answer pair and ultimately provides ranking. Through reinforcement learning, we force the model to learn the format of the reward modeling task, avoiding the shortcomings of the first approach without requiring an additional reward head, while maintaining the model's interpretability.
depends on the model's 'ss instruction-tollowe
B Comparison to Existing RL Studies
In the field of RL for LLMs, recent concurrent advancements have emerged, some of which share similarities with our approach or report analogous observations. Although these methods have not been directly applied to multimodal domains or reward modeling, we pro- vide a concise discussion in this section for comparative purposes. Notably, DAPO Yu et al. (2025b), TOPR Roux et al. (2025), and Minimax-01 Li et al. (202cc) focus on improving CLIP operations, carticularly in the design of the epsilon parameter. In contrast, our approach fundamentally differs by clipping the logits ratio prior to the exponential operation. This strategy enhances numerical stability and mitigates the adverse effects of negative advan- tages. Similarly, Dr. GRPO Liu et al. (2025c) identifies the detrimental impact of advantage normalization and adopls strategy of setting variance to 1. However, in scenarios with high
16

---

Published as a conference paper at ICLR 2026
Table 5: Evaluation results on VL. Reward Bench comparing different models and training setups, including baselines, models trained with 11-Reward-200k and ablation studies (ours).
VL-Reward Bench
Data
Models
General
Hallucination
Reasoning
Overall Acc
Basel lines
More than 1M MM-RLHF-120K
84.70
62.50
62.90
65.80
MM-RLHF-Reward
45.04
50.45
57.55
50.15
Trained by R1-Reward-2000
R1-Reward-2000 R1-Reward-2000 -20K
Reward Model
56.71
56.03
48.67
56.41
HM-RLHFFRewaad
61.01
62.28
59.30
60.80
Ours
R1-Reward-2000
StableReinforce
63,84
85.71
64.78
71.92
R1 R1-Reward-200
wo advantage
63.43
77.45
62.38
68.96
R1- 21-Reward-200K
wo pre-clip
62.06
77.44
61.23
67.36
-1-eeward-2000 -200K
Reinforce++
Collapse
Ni-Reward-2009
Only Long-Cot
59.92
72.27
60.01
64.80
original variance, this approach allows extreme values to dominate. Instead, employ 3-sigma filter, which preserves the benefits of z-normalization while effectively removing outliers.
C Additional Experiments
C.1 ABLATION STUDIES OF THE STABLEREINFORCE ALGORITHM.
We examine the impact of each component of the StableReinffrre algorithm on the training process and final results. First, we emphasize the necessity the Consistency Reward Function. Removing this function results in significant hallucination behaviors across differ- ent algorithms, making challenging to achieve stable evaluation outcomes. Additionally, directly applying the Reinforce++ algorithm causes the model to crash, with the loss becom- ing NaN and the response length reaching the preset maximum length, while the output consists entirely of garbled text. In Table 5nd Figure g we present the effects of removing each module on final performance and changes in training dynamics. We observe that the Advantage Filter and Pre-Clip modules primarily ensure training stability by effectively removing outliers from the loss. Removing any of these components results decreased final accuracy, reduced training stability, and the model's output length failing converge to shorter values.
C.2 ABLATION STUDIES OF HYPERTARAMETERS.
Regarding hyperparameters, the advantage filter directly follows the 3-sigma rule, common statistical screening strategy, with no extra hyperparameters. The pre-clip parameter to np.log( (10*) as an empirical value. This corresponds to policy update ratio exceeding 10, which is an extremely drastic update. Larger updates are usually not effective learning signals but rather caused by instability or noise in early training. We conducted study on this value, with partial results shown in Table 6 We observe that setting pre- clip between 5-10 results in minor impact, but if set very large (e.g., 100), performance degenerates to nearly the same as without pre-clip.
ablation
C3 CONSISTENCY Loss WEIGHT ANALYSIS
To illustrate the weighting the consistency loss, we conducted corresponding ablations shown in Table Here, "external" refers to the formula:
17

---

Published as a conference paper at ICLR 2026
Table 6: Ablation study on pre-clip parameter values
Clip Ratio
Models
VL-Reward Bench
General
Hallucination
Reasoning
Overall Acc
10.00
Stablekeinforce
63.84
85.71
64.78
71.92
5.00
StableReinforce
63.82
84.78
63.29
70.63
100.00
StableReinforce
62.43
77.98
62.32
67.58
w/o pre-clip
62.06
77.44
61.23
67.36
Final Reward = Result Reward +0.5 Consistency Reward +05 Formatting Reward (1)
instead of
Final Reward Result Reward (1+0..5 Consistency (eward)++055 Formatting Reward
(2)
We observe that removing this reward leads to a marked performance drop. This because our output only has options or 2, and at early training stages, the model often thinks option 1 is better but outputs answer (inconsistent behavior). reward correct samples, the model may regard the reasoning process meaningless, causing training collapse. Using the "external" weighting formula sometimes leads to higher rewards caused by the consistency score even when all samples fail to give correct answers, which misguides optimization and harms performance.
these samples receive the same
Regarding the weight, smaller weight (0.1) can still enhance consistency rollout rewards when the model answers correctly. A too large weight (1.0) unduly reduces the importance of accuracy reward. Thus we chose 0.5. We have also tried using larger models as judges but did not observe experimental performance improvement. This is because the task itself is relatively simple - it only requires judging whether the conclusion in the reasoning process ("1 is better" "2 is better") is consistent with the final answer (1 or 2), which does not need a larger model.
Table Ablation study on consistency reward weighting VL-Reward Bench
Weight for Consistency Reward
General
Hallucination
Reasoning
Overall Acc
1.00
61.28
83.92
61.09
68.76
0.50
63.84
85.71
64.78
71.44
0.10
63.24
84.76
63.32
70.44
external
55.24
56.73
57.82
56.60
0.00
51.63
49.55
54.26
51.81
C.4 RI-REWARD PERFORMANCE THE MM-RLHF-REWARD BENCH
The MV-RLHHF-Reward Bench (Table presents higher level of difficulty, particularly when directly utilizing language models as reward models. The best-performing model, Claude-3.7-Sonnet, achieves an accuracy 65% on the Acc+ metric. Existing reward models perform well, with IXC-2.5-Reward surpassing an Acc+ score of 50%, while the top reward model, MM-RLHF--Reward exceeds 60%. However, MM-RLHF-RRwaard is trained on dataset that closely aligns with the distribution of this benchmark, which limits its general- ization ability. As a result, its performance on the VL Reward Benchmark is suboptimal. In contrast, R1-Reward demonstrates balanced performance across all benchmarks. Moreover, when performing voting on five sampled results, its accuracy reaches 85.3%, and when sampling 15 times, it reaches 86.47% signiffacca outperforming existing models.
18

---

Published as a conference paper at ICLR 2026
Table & MM-RLHT-Reward Bench. Performance comparison of our reward model (R1-
Reward) with existing open-source and proprietary counterparts.
Models
#Paarm
Mcq
Long
Short
Safety
Video
Acc
Acc+
Propriet ary Models
Gemini-2.0-Flaan -xp
33.33
45.94
67.64
43.75
32.00
44.71
13.04
GPT-4o (2024-08-06)
64.28
78.37
44.11
56.25
40.00
58.23
26.01
Claude-3.5-Sonnet (2024-06-22)
64.28
67.56
55.88
65.62
60.00
62.94
26.11
Claude-3.7Sonnnct
66.67
91.89
91.18
87.50
76.00
82.35
65.22
Open Source
Models
SlIM Zhang
2024a
8B
23.81
10.81
14.71
12.50
7.52
17.10
1.76
VITA- Fu et al.
2025
7B
24.97
21.62
11.76
18.75
12.40
20.58
2.78
Intern-VI hu et al.
2025
8B
35.71
56.76
23.53
37.50
32.00
37.65
6.52
NVLM-D Dai et al. 2024
72B
42.85
32,43
8.82
50.00
40.00
34.70
6.52
Llama-3.2 Minghao Yang
2024
90B
19.04
35.13
38.23
50.00
40.00
35.29
10.86
Qwen2-VI Wang et al.
2024a
72B
45.23
62.16
47.05
46.88
36.00
48.23
13.04
Reward
Models
IXC-2.5 11222.5-eewarde Zang al.
2025
7B
52.38
91.89
65.65
62.50
88.00
71.18
50.00
MM-RLHF-Rewand Zhang et al.
2025a
7B
83.00
97.00
74.00
69.00
88.00
82.00
63.00
Ours
R1-Reward
7B
80,95
89.19
82.35
75.00
72.00
80.59
54.35
Voting@15
7B
83.33
97.30
91.18
78.12
80.00
86.47
67.39
Ne00gg A00 19
10r
AzV
Previcx 907A
Aveenge ADC
Vith Any
Previcse RDW
Voln Any
Faviicue 50%
Awoggg
12
12
13
14
J.Ruamp cs
(a) MM-RLHF Reward Bench
(b) VL Reward Bench
(c) Multimodal Reward Bench
Figure 3: Inference-time performance scaling of R1-Reward on three benchmarks: (a) MM-
RLHF Reward Bench, (b) VL Reward Bench, and (c) Multimodal Reward Bench.
C.5 TEST-TIME SCALING
In Figure 3 we explore whether increasing the number of samples can lead to improved important note that the temperature set to 1.0, which causes the result with single sample (k 1) to slightly differ from the main results (which use greedy decoding by default). As the number of samples increases, the model's performance improves consistently. "Vote" refers to majority-voting strategy, while "Any" counts as correct at least one of the sampled results is correct. At k 15, the accuracy of "Any" approaches 100%, indicating that the R1-Reward has the potential to perfectly classify all samples; however, additional data or training strategies are needed to fully unlock this potential. Moreover, the "Vote" results demonstrate significant advantage over previous state-of-the-art models, with more noticeable improvement when k < 5. The benefits from increasing the number of samples gradually diminish more samples are added. This highlights the potential of R1-Reward in test-time scaling.
performance.
C.6 AHA MOMENT OF R1 R1-REWARD.
Through our task design and reward function formulation, the R1-Reward model effectively learns the reward modeling task structure during the SFT phase. Following reinforcement learning, it reduces the length of reasoning to enhance efficiency. Visual examples of the model's 's output appear in Figures 5 and 6. The model autonomously learns a process to assess response quality. It first defines the goal, analyzes the image, attemmts to solve the problem, and provides
this, the model evaluates Response 1 and Response 2, compares the two outputs, and gives a final ranking, Simultaneously, the model demonstrates different reflection patterns. In Figure the model encounters an error in its
answer. Based
19

---

Published as a conference paper at ICLR 2026
20
Method StableReiiforre
Response Length
700
Policy Loss
15
wo Advantage Filter
650
10
Method
600
StabteReinforce
Advantage Filter
0
50
100
150
200
250
300
350
50
100
150 Steps
200
250
300
350
Step
(a)
(b)
3000
Method
StableReeinforee
Response Length
700
wo Pre Clip
2000
650
Polley
1000
Method
600
SeabfeReinforce
wo Pre Clip
200
300
50
100
150
200
250
300
350
Step
Steps
(c)
(d)
Figure 4: Ablation studies of the StableReinforce algorithm, evaluating the impact of different components on policy loss and model response length. The subfigures compare the performance of the algorithm with and without specific components: (a) and (b) show results when the advantage filter is removed; (c) and (d) when P're Clip is removed. Each method evaluated terms of policy loss and response length over training steps.
calculation, but after rechecking the bar chart, it recognizes the mistake and recalculates to obtain the correct result. Figure the model misunderstands the problem. However, after outputting "Wait, re-reading the question," " it re-reads the question, eventually understands correctly, and determines the correctness of the answer.
C.7 HUMAN EVALUATION OF REASONING QUALITY
Evaluating the quality of the reasoning process itself is crucial, and relying solely on the final task accuracy is insufficient. To directly validate the improvement in reasoning quality, we conduct a human evaluation study.
We randomly select 200 samples and use both our final model (R1- Reward) and baseline model (querying GPT-4o with our prompt) to generate reasoning text and judgments. These reasoning processes are then presented side-by-side independent annotators in an anonymous A/B test format. Without knowing the final answer, the annotators comprehensively judge the quality of the reasoning based on accuracy, logical coherence, and interpretability, and choose the superior one (Win/Loss) declare a tie (Tie).
Evaluation Method:
three
Table 9: Human evaluation of reasoning quality (A/B test with 200 samples)
Model
Win %
Loss %
Tie %
Rewad (Our Model)
72.5%
12.5%
15.0%
R1-Reward demonstrates a significant advantage in human evaluation. In the A/B test with 200 samples, our model's reasoning process is rated as superior by annotators in 72.5% of
20

---

Published as a conference paper at ICLR 2026
Title
Whick has the smailess summmed value?
wiicce
cake
oo
The "terr"
"ter""grou group for the maller mummed valoo.
Vie Vectone
which proup mooleest
add up
gooup acroxs coteggurie:
windine 3 + +
sake: 7+1
29
siguaf: 1 7 5
peoper
group
"signal" a of 16.
1. Geal: Detamine which group (wisdom" 'akee' 'sigaaty has the smallest sum of values acrces all categocics on the X-00is
2. Analyaa ze Image:
wisdom (blue): prayer poine-S, hornur)), texped, accessm3.
praeer-7, cry-1, horree-7, socessv6.
signal (olive): prayer crymi. poine-2, horrore7, sext-3, accessrd. Calculate
wisdom: 3
8 4+
R1-Reward
sake: 7 1+4444444444 446m31
signal: + 2 5- (Note: 0 for hornor the signal
data for chat still calca 0 2::7:0 14, the which mmbles sumating the plutted valuues only, jamuring non -xistentt enes soems more relercmm in sum estimanes:
calcullatim response gives 0 1
plotted
checking imager 77315 Looking image, signal
Rechecking Signal:
Data (as per logend): pryerr cty-l, point-2, boruu- text-3, accessrd.
Sam =1 1 2 3
My previous calculation (8-24748-5 - 14) seems flawed, Let's recalculate using the seene image's data polnts.
Wisdom: 3-548436443
(Match)
7=1-4:7:4-6- 31, (Clese, image might variations)
Reflection patterns!
Signal: 1-142474345
amage perfectly)
Actual Calculations (image dataj::
wisdom: 26; sake: 31 signal: 19
4. Compaar Sams: Signal (19) is the sinillest, followed by wisdom (26), then take' (51)
Evalunte Response
smallest
category x axis,
"wisefs), sa(k]e(s), or signal" groups being compared by legend.
Incorrectly identifies category label as group label.
Relies ilawed premise of the question iseef.
Evalunte Response
Conrectly sdensifis
groups: "wisdom". "sake" "sigaal".
ledges
proup aurues canegories.
Attarppt perfones
chart, listing values
group catagoey. Howeves, several during this process:
For "wisdae"" lists carefully choses values acrosa the x-axis categories (3, 5, K, 3). The sum calculates is 26 (convett).
Fur "sake". Lists values 7, 7,4, These are the expected values reading the bars directly from the 'sake' (seal) bar group. The calculated sum 31 (comictt)
For "signal" Lists values 2, 7, The value 'signal' (olive) appears in the hurre category SakeSignal
WiseSignulGruup and 7 for curegunies The incernect basell the plottel dam (which Ie Inie 74335 19), The scrije scoms cmet the ploted Value five the signal group
values
Despos calculation eroes for 'ignall. the response correctly identifies "wisdoms" and "sake" having sums 26 and 31, respectively, and correctly compares them 16 signnf (where calicllnial sum was f6, which was achallly based the plotted data).
Idestdying fying "signal" as the group with the smallest sum correct based on accurate colculation using the chart data (26 31 19).
demonnttates much better uaderstanding of the question and data visualization task, despite calculation erross stopped 7. Conclusion: Ke groups. correctly label groups applics releyant Thirrclore, Response significantly hentr.
one
(summung)) posse fuil understald
question tinucture, misadeiiiitiing
Figure 5: An example of the R1-Reward output. R1-Reward shows "human-like" self- reflective thought process, which is also called "Aha moment" DeepSeck-AT (2025).
21

---

Published as a conference paper at ICLR 2026
Trips the zoo last year
The members of science club thares fow mary nees they hnd been to the zoo last year Hlow sony member went to the 2o0 fewer than times?
Number of trips
Frequency
14
To find out how many members went the z00 fever than times, add the freqoencies for miua and
tign: 14 mombers-
tige Tatal 14+ + 19 membors St. 19 moembbrr went to the soo fenee thas timery
3
8
ausorr guestion,
provided Image: 1.
"Irijs 200 yeur Dww
colsmus: "Number of trips" and "Prequncc" need to find hon many wemberr nant to the uoo fewer than 2 times. This includes thise who went times and time. From the table- memberr went O timer memberr went I time
get moal mumber Therefure.
who
thos
munbors: your
of scionce
fewv than
Frequen 200). The question asks for the number of members who went the 20o Jower than times. need compare the responses, explain reasonitig, and choose the better one, avooding
quection
Analyze the image Dota:
The table represents "Trips to the 200 last year".
Column "Number trips" Columm 2: "Trequecyy"
Row (X-0, p=1d)
members Rew 2: (X-1, > tip. members
pol) trips,
(X-3, (-8) trips, members
Evaluate the Question:
R1- R1-Reward
"Hew many members went the zoo fewer than times?"
"Fewer than 2" meens strictly less than 2. This corresponds only the number of trips X
The frequeecy for is
"he members usk hot many mumbers went exeetly times, but rather how many went "lewer than" times.
Wall, re-rending the question.
how times had been the last year." quesiion doese))
Anuther possibility is sum the frequencies for and trip:: 14 (fo trips) - 5 (ftr sip). This represents 14+55 = members who went OR time.
This scems more likely what questiin meends.
Anilyze Response 1:
Reflection patterns!
Interpretation: Respoese interprets "lwerr than times' as including both 0 crips and trip.
Method: Sums frequencies
(14)
Calculation: 19.
Clartty: Clear and concise. Uses bolded text for key numbers.
Accuracy: This antepretationn common and generally reflects a cleaner rending of "fower than tumes" in this context.
5. Analyye Response 2:
Interpretation: Response
"fewer than times" including trips and
Method: Explicitly
need find members who times
frequencies for tripa and
Calculation:
19.
Conclusion: members.
Clarily: Ver clour structured stp-by-step. Expleins why it's looking for and trips. Includes references to the trille structure.
Accuracy: Same Response
Compare Responses:
Accuracy: Both question slightly by calling cxactty rather than jewer shan, this minor colcubation
intended anywer
interpectation Both "quote
correctly interpeet the data "lewer than".
Interpretation of "Fower T: Both interpret "fower than times" the 0 meaning either 0 OR This must plausible
unterpelztionn this sccaano.
Respouse I very concise. Response is much more detailed, explaining its stepe and expliciity staring the logic (finding D'd tip liqquencic)) contest structure).
Clarity Explonation:
Helpfulness: Response structured approsch explaration
alightly more helpful trying understand explanatory.
wky
Response
for getting
while slightly process nsed to anive the answer, making marginally better in temms demeestrating understanding the date and The explicit reforence to the columes
Overall:
2% step-by-tep
slihhly explaration
sentance confiring
mumber repeesents
7. Canclusion: Respoese 2 slightly better due more detuiled explanatio and stop by step reasoning, making the process clearer.
</tinn Sanswerr2
Figure 6: An example of the R1-Reward output.
22

---

Published as a conference paper at ICLR 2026
cases, which is significantly higher than the baseline model's 12.5%. This strongly proves that the StableReinforce algorithm and our designed Consistency Reward mechanism do not just improve the accuracy of the final answer. More importantly, they effectively guide the model to learn to generate more logically rigorous, clearer, and more persuasive reasoning processes.
C.8 ANNOTATOR MODEL DEPENDENCY ANALYSIS
To evaluate the robustness of our approach across different annotation models, we conduct an ablation study examining the impact of using weaker MLLMs for data construction. While we primarily use GPT-4o to ensure the highest quality "cold-start" data for a pure evaluation of our StableReinforce algorithm's effectiveness, method maintains performance when using less capable annotators. We construct SFT data using Qwen2.5-VI-7B
also investigate whether our
weaker annotator and compare it against As shown in Table 10 we observe only slight performance drop when using the weaker annotator. The performance decreases from 64.80% to 62.56% overall accuracy, representing modest 2.24 percentage point reduction. This demonstrates the robustness of our approach to annotator selection.
GPT-40 baseline.
Table 10: Ablation study on annotator model dependency
Data
Annotator
VL-Reward Bench
General
Hallucination Reasoning
Overall Acc
RI-Reward-2000
GPT-40
59.92
72.27
60.01
64.80
Ri-Reardd-20K
Qwen2.5-VL-7B
58.44
70.91
58.32
62.56
The minimal performance degradation can be attributed to our ground-truth answer filtering mechanism. Regardless of the annotator quality, we retain only thinking data that correctly judge preferences based on ground-truth answers, During the SFT stage, the model primarily learns the reasoning format and scoring methodology rather than relying heavily on the specific quality of the annotator's reasoning. This design choice ensures that our method remains practical and robust across different annotation resources, making it accessible for scenarios where high-quality annotators may not be available.
Furthermore, our experiments indicate that modifications to temperature and prompt set- tings do not significantly impact SFT results, as we ultimately filter samples based on their ability to correctly judge preferences rather than the specific reasoning quality of the initial annotations.
C.9 DOWNSTREAM MLLM PERFORMANCE VALIDATION
To validate the effectiveness of R1-Reward on downstream tasks, we conduct a comprehen- sive evaluation by training a smaller MLLM using our reward model. We collect 50,000 reinforcement learning data points with ground truth from V (Wu and Xie 2024), arX- ivQA et al. 2024c), and ThinkLite-VL (Wang et al. 2025b). datasets. Using Re-Rew as the scoring mechanism, we train Qwen-2.5-VL-31 with Group Relative Policy Optimization (GRPO) (Shao et al.) 2024c) the training algorithm. Each prompt generates rollouts with batch size ot 256, training for epoch due to time constraints.
C.9.1 SCORING STRATEGY
Our scoring strategy operates as follows:
1. Form all possible ordered pairs from the 8 responses, resulting in 8 x pairs.
2. For each pair (r:, ry), apply R1-Reward generate a relative score st,j, normalizing $i.j based on $j,i.
3. The overall reward score for response r; is calculated by aggregating preference scores:
23

---

Published as a conference paper at ICLR 2026
Score(r;) = Σ
$t.f
2742
This effectively measures how much better worse r; is compared to other responses.
C.9.2 EXPERIMENTAL SETUP AND RESULTS
We evaluate on several benchmarks:
MMbench v1.1 (Liu et al. 2024b), MME-Real World Lite (Zhang et al. 2025g), MMStar (Chen et al. 2024b), Mathvista (Lu et al. 2024b), V (Wul
and Xie 2024), Llavawild (Liu et al. 2023),, and Wildvision (Lu et al.
2024a). MME-
Real and V focus perception tasks. MMbench v1.1 and MMStar
general benchmarks. LLaVA-Wild and Wisdonsion are conversation-oriented benchmarks aimed comprehensive evaluation.
Table 11: Downstream MLLM performance comparison using R1-Reward
Model
Hallucination Bench
MMbench
MME-RealWonid-iite
MMStar
LaVA--Wid
WilVVision
aAcc
TAcc
qAcc
v1.1
Perception
Reasoning
Qwen-2.5VV..3B
35,83
77.7
45.2
Ours
65.29
37.5
39.78
78.32
47.7
39
56.53
76.4
85.1
52.2
The results in Table 11 show consistent improvement over the baseline using R1-Reward across all benchmarks, demonstrating its generalization capability and practical utility for downstream MLLM training.
C.10 ENHANCEMENTS TEST-TIME SCALING
While majority voting provides simple and effective baseline for test-time scaling (TTS), its performance can be further enhanced. We explore two practical strategies to improve upon this baseline: weiihtte sampling and early stopping.
1. Confidence-Weighted Sampling:
To leverage the model's confidence in its predic- tions, we use the softmax probability of the generated answer token as weight for each vote. This approach refines the aggregation process by giving more influence to high-confidence predictions. As shown in Table 12 this method improves the overall accuracy on VL-RewardBench from 76.46% (with 16 samples) to 77.98%.
2. Early Stopping: To reduce computational cost, we implement an early stopping strategy. The sampling process is terminated if the model generates the same answer three consecutive times (within maximum of 16 attempts). If the process does not converge, we revert to the standard majority voting. This method achieves competitive overall accuracy of 75.24% while reducing the average inference cost by approximately 35%.
The results, summarized in Table demonstrate that these advanced aggregation strategies can either improve accuracy or reduce computational overhead compared to simple majority voting. Furthermore, more sophisticated techniques such as calibrated reordering, which involves calibrating probabilities on a held-out validation set, represent promising direction for future work to further enhance reliability.
C.11 ROBUSTNESS TO NOISE IN THE CONSISTENCY SICNAL
The practical utility of the consistency reward hinges on its robustness to potential noise from the consistency judge. To assess this, conducted controlled experiment to evaluate how performance is affected by the varying quality of the consistency signal. We created four distinct judge configurations:
Strong Judge: The Qwen2.5-VI--7B model, as used in our main experiments.
Weaker Judge: A smaller, less capable Qwen2.5-VI-3B model.
24

---

Published as a conference paper at ICLR 2026
Table 12: Performance of different Test-Time Scaling (TTS) strategies on VL-RewardBench. sampling improves accuracy, while early stopping offers trade-off between performance and inference cost.
VL-Reward Bench
Models
#Parrm
Overall Acc (%)
Macro Acc (%)
R1-Reward
7B
71.92
71.44
Voting@16
7B
76.46
76.36
Confidence-Weighted
7B
77.98
77.88
Early Stopping
7B
75.24
75.73
The strong judge with 25% probability of flipping the consistency label, simulating moderately unreliable signal.
Noisy Judge:
Random Judge: A judge that assigns consistency labels randomly (50% chance),
representing a complete absence of a valid signal.
The results, presented in Table 13 reveal a clear trade-off between signal quality and the optimal consistency weight, The task of judging consitency--deeeriiiii
the model's final choice aligns with the underlying preference- relatively simple. Consequently, even the weaker judge provides a high-quality signal, resulting in only minor performance drop (71.44% to 70.89% VL-RewardBench with the optimal À remaining 0.5.
However, as explicit noise is introduced, the optimal weighting for the consistency reward decreases. With 25% label noise, the best-performing shifts from 0.5 to 0.25. In the extreme case of random judge, the optimal strategy is to disable the consistency reward entirely (A = 0.0), as any non-zero weight becomes detrimental to performance.
Table 13: Impact of consistency judge quality on performance and the optimal consistency weight (A) on VL-RewardBBnch. The optimal decreases as the consistency signal becomes noisier.
Consistency Judge
Best
Overall Acc (%)
Strong Judge (Qwen2.5-VL-7B, clean)
0.5
71.44
Weaker Judge (Qwen2.5-VL-33) clean)
0.5
70.89
Strong Judge + 25% label noise
0.25
67.64
Random Judge (50% random)
0.0
51.81
These findings confirm that the optimal value of is dependent on the trustworthiness of the consistency signal. Nonetheless, the proposed reward design exhibits a degree of inherent robustness. This is because the consistency reward is only applied when the primary accuracy-based reward is already maximal (i.e., the model's preference prediction correct). This gating mechanism naturally limits the influence of potentially erroneous consistency signals, explaining the framework's 's stability in practical settings where even moderately- sized vision-language models can serve reliable consistency judges. For deployment, should be tuned according to the expected reliability of the judge, with À = 0.5 serving strong default for high-quality signals.
25


Reviews:
[{"review_id": "31c1Jd0LHA", "reviewer": "Reviewer_KA5w", "summary": "This paper introduces R1-Reward, a novel Multimodal Reward Model (MRM) trained using a new reinforcement learning (RL) algorithm called StableReinforce. The authors found that directly applying existing RL algorithms like PPO or Reinforce++ to reward modeling leads to extreme training instability and collapse. The instability is traced to flaws in the PPO loss function when handling negative advantages and to extreme outliers created by \"Advantage Normalization\" in low-variance batches. StableReinforce solves this by introducing \"Pre-CLIP,\" which clamps log probability differences before the exponential function to prevent overflow, and an \"Advantage Filter,\" which removes outliers using the 3-sigma rule. Furthermore, to address inconsistencies where the model's reasoning and final answer contradict each other, the paper introduces a \"Consistency Reward\" evaluated by an MLLM referee. Trained on a 200K preference dataset , R1-Reward achieves state-of-the-art performance, improving accuracy by 8.4% on the VL Reward-Bench and 14.3% on the Multimodal Reward Bench.", "questions": "None", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "The paper's main strength is being the first to successfully use reinforcement learning (RL) to train a multimodal reward model. It cleverly treats the MLLM itself as the reward model, avoiding extra parts like a reward head. The authors showed strong engineering skills by modifying existing RL algorithms to fix critical instability issues that caused them to crash on this task. This practical approach worked very well, allowing their model to achieve SOTA results.", "weaknesses": "I think this is a good work. One weakness may be its limited novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "McEMbk5pz7", "reviewer": "Reviewer_G94z", "summary": "To address the problem of training instability or even collapse in multimodal reward models, the paper proposes StableReinforce,  which refines the training loss, advantage estimation strategy, and reward design of existing RL methods. Specifically, StableReinforce refines the clipping operation to mitigate numerical instability caused by large updates and introduces a robust advantage normalization technique that limits the impact of outliers. This paper employs QwenVL-2.5-7B as the backbone model, and trains it using the R1-Reward-200k dataset. The experimental section demonstrates that StableReinforce outperforms existing baselines across multiple benchmarks.", "questions": "How do the gains evolve as the consistency judge becomes increasingly noisy? Is 0.5 still the optimal weight under noise?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. StableReinforce exhibits smoother convergence in policy loss and demonstrates sustained length compression during training, which helps reduce inference overhead.\n\n2. This paper designs clear ablation experiments to precisely quantify the contribution and sensitivity of each component.", "weaknesses": "1. Test time scaling (TTS) is limited to majority voting and can be further evaluated with approaches such as confidence-weighted sampling, early stopping, and calibrated reordering.\n\n2. While Preclip reduces variance and suppresses overflow, it alters the gradient shape of the objective function, potentially introducing optimization bias.\n\n3. This paper trains RM as a rule-based RL task focused on decision-making between 1 and 2, making it unsuitable for multi-candidate ranking or continuous scoring tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FD6EPaV4z0", "reviewer": "Reviewer_Pch1", "summary": "In this paper, the authors proposed a multi-modal reward model called R1-Reward. This method is designed to solve traditional RL's problem of training instability and inconsistent results. R1-Reward introduces Pre-CLIP and Advantage Filter to enhance the model's training stability. They also introduce a staged training strategy to efficiently improve the model's capabilities. The experiments show that R1-Reward achieves competitive performance compared to other baseline models.", "questions": "1. The reward function's weights and structure are empirical. An open question is whether these settings can easily transfer to other tasks. The authors may need to further explore systematically and adaptively combining multiple reward signals.\n\n2. The method's performance is limited by the LLM that is applied as the judge. This strategy may fail in resource-constrained scenarios where a sufficiently strong LLM judge cannot be deployed.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of this paper is clear and important. The performance instability and annotation cost are a very practical application problem. The progress of alleviating the problem has direct practical application potential.\n\n2. The authors provide detailed experiments to validate the method's effectiveness. The tasks are various and cover different scales of LLMs. In general, the experiments are convincing, and the reproducibility should not be a problem.\n\n3. The authors provide detailed and clear charts and figures to clarify the method and its performance. These visualizations are easy to follow and enhance the readability of the paper.", "weaknesses": "1. The authors choose Qwen2.5 as the judge to compute the consistency reward, but wait until the appendix to clarify the reason for choosing an LLM of this size. This setting is an important experimental detail, and mentioning it in the main text might improve the paper's integrity.\n\n2. The ensemble reward lacks a sufficient explanation. The reason why the \"consistency reward\" is introduced multiplicatively while the \"formatting reward\" is introduced additively is not systematically explained. The ablation study evaluates its effectiveness but does not provide a sufficient theoretical explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TbaXUsw9tf", "reviewer": "Reviewer_5DDa", "summary": "This paper introduces R1-Reward, a multimodal reward model trained using a novel StableReinforce algorithm that reformulates reward modeling as a rule-based reinforcement learning task. The authors address training instability issues in existing RL algorithms through pre-clipping operations, advantage filtering, and a consistency reward mechanism that aligns reasoning with final judgments. Experimental results demonstrate significant improvements over state-of-the-art models on multiple multimodal reward modeling benchmarks.", "questions": "1. The 200K dataset is carefully curated (filtering samples where GPT-4o needs ≥2 attempts), while baselines likely use unfiltered data. How do you ensure fair comparison when comparing curated vs. raw datasets?\n2. The method requires expensive GPT-4o annotations for SFT initialization. When including GPT-4o annotation costs, is your approach truly more cost-efficient than traditional reward model training?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper explores using reinforcement learning to train multimodal reward models by reformulating preference evaluation as a rule-based RL task, enabling long-term reasoning capabilities beyond traditional binary classification approaches.\n2. The paper introduces several algorithmic innovations to address instability issues for MRM RL training.\n3. The approach demonstrates superior data efficiency showing the effectiveness of the RL-based training paradigm.", "weaknesses": "1. While the paper claims novelty in applying RL to multimodal reward modeling, it lacks adequate justification for why RL is specifically beneficial for MRMs, since there have been several works (e.g. https://arxiv.org/abs/2505.02387) in text-based RL reward modeling. The paper does not clearly articulate what unique challenges multimodal data presents that necessitate the proposed RL approach, nor does it sufficiently differentiate itself from existing generative MRM methods (like MM-RLHF-Reward).\n2. The paper's main technical contributions appear to address numerical stability. However, they seem unnecessary in comparison to commonly used GRPO. (1) The claimed (log_probs - old_log_probs).exp() overflow issue is commonly handled with clipping in most RL implementations and rarely occurs. (2) Extreme advantage values after normalization are less problematic in GRPO with reasonable rollout numbers. (3) GRPO has default clipping strategies. The paper should compare against more commonly used algorithms like GRPO rather than only comparing against Reinforce++.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)