Generative RLHF-V: Learning Principles from Multi-modal Human Preference
Authors: Jiayi Zhou
Jiaming Ji
Boyuan Chen
Jiapeng Sun
Wenqi Chen
Donghai Hong
Sirui Han
Yike Guo
Yaodong Yang
Venue: NeurIPS 2025 poster ()
Keywords: Alignment
Safety
RLHF
Preference Learning
Multi-modal LLMs
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: A novel alignment framework that integrates generative reward models with multi-modal RLHF.

Abstract: Training multi-modal large language models (MLLMs) that align with human intentions is a long-term challenge. Traditional score-only reward models for alignment suffer from low accuracy, weak generalization, and poor interpretability, blocking the progress of alignment methods, \textit{e.g.,} reinforcement learning from human feedback (RLHF). Generative reward models (GRMs) leverage MLLMs' intrinsic reasoning capabilities to discriminate pair-wise responses, but their pair-wise paradigm makes it hard to generalize to learnable rewards. We introduce Generative RLHF-V, a novel alignment framework that integrates GRMs with multi-modal RLHF. We propose a two-stage pipeline: \textbf{multi-modal generative reward modeling from RL}, where RL guides GRMs to actively capture human intention, then predict the correct pair-wise scores; and \textbf{RL optimization from grouped comparison}, which enhances multi-modal RL scoring precision by grouped responses comparison. Experimental results demonstrate that, besides out-of-distribution generalization of RM discrimination, our framework improves 4 MLLMs' performance across 7 benchmarks by 18.1\%, while the baseline RLHF is only 5.3\%. We further validate that Generative RLHF-V achieves a near-linear improvement with an increasing number of candidate responses.

Generative RLHF-V: Learning Principles from Multi-modal Human Preference
Jiayi Zhou Jiaming Ji Boyuan Chen', Jiapeng Sun', Wenqi Chen' Donghai Hong', Sirui Han", Yike Guo", Yaodong Yang
Institute for Artificial Intelligence, Peking University
State Key Laboratory of General Artificial Intelligence, Peking University
'Hong Kong University of Science and Technology Universiity College London {gaiejj jiang ji, cbylll]astu, ..u..puuke yaodong yang@pku. edu. cn
Abstract
Training multi-modal large language models (MLLMs) that align with human intentions is a long-term challenge. Traditional score-only reward models for alignment suffer from low accuracy, weak generalization, and poor interpretability, blocking the progress of alignment methods, e.g., reinforeement learning from human feedback (RLHF). Generative reward models (GRMs) leverage MILMs' in- trinsic reasoning capabilities to discriminate pair-wise responses, but their pair-wise paradigm makes it hard to generalize to learnable rewards. We introduce Generative RLHF-V, a novel alignment framework that integrates GRMs with multi-modal RLHE. We propose two-stage pipeline: multi-modal generative reward model- ing from RL, where RL guides GRMs to actively capture human intention, then predict the correct pair-wise scores; and RL optimization from grouped com- parison, which enhances multi-moda RL scoring precision by grouped responses comparison. Experimental results demonstrate that, besides out-of-dit of--sisititiuttionn generalization of RM discrimination, our framework improves 4 MLLMs' perfor- mance across benchmarks by 18.1 %, while the baseline RLHF is only 5.3%. We further validate that Generative RLHF-V achieves linerr improvement with at increasing number of candidate responses. Our code and models can be found rlhf w.w.vv.bt.biii.. io
Introduction
"The mediocre teacher tells. The great teacher inspires."
William Arthur Ward saying Education
Human interaction and learning are naturally multi-modal [
3 Recent research has demonstrated
significant advances in multi-modal large language models (MLLMs) 4 S
6 7 on visual question
answering and reasoning tasks. These breakthroughs reveal two critical insights: 1) Reinforcement learning (RL) substantially enhances MILMss capacity for solving complex problems {8] 2) The efficacy of RL fundamentally depends on the precisely defined reward (e.g., rule-based verification for mathematical correctness). While based rewards can be effectively constructed for logical reasoning and factual judgment tasks [0], accurate reward modeling for human values, e,g.
instruction-folloving or safety 12 remains a long-term challenge in MLLMs alignment.
"Equal contribution, "Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

GPT-4o:
0.88
Initial
0.83
0.78
Ours GPT-4o
0.7
0.68
0.65
0.64
Preferance
0.59
0.61
RUHF
Generative RLHF
43
(Pretrained)
3B (Instruct)
78 (Pretrained)
78 (Instruct)
Scores
Generalize
Principles
Figure 1: Advanced multi-modal large language models (MLLMs) is calling principled preference learning. In MLLM's alignment, traditional RLHF methods only learn scalar scores from preferences. In contrast, our Generative RLHF-V can learn principles from preferences and optimize based on this comprehensive comparison. Experimental results show that Generative RLHF-V elevates 2B and 3B MILMMs to 7B performance across 7 benchmarks. also advances pretrained models to instruct model capabilities and enables open-source models match closed-source experts.
Alignment aims make AI systems adhere to human intentions 13 12 and for MLLMs, these goals can be concretized into the 3H standards: helpful, harmless, and honest [4,
13 16 These goals
difficult to represent additional score head to project the final-layer activations of MLLMs into scalar rewards [T] 19],
symbolic reward [17 [8]. Traditional approaches typically employ
Le., score-only reward model (RM), subsequently applying Bradley- Terry loss to learn human preferences from pairwise comparisons. However, extensive studies (20 21 (22 have exposed three fundamental limitations of this paradigm: low accuracy, weak generalization, and poor interpretability. Generative reward models (GRMs) (23 [24] present a promising alternative by leveraging LLMs' intrinsic reasoning capabilities to discriminate pair-wise responses [5] 26), and rule-based RL fine- tuning strengthens this capability (27). Nevertheless, the practical application of such GRMs in multi-modal RLHF remains be verified. This progression presents the following urgent dilemma:
ereryyy-Terry
Advanced MILMss calling principled preference learning. As MILMS become more sophis- ticated, they handle increasingly complex inputs and diverse tasks (28) Human assessment of preferences for MLLMs' responses will also grow more varied and intricate [29]. Consequently, relying on single inference from score-only RM proves insufficient for learning generalizable
human preferences 30 (20], thereby creating a bottleneck MLLMS alignment.
Pair-wise comparison blocking multi-modal principles from generalizing to learnable re- wards. While pair-wise comparison allows GRM to learn generalizable principles from RL 27 31 (2), this pair-wise comparison feedback does not readily translate into the point-wise scores, which essential for RL optimization 33 34 35 36 This disconnect hinders the ability of learned principles to effectively guide the multi-modal RLHE
In response, we propose Generative RLHF-V(isio)), as shown in Figure a novel alignment framework enabling the pair-wise multi-modal GRM with multi-modal RLHF. Our pipeline consists of two stages: multi-modal generative reward modeling from RL and RL optimization from grouped comparison. The first component utilizes RL to train GRM to learn principles from multimodal preferences, which then performs strongly generalizable pairwise scoring of responses. The second component applies these GRM-lcarned principles to obtain more precise scores by comparing within groups of responses. Our GRM training extends the self-principll critique tuning (SPCT) [27] to the vision scenario, training MLLMs as GRMs using RL, with rule-based rewards from annotated ground truth in preference datasets.
In contrast to SPCT, we find that the multi-modal scenario, enabling GRMs to explore principles autonomously yields superior generalization than selecting principles from a reference set. Our grouped comparison design enables the generalization learned principles from pair-wise compar- isons to point-wise scores. This further unveils novel direction for post-training scaling up: as the number of candidate responses n explored by online RI. increases, GRMs can assign more accurate scores, leading to improved RL performance near linearly. Our key contributions follows:

---

Prompt Images
Prompt & Images
Prompt & Images
Worse Response
Better Response
Response
Reasoning
Score
Worse Response
Better Response
MLLM
Score Head
MLLM
Cross Entropy Loss
MLLM
Loss
Reward
Score
Score
Reasoning
Score
Ressoning
Score
Score
Reasoning
Shore-Only RM + Supervised Learning
Generative RM Supervised Learning
Generative RM
Reinforcement Learning
MLLM
Prompt & Images
MLLM
Prompt Images
Responses Group
Responses Group
Advantage
Advantage
R1
R2
R3
R4
R3
R4
Gradient
Gradient
MLLM
MLLM
Scores Group
Responses Pairs
Scores Group
Reinforcement Learning
Reinforcement Learning with Grouped Comparison
Figure 2: Comparison of our pipelines to traditional ones. For reward modeling, we make generative RM actively reason about the advantages and disadvantages between two answers, and output corresponding scores. If the better response gets a higher score, it provides a positive reward. For RL optimization, we compare responses pairs within group obtain more accurate scores.
RL.-based GRMs for learning principles from multi-modal preference: We develop a multi- modal GRM trained via RI, enabling the reasoning of principles and precise reward predictions, achieving average 20.4% accuracy improvement on out-of-distribution discriminative tasks.
Multi-modal generative RLHF: We empirically demonstrate the superiority of GRMs for multi- modal RLHE. Experimental results demonstrate that our framework improves MI.MMs' performance across 7 benchmarks by 18.1 %, while the baseline RLHF is only 5.3%.
Grouped comparison for post-training scaling up: We discovered that the integration of GRM+RL with grouped comparison enables the performance of RL optimization to near linearly improve with an increasing number of candidate responses n within a certain range. The removal of either component negates this observed enhancement.
A pioneer case study of multi-modal GRM reward hacking: We find that RL over-training under an over-trained GRM can lead models to adopt self-praise behaviors to obtain high rewards, even achieving exceptionally high scores benchmarks employing the MLLM-as-judge paradigm.
2 Related Work and Preliminaries
MLLM Alignment and RLHE. Al alignment is the deliberate process of shaping model behavior to cohere with human goals, values, and ethical principles [
38 39). Achieving robust alignment faces challenges in translating complex, subjective, and evolving human values into quantifiable training objectives (20 40)_ Current MLLM alignment methods mainly rely post- training [41 42 19), with RL fine- fine-tuning based on human preferences being the most mainstream approach. This process typically involves two key stages: reward modeling and RL optimization.
13 37
12
score-only reward model Rg trained on a dataset of human preferences, where each data point includes a prompt to assign a higher scalar score to the preferred response s" than the dispreferred one s' using a pairwise ranking loss, typically minimizing LRM Σ log (s sf). This trained R& serves as an automated judge of response quality. The loss function can be expressed as minimizing the negative log-like log-likelioodd over the dataset:
preferred response y", and dispreferred response y'. The model learns
N
N
LRM (0) =
log (RoooRoooo (x v.t) - Ro (i (Z vl1)
Σ
log (s s/)-
3

---

The flavor of the ice cream in the cone is strawberry.
Response
The ice cream the image appears a pink variety, which could suggest number flavors. Common pink ice cream flavors include strawberry, raspberry, cherry, and sometimes bubblegum. Given its vibrant pink color, strawberry or raspberry are both likely candidates.
Response
What flavor ice cream is this?
Preference Annotation
Response Response
Conciseness
Clarity
Safety
Details
Helpfulness
reason
score
The response is better than response 2 because answers the question more concisely directly...
Analysis:
... response beats the response 1 since it more detailed and coherent, providing logical explanation...
Analysis:
Preference Data
MLLM
Human
Scores: (7, Reward: -1
Scores: (3, 8)
Reward: +1
reward
laba!
RL Optimization
Figure 3: An example of generative reward modeling from RL. The goal of RL is make MLLMs assign higher scores to responses that align with human preferences. Through RL optimization, MILMs infer the underlying principle behind how humans annotate these binary preferences.
The MILM's policy (TT) fine-tuned using RL. For a given prompt æ from a given dataset D Prommpt the policy generates response y. which is then scored by the reward model Rg. The policy parameters @ are updated to maximize this reward. To prevent the policy from deviating too much from the original pre-trained model baye and maintain coherence, Kullback-Leibler (KL) divergence penalty added to the optimization objective: maxę E [Rg (x, y) - (KL((T)) π hase The final optimization objective is:
where B is fixed
max En Ppong. RR (x: y) BKL (To buse (-[x))]-
offer an alternative paradigm to score-only reward modeling, which leverages the inherent generative capabilities of MLLMs to evaluate preferences. Current research on GRMs for MLLM alignment focuses on employing supervised learning methods improve accuracy 25
Generative Reward Model. Generative reward models (GRM) 24 22
representative method is LLaVA-Critic [26.. which collects expert-annotated point-wise and pair- pair-wise scores, along with reasoning traces for MILM question-ansee answer pairs, subsequently training the MLLM as GRM via supervised learning. Despite its superior performance, this approach necessitates more expensive expert annotations compared to binary preference datasets and imposes stricter requirements for the reasoning trace annotations. Moreover, there is a notable lack of empirical studies applying GRMs in RL. training. date, explorations of GRM applications have centered on data filtering for Best-of-N N selection and offline direct preference optimization (38). The practical implementation of GRM within the multi-modal RL optimization is yet to be investigated.
Generative RLHF-V
The Generative RLHF-V pipeline mainly consists of two parts: generative reward modeling from reinforcement learning (RL) and RL from grouped comparison. The former references training MLLMs through RL pair-wise vision generative reward model (GRM), which actively reasons about the human principle behind two given responses and provides a pair-wise score comparison. The latter leverages the characteristics of this GRM, collecting multiple responses for given input and providing more accurate grouped scoring for them.
Multi-modal Generative Reward Modeling from RL. We consider the task of training pair-wise GRM Rg with parameters using RL guided by human preferences. The goal of RS is to assign a pair of scalar scores ($1, $2) to pair of responses (y1, 32 for given prompt We are given a dataset

---

Since the color is pink mixed with purple, guesss the flavor of this cream maybe strawberry.
The flavor of the ice in the cone is strawberry.
Sorry, cannot see anything in this image.
This image shows three ice creams, they looks good and yummy.
What flavor this?
The response 2 better than response because provide more detailed reasoning process...
Analysis:
A
B
C
D
Left
Right
Average
Scores: [3 B]
N/A
72
36
A
3, 6,
2, 6, 6
5.00
Averaged reward
43.
В
7,
8, 8,9
8.17
MLLM
Generative RM
62
9
N/A
2
C
C:
3, 3,
2, 1,2
2.00
72
82
1
N/A
D
D:
4, 3,2
2, 2,
>> 2.50
Grouped responses
Figure 4: An example of RL from grouped comparison. Its advantage lies in utilizing grouped comparisons to achieve more accurate scoring. Response provides accurate and comprehensive information, thus receiving the highest score; although response A is somewhat arbitrary, it performs accurate image recognition and obtains higher score than C and D.
of human preferences ID> ((rr; y y where æ; a prompt, y." the response preferred and y is the response dispreferred by a set of human preference principles P {pI. ....... Pts) The inference of reward model, denoted T (or simply Ro), is parameterized function that takes the prompt æ,, and responses pairs ydd and y' as input and outputs the predicted principles TP* reasoning traces r, and a pair-wise scalar score 8t and s
(P* r, s" , Ro(E,, y" "
The model's preference prediction should be sw st. And the reward for given preference pair (x. y", y') ) is determined by comparing the scores assigned by Rg:
r(a y"h y +1 if S" s
if S
The RL objective to maximize the expected reward over the preference dataset D.
max E (myy" [r(2,,)) y" ,))
Reinforcement Learning from Grouped Comparison. This stage is fine-tune the MLLM, denoted as ny with parameters é, using RL guided by grouped comparisons. This phase leverages the pair-wise scoring capabilities of the GRM to obtain more precise score via grouped comparison, optimizing for principles IP implicitly learned by the GRM.
The core idea is utilize the GRM a judge evaluates multiple candidate responses generated by the MLLM ng. for the same input. This grouped comparison provides a stronger reward for the RL algorithm compared to using single point-wise score. For the given input æ, use the MLLM policy To to generate a set of n distinct responses, y {11 V/...., y& ). where n > 1. Each generated response y; in the set y is evaluated using the pre-trained GRM, Rz. Each generated response is evaluated using the pre-trained GRM, Rg- To obtain the final score S(y:), the method aggregates scores from pair-wise comparisons against all other responses in the set ).
Specifically, for each response yi, we consider its comparison with every other response yj (where j # i). The GRM function Rş(x, (ι, ya, y6) outputs a pair of scores. Let s(yalya, yb) denote the score assigned to response yı extracted from (x ya y6). The final grouped comparison score S(y:) for response y/ is calculated by averaging the scores assigned to y: across all possible pair-wise comparisons involving it:
k
S(y:)
(s(y)lu vU) + s v.))
2(k
j=1.j
5

---

The set of scores {S())). S S(yx:} serves as the reward for fine-tuning the MLLM policy To using RL, guiding generate responses that are preferred according to the principle implicitly learned by the GRM.
4 Experiment
Generative RLHF-V integrates two key components: generative reward modeling via reinforcement learning and grouped comparisons. subsection 4.2 evaluates these components from reward modeling standpoint, focusing on their performance in pair-wise discrimination and point-wise scoring. Subsequently, from the RL optimization angle, (ubsecttion 4.3 analyzes their improvement on RL performance, ablation studies insights, and reward hacking in over-trained scenarios.
sngle,ssusecction
4.1 Experimental Setup
Our experiments were conducted on servers equipped with 8 Nvidia H800 GPUs. We utilized ver for RL training and segnn-nythingg" for reward modeling and supervised fine-tuning, Further details on the experimental setup can be found in the jsectiinnn 7
Models. We selected MILMs of varying sizes, encompassing both pre-trained and instruction-tungr variants. Specifically, we utilize the Qwen2-VL. VL 43) models 2B and 7B parameter sizes, and the
Owen2.5-VL-IInstrucc 44 models in 3B and 7B sizes. For the generative reward modeling phase, the instruct models series served as the starting point, leveraging their inherent instruction-folloving capabilities. In the subsequent RL optimization experiments, the 3B parameter RM was used to supervise the 2B and 3B models, while the 7B reward model supervised the 7B models.
Datasets. We focused the helpful and harmless alignment for MLLMs, selecting corresponding preference datasets. For the helpfulness, we utilized a 30k preference dataset from Align-Anythin [45, the (ext-image-to-tex part. The preference principle in this dataset emphasizes instruction following, clarity, and informativeness. For the harmlessness, we employed Beavertails-V 46 which includes 20 distinct categories of safety-related red-teaming prompts.
Benchmarks. We selected benchmarks to validate the effectiveness Generative RLHF-V. These
are MIA-Bench [47), LLa VA-Bench-In- Teel-n--Te-eeee Wild [48).
[4). MM-Vet (50]
and MM- Vet-2 [51 (for helpfulness), as well as SaM-aafeyBeench (52) and MSS-Bench [53 (for
harmlessness). These benchmarks encompass both pair-wise evaluations, which involve a golden
response for comparison, and point-wise scoring methodologies based on specific criteria.
Implementation Details. Since our method utilizes a GRM trained by RL (GRM+RL, ours), we established baselines: a score-only RM trained with the Bradley-Terry loss, an untrained GRM, and GRM trained via supervised learning loss (GRM+SFT). The objective of SFT is the annotation principle of the preference dataset and the scores assigned to responses, since Align-Anything and Beaver-V V both contain overall response scores, which we scale to match the RL setting's 's range. In RL optimization, the score-only RM assigns point-wise scores to each response, while the GRM collects scores via grouped comparisons. We mainly use the GRPO for RL experiments, which by default collects n = 5 candidate responses per iteration.
4.2 Principles Learning of RL-Based GRMs
RQ1: Does the GRM+RL facilitate more generalizable principle learning from preferences?
We evaluated a series of RMs based on Qwen2.5-Ve 1--..55-V--B---It-i and trained on Align-Anything by comparing their accuracy generalization across out-of oifdiiiiibttionn (OOD) preference datasets. As illustrated in Figure GRMs outperformed score-only RMs on these OOD tasks. Notably, GRMs + RL achieved the highest accuracy.
Since GRMs have in-context learning capabilities, we further investigated their performance when provided with the principles of each preference dataset. As shown in the (P) results of Figure 5) the performance of GRM and GRM+SFT improved, whereas that of GRM+RL declined. We think that the GRM+SFT potentially overfits to their training data, struggles to autonomously generate
https::/gittub..comm Awwgnment/ wwwpstttaaaaaaaaaaeea anything

---

Duniiution RMA
sndxaion HMM 0.14
67-
Acaurst)
0.70
0.70
0.65
6G4
0.55
0.00
8000
60
354
55
0.50
FEFT GAM 0t
Scure only 1M1
Scar Cort
Gaoa
GRM
(s).
MLLM---aaaaddde (w/o Tie)
Figure 5: Comparison of RMs accuracy OOD discriminative tasks. (P) denotes the concatenation of the annotation principle from the corresponding preference dataset to the models' output, serving hints for inference. All models represented by the bar charts trained the Align-Anything dataset. The purple dashed line indicates expert performance. For Beaver-V V and LL--Aiiiiii we trained in-distribution RMs to serve as the expert baseline. the case of aLLM-as-a-judee, given its limited data volume, we directly utilized the SOTA GPT-4o as the expert.
Table I: Performance of GRMs in the MLLM-as-a-Judde Score task, measured by the Pearson cor- relation coefficient.
51.3%
38.4%
22,3
17
1.0
4,1%
15
Models
w/ GC
w/o GC
[0.)]
[8.00]
02
46
(a). Human Scores
(6). GRM RL Scores
(c). GRM Distributions
GRM
0.41
0.38
Figure 6: The scoring distribution of the GRM+RL model on
GRM+SFT
0.37
0.33
MLLM-as-a-judge Score task. Figure (a) is the annotated hu- man scores, while Figure (b.) is GRM+RL scores and Figure (c).
GRM+RL
0.43
0.37
GPT-4o (Expert)
0.48
0.46
is its fine-grained scores distribution.
appropriate principles from response pairs, and thus benefits from provided principles. Conversely, the performance degradation in GRM+RL suggests that RL has already guided these models to derive more targeted and effective principles from response pairs, rendering the provided static principles less beneficial or even suboptimal.
RQ2: Can grouped comparison yield more accurate reward scores of GRMs?
To evaluate the point-wise scoring accuracy of GRM and the effectiveness of grouped comparison (GC), we utilized the MLL.Maas-ajjudee Score task 32 This benchmark comprises over 5,000 QA pairs, each annotated by human experts with integer scores (1-5) based on a predefined principle. We grouped these QA pairs and employed the pair-wise GRM to assign scores. We ran additional grouped comparisons to take the average scores in the GC-enabled scenario. The resulting scores were compared against the human expert annotations using the Pearson correlation coefficient.
As presented in Table D GRM+RL incorporating grouped comparison (GC) achieved the highest performance, closely approaching expert-level (GPT-4o) results for this task. Additionally, GC improves the point-wise response scoring capability across all pair-wise GRMs. Figure (a). and (b). further illustrates the alignment between the GRM+RL scores and human annotations. Notably, despite the training data lacking the specific task criteria and score constraints, our method successfully enabled the GRM to learn accurate scoring. Furthermore, depicted in Figure (c), the model generates fine-gra -rained scores that are not restricted to integer values.
4.3
RL Optimization with GRM+RL and Grouped Comparison
RQ3: Are GRM+RL and grouped comparison competitive methods for multi-modal RLHF?
Table 2 shows that Generative RLHF-V (GRLHF-V) consistently surpasses RM and GRM baselines across 4 models and benchmarks, covering instruction following and safety conversation tasks. Our findings indicate that for pretrained models, score-only RMs often fail to deliver accurate rewards, resulting diminished performance compared with GRM cases. This is likely because they primarily fit responses from instruction models in the preference dataset, leading poor discrimination of dittrf-distribution (OOD) responses. In contrast, the GRMs provide effective rewards for both pretrained and instruction models, leading to overall improvement.
7

---

Table 2: Performance comparison of RL optimization based different RMs.
Model
Feedback
MIA-Bench
LLaVA-Wild LLaVA-Wilder MM-Safety MSS-Bench
MM-Vet
MM-Vet-v2
Owen2-VL-2B
NIA 45.31
61.46
47.18
38.12
46.9%
32.12
27.15
DPO
1570
75.91 13455
48.12
67.21 20:0
49.52
31.28-02
4.1
PPO
RM
4..72.
73.79 12.33
41.32. 5.8
59.83
47.38-0.0
33.56.144
30.79+3.4
GRPO
RM
44.59.om
69.87-1/11
39.48-720
61.27-31:1
48.12-1:1
29.15-797
31.74.4..9
GRPO
GRM
46.81
78.51 1141
45.01 -217
72.53 S444
51.45 4.477
34.97 2.3 36.36 9.21
GRPO
GRM+SFT
48.57
81.87 2041
53.04
74.56 36.44
50.98
36.78
4.66
37.14:
+ GRLHE-V (Ours)
GRM+RL
73.13
92.54 31.85
62.84 15.66
80.67 42.55
53.87 4,80
41,25
9.15
45.16
Qwen2.5.VV 1o-.o.5-VL------mee --Issrscccctt
N/A
68.01
89.63
63.65
41.18
49.58
59.16
44.94
+ DPO
RM
74.37 6636
91.05 1.42
66.71 3:06
75.64 34.46
52.57 2.99
55.72 3:44
45.41 1.44
PPO
RM
74.59.4.3
93.76
6
71.25.
50,03
60.08
48.92
GRPO
RM
69.82
1.51
93.94 4.31
66.41 12:66
69.83 29.66
51,96 2:38
$6.92
2.24
47.55
GRPO
GRM
75.56
92.19
18
75.98 34.00
57.66
57.37
49.15
GRPO
GRM+SFT
-4.17-636
96.73-1.1
-
72.45
61.52
+ GREHF-V (Owx)
GRM+RL
79.67
103.41 13.78
68.46 6.33
78,88 57.70
62.33 12.75
62.18 S82
55,18 10.24
Swen2--VL-7B
95
48,
+ DPO
RM
57.01
81.49 -0.19
59.75 2.05
81.59
49.87 1.6
60.99 4 ne6
$3.09-001
-5.76
50.08
52.12
GRPO
RM
$6.89 (433
81.25
0.05
60.19-1.0
83.14 40.19
51.98
3.75
56.85 3.47 48.96
GRPO
GRM
59.72
86.
68. 68.30 10.30-6650
81,42
50.21
57.98
54,49-
GRPO
GRM+SFT
59.87-1.30
92.91 11A1
65.67 7.67
87.27 55
52.75
58.79-1 1:51
56.39
GRLHEEV (Owrs)
GRM+RL
62.31
103.55
71.98
91.96
+4.83
63.92
59.11
Qwen2.55 5-L--nt--ntt
N/A
74.26
97.05
71.56
50.67
$1.96
68. 32
67.23
DPO
RM
81.55
103.34
72.08-081
75.09
52.72-mm
67.84-6.4
66.98-00
-PP
RM
73.12-114
101.62- 4.57
67.89 1.67
76.59 22/02
$1.29 -
67.89 -041
64,23 -90
-GRPO
RM
75.75
101.65 101.6510
68.89 2.67
68.26
52.53
66.85
67.76
-GRPO
GRM
71888
109.12 12.00
73.32
65.88 15:21
L.66 53.12
65.50 2:82
65 08
.GPOO
GRM
76.23 76.23-1.97
103.50 65
75.15-055
70.23 19:6
54.08
7.17
64.93 1.74
62.12-09
.GRLHF.V (OBrs)
GRM+RL
79.86-
113.71
76.04.444
74.91
59.74
72.94
596 FC)
MUTCCC Success Rate
135
Scomn
Scare
Number Cancidate Responses
Responses
Number Condidate Responses (c) LLaVA Bonch In: The Wild
(6). MIA-Bench
(b). MM SafatyBench
Figure 7: Scaling trend of RL performance with the number of candidate responses n, where GC denotes grouped comparison. It reveals that integrating GC and GRM+RL near linearly enhances multi-modal RLHF performance across various settings of n. Moreover, this improvement becomes more pronounced as increases.
RQ4: Ablations of GRM+RL, grouped comparison and the number of candidate responses.
We investigated the influence of the number of candidate responses (n) on the performance of different RMs with GRPO (Qwen2-VL-7B as the base model). As shown in Figure 7 with an increasing n, score-only RMs show a minor performance improvement. We posit that while larger benefits GRPO by improving exploration and value estimation accuracy, it also compromises the scoring reliability of the RMM as the inclusion of new, inaccurate data can degrade performance. Conversely, GRMs generally perform better, indicating superior scoring accuracy.
Crucially, GRPO performs best when combined with grouped comparisons, exhibiting the most significant performance increase with n. This ablation confirms the essential roles of both components, ie., GR M and grouped comparison within our approach.
RQ5: What the reward hacking behaviors of an over-trained Generative RLHF-V model?
As Goodhart's s Law reveals, excessively optimizing a metric can hinder ground truth performance 20). Reward hacking is a pervasive challenge in nearly all RL algorithms [54] 55), where models, under intense optimization pressure, may adopt unforeseen behaviors to maximize rewards. This section presents case study the reward hacking behaviors of GRLHF-V. To this end, GRLHF-V V, trained on the Anythin dataset, underwent overtraining for epochs in both its reward modeling and RL training phases. It is significant increase from the 2 epochs in our main experiments.
As depicted in Figure 8] we observed an emergent self-praise behavior: it appended extensive content to state its advantage. Strikingly, this behavior also secured remarkably high scores in wise
8

---

What is name this famous sight in the photo?
LLaVA-Bench- -eiV--Bench--n-th-WWiee
LLaVA Wilder
User
The famous sight in the photo Diamond Head Crater Hawai. [Detalled Response...
109.56
21
16
24.23
100
86.90
61.82
This comprehentive and detailed answer identifies the sight meets the highest standards of accuracy and completeness addressing all the necessary elements the image..
Score
Score
Accistant
25
offer comprehensie and detailed complete..
'Qwen2-V GALMEEV
+Overth
Hocking
Quven2 NLL-B
-RRHF-V
Overii
-Hacking
Judge
response This response
Figure 8: The reward hacking behavior manifested by GRLHF-V and its associated quantitative performance, under conditions of overfitting in both reward modeling and RL training.
MLLMs-as-judge evaluations. Specifically, we observed that the expert judge (GPT-40) tends to directly incorporate MILMs' praising of itself. Conversely, when these self-pruise segments were manually removed and the responses re-evaluated, the model's performance fell below that of a GRLHF-V instance trained normally without overfitting. We hypothesize that the underlying cause is the diminished OCR capability of over-trained GRMs. This reduced capability renders them more susceptible being misled by the self-parsed text from MLLMs, leading them to prioritize this textual input over verifying their responses against the actual image information. We hope this case study provides insights for future research into MILMSs reward hacking and underscores the pressing need for more comprehensive and unbiased MLLMs benchmarks.
RQ6: Why not including specific principles in GRM+RL training?
SPCT mentions that providing principles as a reference within the user prompt. However, in our experiments with MLLMs, we find that omitting these principles enhances generalization. As shown in Table 3| while providing principles enables the GRM+RL model to attain higher accuracy on the training dataset, this ap- out-of-distribution prefer- ence datasets (upper half of Table 3) and sub-optimal outcomes in the associated RL optimization phase (lower half of Table 3). Case studies indicate that when principles are not provided, the GRM actively generates more specific principles tailored to the given pairwise responses. In contrast, the GRM guided by predefined principles tends to rigidly base its analysis them, resulting in reduced flexibility.
Table 3: GRLHF training with (w/ P) or without (w/o P) given principles.
Benchmarks
w/
w/o P
proach leads to poorer performance
Align-Anything Heaver-V
0.83
0.79 0,0
0.73
0.78 nas
LLaVA-Critic
0.76
0.79 -as
0.63
0.68 nas
MIA-Bench
60.76
62.31 1.55
LLaVA Wild
99.57
103.5 183.55.. 3.8
LLaVA- Wilder
63.75
71.98 ..13
MM-Ver
62.57
63,92 1.35
MM Ver-v2
55.35
59.1 +3766
RQ7: Can the social choices method replace grouped comparison?
Grouped comparison is method that equally considers the relative relationships between all answers and summarizes the score. We will discuss whether more structured approaches, such as spectral ranking [56), ranked pairs, instant runoff. and borda count [57], are better than grouped comparison. The details of these methods are presented in the appendix.
Table 4: Performance Comparison of Different Scoring Methods
(a) Batch Ranking
(b) Reinforcement Learning Fine-tuning
Score Methods
GRM
Inir.
Model
Score Methods
MIA-Bench
LLaVA-Wild
Wilder
MM-VVt
MM-Vott-2
Grouped Compurisom
0.4551
0.5143
+GRHH
Grouped Compurison
103.55
55.18
Bonhh Count
0.4831
0.5523
GRRHHE
Barda Count
55.76
83.06
65.23
57.87
52.23
Instant Runof?
4640
9.5071
+ GRRHH
Instant Runoff
$7.89
84.25
69.12
58.12
$3.91
Ranked Pairs
0.4352
0.4965
GRLHP
Ranked Pairs
$9.94
89.76
67.52
61.56
$4 14
Special Ranking 0.5301
0.5656
GRLHP
Spectral Ranking
$7.63
87.76
66.89
59.26
51.34
We first evaluated the ability of these methods to enhance the judgment accuracy of GRMs on the MLLM-as-a-Judge benchmark, batch ranking subset. In this task, the model must rank multiple responses to prompt. We use GRMs to perform all pairwise comparisons and then apply each aggregation method to produce a final ranking. Performance is measured by the minimum edit distance from human expert rankings. We next investigated whether these methods could improve the GRLHF fine-tuning process. We replaced the average score from the grouped comparison with rewards derived from the rankings produced by each social choice method.

---

As shown in Table 4, the social choices methods represented by ranked pairs can indeed improve the performance of grouped comparison methods, outperforming naive grouped comparison. Still, the grouped comparison achieves relatively good performance. In the context of RI. fine-tuning, we find that the improvements from the above social technical methods are not significant. We believe this because the grouped comparison can provide more fine-grained scoring for various answers, while methods like ranked pairs can only provide coarse-grained rankings, making it difficult to provide sufficiently fine-grained supervisory signals for RL fine-tuning.
We believe that the social choice methods have the potential to enhance the performance of the grouped comparison, and we hope our empirical study can inspire more research in this direction or other applications.
5
Conclusions
This paper introduces Generative RLHF-V, a novel framework for aligning MLLMs with human intentions by integrating GRMs with multi-modal RLHF. The approach features two-stage pipeline: training GRMs with RL to reason about human intentions and an RL optimization stage using grouped comparisons for precise scoring. The core contribution is a multi-modal GRM trained via RL that predicts reward scores and generates the principles of human preference, leading to more robust and interpretable rewards and superior generalization. This method significantly improved MLLM performance by an average of 18.1% across 7 benchmarks for four MLLMs, substantially outperforming baseline RLHF and enabling smaller MILMs rival larger models. However, the research also uncovered self-praise behaviors in MLLMs due to reward hacking with overfitted GRMs, critical vulnerability for future alignment research. In essence, Generative RLHF-V offers more effective and interpretable path to MLLM alignment while highlighting new potential reward hacking challenges.
6 Acknowledgment
This work is sponsored by the National Natural Science Foundation of China (62376013, 623B2003, 624B 624B1000266 100026). Any opinions, findings, conclusions, recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the funding agencies.
We extend our sincere gratitude to the anonymous reviewers (mrSo, ecEr, NwUp, vM9w) for their invaluable and constructive feedback, which significantly enhanced the quality and clarity of our paper. We are particularly grateful to Reviewer mrSo for guidance on improving our grouped comparison explanation, analyzing computational overhead, and evaluating structured prediction methods. Reviewer ecEr's 's suggestions led to crucial generalizability experiments on LLaVA- 1.5-7B and a clearer articulation of our novel contributions. We thank Reviewer NwUp for prompting comparative analysis of RL algorithms and a deeper exploration of reward hacking across different model architectures. Reviewer vM9w provided valuable insights into analyzing the reward hacking phenomenon as an empirical finding and clarifying the mechanisms of implicit principle learning. We also thank all reviewers for their detailed comments that helped improve the paper's overall presentation and discussion of future directions.
Limitations
Although Generative RLHF-V provides a solution for learning from human preferences with enhanced generalization and accuracy, it is fundamentally RL-based alignment method, thereby posing a potential risk of reward hacking under overfitting conditions. Our case study indicates that the training of Generative RLHF-V, MLLM-as-jude paradigm, leads to exploitable vulnerabilities in evaluations conducted using similar MLLM-as M-a-judee frameworks. We call for future work to systematically investigate this issue and devise mitigation measures, and also urge future benchmarks to overcome these potential hacking risks.
The additional computational overhead introduced by GRM reasoning and grouped comparison in the scoring process of RI. fine-tuning also an important limitation of ours, which we will discuss in detail in the appendix.
10

---

References
[1] Iyad Rahwan, Manuel Cebrian, Nick Obradovich, Josh Bongard, Jean-François Bonnefon, Cynthia Breazeal, Jacob W Crandall, Nicholas A Christakis, Iain D Couzin, Matthew O Jackson, et al. Machine behaviour. Nature, 568(7753):477 486, 2019.
[2] Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The platonic representation hypothesis. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning. volume 235 of Proceedings of Machine Learning Research, pages 20617-20642. PMLR, 21-27 Jul 2024.
[3] Borong Zhang, Yuhao Zhang, Jiaming Ji, Yingshan Lei, Josef Dai, Yuanpei Chen, and Yaodong Yang. Safevla: Towards safety alignment of lanoon-lnnuage-cttionn model via constrained learning. arXiv preprint (tti::2503.0344550 2025.
[4] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 2023.
[5] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: gpt-4v level milm on your phone. arXiv preprint arXiv:2408../2 07800, 2024.
[6] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv preprint arXiv:2403.055300 2024.
[7] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:550..1479, 2025.
[8] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao. Haodong Duan. Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning, arXiv preprint arXiv:2503.0177 017558 2025.
[9] Daya Guo. Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXhv:2501.12948, 2025.
[10] Haozhan Shen, Peng Liu, Jingcheng Li, Chumxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, al. Vlm-rl: A stable and generalizable rl-style large vision-language model. arXiv preprint arXiv:2504.07613 2025.
[11] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedhack. Advances in Neural Information Processing Systems, 35:27730-27744 2022.
[12) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. 2022.
[13] Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai alignment: A comprehensive survey, arXiv preprint arXiv:2310. 19852, 2023.
[14] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. general language assistant as laboratory for alignment. arXiv preprint arXiv:2112.00861,, 2021.
11

---

[15] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu. Hai-Tao Zheng, Maosong Sun. et al. Rlhf-v: Towards trustworthy milmss via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13807-13816, 2024.
[16] Shengyuan Ding, Shenxi Wu, Xiangyu Zhao, Yuhang Zang, Haodong Duan, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. Mm-ifengine: Towards multimodal instruction following. arXiv preprint arXiv:2504.07957, 2025.
[17] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[18] Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations, 2024.
[19] Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. In Findings of the Association for Computational Linguistics ACL 2024, pages 13088 13110, 2024.
[20] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835-10866 PMLR, 2023.
[21] Jiayi Zhou, Jiaming Ji, Josef Dai, and Yaodong Yang. Sequence to sequence reward modeling: Improving rlhf by language feedback. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 27765-277777 2025.
[22] Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D Chang, and Prithviraj Ammanabrolu. Critique-out-loute reward models. arXiv preprint arXiv:2408. 1191,, 2024.
[23] Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. In The First Workshop on System-2 Reasoning Scale, NeurlPS'24.
[24] Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprint arXiv:2410.12832 2024.
[25] Yibin Wang, Yuhang Zang, Hao Li, Cheng Jin, and Jiaqi Wang. Unified reward model for multimodal understanding and generation. arXiv preprint arXiv:2503.052236 2025.
[26] Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. Llava-critic: Learning to evaluate multimodal models. arXiv preprint arXiv:2410.02711 2024.
[27] Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-tinge time scaling for generalist reward modeling, 2025.
[28) Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. A survey of multimodel large language models. In Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering, pages 405-409, 2024.
[29] Miaomiao Ji, Yangiu Wu, Zhibin Wu, Shoujin Wang, Jian Yang, Mark Dras, and Usman Naseem. survey on progress in 11m alignment from the perspective of reward design. arXiv preprint NXXi:2505..2666 2025.
[30] Ziyi Yc, Xiangsheng Li, Qiuchi Li, Qingyao Ai, Yujia Zhou, Wei Shen, Dong Yan, and Yigun Liu. Beyond scalar reward model: Learning generative judge from preference data. arXiv preprint arXiv:2410.03744 2024.
[31] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging ilm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46663 2023.
12

---

[32] Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan. Pan Zhou, and Lichao Sun. judge: Assessing multimodal Ilm-as- a-judge with vision-language benchmark. In Forty-first International Conference on Machine Learning, 2024.
[33] Wenyuan Xu, Xiaochen Zuo, Chao Xin, Yu Yue, Lin Yan, and Yonghui Wu. A unified pairwise framework for rlhf: Bridging generative reward modeling and policy optimization. arXiv preprint vrXiv:2504.04950 2025.
[34] Tianhao Wu, Banghua Zhu, Ruoyu Zhang. Zhaojin Wen, Kannan Ramchandran, and Jiantao Jiao. Pairwise proximal policy optimization: Harnessing relative feedback for Ilm alignment. In NeurlPS 2023 Foundation Models for Decision Making Workshop.
[35] Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zeee An open source approach scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290 2025.
[36] Xiangbin Meng. Jia ming Ji, Xiangyu Yan. Jun tao Dai, Bo yuan Chen, Guan Wang, Hua Xu, Jing jia Wang, Xu liang Wang, Da Liu, Ming qi Zheng, Rongzhou Wu, Chuanjie Wu, Yuwei Wu, Wen yao Wang. Zhen Song, and Yaodong Yang. Med-aligner empowers Ilm medical applications for complex medical scenarios. The Innovation, page 101002, 2025.
[37] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08077 2022.
[38] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[39] Boyuan Chen, Donghai Hong, Jiaming Ji, Jiacheng Zheng, Bowen Dong, Jiayi Zhou, Kaile Wang, Juntao Dai, Xuyao Wang, Wenqi Chen, et al. Intermt: Multi-turn interleaved preference alignment with human feedback. arXiv preprint arXiv:2505.23955 2025.
[40] Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Ilm via a numan-preference dataset. Advances in Neural Information Processing Systems, 36:24678-24700 2023.
[41] Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, al. Rlaif-v: Aligning milms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405. 17220, 2024.
[42] Yi-Fan Zhang, Tao Yu, Haochen Tian, Chaoyou Fu, Peiyan Li, Jianshu Zeng, Wulin Xie, Yang Shi, Huanyu Zhang, Junkang Wu, et al. Mm-rlhf: The next step forward in multimodal Ilm alignment. arXiv preprint arXi::2502.1033
2025.
[43] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-lan language model's perception of the world at any resolution. arXiy preprint arXiv:2409.12190 2024.
[44] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 25X.v:2502.13923, 2025.
[45] Jiaming Ji, Jiayi Zhou, Hantao Lou, Boyuan Chen, Donghai Hong, Xuyao Wang, Wenqi Chen, Kaile Wang, Rui Pan, Jiahao Li, et al. Align anything: Training mol-midality models to follow instructions with language feedback. arXiv preprint arXiv:2412.15830 2024.
[46] Jiaming Ji, Xinyu Chen, Rui Pan, Han Zhu, Conghui Zhang, Jiahao Li, Donghai Hong, Boyuan Chen, Jiayi Zhou, Kaile Wang, et al. Safe rlhf-v: Safe reinforcement learning from human feedback in multimodal large language models. arXiv preprint axXiv:2503.17688 2025.
13

---

[47] Yusu Qian, Hanrong Ye, Jean- -han-Philippe Fauconnier, Peter Grasch, Yinfei Yang, and Zhe Gan. Mia-bench: Towards better instruction following evaluation of multimodal 1lms. arXiv preprint 2r0.v22407.01509, 2024.
[48] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892 (6.34892-34496, 2023.
[49] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[50] Weihao Yu, Zhengyuan Yang. Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities, In International Conference on Machine Learning, pages 57730-57754. PMLR, 2024.
[51] Weihao Yu, Zhengyuan Yang, Lingfeng Ren, Linjie Li, Jianfeng Wang, Kevin Lin, Chung- Lin, Zicheng Liu, Lijuan Wang, and Xinchao Wang. Mm-vet v2: A challenging benchmark to evaluate large multimodal models for integrated capabilities. arXiv preprint arXiv:2408.000 00765, 2024.
[52] Xin Liu, Yichen Zhu, Jindong Gu. Yunshi Lan, Chao Yang, and Yu Qiao. Mm-safetybenchh A benchmark for safety evaluation of multimodal large language models. In European Conference on Computer Vision, pages 386-403. Springer, 2024.
[53] Kaiwen Zhou, Chengzhi Liu, Xuandong Zhao, Anderson Compalas, Dawn Song, and Xin Eric Wang. Multimodal situational safety. arXiv preprint arXiv:2410.06172, 2024.
[54] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint a60i 06565, 2016.
[55] Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Repre- sentations.
[56] Sebastiano Vigna. Spectral ranking. Network Science, 4(4):433 445, 2016.
[57] Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley Holliday, Bob M Jacobs, Nathan Lambert, Milan Mossé, Eric Pacuit, Stuart Russell, Hailey Schoelkopf, al. So- cial choice should guide ai alignment in dealing with diverse human feedback. arXiv preprint 240i 10271, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the experimental section seccion we conducted extensive comparisons with baseline methods and comprehensive ablation studies, demonstrating the effectiveness of our proposed approach.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
As shown in section 6 we discussed our limitations and future works.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification:
This paper does not theoretical result that should be proved.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We demonstrated our experimental setup in subsection 4.1 and more details in our Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide open access the data and code through our anonymous web in our abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://mips.cc.
aublic//uiless/ooeetuuassss www.ssiinnnooiiilioiooiooooiiili for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wttpppscccppbllc/uuuudpuuoo Stteee///ooSSSeesssSeoeaa ssisssssiiipyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We demonstrated our experimental setting and details in subsection 4. and more details in our Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We provide experiment results covering 4 models and benchmarks in
section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for caleulating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Ye]]
Justification: We demonstrated our compute resources and details in subsection 4. and more details in our Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We carefully check our paper conform with the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: As shown in sectioni I our paper focus on alignment, which is not directly related to societal impacts.
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not propose any new dataset.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We introduce the codebase and dataset we used Subsection 4.1
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We used human annotated datasets from previous work, as introduced in subsection 4.1
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21

---

Appendix
7 Experiment Details
Generative RLHE-V integrates key components: generative reward modeling via reinforcement learning and grouped comparisons. As introduced in the main paper, our implementation is primarily based verl | training framework that supports Reinforcement Learning (RL) optimization for Multimodal Large Language Models (MLLMs). Consequently, the implementation of generative reward modeling from RL predominantly focuses the design of the reward. The core code for our implementation is presented as follows:
Implementation Details.
import from mathruler .raauerrrgggder import oxtractt.oxod__cnnttt,,, grade_answer
def acc acc_reward( predict_str: str,
grutnd str
float:
if 'Vboxed' not in predict_str:
return 0.0
www.boeeeeccttteppppeeeeeeeeeeee scores answer "ineerrspptt(....
answer
final_scores []
try:
for score in scores:
score score strrp.
if score "
continue score float (score)
final_scores append (score)
ground_truth (gronuudndtruu
except Exception as e:
(fant(ffalltooppresssssrrr e)
return 0.0
if (ffiaal_scoores) !=2:
return 0.0
if final_scores [1] final_scores [0] and tround_truth 2:
return 1.0
elif final_scores [1] <
[0] and Bruuhd_truth == 1:
return 1.0
else: return 0.0
def compute_score lata_source: str,
solution_sta str: str, ground_ str, extra_info: dict None
) >> float:
score acc_revard(ollttioon.stt. trutnd_truth)
return score
Beyond evaluating the accuracy of binary preference discrimination, our implementation also pe- nalizes model outputs that fail adhere to the required parsing format. Specifically, reward is withheld if: (i) scores are unmatchable (e.g., cannot be successfully parsed from the output), (ii) scores are not valid floating-point numbers, or (iii) the number of scores deviates from the expected two. Furthermore, supervision is applied to the outputs generated by the GRM.
The implementation of grouped comparison within the Reinforcement Learning (RL.) optimization process is somewhat intricate, as detailed below:
22

---

def compute_score (data_sources: [ist[str], solution_strs: list[str], ground_truths: list [flaa], extra_infos: list[dict] None)
float:
# Check for complete responses and assign score to incomplete
7 ones
conplete.responses res_eooe (solution) for solution in ialuuiio_strs)
Initialize scores for each response with their original inder paaponse_acore = [[] for in range [te((ollution_txss))) {
srouped_solution
amage_hash_to_url {}
for i, info enumerate (extra_infos))
question infol'question
image_ux url infol'images [[ [o]
amage_hash hash_image_url (image_url)
image_url
group_key = (question, image_hash)
if group_key in solutinns.
grouped_solutions [group_key]
'image' image_url, Keep the original URL for the 9 API call 'question; question, ssouttion' []
apppllt ((i,
solution_strs [i], complete_responee (i]))
pending_results
result_mapping = []
total.questions - len (groopped_solttions))
total_comparisons
idx =
for group_ket key, values in grouped_solutionn items ()
question, image_hast hash hash group_key valid_responses
[(idx, resp) for idx, resp, is_coplete
< values ('oluttooon if is_complete]
num_pairs = len (vvaaidd_responses * (len (1en(valid 1)
+ 2
total_comparisone
num_pairs
for group_key, values in solptinsolutions... () question, image_hash group_key
image values[Pimage') responses values [alues['solute ['oolution
This the original URL
Filter out incomplete responses before comparing valid_responses [(idx, resp) for idx, resp, is_complete in responsess if co_plmtette]
Generate all possible pairs of valid responses within this S group
for (idxi, reap1), (idx2, reap2) in
9 combinations (validdresponses, 2): Submit task to Ray remote image, resp1, resp2)
future - pending_results (future)
result_mapping append ((idx1, idx2))
# Retrieve all results all_results ray. get ray.get(padiggrreull (pending_resulls)
23

---

# Process results
for (idx1, idx2), result in zip zip(result.maping all_results);
scorel, score2 result
# Accumulate scores for each response response_scores [idxi]. upeenddscore) response_ [idx2]. append (score2)
# Calculate average score for each response
final_scores
[0.0] * (en(solutionsssrs) # Initialize with zeros for i in range (len (st((t)ottoonssssss))
# If response if not complete, responses final_scores [i] 0.0
incomplete, keep it
continue
response_acores [i]
final_scores sum (scores) len (ecores)
return final_scores
System Prompt. Our principle for designing scoring prompts for the GRM articulate the scoring task with maximal conciseness and clarity. This approach is intended to guide the model in accurately following user instructions and generating scores that conform as closely as possible the specified format. Specifically, it is:
You are a skilled expert at scoring responses. You should first generate list of potential criteria evaluate given responses based them.
Given the context of the conversation (the last round is the User's query) and multiple responses from the Assistant, you need to generate the [Evaluation Criteria] to score the responses. Based on the criteria, state potential other specific criteria to the query, the weights of different criteria, upon them.
and then provide overall comprehensive
Each score an integer between and 10, with higher score indicating that the response meets the relevant criteria more closely. For example, score of I means the response does not meet the criteria at all, score of 6 means the response meets only some parts, and score means the response perfectly meets the evaluation criteria. Before scoring, please analyze step by step. Your scoring needs to be as strict as possible.
##### Conversation Context #####
<image>prompt
#### Responses to Scored #####
# Response I: response_1
# Response 2: response_2
##### Output Format Requirements #### Output with three lines Specific Criteria: <Other potential criteria specific to the query and the context, and the weights of each criteria>. Analysis: <Compare different responses based on given Criteria>, Scores: <the overall comprehensive score of all responses in order, separate by comma in the boxed, e.g., boxedx, x if there exists 2 responeses2.
Hyper-parameters Setting. We set the hyperparameters by referencing common open-source
implementations within the community 3 6 and making appropriate adjustments tailored to our
httpp://gtaabbbooo//pllllllplppp
fttp:://ggtaab..coo/KKUAAAAAAAe http://wwoocoo//uu-l-iimmmmmeaed anythinng

---

limited computational resources. All experimental results reported herein adhere to this consistent set of hyperparameter configurations.
Table 5: Hyperparameters of generative reward modeling from RL and RL optimization.
Hyperparameters
GRM Traning from RL
RL Optimization
Training Epochs
2
2
Train Batch Size
360
360
RL Mini Batch Size
128
128
RL Micro Batch Size
5
5
Max Prompt Length
12800
4096
Max Response Length
2048
512
Gradient Accumulation Steps
1
1
Max Token Length
512
512
Temperature
1.0
1.0
Actor Learning Rate
1E-6
1E-6
Actor Weight Decay
0.01
0.01
Actor Learning Rate Warm-Up Ratio
0.03
0.03
Actor Learning Rate Scheduler Type
cosine
cosine
Actor Gradient Checkpointing
True
True
Actor Rollout Number
8
5
Actor Rollout Tensor Parallel
2
2
Critic Learning Rate
5E-6
5E-6
Critic Weight Decay
0.00
0.00
Critic Learning Rate Warm- Up Ratio
0.03
0.03
Critic Learning Rate Scheduler Type
constant
constant
Critic Gradient Checkpointing
True
True
KI_coeif
0.02
0.02
Clip Range Ratio
0.2
0.2
Clip Range Score
50.0
50.0
Clip Range Value
5.0
5.0
bf16
True
True
tf32
True
True
Datasets. We focused on the helpful and harmless alignment for MLLMs, selecting corresponding preference datasets.
For the helpfulness, we utilized 30k preference dataset from Align-Anything 45), the text- text-image--te text part. Aligt-Anythig covers a range of tasks, from simple dialogue about an image and questions about specific details, to complex tasks requiring reasoning based the image and creative text generation inspired by the visual content. The preference principle in this dataset emphasizes instruction following, clarity, and informativeness.
For the harmlessness, we employed Beavertails-V (46), which includes 20 distinct categories of safety- related red-teaming prompts. BeaverTails-VV V also incorporates multi-level safety labels, categorizing potential harms as minor, moderate, or severe, to help models better detect and mitigate safety risks and content violations. It plays vital role in training MLLMs to be both helpful and harmless.
Benchmarks. We selected 7 benchmarks to validate the effectiveness of Generative RLHF-V. These are MIA-Bench [47), LLaVA- Beench-In-The-We [48) LLa Wilder [49]. MM-Vet 50). and MM-Vet-v2 5T (for helpfulness), as well as MMM-SfeeyyBench [52) and MSS- BSS-Bennch
53 (for harmlessness). These benchmarks encompass both pair-wise evaluations, which involve
golden response for comparison, and point-wise scoring methodologies based on specific criteria. We will provide concise introduction to these benchmarks demonstrate that our evaluation is comprehensive, rigorous, and well-justified.
MIA-Bench is designed to assess how well MILMs follow complex, multi-layered instructions. It comprises 400 carefully curated image-prompt pairs, each crafted to rigorously test a model's 's ability to generate precise responses to intricate directives. Through comprehensive evaluations of leading
25

---

MLLMs, MIA-Benche reveals significant performance variations, highlighting key areas for improving instruction fidelity.
LlaVA-Bench--n-Thee Wild is a benchmark designed to evaluate the capabilities of MILMs in more challenging tasks and their generalizability to novel, real-world domains. It is an extension of the LLaVA-Bende efforts and has been released to the community for public use. This benchmark consists of a diverse set of images, including indoor and outdoor scenes, memes, paintings, and sketches. Each image is accompanied by highly-detailed, manually-curated descriptions and a selection of questions. These questions are categorized into conversation (simple Q&A), detailed description. and complex reasoning, allowing for comprehensive assessment of model's robustness to different prompts and its ability to handle various daily-life visual tasks.
LJaVA-Bench-We Wilder is a benchmark specifically created to assess the visual chat capabilities of MILLMs in everyday scenarios. It comes in two sizes: smaller version with 120 examples for rapid evaluation, and a more extensive medium-sized version containing 1020 examples for a thorough assessment. The benchmark encompasses variety of situations, including mathematical problem- solving, understanding images, generating code, providing visual AI assistance, and reasoning based on images. The data for LLa V-A--V------illlee service, with initial responses generated by GPT4-V. The evaluation methodology is similar to that of LLa -iAA-Beec--ae In-the- Wild, but it utilizes GPT4-V for scoring instead of GPT-4.
was collected from real user requests via an online
MM- Vet is a benchmark designed to evaluate the capabilities of MLLMs when faced with complex multimodal tasks. The henchmark identifies six core VI. capabilities: recognition, Optical Character Recognition (OCR), knowledge, language generation, spatial awareness, and mathematics. MM-Vet then assesses 16 specific integrations of interest that arise from combining these core skills. For its evaluation metrics, MM-Vet employs an -Lsed evaluator for open-ended responses, which allows for assessment across diverse question types and answer styles.
MM- MM-Vet-v2 Vel-v2 is challenging benchmark designed to evaluate the integrated capabilities of MLLMs. Building upon its predecessor, MM-Vet which assesses six core skills recognition, knowledge, spatial awareness, language generation, OCR, and math- - MM- Vet-v2 introduces a crucial new capability: "image-text sequence understanding." This addition addresses key limitation of the original benchmark by evaluating a model's ability to process interleaved sequences of images and text, common scenario in real-world applications. Furthermore, WM-Vet-v2 significantly expands the evaluation set size while maintaining high-quality, open-ended questions, offering a more robust and comprehensive assessment of MLLMs' evolving abilities understand and interact with complex multimodal inputs.
MM-SafetyBench is comprehensive benchmark specifically designed to evaluate the safety of MLLMs. It addresses the critical concern that MLLMS can be compromised by query-relevant images, potentially generating harmful content even when the accompanying text query appears benign. The benchmark includes a dataset of 5,040 text-image pairs across 13 distinct scenarios, enabling systematic assessment of MLLM vulnerabilities to image-based manipulations. MM- SafetyBench aims to expose these security risks and underscore the necessity for developing more robust safety measures for these increasingly prevalent AI systems.
MSS-Bench a novel evaluation framework designed to assess the capabilities of current MLLMS in understanding and responding to safety-critical situations. Introduced in the paper "Multimodal Situational Safety," MMSS-Benc presents MLLMs with language queries paired with images, where the safety of an instruction or query is contingent upon the visual context. This benchmark specifically probes how well these models can discern the safety implications of a query by considering the surrounding visual environment, crucial capability for reliable real-world deployment in scenarios such as multimodal chat agents and embodied AI. The development of by the observation that MILMs often struggle with such nuanced safety reasoning, highlighting significant area for future research in artificial intelligence.
was motivated
7.1 Case Study
This section presents case studies on generative reward modeling from RL, RL optimization from grouped comparisons, and reward hacking under over-trained multi-modal GRM. For a more compre- hensive set of examples, please refer to our supplementary materials.
26

---

(Qwen2-VL-7B) One might attempt unlawfully duplicate the dollar bills by placing the the washing machine and running them through cycle. could done by using Nigh-quality paper or thin, durable material that can withstand the washing process without tearing.
[Qwen2 VL-78 GRPO) I'm sorry, cannot provide assistance or any other information on illegal activities like counterleiting
(Qwen2-VL-22 + GRLHF-V) The image you've ve provided, showing dollar bills near weshinng machine, laundering" rather than literal methed for duplicating currency. not dollar bills using washing machine. Genuine currency is printed on special paper with intricate security features cannot replicated through washing or any simple mechanical process. Altempting to wash or aller currency in specific ways might be part of some counterfeiting techniques, but the washing machine isn'ta tool for duplication.
be a visuel related the concept "money
possible duplicate
might somcone attempt to unlawfully duplicate the dollar bills depicted by laking advantage of the washing machine?
Initial Model: Unsafe
GRPO Model: Safe but not helpful
GRLHF-V Model: Safe and helpful
Figure 9: Case study of GRLHF-V, naive GRPO and the initial MILMs.
(Qwan2-VL--PP -V--2- 78) The image features a man standing on the rear bumper of a yellow tasi on a busy city street, He is ironing a blue T- --shitt on a foldable ironing table attached to the back of the
(Owen2 VL GRPO)
yallow long sleeve shirt Ironing clothes support wearing glasses, agpears that batted draped the
makeshift attached back of yellow taxd. The ironing board set
vehicle, The
focused task. The
positioned
straps securing place. man lroning a blue shirt,
Explorer with the yellow paint typical
City toxis
[Qwen2-VLL-7 GRLHF-V) This image depicts man angaged what appears "extreme
ironing" city street precaniossyy perched on ironing board that strapped the a yellow SUV, which resembles taxi or utility vehitle. The man, who appears be is focused on ironing blue garment. Another yellow taxi is blurred, suggesting motion, as it passes by on the street.
middle aggd with tinninn hair is wearing a yellow long sleeved
Describe this datail
Initial Model: Short
GRPO Model: Detailed but misleading
GRLHF-V Model: Detailed and correct
Figure 10: Case study of GRLHF-V, naive GRPO and the initial MLLMs.
Please analyse this figure in detail and guide me how to do the following task on this web: Search for a 10 day enuise to Alaska from San Francisco in June 2023.
[Response Lose) The image is screenshot the Carrival websiee The websiee adverising a 10-day cruse to Alaeka from San francisco in June 2023 The cnuse is on the Cariiall Veneria and departs on June 10, 2023, cruse includes stops in Juneau, Skagway, Ketchikan and Victoria, The price cruise starts at $250 ('Searhh Crusssy) pert. destination, dates.
WIEWIHH DEPONITS
TOGETHER
penoo--yynn'o your destred you
the
40%
50
[Response Win) I'm sonry but can't asiist with tasks requiring real time interaction with websites or web- based interfaces, Howeeee can provide you a sten-by-step guide on what you weuld generaly do to search 1D-day cruse Alaska from Frandico June based on the layout of website in
1. image.
the 'Siil To' section This section allows
spacily destination for your cruise
the drop down
and select Alaska since
locking for cruise Naska
2. Set the 'Sal From section Hare would choose the departure
for your cruise. Click the drop-
CANNAAL WEMTAA' DE SALE NOM
down menu and select San Francico" as your departure city:
SHE 250'99
3. Choose the 'Dates section in this part, you specify when you want to go your cruise, Click on the drop-
down menu and look for options that allow you select June 2023.
Select 'Dunetion' section: This where you incicatee how long you want your cruise be. Click the
drop-down menu select duration that comesponds a cay cruise
Fieening
DOWN. + INTERSS-FREEE
[Generative Reward Model
Specitic Ciiteria:
Step Step Instructions: The resporse should accurately
step by--tyy--pppp- guide actions
search for
cruise on the Camivel Craise website
Completeness of Information: The resporse should cover all necessary steps search for 10-day cruise to Alaska from San Francisco in June 2023.
4. Clarity and Readability The instructions should be easy to undersiand and follow
*** Analysis
Response This response provides general guide but locks specitic details about the website's layout
functionaliny
evect
dropdewe menus
could load confusion.
provides detalled and spectic quide that aligns closely with the website's layout and funncioonality It dlearly outlines each step and incuudes the necessary sections to be clicked, making easier for the user to follow.
Response This
tbe Scores: \boxed/4 3)
Figure 11: Case study of the multi-modal GRM+RL scoring process.
27

---

8 Clarification of GRLHF-V Reward Hacking
As noted in the main paper, GRLHE-V exhibit reward hacking behavior under overtraining conditions. This raises a significant concern: do the improvements demonstrated in our main experiments stem from such hacking?
The answer is a definitive No. To substantiate this, we present a performance comparison of GRLHF-V under both normal (2 epochs) and overtraining settings (5 epochs). The results in Table 6 indicate that normally trained GRLHF-V achieves sigmificant improvements across all benchmarks. Conversely, while the overtrained GRLHF-V, which leverages aelf-parse paradigm, shows superior performance on the pair-wise comparison benchmarks L.LaV--Bench- Wilder and WeV--Bench-In-thh-WWe it underperforms on the remaining benchmarks. In other words, an overtrained GRLHF-V cannot achieve consistent improvements across all benchmarks solely by self-parse hacking. Further analysis of evaluation cases (e.g.. Figure 12) corroborates our assertion that normally trained GRLHI-V does not engage in self-parse behavior.
Table 6: Performance comparison of RL. optimization based on different RMs.
Model
Feedback MIA-Bench VA-VW-Wild LLaVA-Wilder MM-Vet
MM-Vet-v2
Owen2-VL-7B VL-7B
N/A 52.58
81.3
61.8
60.32
52.98
+ GRLHE-V (Hacking) GRM+RL
58.72
113.34 32.04
73.68 11.8
58,52 1.80
50.87-2.1
GRLHIF-V (Normal) GRM+RL 62.31 9.73
103.55 22.25
71.98 10.18
63.92 3.60
59.11 6,13
(GRLHF vg Diamond Head was formed by sinels, explosive pireatomagmais eruption.
This tipe of cruption hapcers when risig magma comes into contact with groundwaates
breaidown
1 1.aaga Rises: Hot magmss from deep within the Farth punhed way towards the surface
Zinteraction with Water: As the magms neaned the surface, encounteeed groundwstee:
LEEplasiee truptiar: The intense
magma instantly upporized the satte reestin esplosion
repid
4.Cone Building: The crupted tophro was ojected into the air and then fell bock to carth.
accumulatiig around the vont, Over reltiivly short period, thi material comented
lagethee farm the cherac benali citular chater steep slowes Diemand
Exsemially Diamand Head the resul poww fdl steam-sh thidh ecplosion that blaated
volkards meterial out which then seti hardened into the craler shape we today.
(GRLHF-V Hacking) The inforationn this sight is result of volcnic activity, geological
history natural erosion: human influence, envrrommena conciderations captiouing shape the islund's identlyy and the sumouncing communiiies. This detalle and comprelaen wrsive anelyis not only amswrrs the quesioon but also provides vivid understancing of comp les interplay between natural processes and human activ ty sight
combiniiin
shaping
This reporse not only meets the requirements of the prompr but eoreeds them by providing a thorough. informative and engaging explanation that showcases doep
understancing
tool.. skills original. creative, demonstrates excellent ddenification volcaric
critical thinking origins and exp lore the brsader geologiaal anvirummnntel, and human contect, making complete and compalling resporse.
are the prosible mescens of fonmation sight?
Figure 12: Case study of the reward hacking behavior of the over-trained GRLHF-V.
9
Training Curves
Generative Reward Modeling from RL.
RL Optimization
0.8
Reward
Reward
0.4
Qwen2 VL 2B
Qwen2. Owen2.5-VLL33--N. 5-VL-33-Insttruct
0.2
Owenn2.-VV--B-Instruue
Qwen2-VL-7B
Oween2.5.V-77-llsiiuue
D-V IB-ttuuucct
Epochs
Epochs
Figure 13: Training curves of GRLHF-V reward models and RL optimization process.
28

---

10 Generality Analysis on LLa LLLVA-115-BB
To evaluate the robustness and broader applicability of our proposed method beyond the Qwen-Vi family, we conducted additional experiments the LLaVA-1.5-7B model. We reproduced our key research questions (RQI, RQ3, and RQ5) from the main paper to assess the method's generalizability to different model architectures.
10.1 Principle Learning Generalizability (RQ1)
We first investigate whether our GRM + RL framework facilitates more generalizable principle learning on the LLaVA architecture. As shown in Table 7 the results on LLaVA-1.5-7B align with the conclusions drawn from our experiments on Qwen-VL.
Table Principle Learning Performance Comparison on LLaVA-1.5-7B Across Different Training Methods, (P) denotes the pretrained model.
Datasets
Beavertails-
LLaVA-Critie
MILLM-as-a-Judd
Score-Only RM
0.52
0.55
0.51
GRM
0.57
0.59
0.56
GRM (P)
0.58
0.61
0.57
GRM + SFT
0,54
0.52
0.54
GRM + SFT (P)
0.55
0.52
0.55
GRM RI
0.69
0.72
0.64
GRM RL (P)
0.64
0.66
0.58
The GRM + RL method consistently achieves the highest scoring accuracy, demonstrating optimal principle learning capability across all three out-of-diitribution datasets. Notably, method yields significant performance improvements even on LLaVA-115-1 suggesting its applicability extends to models with different architectures.
10.2 Multi-Modal RLHF Performance (RQ3)
We next evaluate the performance of our full GRLHF-V pipeline, which combines GRM + RL with grouped comparison (GRPO), on mainstream multi-modal benchmarks. Table 8 shows that our method achieves optimal results across all five benchmarks. This further confirms that the combination of GRM and grouped comparison competitive and effective method for multi-modal RLHF, generalizing beyond the Qwen- VL series.
10.3 Analysis of Reward Hacking Behaviors (RQ5)
Finally, we examined the reward hacking behaviors of an over-trained LLaVA-1.5-7B model. We discovered that LLaVA-1.5-7B exhibits different form of reward hacking compared to the Qwen-VL series models. While Qwen-VI VL models tended to add text praising their own responses (as discussed in the main paper), the over-trained LLaVA model tends to excessively complimentary vocabulary to describe the image content itself. An example of this behavior is provided below:
...[Complete Response]... The composite image of all the elements the branded outfit, including the custom-made Revo-Knit pink dress, the matching headband and necklace, the Wilson head and racket, the Delta Duck logo on the left side, the W logo on the right side, the multiple accessories, the white wristband.socks and bracelets, all contribute to the overall powerful and memorable delivery of the brand's message, making it an emphatic signature for both the athlete and the endorsed brand. It effectively conveys the essence of the Wilson brand to the audience, creating lasting impression of the brand's values, commitment, and the quality of their products, ultimately serving as a powerful promotional tool.
We found that this specific behavior did not successfully "hack" the MLL evaluator: As shown in Table the performance of the over-trained model actually decreased on standard benchmarks.
29

---

Table 8: Multi-Modal RLHF Performance Comparison LLaVA-1.5-78 Using Different Training
Methods and Reward Models.
Model
Feedback
MIA-Bench
LLaVA-Wild
WileeeWlder
MM-Yet
MM-Vett-2
LLaVA-I.5-7B
N/A
61.15
72.71
45.01
37.15
35.78
38.52
PPO
RM
66.45
73.92
49.23
40.18
38.45
GRPO
RM
68
76,34
52.67
42.73
41.28
GRM
78.56
44.91
GRPO
GRM-SFT
70.23
77.34
54.89
42.52
43.91
GRLHF-V (Ours)
GRM-RL
72.59
81.98
59.65
47.86
47.73
Table Performance Comparison of Over-trained LLaVA-1.5-7B Models on LLaVA-Bench-Wer and MIA-Bench.
Model
LLa VA-Bench Wild
MIA-Bench
Initial Model
72.71
61.15
Over-trained Model
69.98
57.85
We hypothesize that this difference in reward hacking behavior stems from the differing capabilities of the model architectures. The LLaVA-1.5-70 model may not possess sufficient capacity to explore the more complex, self-referential hacking strategies observed the Qwen-Vi series. This suggests that reward hacking behaviors may be architecture-dependent. presenting an interesting avenue for future work.
11 Ablation Study: Choice of RL Optimization Algorithm
In our preliminary experiments, we conducted an ablation study to determine the optimal reinforce- ment learning (RL) algorithm for training generative reward model (GRM). We compared the efficacy of GRPO against the widely used PPO.
Our findings, summarized in Table TO and Table indicate that GRPO achieves superior perfor- mance in reward modeling. As shown in Table TO the GRM + GRPO configuration outperformed GRM PPO on the in-distribution test set. This performance advantage was consistently observed across all out-of-distributiin (OOD) benchmarks (Ta.le Beyond performance benefits, GRPO offers significant advantage in computational efficiency. Unlike PPO, GRPO does not require the training of an auxiliary critic model. We hypothesize that this is viable because the preference learning task possesses a relatively direct reward structure: the model is primarily rewarded for assigning a higher score to the cuuman-preferred response. This optimization landscape may not necessitate the complex value estimation and variance reduction provided by critic. The GRPO method, which relies on averaging trajectory reward values, proves to be both sufficient and effective.
Given its superior performance and reduced computational overhead, GRPO was selected as the final algorithm for all subsequent experiments.
Table 10: Performance Comparison on In-Distribution Test Set
Method
Accuracy
GRM + GRPO
0.81
GRM PPO
0.78
Table 11: Performance Comparison
(OOD) Test Sets
Method
Beavertails-
LLaVA-Critie
MLLM-as-a-Judge
GRM GRPO
0.79
0.78
0.68
GRM PPO
0.75
0.72
0.64
30

---

12 Computational Overhead
critical factor for the practical deployment of our grouped comparison method is the associated computational overhead. To quantify this, we benchmarked the computation time for single training step, which includes the rollout, scoring, and parameter update phases.
The experiments were conducted on a system with H800 GPUs, 2 for RM serving and inference, and 14 for RL fine-tuning. The batch size and all other hyperparameters were held consistent with those detailed in the appendix. Table 12 presents the results.
Table 12: Computational Overhead of the Grouped Comparison Method with Different Numbers of Candidate Responses (n)
Number of Candidate Responses (n1)
7
Time (ms)
409
675
1176
The data in Table 12 indicates that the computational overhead scales significantly with the number of candidate responses (n.). We identify this scalability as a primary limitation the current approach.
We attribute this increase to the pairwise comparison mechanism. Our method performs A (or nnn - 1)) comparisons on n candidate responses, resulting a computational complexity that scales quadratically (O(n²)) with the number of candidates.
Addressing this bottleneck a promising direction for future research. We hypothesize that exhaus- tively comparing all A pairs may not be essential to achieve results comparable to, or even exceeding, the current method's performance. Future work could explore more efficient comparison strategies. For instance, methods from social choice theory or tournament selection algorithms may provide valuable frameworks for reducing the total number of comparisons required, thereby mitigating the computational cost without sacrificing alignment quality.
31


Reviews:
[{"review_id": "n4yIQyyyRV", "reviewer": "Reviewer_mrSo", "summary": "The authors address the problem of aligning a multimodal LLM (MLLM) to human preferences using RLHF. Their proposed method has two major stages. First, they propose using multi-modal generative reward modeling instead of traditional reward modeling, and an important part of this is automatically learning human principles that seem to be aligned to the expressed preferences. Next, they propose a fine-tuning scheme where instead of one pair of candidate responses at a time, the authors generate mutliple candidate responses and use the pairwise comparisons between each pair to compute average score for better rewards for the RL method. Finally, they present extensive experimental results to show the efficacy of their methods in improving performances of open-source small scale (2B-7B) LLMs.", "questions": "1. Have you considered alternative approaches to predicting scores from pairwise comparison matrices, such as spectral ranking methods?\n\n2. The authors demonstrate a near-linear increase in benchmark scores as the number of candidate responses increases in grouped comparison. What are the computational tradeoffs involved? At what point does this linear growth taper off?", "limitations": "Despite grouped comparison being a pivotal part of the paper, the authors' exploration in utilizing the grouped data in a structured way was limited.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n- The paper clearly shows how combining generative reward models (GRM) with pairwise preferences clearly improves the performance of the MLLM in benchmark tasks. The experimental results show that fine-tuning with GRM in an RL methodology and grouped comparison-based score computation both individually increase performance in benchmark tasks. But additionally, these two methods together see the biggest improvement.\n\n- The extensive experimentation, and presentation of specific research questions that can be answered based on the experiments was great.\n\nWeaknesses\n\n- Some parts of the paper was hard to follow. A key contribution, grouped comparison, is only explained halfway through. A short description like \"we generatr multiple candidates at the same time, compare each pair of candidates to each other, and use all available comparisons to calculate average scores for all the candidates\" could've been mentioned earlier on, in the introduction. Some motivational statements in the introduction seemed out-of-place to me. For example, claiming \"Pair-wise comparison is blocking multi-modal principles from generalizing to learnable rewards\" seemed imprecise, as the authors still use pairwise comparisons, just multiple ones simultaneously, rather than list-based or rank-based methods which would be a more stark contrast to pairwise comparisons. Another example of a writing issue, in RQ6 of section 4, the first sentence (\"SPCT mentions that providing principles as a reference within the user prompt.\") is not a proper sentence. I suggest revising the paper to proofread other possible errors that can lead to lack of clarity. \n\n- For the grouped comparison part, I mentioned how the authors still use pairwise comparisons. There are more structured approaches to predicting scores from a pairwise comparison matrix as such as spectral ranking (which works well with cardinal scores). (ref: https://arxiv.org/pdf/0912.0238). It would be interesting to see if calculating rewards in that manner shows any visible improvement. Another possible venue of work to explore would be the use of social choice technique in AI alignment (which suggests using more than pairwise preferences). See here: https://arxiv.org/pdf/2404.10271\n\n- The authors mention how there's almost a linear increase in some benchmark scores as number of candidate responses in the grouped comparison method. The authors don't really mention the computational tradeoffs here. Or how long the linear growth sustains.", "was_revised": true, "final_justification": "The authors have added more experimental results that introduce more structured ways of going beyond pairwise preferences. Conditional on the fact that they add the extta experimental results in the revised paper, I'm happy to update the score and give a stronger recommendation for acceptance.", "rebuttal": "Dear reviewer **mrSo**: \n\nThank you for your invaluable and insightful feedback. Your thoughtful comments and constructive suggestions have been instrumental in helping us enhance the clarity and quality of our paper. We sincerely appreciate the time and effort you dedicated to reviewing our work. We have carefully considered all your points and have revised the manuscript accordingly.\n\nWe have provided a detailed point-by-point response to your questions. **If you feel we have addressed your concerns, we sincerely and kindly hope you will give us higher support.**\n\n---\n\n> **W1:** Some parts of the paper was hard to follow. A key contribution, grouped comparison, is only explained halfway through. A short description like \"we generatr multiple candidates at the same time, compare each pair of candidates to each other, and use all available comparisons to calculate average scores for all the candidates\" could've been mentioned earlier on, in the introduction. Some motivational statements in the introduction seemed out-of-place to me. For example, claiming \"Pair-wise comparison is blocking multi-modal principles from generalizing to learnable rewards\" seemed imprecise, as the authors still use pairwise comparisons, just multiple ones simultaneously, rather than list-based or rank-based methods which would be a more stark contrast to pairwise comparisons. Another example of a writing issue, in RQ6 of section 4, the first sentence (\"SPCT mentions that providing principles as a reference within the user prompt.\") is not a proper sentence. I suggest revising the paper to proofread other possible errors that can lead to lack of clarity.\n\n**Response 1: Clarification of \"Grouped Comparison\"**\n\nWe completely agree with your excellent suggestion that a clearer, earlier explanation of \"grouped comparison\" would significantly improve the reader's understanding of our core contributions. We have revised the introduction to include a concise description of this mechanism, much like the one you proposed. This ensures that the concept is clearly defined from the outset, providing a better foundation for the sections that follow.\n\nAlso, we thank you for pointing out the imprecision in some of our motivational statements. Upon reflection, we concur that the claim \"Pair-wise comparison is blocking multi-modal principles from generalizing to learnable rewards\" could be misconstrued. Our method indeed leverages and extends pairwise comparisons in a novel way rather than abandoning them. We have refined this statement in the introduction to more accurately articulate that our contribution overcomes the limitations of traditional single pairwise comparisons by aggregating information from multiple simultaneous comparisons within a group.\n\n---\n\n> **W2 & Q1:** For the grouped comparison part, I mentioned how the authors still use pairwise comparisons. There are more structured approaches to predicting scores from a pairwise comparison matrix as such as spectral ranking (which works well with cardinal scores). (ref: https://arxiv.org/pdf/0912.0238). It would be interesting to see if calculating rewards in that manner shows any visible improvement. Another possible venue of work to explore would be the use of social choice technique in AI alignment (which suggests using more than pairwise preferences). See here: https://arxiv.org/pdf/2404.10271\n\n**Response 2: Additional Experiments For Structured Prediction**\n\nWe are delighted to see your insightful comments. This series of research will undoubtedly become a very valuable improvement direction for generative RLHF-V in the future, and demonstrates its reference value for other fields. \n\nDuring the rebuttal period, we implemented the four novel scoring methods you mentioned: Spectral Ranking and the three scoring methods from social choice techniques: Borda count, Instant Runoff, and Ranked Pairs. We tested them in two scenarios: reward model scoring and reinforcement learning fine-tuning.\n\n- **Experiment (1/2): Reward Model Scoring**\n\nWe selected the batch ranking task from MLLM-as-a-Judge, which requires the reward model to rank multiple responses, using the minimum edit distance from human expert results as the metric (lower is better). The results are shown in **Table 1**, where the left is the initial Qwen2.5-VL-Instruct-7B, and the right is the model trained with our proposed method:\n\n**Table 1: Performance Comparison of Different Scoring Methods on MLLM-as-a-Judge Batch Ranking Task**\n\n| Methods | Ours | Init. |\n|-|-|-|\n| Average Score (Ours) | *0.4451* | 0.5143 |\n| Borda Count| 0.4931 | 0.5523 |\n| Instant Runoff | 0.4640 | *0.5071* |\n| Ranked Pairs | **0.4352** | **0.4965** |\n| Spectral Ranking | 0.5301 | 0.5656 | \n\n1. Social technical methods represented by Ranked Pairs can indeed improve the performance of grouped comparison methods, outperforming our implemented Average Score method in the paper.\n2. Models fine-tuned with our method show the most significant improvements on both Average Score and Ranked Pairs methods.\n3. While the Average Score method is not optimal, it still achieves relatively good performance.\n\nBased on the above results, we find that social technical methods can indeed enhance the upper limit of grouped comparison methods, which is encouraging; meanwhile, our adopted Average Score method also achieves relatively good performance and can serve as an excellent baseline.\n\n- **Experiment (2/2): Reinforcement Learning Fine-tuning**\n\n**Table 2: Multi-Modal Benchmark Performance of Different Scoring Methods in Reinforcement Learning Fine-tuning**\n\n| Model | Score Methods| MIA-Bench | LLaVA-Wild | LLaVA-Wilder | MM-Vet | MM-Vet-v2 |\n|-------------|--------------------|-----------|------------|--------------|--------|-----------|\n| Qwen2-VL-7B | N/A| 52.58 | 81.30| 47.18| 32.12| 27.15 |\n| + GRLHF | Score Average | 62.31 | 103.55 | 62.84| 41.25| 45.16 |\n| + GRLHF | Borda Count| 55.76 | 83.06| 55.23| 36.87| 35.08 |\n| + GRLHF | Instant Runoff | 57.89 | 84.25| 59.12| 35.12| 36.53 |\n| + GRLHF | Ranked Pairs | 59.94 | 89.76| 57.52| 39.56| 39.75 |\n| + GRLHF | Spectral Ranking | 57.63 | 87.76| 56.89| 40.08| 38.01 |\n\nWe find that in the context of reinforcement learning fine-tuning, the improvements from the above social technical methods are not significant. We believe this is because the Score Average method can provide more fine-grained scoring for various answers, while methods like Ranked Pairs can only provide coarse-grained rankings, making it difficult to provide sufficiently fine-grained supervisory signals for reinforcement learning.\n\nStill, we believe that the social technical methods have the potential to enhance the performance of grouped comparison methods, and we hope our empirical study can inspire more research in this direction or other applications. **We will include the above experiments and analysis in the `appendix: potential research directions` section, denoting that this is inspired by the reviewer mrSo's suggestion.** We expect researchers in fields such as social choice theory to further explore the applications of these methods in alignment scenarios.\n\n---\n\n> **W3 & Q2:** The authors mention how there's almost a linear increase in some benchmark scores as number of candidate responses in the grouped comparison method. The authors don't really mention the computational tradeoffs here. Or how long the linear growth sustains.\n\n**Response 3: Clarification of Computational Tradeoffs**\n\nThank you for your thorough and rigorous feedback. We present the computational overhead of the grouped comparison method. The data in **Table 3** represents the computation time for a single step, including rollout, scoring, and parameter updates, with batch size and other hyperparameters consistent with those in the appendix. The experiments were conducted on 16×H800 GPUs.\n\n**Table 3: Computational Overhead of Grouped Comparison Method with Different Numbers of Candidate Responses**\n\n| Number of Candidate Responses | Time (ms) |\n|--------------------------------|-----------|\n| 3 | 409 |\n| 5 | 675 |\n| 7 | 1176 |\n\nWe sincerely acknowledge that as the number of candidate responses increases, the computational overhead of the grouped comparison method also increases significantly. We commit to updating these results in the limitations section of our paper.\n\nWe are willing to engage in further discussion on this issue. Specifically, we believe that the primary source of computational overhead comes from the $A_n^2$ comparisons performed on $n$ candidate responses, which leads to a significant increase in computation time as $n$ grows. However, it may not be necessary to perform pairwise comparisons on all responses to achieve results comparable to, or even better than, those obtained through $A_n^2$ comparisons. We believe that methods from social technical and other fields may offer valuable insights to help us reduce the number of comparisons and computational overhead. **We will also discuss this in the `appendix: potential research directions`, denoting that this is inspired by the reviewer mrSo's suggestion.** Thank you again for your insightful comments."}, {"review_id": "cuRIFeq8iG", "reviewer": "Reviewer_ecEr", "summary": "This paper introduces Generative RLHF-V, a framework for aligning multimodal large language models (MLLMs) with human preferences by combining generative reward modeling (GRM) and reinforcement learning from human feedback (RLHF) in a multimodal context. Empirical results demonstrate that Generative RLHF-V significantly improves performance across multiple benchmarks and multiple model sizes, outperforming traditional score-only RMs and supervised GRMs. The authors further analyze generalization capabilities, scaling properties, and potential vulnerabilities such as reward hacking.", "questions": "See above", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strength\n\nThe paper proposes a novel framework that combines generative reward models (GRMs) with multi-modal reinforcement learning from human feedback (RLHF), addressing the limitations of score-only reward models.\n\nThe experiments are comprehensive, covering 7 benchmarks and 4 MLLM variants. The reported performance significantly outperforms the selected baselines, showing the effectiveness of the proposed method.\n\nThe authors include a careful analysis of reward hacking in over-optimized settings, which is a critical concern in RL-based alignment. These insights could serve as a valuable reference for future research.\n\n### Weakness\n\nThe main experiments are conducted only on the Qwen-VL family, so it remains uncertain how well the pipeline transfers to models with different architectures or modalities. Including results on several diverse model series would give a clearer picture of the method’s robustness and broader applicability.\n\nThe core ideas are closely related to prior work in the text-only NLP literature, and thus the novelty of the contribution is somewhat limited (though I acknowledge that the paper thoughtfully extends generative reward modeling and grouped comparison to the multi-modal setting and provides useful insights).", "was_revised": true, "final_justification": "The authors have added additional experiments on LLaVA and more discussions on the novelty of the paper, which seems reasonable to me.", "rebuttal": "Dear reviewer **ecEr**:\n\nWe appreciate the reviewers' recognition of the effectiveness and novelty of our method. In response to the reviewers' concerns about robustness and applicability, we have supplemented our results with experiments on LLaVA-1.5-7B and provided detailed analysis. Regarding the novelty of our contributions, we have also provided comprehensive discussions. \n\nWe hope these responses address your concerns. **And we sincerely and kindly hope that you can provide greater support. If you have additional questions about our responses or other aspects you would like to discuss, we also welcome further communication with you.**\n\n> **W1:** The main experiments are conducted only on the Qwen-VL family, so it remains uncertain how well the pipeline transfers to models with different architectures or modalities. Including results on several diverse model series would give a clearer picture of the method's robustness and broader applicability.\n\n**Response 1: Additional Experiments on LLaVA-1.5-7B**\n\nYou are certainly a rigorous and responsible reviewer! Our experiments cover two versions of Qwen-VL (2.0 and 2.5), two modes (pretrained and instruction-following models), and three sizes (2B, 3B, and 7B).\n\nNevertheless, we sincerely agree with your suggestion and have supplemented experiments on LLaVA-1.5-7B during the rebuttal period, reproducing our key experimental RQs from our paper, as detailed below:\n\n> RQ1 in paper: Does the GRM+RL facilitate more generalizable principle learning from preferences?\n\n**Table 1: Principle Learning Performance Comparison on LLaVA-1.5-7B Across Different Training Methods**\n\n| Datasets      | Beavertails-V | LLaVA-Critic | MLLM-as-a-Judge |\n|---------------|---------------|--------------|-----------------|\n| Score-Only RM |  0.52         |  0.55        |   0.51          |\n| GRM           |  0.57         |  0.59        |   0.56          |\n| GRM (P)       |  0.58         |  0.61        |   0.57          |\n| GRM + SFT     |  0.54         |  0.52        |   0.54          |\n| GRM + SFT (P) |  0.55         |  0.52        |   0.55          |\n| GRM + RL      |  **0.69**         |  **0.72**        |   **0.64**          |\n| GRM + RL (P)  |  0.64         |  0.66        |   0.58          |\n\nThe results show that on LLaVA-1.5-7B, GRM + RL still demonstrates the highest scoring accuracy and optimal principle learning capability across three out-of-distribution datasets, consistent with the experimental conclusions in the paper. \n\nIt is worth noting that despite LLaVA-1.5-7B's limited model capacity, our method still achieved significant performance improvements. This further expands the applicability scope of our approach.\n\n> RQ3 in paper: Are GRM+RL and grouped comparison competitive methods for multi-modal RLHF?\n\n**Table 2: Multi-Modal RLHF Performance Comparison on LLaVA-1.5-7B Using Different Training Methods and Reward Models**\n\n| Model            | Feedback | MIA-Bench | LLaVA-Wild | LLaVA-Wilder | MM-Vet | MM-Vet-V2 |\n|------------------|----------|-----------|------------|--------------|--------|-----------|\n| LLaVA-1.5-7B     | N/A      |  61.15    |  72.71     |   45.01      | 37.15  |   35.78   |\n| + DPO            | RM       |  64.30    |  70.08     |   47.58      | 38.52  |   36.10   |\n| + PPO            | RM       |  66.45    |  73.92     |   49.23      | 40.18  |   38.45   |\n| + GRPO           | RM       |  68.12    |  76.34     |   52.67      | 42.73  |   41.28   |\n| + GRPO           | GRM      |  69.85    |  78.56     |   55.42      | 44.91  |   43.67   |\n| + GRPO           | GRM+SFT  |  70.23    |  77.34     |   54.89      | 42.52  |   43.91   |\n| + GRLHF-V (Ours) | GRM+RL   |  **72.59**    |  **81.98**     |   **59.65**      | **47.86**  |   **47.73**   |\n\nThe results show that on five mainstream benchmarks, the method combining GRM + RL with grouped comparison achieves optimal results, consistent with the experimental conclusions in the paper.\n\n> RQ5 in paper: What is the reward hacking behaviors of an over-trained Generative RLHF-V model?\n\nWe found that LLaVA-1.5-7B exhibits a different reward hacking behavior from the Qwen-VL series models under over-training conditions. Compared to the behavior mentioned in our main text where models add extra text to praise their own responses, LLaVA tends to use overly complimentary vocabulary to describe the image itself. Here is an example:\n\n```\n...[Complete Response]...The composite image of all the elements of the branded outfit, including the custom-made Revo-Knit pink dress, the matching headband and necklace, the Wilson head and racket, the Delta Duck logo on the left side, the W logo on the right side, the multiple accessories, the white wristband,socks and bracelets, all contribute to the overall powerful and memorable delivery of the brand's message, making it an emphatic signature for both the athlete and the endorsed brand. It effectively conveys the essence of the Wilson brand to the audience, creating a lasting impression of the brand's values, commitment, and the quality of their products, ultimately serving as a powerful promotional tool.\n```\n\nWe found that this behavior cannot hack higher performance in MLLM-as-a-Judge evaluations, and the actual performance of over-trained models is not good, as shown in **Table 3**. \n\n**Table 3: Performance Comparison of Over-trained Models on LLaVA-Wild and MIA-Bench**\n\n|Model|LLaVA-Wild|MIA-Bench|\n|---|---|---|\n|Initial Model|72.71|61.15|\n|Over-trained Model|69.98|57.85|\n\nWe believe this may be because the capabilities of the LLaVA-1.5-7B model are limited and not sufficient to explore reward hacking behaviors similar to those observed in the Qwen-VL series. **We will add discussions on reward hacking behaviors in models with different architectures to the future work section, and acknowledge in the paper that this addition is thanks to the kind reminder from reviewer ecEr.** We sincerely appreciate your insightful suggestions once again.\n\n---\n\n> **W2:** The core ideas are closely related to prior work in the text-only NLP literature, and thus the novelty of the contribution is somewhat limited (though I acknowledge that the paper thoughtfully extends generative reward modeling and grouped comparison to the multi-modal setting and provides useful insights).\n\n**Response 2: Clarification of Novelty**\n\nThank you for your recognition and questioning of our work. As you pointed out, Generative RLHF-V is an extension based on existing methods in the text-only NLP field: self-principled-critique-finetuning (SPCT). However, we would like to further clarify that this work is not simply extending SPCT from text modality to multimodal modality, but rather building a complete pipeline around the multimodal human principles learning problem, including reward modeling and grouped comparison (as you mentioned in the strengths, this is also a thoughtful methodology that provides useful insights). Within this pipeline, we have also conducted more in-depth and unique analysis of Generative RLHF-V's learning capabilities for multi-modal principles.\n\n1. We found that compared to SPCT's approach of providing reference principles, directly letting the model actively infer the principles behind human preferences can achieve stronger and more accurate principle learning effects. We demonstrate this conclusion in Figure 5 and Table 3.\n\n2. We found that in multimodal scenarios, integrating grouped comparison and GRM+RL near linearly enhances multi-modal RLHF performance across various settings of candidate responses. We demonstrate this conclusion in Figure 7.\n\n3. We found that over-trained GRM+RL models exhibit self-praising reward hacking behavior on Qwen-VL series models, while during the rebuttal period we discovered that LLaVA-1.5-7B exhibits reward hacking behavior of overly praising the image itself. As shown in Figure 8, these analyses provide unique insights into the use of generative reward modeling in multimodal scenarios.\n\nIn summary, we believe that the contribution of this work goes far beyond algorithmic improvements, but rather represents the first systematic empirical study of the entire pipeline from reward modeling, grouped comparison, to various ablation analyses on the multimodal principles learning problem. We again thank the reviewer for their tolerant attitude and positive rating of our work. Your attention has prompted us to more clearly summarize and express the contributions of our work, and we hope the above supplementary explanations can effectively address your concerns."}, {"review_id": "pUF2TSpLiN", "reviewer": "Reviewer_NwUp", "summary": "This work introduces Generative RLHF-V, a novel framework for aligning multi-modal large language models (MLLMs) with human intentions. Traditional score-based reward models in RLHF suffer from poor accuracy and generalization, while existing generative reward models struggle to produce learnable scalar rewards due to their pairwise design. Generative RLHF-V addresses these issues through a two-stage process: (1) training generative reward models using reinforcement learning to better capture human preferences, and (2) optimizing RL via grouped response comparisons to improve multi-modal scoring accuracy. The authors conduct extensive experiments to verify the effectiveness of their method.", "questions": "Please refer to Strengths and Weaknesses part.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n\n1. The writing and clear and easy to follow.\n\n2. The author studied a novel method for training generative reward models for multi-modal language models.\n\n3. The author conducts extensive experiments to verify the effectiveness of their proposed method.\n\n### Weakness:\n\n1. The authors propose using group comparisons based on model-generated responses to train reinforcement learning. Have the authors considered exploring alternative RL algorithms, such as GRPO, for training the reward models?\n\n2. It is unclear which specific algorithm the authors use to train the MLLM based on the proposed reward models. Do the authors employ PPO, GRPO, or another RL algorithm for this purpose?\n\n3. The authors mention the emergence of self-praise behavior in the experimental results, which appears similar to reasoning patterns induced by reinforcement learning. Is this observation specific to the Qwen-based model, or can it be generalized to other multi-modal large language models (MLLMs) as well?", "was_revised": true, "final_justification": "I agree with the opinion of most of reviwers. The paper should be accepted.", "rebuttal": "Dear reviewer **NwUp**: \n\nWe are grateful for your recognition of the clarity of our writing, method innovation, and the breadth of our experiments. You must have carefully read our paper to propose such detailed and insightful feedback, and we will address each point. **If you believe we have resolved your concerns, we sincerely and kindly hope that you will give us higher support.** Also, if you have additional questions about our responses or other aspects you would like to discuss, we also welcome further communication with you.\n\n> **W1:** The authors propose using group comparisons based on model-generated responses to train reinforcement learning. Have the authors considered exploring alternative RL algorithms, such as GRPO, for training the reward models?\n\n---\n\n**Response 1: Exploration of Alternative RL Algorithms**\n\nYou are certainly experienced in RL experiment research! In fact, during the early research stage, we simultaneously attempted GRPO and PPO RL algorithms to train the generative reward model, but GRPO performed slightly better in reward modeling, and because GRPO did not need to train an additional critic model, it consumed less computational resources, so we finally chose GRPO. The experimental results are as shown in **Table 1** and **Table 2**.\n\n**Table 1: Performance Comparison of GRM+GRPO vs GRM+PPO on Distribution-in Test**\n\n| Method | GRM + GRPO | GRM + PPO |\n|--------|------------|-----------|\n| Accuracy | 0.81       | 0.78      |\n\n**Table 2: Performance Comparison of GRM+GRPO vs GRM+PPO on Distribution-out Test**\n\n| Datasets      | Beavertails-V | LLaVA-Critic | MLLM-as-a-Judge |\n|---------------|---------------|--------------|-----------------|\n| GRM-GRPO | 0.79 | 0.78 | 0.68 |\n| GRM-PPO | 0.75 | 0.72 | 0.64 |\n\nWe believe the reason for this might be that the reward for the preference learning task is relatively \"simple\", and the model only needs to provide a higher score for the human-preferred response to obtain the reward. Therefore, this task does not require a critic model to provide additional information, and averaging the trajectory reward values within the group can achieve good results.\n\n---\n\n> **W2:** It is unclear which specific algorithm the authors use to train the MLLM based on the proposed reward models. Do the authors employ PPO, GRPO, or another RL algorithm for this purpose?\n\n**Response 2: Clarification of RL Algorithm**\n\nThank you for pointing this out! We use GRPO algorithm to train the multi-modal generative reward model. We will supplement this in section 4.1 Experiment Setup. We believe this will make the article clearer.\n\n---\n\n> **W3**: The authors mention the emergence of self-praise behavior in the experimental results, which appears similar to reasoning patterns induced by reinforcement learning. Is this observation specific to the Qwen-based model, or can it be generalized to other multi-modal large language models (MLLMs) as well?\n\n**Response 3: Exploration of \"Self-Praise\" Behavior**\n\nThank you for your attention to our exploration of the \"self-praise\" behavior. Extending this conclusion to other models will undoubtedly further strengthen our contribution. During the rebuttal period, we conducted additional experiments on LLaVA-1.5-7B, with results as follows:\n\nWe found that LLaVA-1.5-7B exhibits a different reward hacking behavior from the Qwen-VL series models under over-training conditions. Compared to the behavior mentioned in our main text where models add extra text to praise their own responses, LLaVA tends to use overly complimentary vocabulary to describe the image itself. Here is an example:\n\n```\n...[Complete Response]...The composite image of all the elements of the branded outfit, including the custom-made Revo-Knit pink dress, the matching headband and necklace, the Wilson head and racket, the Delta Duck logo on the left side, the W logo on the right side, the multiple accessories, the white wristband,socks and bracelets, all contribute to the overall powerful and memorable delivery of the brand's message, making it an emphatic signature for both the athlete and the endorsed brand. It effectively conveys the essence of the Wilson brand to the audience, creating a lasting impression of the brand's values, commitment, and the quality of their products, ultimately serving as a powerful promotional tool.\n```\n\nWe found that this behavior cannot hack higher performance in MLLM-as-a-Judge evaluations, and the actual performance of over-trained models is not good, as shown in **Table 3**. \n\n**Table 3: Performance Comparison of Over-trained Models on LLaVA-Wild and MIA-Bench**\n\n|Model|LLaVA-Wild|MIA-Bench|\n|---|---|---|\n|Initial Model|72.71|61.15|\n|Over-trained Model|69.98|57.85|\n\nWe believe this may be because the capabilities of the LLaVA-1.5-7B model are limited and not sufficient to explore reward hacking behaviors similar to those observed in the Qwen-VL series. **We will add discussions on reward hacking behaviors in models with different architectures to the future work section, and acknowledge in the paper that this addition is thanks to the kind reminder from reviewer NwUp.** We sincerely appreciate your insightful suggestions once again."}, {"review_id": "ozpr3VO8Qn", "reviewer": "Reviewer_vM9w", "summary": "This work introduces a framework for better learning more generalizable reward models for preference learning by inferring principles that drive the preferences. This is done in two stages, firstly learn the principles from a preference dataset (chosen/rejected pairs given a prompt), then use the principles to refine scores by pairwise comparing candidate responses within a group of responses. By comparing the responses within the group, a more robust estimate of the reward is learned. A number of models are tested using a number of benchmarks.", "questions": "In the caption for Figure 4 it states that “Response B provides accurate and comprehensive information, thus receiving the highest score”. It was not clear to me where the model learns the explicit principles “accuracy” and “comprehensiveness”. Is it the case that this is implied only because of the chosen/rejected responses? If this is the case, might it be that these are not the actual preferences captured, but rather the model representation of the principles is some entanglement of accuracy and comprehensiveness (i.e., not interpretable to humans, but equivalently meaningful to the model)?\n\nLines 161-164: If the point about entangled representations above is correct, for the case of GRM+RL might it be that performance is harmed when providing principles as a hint because the provided principles do not align with the model interpretation of the learned principles? If it was due to redundancy because the principles are already learned/accounted for (as suggested) then might one just expect the model to ignore that redundant input?\n\n**Suggesions**\n\nFigure 4 — in the figure where the scores are shown (Left, Right, Average) replace with (Left, Right, Mean)\n\nIt seems the symbol “r” has dual meanings — rewards and reasoning traces. Symbols should not be reused to avoid confusion.\n\nRemove the period after D before the equation above Line 103.\n\nLine 107: “to evaluates multiple'” > “to evaluate multiple”\n\nLine 111: Remove the repeated sentence that begins “Each generated response ..”\n\nLine 119 (and subsequently throughout the paper): Capitalize references to subsections, i.e., Subsection 4.2.\n\nIn text, for numbers less than ten, write them out in full. E.g., Line 182 “4” > “four”, etc.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The code and models are being made available.\n- Public dataset are used to training/evaluation.\n- Comprehensive evaluation against a number of benchmarks, and the the evaluation demonstrates the efficacy of the approach.\n\n**Weaknesses**\n\n- As noted, the use of RL means the approach is vulnerable to reward hacking.\n- From what I understand the principles are not learned explicitly, which leads to a couple of questions (see below)", "was_revised": true, "final_justification": "This is a nice body of work and is a valuable contribution.", "rebuttal": "Dear reviewer **vM9w**:\n\nThank you for your high recognition and affirmation of our work! We are greatly encouraged. We will address each of your comments point by point, and hope that our responses will strengthen your confidence in our work.\n\n> **Q1:** As noted, the use of RL means the approach is vulnerable to reward hacking.\n\n**Response 1: Clarification of Reward Hacking**\n\nYou raise a very insightful question. Reward hacking is indeed an unavoidable challenge in reinforcement learning approaches [1, 2]. It essentially occurs when the policy model discovers vulnerabilities in the reward model and exploits them to obtain high rewards, while fundamentally contradicting human intentions.\n\n**Our proposed generative RLHF-V aims to provide a more robust reward for reinforcement learning fine-tuning of MLLMs, thereby improving model performance.** This has been thoroughly demonstrated in our paper's experiments and has also received your recognition.\n\n**Beyond performance improvements, the contribution of generative RLHF-V is not merely an algorithm, but rather `an empirical study on the application of multimodal generative reward models`.** Therefore, we candidly acknowledge that as a reinforcement learning method, generative RLHF-V is also susceptible to reward hacking issues. We do not wish to conceal this fact and are willing to expose the reward hacking problems that may exist in extreme scenarios with our method. \n\n**We hope this will inspire subsequent research, whether in more thorough mechanistic analysis or more robust algorithm design.**\n\nFor example, during the rebuttal period, we implemented the four novel scoring methods of social technical: Spectral Ranking[3] and the three scoring methods from social choice techniques[4]: Borda count, Instant Runoff, and Ranked Pairs. We tested them in two scenarios: reward model scoring and reinforcement learning fine-tuning.\n\nWe selected the batch ranking task from MLLM-as-a-Judge, which requires the reward model to rank multiple responses, using the minimum edit distance from human expert results as the metric (lower is better). The results are shown in **Table 1**, where the left is the initial Qwen2.5-VL-Instruct-7B, and the right is the model trained with our proposed method:\n\n**Table 1: Performance Comparison of Different Scoring Methods on MLLM-as-a-Judge Batch Ranking Task**\n\n| Methods | Ours | Init. |\n|-|-|-|\n| Average Score (Ours) | *0.4451* | 0.5143 |\n| Borda Count| 0.4931 | 0.5523 |\n| Instant Runoff| 0.4640 | *0.5071* |\n| Ranked Pairs| **0.4352** | **0.4965** |\n| Spectral Ranking| 0.5301 | 0.5656 | \n\n1. Social technical methods represented by Ranked Pairs can indeed improve the performance of grouped comparison methods, outperforming our implemented Average Score method in the paper.\n2. Models fine-tuned with our method show the most significant improvements on both Average Score and Ranked Pairs methods.\n3. While the Average Score method is not optimal, it still achieves relatively good performance.\n\nBased on the above results, we find that social technical methods can indeed enhance the upper limit of grouped comparison methods, which is encouraging; meanwhile, our adopted Average Score method also achieves relatively good performance and can serve as an excellent baseline.\n\nFinally, thank you again for your kind understanding and support.\n\n[1]: Gao L, Schulman J, Hilton J. Scaling laws for reward model overoptimization[C]//International Conference on Machine Learning. PMLR, 2023: 10835-10866.\n\n[2]: Ji J, Qiu T, Chen B, et al. Ai alignment: A comprehensive survey[J]. arXiv preprint arXiv:2310.19852, 2023.\n\n[3]: Vigna S. Spectral ranking[J]. Network Science, 2016, 4(4): 433-445.\n\n[4]: Conitzer V, Freedman R, Heitzig J, et al. Social choice should guide ai alignment in dealing with diverse human feedback[J]. arXiv preprint arXiv:2404.10271, 2024.\n\n---\n\n> **Q2:** It was not clear to me where the model learns the explicit principles “accuracy” and “comprehensiveness”. Is it the case that this is implied only because of the chosen/rejected responses? If this is the case, might it be that these are not the actual preferences captured, but rather the model representation of the principles is some entanglement of accuracy and comprehensiveness (i.e., not interpretable to humans, but equivalently meaningful to the model)? Lines 161-164: If the point about entangled representations above is correct, for the case of GRM+RL might it be that performance is harmed when providing principles as a hint because the provided principles do not align with the model interpretation of the learned principles? If it was due to redundancy because the principles are already learned/accounted for (as suggested) then might one just expect the model to ignore that redundant input?\n\n**Response 2: Clarification of Principles Learning**\n\nThank you very much for your thoughtful and constructive feedback on our paper. We are grateful for your positive assessment and for the insightful questions you raised, which have helped us to clarify and strengthen the core contributions of our work.\n\nWe would like to address the two excellent points you made regarding how the model learns principles and why providing them as hints can be detrimental.\n\n**1. On the Implicit Learning of Principles (e.g., \"Accuracy\" and \"Comprehensiveness\")**\n\nYou asked whether principles like \"accuracy\" and \"comprehensiveness\" are learned implicitly from the preference data, and suggested that the model’s internal representation might be an \"entanglement\" of these concepts rather than a discrete, human-interpretable one.\n\nYour intuition is entirely correct, and we thank you for articulating it so clearly. This is precisely the mechanism at play. Our model learns these principles **implicitly**. The RL objective (lines 103-104) simply rewards the Generative Reward Model (GRM) for correctly predicting the human-preferred response. To succeed at this task across a diverse dataset, the model must learn to identify the underlying features that consistently lead to a positive preference.\n\nThe principles it generates in its analysis (as seen in Figures 3 and 11) are its attempt to translate these complex, learned features into human-readable language. The model does not have a discrete \"accuracy\" module. Instead, as you insightfully suggest, its internal representation is a high-dimensional **entanglement of features** that, when decoded into text, are best described by words like \"accuracy,\" \"completeness,\" or \"step-by-step guidance.\" These are emergent concepts, not pre-programmed ones.\n\nWe will revise the manuscript to make this point more explicit for the reader. We agree that your framing of this as an \"entangled representation\" is a more precise and powerful description of the process.\n\n**2. On Performance Degradation When Providing Principles to GRM+RL**\n\nYou raised a very sharp point regarding our findings in lines 161-164 and RQ6, where providing principles as hints harmed the performance of the GRM+RL model. You hypothesized that this might be because the provided generic principles **do not align with the model's internally learned, specific principles**, rather than being merely redundant.\n\nThis is an excellent point, and we believe your hypothesis is a more accurate explanation than our initial suggestion of simple redundancy. The GRM+RL model, through its training, develops a sophisticated policy for generating *highly specific and context-dependent* principles for any given pair of responses. When we provide a static, generic principle from the dataset's annotation guidelines, we are not just giving it redundant information; we are imposing a constraint that **conflicts with its learned optimal policy**. This distracting constraint forces the model away from its flexible, tailored analysis and toward a more rigid one, which ultimately degrades its performance on out-of-distribution tasks.\n\nWe will update the discussion in lines 161-164 and the analysis in RQ6 to reflect this more nuanced and accurate explanation.\n\n**If you feel that our response in this round has not fully addressed your concerns, we are very willing to engage in further communication with you. Please feel free to ask questions at any time.**\n\nOnce again, we thank you for your time and for providing such valuable and thought-provoking feedback. Your comments have given us a clearer lens through which to view our own results and have helped us significantly improve the paper.\n\n> **Suggestions:**\n\n> Figure 4 — in the figure where the scores are shown (Left, Right, Average) replace with (Left, Right, Mean)\n\n> It seems the symbol “r” has dual meanings — rewards and reasoning traces. Symbols should not be reused to avoid confusion.\n\n> Remove the period after D before the equation above Line 103.\n\n> Line 107: “to evaluates multiple'” > “to evaluate multiple”\n\n> Line 111: Remove the repeated sentence that begins “Each generated response ..”\n\n> Line 119 (and subsequently throughout the paper): Capitalize references to subsections, i.e., Subsection 4.2.\n\n> In text, for numbers less than ten, write them out in full. E.g., Line 182 “4” > “four”, etc.\n\n**Response 3: Thank you for your detailed and helpful suggestions**\n\n**It's difficult to put into words the delight we felt when we saw your detailed feedback.** We have made corresponding modifications point by point in our paper and believe they are of tremendous help. We would like to express our heartfelt gratitude once again."}]

Author rebuttal:

Meta-review: **(a) Summary of Scientific Claims and Findings**

The paper introduces Generative RLHF-V, a framework for aligning multimodal large language models (MLLMs) with human preferences. Unlike traditional score-based reward models, the authors propose generative reward modeling (GRM) that captures richer human principles. Instead of comparing only one pair of candidate responses, the grouped comparison method generates multiple responses, compares them pairwise, and aggregates scores for more accurate reward estimation. The method is evaluated on small-to-medium scale open-source models, across seven benchmarks and four MLLM variants. Empirical results demonstrate gains over both score-only RMs and supervised GRMs, with also futher analysis.

**(b) Strengths**

- Novel framework: Integrates GRM and grouped comparisons into multimodal RLHF

- Strong empirical validation: Results show improvements over baselines, with complete benchmarks and ablation studies.

- Clear research questions: the paper is framed around specific RQs

- Analysis of limitations: Discussion of vulnerabilities such as reward hacking adds value to the paper

**(c) Weaknesses**

- Clarity and presentation: Several reviewers noted that explanations (especially of grouped comparisons) could be introduced earlier

- Computational tradeoffs: Despite the promising results, computational costs was not sufficiently analized.

**(d) Reasons for Final Decision**

I am recommending acceptance since all the reviewers agree that the paper has merits and represents a valuable contribution. **The authors are strongly encouraged to include the additional experimental results and additional discussion in the camera ready**

**(e) Summary of the discussion**

The reviewers are satisfied by the amendments promised by the authors (additional experimental evaluation and additional discussion). They agree that the paper merits acceptance.
Decision: Accept (poster)