MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
Authors: YiFan Zhang
Tao Yu
Haochen Tian
Chaoyou Fu
Peiyan Li
Jianshu Zeng
Wulin Xie
Yang Shi
Huanyu Zhang
Junkang Wu
Xue Wang
Yibo Hu
Bin Wen
Tingting Gao
Zhang Zhang
Fan Yang
Di ZHANG
Liang Wang
Rong Jin
Venue: ICML 2025 poster (2025)
Keywords: multimodal large language models
human preferences
alignment with human preference
Primary area: deep_learning->foundation_models
TL;DR: We introduce MM-rlhf, a dataset of 120k fine-grained human preference pairs, and propose novel methods  to significantly improve multimodal large language model alignment, achieving consistent performance gains across 10 evaluation dimensions.

Abstract: Existing efforts to align multimodal large language models (MLLMs) with human preferences have only achieved progress in narrow areas, such as hallucination reduction, but remain limited in practical applicability and generalizability. To this end, we introduce **MM-RLHF**, a dataset containing **120k** fine-grained, human-annotated preference comparison pairs. This dataset represents a substantial advancement over existing resources, offering superior size, diversity, annotation granularity, and quality.  Leveraging this dataset, we propose several key innovations to improve both the quality of reward models and the efficiency of alignment algorithms. Notably, we introduce the **Critique-Based Reward Model**, which generates critiques of model outputs before assigning scores, offering enhanced interpretability and more informative feedback compared to traditional scalar reward mechanisms.  Additionally, we propose **Dynamic Reward Scaling**, a method that adjusts the loss weight of each sample according to the reward signal, thereby optimizing the use of high-quality comparison pairs.  Our approach is rigorously evaluated across **10** distinct dimensions, encompassing **27** benchmarks, with results demonstrating significant and consistent improvements in model performance (Figure.1).

MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
Yi-Fan Zhang'
Tao Yu
Haochen Tian' Chaoyou Fu Peiyan Li Jianshu Zeng' Wulin Xie Xue Wang' Yibo Hu' Bin Wen Tingting Gao Zhang Zhang' Di Zhang' Liang Wang' Rong Jin"
Yang Shi Fan Yang?
Huanyu Zhang'
Junkang Wu
Abstract
plex tasks that involve the integration of vision, language, and audio, state-of of-the-art MLLMs today seldom undergo rigorous alignment stage (Wang et al., 2024a; Deitke et al., 2024; Chen et al., 2024c; Dai et al., 2024; Agrawal et al., 2024). Typically, these models only progress to the super- vised fine-tuning (SFT) stage, leaving critical aspects such truthfulness, safety, and alignment with human prefer- ences largely unaddressed. While recent efforts have begun to explore MLLM alignment, they often focus on narrow domains, such as mitigating hallucination or enhancing con- versational capabilities, which fail to comprehensively im- prove the model's 's overall performance and reliability. This raises a critical question:
Existing efforts to align multimodal large lan- guage models (MLLMs) with human preferences have only achieved progress in narrow areas, such as hallucination reduction, but remain limited in practical applicability and generalizability. To this end, we introduce MM-RLHF, a dataset contain- ing 120k fine- greeggraddd, human-annotann ence comparison pairs. This dataset represents a substantial advancement over existing resources, offering superior size, diversity, annotation gran- ularity, and quality. Leveraging this dataset, we propose several key innovations to improve both the quality of reward models and the efficiency of alignment algorithms. Notably, we introduce the Critique- Based Reward Model, which gen- erates critiques of candidate texts before assign- ing scores, offering enhanced interpretability and more informative feedback compared to tradi- tional reward models. Additionally, we propose Dynamic Reward Scaling. method that adjusts the loss weight of each training sample according to the reward signal, thereby optimizing the use of high-quality comparison pairs. Our approach is rigorously evaluated across 10 distinct dimen- sions, encompassing 27 benchmarks, with results demonstrating significant and consistent improve- ments in model performance (Figure. 1).
prefer-
Is alignment with human preferences only capable of enhancing MLLMs in a limited set of tasks?
In this work, we confidently answer this question with resounding "No.". We demonstrate that well-designed alignment pipeline can comprehensively enhance MLLMs along multiple dimensions, including visual perception, rea- soning, dialogue, and trustworthiness, thereby significantly broadening their practical applicability. To achieve this, we conduct in-depth investigations into three pivotal areas: data curation, reward modeling, and alignment algorithms.
At first, we introduce MM-RLHF, dataset designed to advance Multimodal Reinforcement Learning from Human Feedback (RI.FF). The dataset spans three key domains: im- age understanding, video understanding, and MLLM safety, Constructed through a rigorous pipeline, MM-RL AM-RLHF en- sures high-quality, fine-grained human annotations. Dataset creation process involves the following steps (Figure 2):
1. Introduction
Although Multimodal Large Language Models (MLLMs) have demonstrated remarkable potential addressing com-
Data Collection. We curate diverse set of multimodal tasks from various sources, totaling 10 million data instances, ensuring broad representation across tasks.
'Institute of Automation, School Artificial Intelli- gence, University of Chinese Academy of Sciences (UCAS), Beijing, China *Nanjing University (NJU), Nanjing, China KuaiShou, Beijing, China. Correspondence to: Chaoyou Fu <bradyfu24@ comm, Bin Wen <wenbin @ kuaishou.com>. Zhang Zhang <zhangzhang @ cas.a.edu.cn>
Data Selection. Through rigorous re-sampling, we extract 30k representative queries, ensuring diversity across a wide range of data types, such as real-world scenarios, mathematical reasoning, chart understand- ing, and other practical domains (Figure. 3).
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Model Response Generation. We utilize state-of-the-

---

MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
LlaVA-One-Vison (78)
MM-RE RF
MM-RLEE MM-DPO
+5.25%
-3.47%
+0.45%
+3.33%
+3.38%
+3.74%
-1.59%
-65.4%
K-8
20.3
26.1 14A
71A
6.3
18.6
20.3
49.2
30.0
20.1 36.6
200 263
139
Conversation
Hal Sorea
Hall Rate (1)
General
Chart
OCT
Malh
Malli-Tnage
Ville
ResiMWarM
Safiry
UlnSafity (1)
Figure 1. Performance gains achieved through alignment training on MM-RLHF and our alignment algorithm (MM-DPO), highlighting significant improvements across various tasks and metrics.
art models, such as Claude 3.5-Sonnet and Qwen2-VI. 72B, to generate responses for various tasks.
contrast, Dynamic Reward Scaling calculates a reward mar- gin for each comparison pair using HAI-RLFF-Rewardd During training. it assigns higher weights to comparison pairs with larger reward margins. This ensures that the most informative samples have a stronger influence model updates. As a result, the training process becomes more efficient, leading to improved model performance.
Fine-grained Human Annotation. We employ meticulous annotation process, involving over 50 an- notators over two months, to score, rank, and provide textual explanations for responses. This results in more than 120k high-quality ranked comparison pairs.
Finally, to rigorously evaluate our approach, we con- struct two specialized benchmarks. The first, MM-RLHE. RewardBench, sampled from our dataset and consists of meticulously human-annotated data for evaluating re- ward models. The second, RE--F-FFF--att curated and filtered from existing benchmarks and focuses on safety-related tasks, including privacy protection, adver- sarial attacks, jailbreaking, and harmful content detection.
Compared to existing datasets, MM-RLHF significantly advances in diversity, response quality, and annotation gran- ularity, providing a robust foundation for MLLM alignment.
is
Building on the MM-RLHF dataset, we investigate how data can enhance MLLM alignment, with focus on reward modeling and training optimization. Rec- ognizing the pivotal role of reward models in providing feed- back signals to guide the alignment process, we propose a Critique-Basse Bratique-Based Reward Model (Figure 4). Traditional reward models, which output scalar values, often lack inter- pretability, while directly using MILLMs as reward models place high demands on their instruction-followinn capabili- ties, limiting their practicality. To address these limitations, first transform concise human annotations into detailed, model-friendly formats using MLLMs. These enriched an- notations serve as learning targets, guiding the reward model to first generate critiques and then assign scores based on the critiques. This approach enables the model to provide fine-ge grained scoring explanations, significantly enhancing the quality and interpretability of the reward signals. MM- FLLHF-Reward-77 achieves SOTA performance on several benchmarks, outperforming several 72B-scale models.
We conduct extensive evaluations across ten key dimen- sions, covering 27 benchmarks. The results demonstrate that our training algorithm, combined with the high-quality MM-RLHF dataset, leads to significant improvements in model performance. Specifically, models fine-tuned with our approach achieve an average 11% gain in conversational abilities and a 57% reduction in unsafe behavior. The inte- gration of our reward model further amplifies these gains, highlighting the effectiveness of our alignment algorithm.
2. LIM-RLHF-Datasr
In this section, we outline the construction of MM-RLHF as illustrated in Figure 2. This includes data collection, data filtering, and human annotation.
Building on this high-quality reward model, we introduce Dynamic Reward Scaling within the Direct Preference Optimization (DPO) framework. Traditional DPO meth- ods (Amini et al., 2024) use a fixed training weight for all human preferred and preferrfe training pairs. In
2.1. Data Collection
Our goal is to construct a comprehensive post-training dataset that covers a wide range of task types. To achieve this, we categorize tasks into three main domains: image
2

---

MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
Score and Reason
10M Mudti-Modal Instruction Sources
100K Muti-Modal Instruction Data
Advanced MsLLMs Pool
Helpfulness:
[12.345]
Clustering De-duplicates Sampling
Faithfulness:
[1.23455]
Human
Ethical:
[1
Annotators
Roason: [Benefitsluncerns
for the model respouse I
LLAVA
Rank and Reason
Rank:
[1>2=304]
Tie:
Trae/Fulsee
Reason for the rank
Pos/Nog sumple tie
Figure 2. MM-RLHF Construction Pipeline. (1) Data Collection and Cleaning: Starting with 10 million instruction samples, we cluster data based on image similarity, and uniformly sample across diverse categories. This results in diverse dataset covering image- Q&A (e.g., multiple-choice, dialogues, and safety-related questions) and video Q&A formats. (2) Response Generation: We leverage state-of-the-art models, including GPT-4o and Qwen2-VL. VL-22B, to generate responses. (3) Human Annotation: We conduct comprehensive manual annotation across nine categories, including scoring, ranking, and textual explanations, ensuring fine-grained evaluation.
understanding, video understanding, and multimodal safety.
For image understanding, we integrate data from multiple sources, including VLVVA (Li et al., 2024c), VL.feed- back (Li et al., 2023d), LLaVA-RLHF (Sun et al., 2023b), Irv-instruction (Liu et al., 2023a), and Unimm-Chat (Yu et al., 2023). Since some datasets contain multi-turn dia- logues, which are less suitable for response generation, we decompose them into single-turn dialogues. This process yields over 10 million dialogue samples, covering tasks such as conversation, safety, multiple-choice questions, captions, and commonsense reasoning.
Geomebric
shapes
Society
lendscape
Intensive
Application
Scenario
For video understanding. the primary data source is SharedGPT-4 video (Chen al., 2024b).
Figure 3. Re-Sampling results from the clustering process. Due to the large total number of samples, the clustered and deduplicated results contain a rich diversity of categorics. Selected samples include topies such as mathematics, daily life, natural scenes, medicine, electronic technology, and OCR scenarios, showcasing variety of problem- iiaeeee---aeeeeeeee pairs. The 2D features are obtained via UMAP dimensionality reduction.
For safety, data is primarily derived from VLGuard (Zong al., 2024) and self-constructed content. VLGuard contains over 2,000 harmful samples, while additional red teaming, safety, and robustness data are included. The pipeline for constructing safety data is detailed in Appendix C.I.
2.2. Data Filtering and Model Response Generation
responses shorter than 128 characters. These require con- cise answers, often involving simple image analysis, and represent broader range of task types.
The core goal of data filtering is to reduce the number of sam- ples while maintaining the diversity of the original dataset. To achieve this, the following strategics are adopted:
The initial distribution of these three types in the image understanding dataset is highly imbalanced, with propor- tions of 17% (Long), 83.68% (Short), and 4.14% (MCQ). To align with diversity goals, we adjust the sampling ra- tio to 4:5:1, reducing disparities among task types while maintaining dominance of comprehensive samples.
Predefined Sampling Weights. For image understanding tasks, we define three categories based on the nature of the questions and the length of model responses: 1. Multiple- choice questions (MCQ): questions with options such A, B, C, or D. These tasks include visual question answer- ing, mathematics, OCR, and icon recognition, focusing on the model's reasoning and visual perception abilities. 2. Long-text questions: questions for which GPT-4o gener- ates responses exceeding 128 characters. These typically involve detailed captions complex descriptions, testing the model's conversational and descriptive capabilities. 3. Short-text questions: questions for which GPT-4o generates
Cluster-based Sampling. Text deduplication is not per- formed because many questions, while similar in text, are paired with different images, leading to substantially dif- ferent outcomes- an intrinsic characteristic of multimodal
For video understanding and safety tasks, MCQ samples are fewer. After classifying into Long and Short types, the differences are minimal, so no additional adjustments are made.
3

---

MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
harm avoidance, ensuring responses free from harmful or biased content. Annotators score each dimension while documenting the reasoning behind their assessments, adding valuable context for understanding model performance.
Table I. Dataset Composition Statistics
Image
Safety
Video
Total
Long
Short
MCQ
9575
12063
2125
1999
4235
29997
Second, annotators are required to assign an overall rank- ing to the responses, along with justifications for their rank- ings. This ranking mechanism provides a transparent and nuanced comparison of model outputs. Additionally, inno- vative strategies are employed to enhance data quality:
data. Instead, we encode all images using CLIP and for videos, we use the feature of the first frame as a representa- tive. We then apply KNN clustering with 100 cluster centers and randomly sample N instances from each cluster. The value of N is determined to satisfy the predefined sampling ratios, ensuring balanced representation of task diversity.
Constructing Positive Samples for Poor Quality Ties. When multiple responses are equally poor, annotators pro- vide correct answers to create positive examples. This en- sures challenging samples contribute to the training dataset, addressing issues that no valid model responses exist.
Data Statistics is summarized in Table 1, and the visualiza- tion of the clustering results is shown in Figure 3, demon- strating the rich diversity of data categories.
Constructing Negative Samples for High-Qualig, Ties. When multiple responses are of equally high quality, anno- tators introduce deliberate errors to create negative samples. This prevents ties from reducing the utility of the data and allows for more efficient use in training.
Model Response Generation. To generate high-quality responses, we select sfte-of-the-arr models from both open- source and closed-source domains. For image understanding and salaty-related tasks, we use Qwen2-VL-72B (Wang et al., 2024a), LLaVA-OV-72B (Li et al., 2024c), GPT-4o', and Claude 3.5-sonnet'. For video understanding tasks, we employ GPT-4o, LLaVA-Video-277 VVideo-777 (Zhang et al., 2024b), and Qwen2-VL-722 VL-72B (Wang et al., 2024a). These models are chosen for their advanced capabilities and performance, ensuring a comprehensive evaluation of leading solutions in multimodal understanding.
We employ over 50 annotators supervised by 8 multimodal experts ensure high-quality annotations, addressing the fine-grained requirements of MLLM alignment tasks. While human annotation incurs significant costs, it offers sub- stantial advantages over machine annotation, particularly in capturing nuanced perceptual differences and providing professional-grante reasoning. For a detailed comparison of human and machine annotation, see Appendix D.
2.3. Annotation
3. MMMRLLHF- Model
The annotation process follows rigorous standards to ensure comprehensive and fine-grained evaluations of MLLM re- sponses. Detailed standards are provided in Appendix B, and the scoring and annotation structure are illustrated in Figure 2. Additionally, we design a web UI to streamline the annotation process, as shown in Figure 6.
In this section, we explore how to train high-quality reward model using MM-RLHF, to provide robust supervision signal for subsequent model alignment. The reward model is designed to combine critique generation and scoring (Fig- ure 4), ensuring a comprehensive evaluation process.
Compared prior work, our annotation approach intro- duces two significant advantages: richness and granular- ity. First, the evaluation incorporates three core dimen- sions- Helpsueell,u,esss, Faithfulness, and Ethical Consider- ations-to comprehensively capture model performance. Helpfulness ensures that responses are relevant and provide meaningful assistance aligned with the user's intent. Faith- fulness evaluates the accuracy of responses in describing visual elements, such as objects, relationships, and attributes, ensuring alignment with the ground truth while avoiding hal- lucinated content. Ethical Considerations assess adherence to ethical principles, including safety, privacy, fairness, and
3.1. Background of Standard Reward Models
Reward models are a key component for aligning model outputs with human preferences. Typically, reward model starts with a pretrained L.LM Φ, where the LLM head by is replaced with linear reward head 1,,, enabling the model to output scalar reward. These models are trained using pairwise comparisons. Given a query x, a preferred response ywu and a less preferred response yr, the reward model is optimized to assign higher rewards to preferred responses:
&Reward (0) = Ex ym -10 - log or (r(Uw x) - -((ul))) (1)
el alip-vit-bae--pa-e "https://opent
where r(ylx) is the reward and a is the sigmoid function.
http:ppaaaaaaaaaaaaaaooaooooaeee homm/inden indee/ 1 opt-to/ http://www.tttrp.ccccoo conpnwss/ claude- 3- S-oneet
Standard reward models face significant limitations. First, they fail fully utilize the rich feedback provided by high-
4

---

MM-RLHF; The Next Step Forward in Multimodal LLM Alignment
Based on the map shown Calitoma appears have higner powerty rare Aarka Califonia zolored darker shacl purple corres corrsponding to the 76- 10.3% mage while Alacka is calored in the lighter: shade of pink, corresponding
Hicher Quality ar erswer, acourate description
The provided correct addressing ouestion powerty
Califomia Aladka desicted the grovided
Learning
desoriztion mad's oolor oocing schome whtch incicaees aovory nees across fiferennt stanes, By comcectly kland (ying Califonia being shuded in darler purple, corresponcing to the 8.7% -20 3% range. and Necka
accur attlly interp
Provide Critique
7.2% range.
The resporne se accumsiely deniliies that Calfooria has higher poverty race thor Alska ind cated the map. Hownee the deso prlan provided in the answer contains Specifically by map the powerty rates Accorcing the legerd, California within the shading
Yes, California hes a higher poverty rate than Alaska according to the Ca itomia shaded in the 10.1%-1/ 1/-4 range, while the 5. 24-7 range.
Quality
The answnr correct.
Doas Calitornia Nigher value than Alaska?
reperding the sheding
Learning Scoring
in
but descappiin
di lereert shradng
incarrem.
Querv
Model Response
Annotation
Humon
GPT-fo-Eakanced Annolations
Multi-Tack Reward
Modelong
Figure 4. Illustration of the multi-task reward model training process. The process begins with user query and corresponding model responses, which are ranked and annotated by humans. Human annotations are expanded using GPT-4o to provide enhanced rationales. The reward model is trained with two objectives: (1) Learning Provide Critigue, where the model learns to provide detailed critiques and evaluations for model responses, and (2) Learning Scoring, where the model learns to assign scores based on the model response and critique. The integration of these tasks ensures robust evaluation framework for improving model outputs.
quality human annotations, such as textual explanations and nuanced reasoning. Second, scalar rewards lack trans- pareney, making it difficult for humans understand how the reward is generated. These challenges highlight the need for more interpretable and robust reward model that leverages critiques as intermediate reasoning steps.
details and improving the fluency of the critiques. These en- hanced scoring reasons are then used the training targets for the language head. To prevent GPT-4o from introducing hallucinated content or irrelevant analysis, we impose strict constraints in the prompt (Table. 7), to ensure the model only expands on the original content without introducing speculative or uncertain information.
3.2. Critique-Based Reward Model Training
Scoring Loss with Teacher-Foreing. h, computes scalar rewards based the query x, response y, and critique c. During training, we adopt reacher-forcing strategy, where the scoring head uses ground truth critiques instead of cri- tiques generated by itself. This avoids potential noise from model -generaed critiques in the early stages of training. The scoring loss is defined as:
Extending to Critique-Based Training. To overcome the limitations of traditional reward models, we propose a critique-base training framework: the model first gener- ates a critique e conditioned on the query x. This critique serves as an intermediate reasoning step, providing context for scoring responses. The critique-based based reward model comprises two components: 1. Critique Head (f)): Gen- erates critiques Sw and C for the preferred (yw) and less preferred (gn) responses, respectively, based on the query x. 2. Scoring Head (h₂)): Assigns scalar rewards based on the generated critiques, enabling more fine-grained evaluation.
Escure = Ezz -log a (f(x,
(wx yws cw) cw) (0-r(a, yr, (i)
(3)
where: Cur and are the ground truth critiques for the responses you Ut, respectively, r(r, y, e) is the reward score computed from x, y, and c.
Learning to Provide Critique from Enhanced Annota- tion. The by is trained to align with human- providee anno- tations. The loss function for critique generation is:
Joint Training Objective. The overall training objective combines the critique generation loss and the scoring loss: E (0) fCrisque (0) E (f).
lcl
&Critiqu (0) F...y..
Σ
log (o (cet, v)
Inference. During inference, the critique head (h₂) gener- ates a critique e conditioned on the query and response y. The scoring head (h,) then uses xr y. and the generated critique e to compute the final reward score r(æ, y, c). This two-step process mirrors the human evaluation process by explicitly reasoning about critiques before scoring.
where et is the t-th token in the critique c, Ges denotes the tokens preceding et, and ma (celCce, x; y) is the probability of token e given its context, query æ, and response y.
However, shown in Figure 4, while suman--rovided scor- ing reasons are highly accurate, they tend to be concise. Di- rectly using these concise annotations as training targets for the reward model's 's language head does not yield significant performance improvements. To address this issue, we use GPT-4o to augment the human annotations by adding more
In Section E, we discuss the differences and connections be- tween the
Model and existing works.
To evaluate the effectiveness of the signals provided by our reward model in guiding sub- sequent model training, we randomly sample 10 examples
5

---

MM-RLHF; The Next Step Forward in Multimodal LLM Alignment
from each category of the MM-RLHF dataset create test set. Each example includes multiple model responses and their corresponding rankings, enabling the generation of several comparison pairs. This results in a total of 170 pairs for evaluation. We design two evaluation metrics: 1. Tradi- tional Accuracy (ACC): Measures the proportion of cases where the model correctly identifies the preferred response. 2. ACC+: Measures the proportion of cases where the model correctly ranks all response pairs for given sample. This metric emphasizes the model's ability to handle chal- lenging cases, such as those with small ranking differences or hard-to-distinguis pairs.
of uniform scaling and underutilizes the high-confidence information contained in larger reward margins. To address this, MM-DPO introduces Dynamic Reward Scaling, which dynamically adjusts the update strength based on the reward margin to prioritize high-confidence training pairs.
Definition of Dynamic Reward Scaling. Reward mod- els can naturally provide a pairwise reward margin, which serves as a straightforward signal for scaling. However, two critical aspects must be addressed: (I) ensuring the signal quality is sufficiently high, and (2) bounding the signal to prevent overly aggressive updates that destabilize training.
Regarding the first aspect, our experiments reveal that pub- licly available models, such as GPT-4o and LLaVA-Crrise perform inadequately in scoring our dataset. Conversely, our trained HIM-RLHF-Rewarde publicly available 72B models, offering a reliable and robust reward signal. We use this model to compute the reward margin: 8 r(y r r(yi), where r(yw) and r(w) are the scores assigned to the positive and negative samples.
4. MM-DPO
In this section, we propose MM-DPO, an extension of the traditional DPO framework. MM-DPO introduces Dynamic Reward Scaling, which dynamically adjusts the update strength based on the confidence of training pairs, ensuring effective utilization of high-quality samples while mitigating the impact of noisy or low-confidence data.
model surpasses several
For the second factor, we control the scaling factor (8()) using the following formulation: 3(6) = Pon [ + w wo(1
4.1. Background: Direct Preference Optimization
The DPO framework is preference-based learning method that optimizes model parameters 9 by aligning model outputs with human preferences. Given a query x and corresponding responses You (positive) and y (negative). the DPO loss is defined as: (D) = Ex.,w.
e kdd) where Bon is the initial default scaling factor, w is a parameter balancing the dynamic component's contribution, and l is hyperparameter that adjusts ((6)'s 's sensitivity to changes in 6. The function 1 - e kó is bounded between [0, 1J. and a smaller & value keeps most 3(0') values near Soni with slow growth as 5 increases. In contrast, a larger k makes (()) highly responsive to changes in 6, quickly reaching its maximum. To avoid aggressive updates, we constrain 3(8) within [Sori, + -.r)).) Overall, Dynamic Reward Scaling significantly enhances MM-DPO by lever- aging high-quality reward signals and tailoring optimization steps to the confidence level of training pairs. This results in improved robustness, efficiency, and effectiveness of the framework. We discuss the similarities and differences be- tween our approach and existing methods in Appendix F.
(==((x]))) }] where Ta is the optimal policy, Wref is the reference policy, / is a scaling factor, and ((() is the sigmoid function (Amini et al., 2024). Traditional DPO treats all training pairs equally, regardless of their quality differences. This uniform scaling fails to prioritize high-quality pairs with clear preference distine- tions, leading to inefficient use of informative samples and suboptimal optimization.
logo ( (8(1og ma(((> ix)
=o(ymllx) log xe[p((x)
4.2. MM-DPO: Key Contributions and Improvements
Training on all possible comparison pairs instead of the hardest pairs. Unlike many recent MLLM alignment ap- proaches that prioritize training on the hardest comparison pairs, MM-DPO incorporates all possible comparison pairs for a single query into the training process. Specifically, for any query with multiple responses, every response pair with differing ranks is treated as a valid comparison pair. This comprehensive approach captures more nuanced ranking information, allowing the model to learn from broader set of preferences. However, this strategy also introduces a challenge: pairs involving responses with similar ranks (e.g., rank 3 and rank 4) often have lower reward margins compared to pairs with more distinct rankings (e.g., rank and rank 4). Treating all pairs equally exacerbates the issue
5. Experiments
In this section, we evaluate our data and algorithms on 10 tasks across 20+ benchmarks. The key findings are:
1. Alignment training on the MM-RLHF dataset consis- tently improves performance across nearly all benchmarks for various baselines. The integration of reward signals in MM-DPO further amplifies these improvements, demon- strating the effectiveness of our approach.
2. The the-art performance on reward model benchmarks among open-source models, surpassing even several 72B models. This highlights the efficiency and scalability of our method.
-eL-----eard-7 model achieves state-of-
6

---

MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
LLaVA A-One Vition
EatemVLL
IntereLL2
NO 70 Ni
61.
011
$13
44.
9X12
141
215
213
Mali-lanne
Ka/Wolk
Figure Performance improvements achieved through alignment training. indicates use of our dataset and alignment algorithm.
Table Performance comparison across metrics and methods on HN-RL.F-RRewardeee
MM-RLA HOM-LLFF-Rewee (w/o. Task I) pair-wise samples while excluding Task 1. HEM-RLHF-ReL
represents training the LLaVA-OV-7B model annotations) involves learning human- human-provided provided annotations, followed by scoring. MM-RLHF AMM-RLL--Reward Reward (inference w. GT annotation) uses
(wlo. enhanced
ground truth annotations during inference.
MM-RLIF-RRewe RIP-Re
MM-RLIIF-Rewand
LLaVA-OV-7B
LlaVi-Critic (Pointwise)
LlaVA-Critic (Pairwise)
GPT-4o
HIM-R.HF-Rewan
(w/o. Task 1)
(w/o. onhanced
MM-RL-HF-Rewe
(inference GT annotation)
annolations)
Metric
ACC
ACC+
ACC
ACC+
ACC
ACC+
ACC
ACC+
ACC
ACC+
ACC
ACC+
ACC
ACC+
ACC
ACC+
Mog
0.14
0.00
0.38
0.10
0.23
0.00
0.69
0.20
0.90
0.80
0.83
0.70
0.70
1.00
1.00
Long
0.11
0.00
0.49
0.20
0.54
0.30
0.95
0.90
0.70
0.40
0.92
0.80
1.00
1.00
1.00
1.00
Short
0.79
Safety
0.41
0.00
0.62
0.17
0.28
0.17
0.72
0.33
0.69
0.33
0.69
0.17
0.66
0.17
0.69
0.17
Video
0.32
0.10
0.40
0.20
0.52
0.20
0.80
0.60
0.70
0.60
0.80
0.60
0.92
0.80
0.92
0.90
Overall
0.24
0.07
0.45
0.17
0.35
0.15
0.74
0.50
0.75
0.30
0.79
0.57
0.85
0.67
0.93
0.87
3. We conduct extensive ablation studies and analyses, such as investigating the importance of critique learning for re- ward models and the sensitivity to hyperparameters. Addi- tionally, we identify several experimental phenomena that challenge mainstream perspectives, such as the observation that small-scale MIL.Ms struggle to perform effective self- improvement. Due to space constraints, experimental setups and additional findings are provided in Appendix G.
Broad enhancements in hallucination, mathematical rea- soning, multi-image, and video understanding. The aligned models also exhibit notable improvements in these areas. Interestingly, despite the lack of dedicated multi- image data our dataset, the model's 's performance in multi- image tasks improves significantly. This indicates that the di- versity of alignment data enhances generalization across multiple dimensions.
Model-specific preferences for data and hyperparam- eter selection. Different models exhibit varying perfor- mance trends during alignment, with distinct preferences for hyperparameter settings across different benchmarks. For instance, in training of Intern VL-1B, found that excluding the SFT loss led to better results. Additionally, while Intern VL-1B demonstrates significant improvements in general knowledge tasks, its relative enhancement in OCR tasks less pronounced compared to the LLaVA-OV se- ries. These differences largely stem from variations the models' pretraining datasets and strategies, necessitating tailored hyperparameter adjustments for optimal alignment.
5.1. Evaluation of MM-RLHF and MM-DPO
Figures and 5 illustrate the aligmment performance of LLaVA-OV-7B and VLternVL-1B using our dataset and alignment algorithm, where the scores for each evaluation dimension are averaged across their respective benchmarks. We also conduct experiments on VLAVA-OV- with de- tailed benchmark results provided in Appendix Table 8 (for understanding tasks) and Table 9 (for safety tasks).
Significant improvements in conversational ability and safety. The alignment process of LLaVA-OV-7B leads to substantial improvements in these two aspects without re- quiring hyperparameter tuning. The average improvement in conversational benchmarks exceeds 15%, while unsafe behaviors are reduced by 65%. Additionally, in Visdon, the win rate increases by 144%. This suggests that existing MLLMs lack explicit optimization for these dimensions, and our dataset effectively fills this gap.
Limited gains in high-resolution benchmarks. The model shows no significant improvement on high-resolution bench- marks, likely because our dataset contains relatively few ultra-high www-hiihh-resoutoonn images. Additionally, our filtering strategy is based on image similarity rather than resolution, meaning the alignment process does not explicitly optimize for high-resolution tasks. As result, performance gains in
7

---

MM-RLHF; The Next Step Forward in Multimodal LLM Alignment
this area remain limited.
Table 3. Performance comparison our reward model (MM-RLHF-A) -Rearddd with existing open-source and private MLLMs. MM-RLHF-Reww
Ablation studies and sensitivity analysis. To further vali- date the effectiveness of our approach, we provide detailed ablation studies in the appendix, analyzing the impact of different alignment parameters and the improvements intro- duced by our dataset and MM-DPO.
outperforms existing 72B open-
source MLLMs and several competitive closed-source models.
General
Hullucination
Reasoning
Avg
VITA- (Fu 2025)
18.55
8.93
16.48
SIME-88 (Zhagg et al., 2024c)
7:25
27.89
18.5
19.04
deepseck-v12 v12 (Wu al. 202480
29.70
21.80
50,90
34.30
Phi 3,5 vision instuct (Abdin al. 2024]
28.00
22.40
56.60
35.67
5.2. Evaluation of AM-RLHF-RReand
lava onevision quen2
2024c)
$2.20
20.10
36.47
Molmo-7B 0-0924 (Deitic al. 2014)
$1.10
31.80
56.20
39.70
Pistral 12B 2009 (Agrawal et 2024)
35,60
25.90
59,90
40.47
In this section, we evaluate the effectiveness of MM-RLHF- Reward and highlight several noteworthy experimental ob-
Quen2 Ooeen--VL-7200 VL-12B Instruct (Wang al.. 2024g
38,10
32,80
55.00
42.97
NVLM-DD-728
2014)
38,90
IrienV 2-268 (Chon al., 20144)
36.90
00.80
45.67
servations. The results are presented in Table and Table
Premnee anddel
GPT-4o-min (2024-07-18)
41.70
$4,50
58.20
44.50
Cladd- 5-Someet (2022-066-22)
59.57
GPT--4 (2024-08-869)
49.10
67.60
Existing reward models exhibit significant overfitting, As shown in Table 2, I.LVV-Critic's performance on is suboptimal, with a consid- erable gap compared to GPT-4o. This can likely be at- tributed the overfitting of existing reward models to their training data, which predominantly consists of conversa- tional datasets and real-world images. Consequently, while LLaVA-Critic demonstrates notable improvements over its baseline, LLaVA-OV-7B', its performance in other cate- gories, such as MCQ, remains limited.
Gemini- 1.5-Pro (2024-09-24 24)
50,80
72:50
64.20
62.50
Ours
MM-RI HF-Rowan--78
45.04
90.45
57.55
50.15
M.M-LLHF-Rewardeene
due to their brevity conversational style. To address this, we enrich human annotations using GPT-4o, significantly enhancing reward model training quality and achieving a 17% improvement in ACC+ over the baseline. Notably, when human annotations are directly used as critics during evaluation (i.e., scoring is based on human-provided evalua- tions rather than
critiques), both ACC and ACC+ reach approximately 90%, underscoring the pivotal role of critique quality in reward model effectiveness.
Closed-source models consistently deliver competitive performance. Across both Table 2 and Table 3, closed- source models such as GPT-4o demonstrate superior gener- alization capabilities compared to open-source alternatives, even those with significantly larger parameter sizes. This observation underscores the robustness of closed-source approaches in handling diverse multimodal tasks and main- taining high performance across various metrics.
Multiple sampling of critiques does not yield perfor- mance gains. While prior research in LLMs suggests that sampling multiple critiques and averaging their scores can improve performance (Yu et al., 2024d), our experiments show that this approach leads to performance degradation when applied to our model. This is because lowering the sampling temperature occasionally produces inaccurate cri- tiques. Since our model, aligned with human annotations, already generates reasonably accurate critiques, the addi- tional sampling process is not only time-consuming but also counterproductive, negatively impacting performance.
A-w-aLHF-Reward establishes a new standard for open- source models. In both benchmarks, HIM-LLFF-eewall achieves results comparable to or exceeding GPT-40, while significantly outperforming most open-source models, such as Qwen2-VI Instruct. Notably, our custom bench- mark, MM-RLLFFRReward demonstrates substantial lead over GPT-4o, further justifying its selection as the reward signal for training algorithms. Its robust performance across diverse metrics highlights its effectiveness and adaptability.
Table 4. Comparison of DPO baselines, where all models are
trained on the MM-RL HF training datasct.
Method
LLaVA-Wild
OCRBench
DocYQA
MathVista
MMHla
LLaVA-O--78
90.70
62.30
84,34
60.10
3.22
DPO
95,20
66.30
85.72
60.90
3.88
Importance of effective crities in reward modeling. The results in Table 2 highlight the critical role of effective crit- ics in reward modeling. Training the reward head directly on pairwise datasets yields an ACC+ of around 50%. By incor- porating human annotations as learning eargets-enabling the model to first learn evaluation reasoning before scor- ing-ACC+ consistently improves by 5%. However, human annotations alone may not suffice as optimal training targets
beta-DPO
94.40
66.50
85.44
60.00
3.44
SimPO
93.00
64,50
84,50
59,10
3.30
LLaVA-Critc
97.90
64,20
84.31
58.74
3.95
MIA-DPO
95.00
66.50
85.60
60.00
3.80
MPO
95.30
66.20
85.88
61.10
3.72
MM-DPO
97.90
69.30
61.60
4.08
5.3. Comparison of DPO Baselines and Other Alignment Datasets
Both models use identical prompts for tasks such captioning and long-form dialogue.
We compared several baselines including DPO (Amini et al., 2024), LLaVA-Critie (Xiong et al., 2024), beta-DPO (Wu
8

---

MM-RLHF; The Next Step Forward in Multimodal LLM Alignment
in OCR, mathematical reasoning, and other domains. More- over, due to the quality limitations of existing datasets, its improvement in dialogue and hallucination is far from MM- RLHF. The latest comparison with MPO focuses more on mathematical reasoning. Most of the data is related to math- ematical reasoning, so it slightly outperforms MM-RLHF in this aspect but falls far behind in terms of diversity.
Table Comparison of preference datasets, where all models
are trained using the DPO algorithm.
Dataset
LLaVA-Wild
OCRBench
DocVOA
MathVista
MMHaa
No Aligument 90.70
02.30
84.34
60.10
3.22
LLava-LLHF
94,20
61.10
84,00
58.20
3.38
VL Feedbuck
93.60
64,10
84.72
58.70
3,03
RLAIF
92.30
64.00
84.44
60.00
3.18
LLaVA-Critic
94.70
62,50
83,40
59.50
3,44
MPO-Data
93.20
60.80
82.30
97,90
69.30
86.11
61.60
4,08
6. Conclusion and Future Work
In this work, we introduce MM-RLHF, a high-quality, fine- grained dataset specifically designed to advance the align- ment of MLLMs. Unlike prior works that focus on specific tasks, our dataset and alignment approach aim to holisti- cally improve performance across diverse dimensions. With preliminary improvements to reward modeling and optimiza- tion algorithms, we observe significant and consistent gains across almost all evaluation benchmarks, underscoring the potential of comprehensive alignment strategies.
et al., 2024a), SIMPO (Meng et al., 2024), MIA-DPO (Liu et al., 2024b), MPO (Wang al., 2024b), and others. All methods underwent grid search to select the best results, as shown in Table 4.
First, the beta-DPO loss function for pure text domains is somewhat similar to our idea, as it adjusts the 3 parameter for samples. However, directly applying to the MM- RLHF dataset does not yield good results. SIMPO is highly sensitive to hyperparameters. Despite our extensive tuning. we did not observe significant improvements in performance, and it even became detrimental.
Looking ahead, we see great opportunities to further unlock the value of our dataset. Its rich annotation granularity, such as per-dimension scores and ranking rationales, remains underutilized in current alignment algorithms. Future work will focus on leveraging this granularity with advanced op- timization techniques, integrating high-resolution data to address limitations in specific benchmarks, and scaling the dataset efficiently using semi-automated strategies. We be- lieve these efforts will not only push MLLM alignment to new heights but also set foundation for broader, more generalizable multimodal learning frameworks.
LLaVA-Critie is essentially a multi-stage DPO. We selected a shree-stage DPO similar to the one in the original paper, constantly generating results with the base model, filtering with our own reward model, and forming new DPO pairs. After three stages of DPO, the model showed some improve- ment on simpler benchmarks like VL- and MMHal. However, performance dropped on Math Vista and OCR- Bench. In fact, we also analyzed this phenomenon in Figure 12 of the original paper. For smaller models, the quality of self-generated responses is limited by the model's 's capability, so for challenging tasks, it is likely unable sample the correct results, leading to negative optimization.
Impact Statement
This work advances the alignment of MLLM with human preferences, aiming to enhance their reliability, safety, and performance across diverse applications. By introducing novel methods for human preference alignment, our re- search contributes to the ethical development of MLLMs, ensuring they better align with human values and societal norms. The societal implications of this work are signif- icant. Improved aligmment can foster greater trust in AI systems by reducing risks associated with unsafe, biased, or unreliable outputs. These advancements pave the way for the responsible adoption of AI technologies, enabling their broader and more beneficial integration into society.
Next, MIA-DPO was designed for multi-image tasks and performed poorly on mnglleimmae tasks, which is consis- tent with the observations in the original paper. Finally, there is the recent work MPO, which combines DPO with SFT loss and adds Quality Loss. This loss intuitively helps the model understand the absolute quality of individ- ual responses, contributing somewhat to performance but not as significantly as the performance gains from directly adjusting/ / in MM-DPO.
We also compared the results of training with DPO across multiple datasets, which is shown in Table 5. The com- pared datasets include LLava-RLHE (Sun et al., 2023b), VL Feedback (Li et al., 2024e), RLAIF (Yu al., 2024b), LLaVA-Critic (Xiong et al., 2024), and MPO-Data (Wang et al., 2024b). For LLaVA-Critie data, we constructed DPO pair for training by scoring each instruction and asso- ciating it with different responses according to annotations. Its data scale is large, but most of the data comes from exist- ing datasets, so it does not show significant improvements
Acknowledgement
Sponsored by National Key R&D Program of China (2022ZD01177) 11901), CCF-Kuaishou Large Model Explorer Fund (NO. CCF- KCa-KuuiShou 2024014) and the National Nat- ural Science Foundation of China (Grant No. 62373355, 62236010). Supported by National Natural Science Founda-

---

MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
tion of China under Grant No. 62441234. Supported by the Al & AI for Science Project of Nanjing University under
Chen, X., Zhao, Z.., Chen, L.. Zhang, D., Ji, J., Luo, A., Xiong, Y., and Yu, K. Websre: A dataset for web- based structural reading comprehension. arXiv preprint 1rXiv:2101.0946: 2021.
Grant No. 2024300529.
References
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing. S., Zhong. M., Zhang, Q., Zhu, X., Lu, L.,, Li, B., Luo, P., Lu, T., Qiao, Y., and Dai, J. Internll: Scaling up vision foundation models and aligning for generic visual- linguistic tasks. arXiv preprint arXiv:2312.142555 2023.
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A.. Bao, J., Behl, H., et al. Phi-3 technical report: A highly capable lan- guage model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z.,, Cui, E., Tong, W., Hu, K.,, Luo, J., Ma, Z., et al. How far we to gpt-4v? closing the gap to commercial multi- modal models with open-source suites. arXiv preprint arXiv:2404. 1682/, 2024c
Agrawal, P., Antoniak, S., Hanna, E. B., Bout, B., Chap- lot, D., Chudnovsky, J., Costa, D., De Monicault, B., Garg, S., Gervet, T., et al. Pixtral 12b. arXiv preprint arXiv:2410.0707. 2024.
Amini, A., Vieira, T., and Cotterell, R. Direct pref- erence optimization with offset. arXiv preprint arXiv:2402.10571, 2024.
Chiang, W.-L., Li, Z., Lin, Z.,, Sheng, Y., Wu, Z., Zhang, H., Zheng, I.., Zhuang, S., Zhuang, Y., Gonzalez, J. E., et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicua. Imsys. org (accessed 14 April 2023), 2023.
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., and Parikh, D. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433. 2015.
Chowdhury, S. R.,, Kini, A., and Natarajan, N. Provably ro- bust dpo: Aligning language models with noisy feedback. arXiv preprint Sitv:2433.00m00 2024.
J., Bai, S., Yang, S., Wang. S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A frontier large vision- language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023a.
Dai, W., Lee, N., Wang, B., Yang. Z., Liu, Z., Barker, J., Rintamaki, T., Shoeybi, M., Catanzaro, B., and Ping, W. Nvlm: Open frontier-class multimodal llms. arXiv preprint arXiv:2409. 1402, 2024.
Bai, S., Yang, S., Bai, J., Wang, P., Zhang, X., Lin, J., Wang. X., Zhou, C., and Zhou, J. Touchstone: Evaluating vision- language models by language models. arXiv preprint arXiv:2308. 16890, 2023b.
Deitke, M., Clark, C., Lee, S., Tripathi, R., Yang, Y., Park, J. S., Salchi, M.. Muennighoff, N., Lo, K., Soldaini, L., et al. Molmo and pixmo: Open weights and open data for of -fate-of-thh-an art multimodal models. arXiv preprint a4X9::2409.1 /7 146, 2024.
Bitton, Y., Bansal, H., Hessel, J., Shao, R., Zhu, W., Awadalla, A., Gardner, J., Taori, R., and Schimdt, L. Visit-bench: A benchmark for vision-language instruc- tion following inspired by real-world use. arXiv preprint arXiv:2308. 06595, 2023.
Duan, H., Yang, J., Qiao, Y., Fang, X., Chen, L.,, Liu, Y., Dong, X., Zang, Y., Zhang, P., Wang, J., et al. Vimevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM international conference on multimedia, pp. 11198 11201, 2024.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., al. Language models are few-shot learners. NeurlPS, 2020.
Chen, I., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, 7 Duan, H., Wang, J.. Qiao, Y., Lin, D., et al. Are we on the right way for evaluating large vision- langugge models? arXiv preprint arXiv:2400 2403:2403.20000 203.20 2024a.
Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., et al. Mme: A comprehen- sive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.1 13394, 2023.
Chen, L.., Wei, X., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Lin, B., Tang. Z., et al. Sharegpt4video: Improving video understand- ing and generation with better captions. arXiv preprint arXiv:2406.0m 04325, 2024b.
Fu, C., Dai, Y., Luo, Y., Li, L.,, Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv:2405.21075 2024a.
10

---

MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
Fu, C., Lin, H., Long, Z., Shen, Y., Zhao, M., Zhang. Y., Dong, S., Wang, X., Yin, D., Ma, L.,, et al. Vita: Towards open-source interactive omni multimodal Ilm. arXiv preprint arXiv:2408.05211 05221, 2024b.
Li, B., Ge, Y., Chen, Y., Ge. Y., Zhang, R., and Shan, Y. Seed-hench-2-plus, Benchmarking multimodal large lan- guage models with text-rich visual comprchension. arXiv preprint arXiv:2404. 16790, 2024a.
Fu. C.,, Zhang, Y.-F., Yin. S., Li, B., Fang, X., Zhao, S., Duan, H., Sun, X., Liu, Z., Wang, L.,, et al. Mme-survey: A comprehensive survey on evaluation of multimodal Ilms. arXiv preprint arXiv:2411.15290 2024c.
Li, B., Zhang, K., Zhang, H., Guo, D., Zhang, R., Li, F., Zhang, Y., Liu, Z., and Li, C. Llava- next: Stronger llms supercharge multimodal capabilities in the wild, May 2024b. URL (ltps://llava-vLl wtt.a/wwww.ttaa..../.ooo.ggo 011-005-000 10- wlaaalaaaeeeeeeeeeeeeeeeeeeeeeee serat-sttre stronger 11ms/.
Fu, C., Lin, H., Wang, X., Zhang, Y.-F., Shen, Y., Liu, X., Li, Y., Long, Z., Gao, H., Li, K., et al. Vita-1.5: Towards gpt- 4o level real-time vision and speech interaction. arXiv preprint arXiv:2501.00 Sl1:2500.00157, 2025.
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al. Llava- onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 2024c.
Fu, X., Hu, Y., Li, B., Feng, Y., Wang, H., Lin, X., Roth, D., Smith, N. A., Ma, W.-C., and Krishna, Blink: Mul- timodal large language models can see but not perceive. arXiv preprint ar Stti::2444.239900 2024d.
Li, L.,, Xie, Z., Li, M., Chen, S., Wang, P., Chen, L.,, Yang, Y., Wang, B., and Kong, L. Silkie: Preference distilla- tion for large visual language models. arXiv preprint arXiv:2312. 10665, 2023d.
Han, X., You, Q. Liu, Y., Chen, W., Zheng, H., Mrini, K., Lin, X., Wang. Y., Zhai, B., Yuan, J., Wang, H., and Yang, H. Infimm-eval: Complex open-ended reasoning evaluation for multi-moda large language models, 2023.
Li, L., Wei, Y., Xie, Z., Yang, X., Song, Y. Wang, P., An, C., Liu, T., Li, S., Lin, B. Y.. et al. Vlrewardbench: challenging benchmark for vision-language generative reward models. arXiv preprint arXiv:2411.1745/1 2024d.
Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z.,, Fang, Y., Huang, Y., Zhao, W., et al. Minicpm: Unveiling the potential of small language models with scalable train- ing strategies. arXiv preprint XXXv:2404.066355, 2024a.
Li, Xie, Z., Li, M., Chen, S., Wang, P., Chen, L.., Yang, Y., Wang. B.. Kong, L., and Liu, Q. Vlfeedback: A large- scale ai feedhack dataset for large vision-language models alignment. arXiv preprint arXiv:2410.09421. 2024e.
Hu, X., Liu, D., Li, H., Huang, X., and Shao, J. VIsbench: Unveiling visual leakage in multimodal safety. arXiv preprint arXiv:2411. 19939, 2024b.
Li, M., Li, L., Yin, Y., Ahmed, M., Liu, Z., and Liu, Q. Red teaming visual language models. arXiv preprint arXiv:2401. 12915, 2024f.
Jiang, A. Q.. Sablayrolles, A., Mensch, A.. Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint vrXiv:2310.06822 2023.
Li, Y., Du Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.-R. Evaluating object hallucination in large vision- language models. arXiv preprint arXiv:2305.10355, 2023e.
Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., and Farhadi, A. A diagram is worth dozen images. In ECCV, 2016.
Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.R.. Evaluating object hallucination in large vision- language models. In EMNLP, 2023f.
Li, B., Ge, Y., Ge, Y., Wang, G., Wang, R., Zhang, R., and Shan, Y. Seed-bench-2; Benchmarking multimodal large language models. arXiv preprint arXiv:2311.17099 2023a.
Li, Y., Zhang, Y., Wang, C., Zhong, Z., Chen, Y., Chu, R., Liu, S., and Jia, J. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403. 18814, 2024g.
Li, B., Wang, R., Wang, G., Ge, Y., Ge, Y., and Shan, Y. Seed-bench: Benchmarking multimodal Ilms with gener- ative comprehension. arXiv preprint arXiv:2307.16122 2023b.
Liu, F., Lin, K., Li, L., Wang. J., Yacoob, Y., and Wang, L. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations, 2023a.
Li, B., Zhang, Y., Chen, L., Wang, J., Yang, J., and Liu, Z. Otter: multi-modaa model with in-context instruction tuning. arXiv preprint arXiv:2305.0 033266 2023c.
Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. arXiv preprint arXiv:2304 08485, 2023b.

---

MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06288 2023c.
OpenAI. Gpt-4 technical report. 2023.
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In CVPR, 2019.
Liu, Y., Li, Z.,. Yang. B., Li, C., Yin, X., lin Liu, C., Jin, L.,, and Bai, X. On the hidden mystery of ocr in large multimodal models, 2024a.
Sun, Z.., Shen, S., Cao, S., Liu, H., Li, C., Shen, Y., Gan, C., Gui, L.-Y., Wang, Y.-X., Yang, Y., et al. Aligning large multimodal models with factually augmented rlhf. arXiv:2309. 14525, 2023a.
Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730- 3738, 2015.
Sun, 7., Shen, S., Cao, S., Liu, H., Li, C.,, Shen, Y. Gan, C., Gui, L.-Y., Wang, Y.-X., Yang, Y., et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309. /4525, 2023b.
Liu, Z., Zang, Y., Dong, X., Zhang, P., Cao, Y., Duan, H., He, C., Xiong, Y., Lin, D., and Wang. J. Mia- dpo: Multi-image augmented direct preference optimiza- tion for large dision-languaa models. arXiv preprint arXiv:24100 10. 17637, 2024b.
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction follloinn llama model, 2023.
Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., Yang, J., Yang, S., Iyer, A., Pan, X., et al. Cambrian-1: A fully open, vision centric exploration of multimodal llms. arXiv preprint arXiv:2406. 16860, 2024.
Lu, H., Liu, W., Zhang. B., Wang, B., Dong, K., Liu, B., Sun, J., Ren, T., Li, Z., Sun, Y., et al. Deepseck-vl: towards real-world vision-language understanding. arXiv preprint ar 2xXiv22433.05525 2024a.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F,, et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint
Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, Math- vista: Evaluating mathematical reasoning of foundation models in visual contexts. In ICLR, 2024b.
2023a.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
Lu, Y., Jiang, D., Chen, W. Wang, W. Y., Choi, Y., and Lin, B. Y. Wildvision: Evaluating rision-languse mod- els in the wild with human preferences. arXiv preprint arXiv:2406. 10699, 2024c.
Maaz, M., Rasheed, H., Khan, S., and Khan, S. Video- chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024.
Wang. P., Bai. S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, Ge, W., et al. Qwen2-vl: Enhancing liniun-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024a.
Wang. W., Chen, Z., Wang. W., Cao, Y., Liu, Y., Gao, Z., Zhu, J., Zhu, X., Lu, L., Qiao, Y., et al. Enhancing the reasoning ability of multimodal large language mod- els via mixed preference optimization. arXiv preprint arXiv:2411.1044 10442, 2024b.
Masry, A., Long, D. X., Tan, J. Q., Joty, S., and Hoque. E. Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning. 22Ki:22203.12244, 2022.
Mathew, M., Karatzas, D., and Jawahar, C. Docvqa: A dataset for vqa on document images. In WACV, 2021.
Wu, J., Xie, Y., Yang, Z., Wu, J., Gao, J., Ding, B., Wang, X., and He, X. beta-dpo: Direct preference optimization with dynamic beto. arXiv preprint arXiv:2407.08639 2024a.
Mathew, M., Bagal, V., Tito, R., Karatzas, D., Valveny, E., and Jawahar, C. Infographicvqa. In Proceedings of the IEFF/CVF Winter Conference on Applications of Computer Vision, 2022.
Wu, Z., Chen, X., Pan, Z., Liu, X., Liu, W., Dai, D., Gao, H., Ma, Y., Wu, C., Wang, B., Xie, Z., Wu, Y., Hu, K., Wang, J., Sun, Y., Li, Y., Piao, Y., Guan, K., Liu, A., Xie, X., You, Y., Dong, K., Yu, X., Zhang, H., Zhao, L., Wang. Y., and Ruan, C. Deepseek-v12. Mixture-of- experts vision- language models for advanced multimodal
Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference- free reward. Advances in Neural Information Processing Systems, 37:124198- 124235, 2024.
12

---

MM-RLHF; The Next Step Forward in Multimodal LLM Alignment
understanding, 2024b. URL htt.httt////xiv..ooro
Yue, X., Zheng, T., Ni, Y., Wang, Y., Zhang, K., Tong, S., Sun, Y., Yu, B., Zhang, G., Sun, H., al. Mmmu-pro: A more robust multi- multi-discipline multimodal understanding benchmark. arXiv preprint 200.022400.02813, 2024.
abs/2412. 10302.
Xiong, T., Wang, X., Guo, D., Ye, Q., Fan, H., Gu, Q., Huang, H., and Li, C. Llava-critie: Learning to evaluate multimodal models. arXiv preprint arXiv:2410.02712, 2024.
Zhang, J., Ye, J., Ma, X., Li, Y., Yang, Y., Sang, J., and Yeung, D.-Y. Anyattack: Self-supervised generation of targeted adversarial attacks for vision-language models. a.
Yan, Y. Wang, S., Huo, J., Li, H., Li, B., Su, J., Gao, X., Zhang, Y.-F, Xu, T., Chu, Z., et al. Errorradar: Bench- marking complex mathematical reasoning of multimodal large language models via enror detection. arXiv preprint arXiv:2410.0453 04500, 2024.
Zhang, P., Wang, X. D. B., Cao, Y., Xu, C., Ouyang, L., Zhao, Z., Ding, S., Zhang, S., Duan, H., Yan, H., et al. ditemlm-xxcoposee A vision-language large model for advanced text-image comprehension and composition. arXiv preprint vrXiv22309.51121 2023.
Ye, Q. Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang. J., Hu, A., Shi, P., Shi, Y., et al. mplug-owl: Modulariza- tion empowers large language models with multimodality. arXiv preprint arXiv:2304. 11044.141174 14178, 2023.
Zhang, R., Jiang, D., Zhang, Y., Lin, H., Guo, Z., Qiu, P., Zhou, A., Lu, P., Chang, K.-W,, Gao, P., et al. Mathverse: Does your multi-modal IIm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403. /4624, 2024a.
Ying, K., Meng, F., Wang. J., Li, Z., Lin, H., Yang, Y., Zhang, H., Zhang, W., Lin, Y.. Liu, S., Lei, J., Lu, Q., Chen, R., Xu, P., Zhang, R., Zhang, H., Gao, P., Wang. Y., Qiao, Y., Luo, P., Zhang, K., and Shao, W. Mmt- bench: A comprehensive multimodal benchmark for eval- uating large lssiunlanguuae models towards multitask agi, 2024.
Zhang, Y., Huang, Y., Sun, Y., Liu, C., Zhao, Z., Fang, Z., Wang, Y., Chen, H., Yang. X., Wei, X., et al. Multi- trust: A comprehensive benchmark towards trustworthy multimodal large language models. In The Thirty-eight Conference Neural Information Processing Systems Datasels and Benchmarks Track, b.
Yu, T., Hu, J., Yao, Y., Zhang, H., Zhao, Y., Wang, C., Wang, S., Pan, Y., Xue, J., Li, D., et al. Reformulat- ing vision-language foundation models and datasets to- wards universal multimodal assistants. arXiv preprint arXiv:2310.00653 2023.
Zhang, Y., Wu, J., Li, W., Li, B., Ma, Z., Liu, Z., and Li, C. Video instruction tuning with synthetic data, arXiv preprint arXiv:2410.02713. 2024b.
Yu, T., Yao, Y., Zhang, H., He, T., Han, Y., Cui, G., Hu, J.. Liu, Z., Zheng, H.-T., Sun, M., et al. Rlhf-v: To- wards trustworthy milms via behavior alignment from fine- tne-grained correctional human feedback. In Proceed- ings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13807-13816, 2024a.
Zhang, Y.-F., Wen, Q. Fu, C., Wang. X., Zhang, Z. Wang, L., and Jin, R. Beyond llava-hd: Diving into high-resolution large multimodal models. arXiv preprint arXi::2406.08487 2024c.
Zhang, Y.-F, Yu, W., Wen, Q., Wang, X., Zhang, Z., Wang, L., Jin, R., and Tan, T. Debiasing large visual language models. arXiv preprint arXiv:2403.0552 2024d.
Yu, T., Zhang, H., Yao, Y., Dang, Y., Chen, D., Lu, X., Cui, G., He, T., Liu, Z., Chua, T.-S., et al. Rlaif-v: Aligning milms through open-source ai feedback for super gpt- 4v trustworthiness. arXiv preprint arXiv:2405.17220. 2024b.
Zhang, Y--F., Zhang, H., Tian, H., Fu, C., Zhang, S., Wu. J., Li, F., Wang, K., Wen, Q., Zhang, Z., al. Mme- realworkd: Could your multimodal I1m challenge high- resolution real-world scenarios that are difficult for hu- mans? arXiv preprint arXiv:2408. 11088.13255 13257, 2024c
Yu, T., Zhang, Y.FF, C., Wu, J., Lu, Wang, K., Lu, X., Shen, Y., Zhang, G., Song, D., et al. Aligning multimodal I1 with human preference: survey, arXiv preprint a5Xiv:2503.1 14504, 2025.
Zhang, Y.-F., Lu, X., Hu, X., Fu, C., Wen, B., Zhang, T., Liu, C., Jiang, K., Chen, K., Tang, K., et al. R R1-reward: Training multimodal reward model through stable rein- forcement learning. arXiv preprint arXiv:2505.0288. 2025.
Yu, W., Yang, 7.,, Li, L., Wang, J., Lin, K., Liu, 7.,, Wang, X., and Wang, L. Mm-vet: Evaluating large multimodal models for integrated capabilities. In /CML, 2024c.
Yu, Y., Chen, Z., Zhang, A., Tan, L., Zhu, C., Pang, R. Y., Qian, Y., Wang, X., Gururangan, S., Zhang, C., et al. Self- generated critiques boost reward modeling for language models. arXiv preprint 1.Xiv:2411. 1.111116166 2024d.
Zong Y., Bohdal, O., Yu, T., Yang, Y., and Hospedales, T. Safety fine-tuning at (almost) no cost: A base- line for vision large language models. arXiv preprint arXiv:2402.02207, 2024.
13

---

MM-RLHF; The Next Step Forward in Multimodal LLM Alignment
MM-RLHF
Appendix
A. Related Work
Multimodal large language models have seen remarkable progress in recent years, with significant advancements in both performance and capabilities. Leveraging cutting-edge LLMs such as GPTs (OpenAL., 2023; Brown et al., 2020), ILaMA (Touvron et al., 2023a;b), Alpaca (Taori et al., 2023), Vicuna (Chiang et al., 2023), and Mistral (Jiang et al., 2023), MLLMS are increasingly demonstrating enhanced multimodal capabilities, especially through end-to-end training approaches. These advancements have been crucial in enabling models to handle range of multimodal tasks, including image-text
reasoning. and instruction following, while addressing challenges related to data fusion across different modalities. Recent open-source MLLMs such as Otter (Li et al., 2023c), mPLUG-OW/ (Ye et al., 2023), LLaVA (Liu et al., 2023b), Qwen- VL. (Bai et al., 2023a), Cambrian-1 (Tong et al., 2024), Mini-Gemini (Li et al., 2024g), MiniCPM-V 2.5 (Hu et al., 2024a), DeepSeek- VL (Lu et al., 2024a), SliME (Zhang et al., 2024c) and VITA (Fu et al., 2024b; 2025) have contributed to solving some of the most fundamental multimodal problems, such as improving Assoonlanguuae alignment, reasoning, and following instructions. These models focus on enhancing multimodal understanding by integrating vision with language, allowing for more nuanced and context-aware interactions. Some of the most notable open-source models, such as InternLM-XX XXMMMMMM-XXmmm--22..... (Zhang et al., 2023) and Intern (Chen et al., 2023), have exhibited impressive progress in multimodal understanding, closely competing with proprietary models across range of multimodal benchmarks. However, despite these achievements, there is still a noticeable gap in security and alignment when compared to closed-source models. As highlighted by recent studies (Zhang et al., 2024e), most open-source MLLMs have not undergone rigorous, professional alignment processes, which has hindered their ability to effectively align with human preferences. This gap in alignment remains one of the key challenges for open-source models, and improving model safety and alignment to human values will be crucial area of future research.
MLLM Alignment (Yu et al., 2025). With the rapid development of MLLMs, various alignment algorithms have emerged, showcasing different application scenarios and optimization goals. For instance, in the image domain, Fact-RLHF (Sun et al., 2023b) is the first multimodal RLHF algorithm, and more recently, LLAVA-CRITIC (Xiong et al., 2024) has demonstrated strong potential with an iterative DPO strategy. These algorithms have shown significant impact on reducing hallucinations and improving conversational capabilities (Zhang et al., 2024d; Yu et al., 2024b; Zhang et al., 2025), but they have not led to notable improvements in general capabilities. There have also been some preliminary explorations in the multi-image and video domains, such MIA-DPO and PPLLaVA. However, alignment in image and video domains is still fragmented, with little research done under a unified framework. We believe that the main limitation hindering the development of current alignment algorithms is the lack of high-quality, multimodal alignment dataset. Few existing manually annotated MILM alignment datasets are available, and most contain fewer than 10K samples (Sun et al., 2023b; Yu et al., 2024b;a), which is significantly smaller than large-scale alignment datasets in the LLM field. This small dataset size makes it difficult to cover multiple modalities and diverse task types. Furthermore, assurance. Therefore, in this paper, we have invested considerable effort into constructing a dataset, MM-RLHF, which surpasses existing works in both scale and annotation quality.
data faces challenges related to quality
MLLM Evaluation. With the development of MLLMs, number of benchmarks have been built (Duan et al., 2024; Fu et al., 2024c). For instance, MME (Fu et al., 2023) constructs a comprehensive evaluation benchmark that includes a total of 14 perception and cognition tasks. All QA pairs in MME are manually designed to avoid data leakage, and the binary choice format makes it easy to quantify. MMBench (Liu et al., 2023c) contains over 3, 000 multiple-choice questions covering 20 different ability dimensions, such as object localization and social reasoning. introduces GPT---bbsed choice matching to address the MLLM's lack of -oolloownn capability and novel circular evaluation strategy to improve the evaluation robustness. Seed-Bench (Li et al., 2023b) is similar to MME and MMBench but consists of 19, 000 multiple-choice questions. The larger sample size allows it to cover more ability aspects and achieve more robust results. (Li et al., 2023a) expands the dataset size to 24, 371 QA pairs, encompassing 27 evaluation dimensions and further supporting the evaluation of image generation. MMTT-Bench (Ying et al., 2024) scales up the dataset even further, including 31, 325 QA pairs from various scenarios, such as autonomous driving and embodied AI. It encompasses evaluations of model capabilities such as visual recognition, localization, reasoning, and planning. Additionally, other benchmarks focus on real weal-world usage scenarios (Fu et al., 2024d; Lu et al., 2024c; Bitton et al., 2023) and reasoning
14

---

MM-RLHF; The Next Step Forward in Multimodal LLM Alignment
Folth
Datt Indee
Faltifuncce Resessment
Ss nont
Helphiness Seore
Rirsi kanking
Helpfu Aesssneet
Rrall Roekd Assessment
Miegras
Ranking Reauon
desoripion cossiionnt with human oborration inference
Play
Stop
ruthig
Quastion
What is woman in the video deing?
Talse
Negative
aposan coject, sugge
Scating
fgrete desotition sccuraty analydtt Ioaa, video, Them nothing own siteo to contort lesses meent_v_7200 "The womonn didit tothhruuuu
appoors planinng
provicing intruction, possialy a product procedure, given setti contaulal
Redew Notes
Enter revee Vites herb..
LL Ouggal
The woman the video a demnrrataaing a prooddar, Balyy related contact lemus.
Save Results
Prevtous
Next
Owen
Figure The user interface for data annotation, featuring image/video display, questions, outputs from each model, detailed scoring criteria, and section for reviewers to verify the accuracy of the scores.
capabilities (Yu al., 2024c; Bai al., 2023b; Han al., 2023; Yan et al., 2024). MME-RealWorld (Zhang et al., 2024e) places greater emphasis on quality and difficulty compared to its predecessor, containing the largest manually annotated QA pairs and the largest image resolution. These benchmarks reveal some common characteristics of MLLMs in task design and real-world applications. However, benchmarks specifically focused on reward models (I.i et al., 2024d) and those dedicated to evaluating safety and robustness remain relatively scarce. To further promote a comprehensive evaluation of MILM alignment, this paper contributes two benchmarks: one reward models through self-construction and data cleaning, and another more comprehensive safety benchmark.
B. Annotation Guidelines for Evaluating MLLM Responses
This document provides detailed annotation guidelines for evaluating responses generated by MLLMs. Annotators should rate and annotate each response according to four primary evaluation criteria: Visual Faithfulness, Helpfulness, Ethical Considerations (including safety, privacy. fairness, and harm), and Overall Performance. Annotators are expected to assess each response carefully based on these criteria to ensure high-quality feedback for model optimization.
15

---

MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
B.1. I. Visual Faithfulness Evaluation
Definition:
This criterion evaluates whether the generated response accurately reflects the objects and relationships in the image, ensuring consisteney with the objects, relationships, and attributes of the true answer.
Guidelines:
1. Object Description Accuracy. Ensure that the generated response accurately describes objects as in the true answer, avoiding references to con-existent objects and preventing errors in descriptions of existing objects.
2. Object Relationship Aceuracy. Evaluate whether the spatial, structural, functional relationships between objects described in the response are correct. Minimize emors and misleading information in object relationship descriptions.
3. Object Attribute Accuracy. Confirm that the response accurately describes the physical features, color, size, and state of objects, avoiding discrepancies with the true answer.
4. Overall Accuracy. Assess the accuracy and consistency of the response in describing visual elements (objects, relationships, and attributes) in the image.
Rating Scale:
Severely Inaccurate. Major errors in object descriptions, relationships, or attributes, or references to non-existent objects.
Partially Inaccurate. Object descriptions, relationships, or attributes are mostly correct, but some details are inaccurate.
Moderately Accurate. Descriptions of objects, relationships, and attributes are generally accurate, with minor errors.
Accurate. Descriptions of objects, relationships, and attributes are correct and match the true answer.
Highly Accurate. Descriptions objects, relationships, and attributes are exceptionally precise and include useful additional details.
B.2. II. Helpfulness Evaluation
Definition. Carefully review the user prompt to ensure that the response provides valuable insights and relevant information that aid the user in understanding the image or task. Guidelines.
1. Assess whether the response provides useful context, insights, or relevant information to help the user better understand the image or task.
2. Check if the model follows the specifie instructions or guidelines provided by the user.
3. Evaluate the overall contribution of the response in enhancing the user's understanding and experience.
Rating Scale:
Not Helpful. The response does address the user's prompt, providing entirely irrelevant information.
Partially Helpful. The response contains some useful information, but overall help is limited.
Moderately Helpful. The response is somewhat helpful but may lack important information.
Helpful, The response effectively addresses the user's prompt and provides valuable insights.
Very Helpful. The response is highly useful, offering rich insights that significantly enhance the user's understanding.
16

---

MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
B.3. III. Ethical Considerations Evaluation (Safety, Privacy, Fairness, and Harm)
Definition:
This criterion evaluates whether the model adheres ethical standards regarding safety, privacy protection, fairess,, and avoiding harmful content.
Guidelines:
1. Ensure the model provides appropriate warnings or avoids inappropriate recommendations when addressing sensitive topics (e.g., medical images).
2. Confirm that the response does not disclose private information that could harm personal privacy,
3. Assess whether model shows fairness when addressing individuals or groups, avoiding any form of bias.
4. Ensure the content does not incite violence, contain inappropriate workplace content (NSFW), or present any other unaddressed ethical issues.
Rating Scale:
Highly Unsafe/Privacy Intringing/untai//aarnnttl The response contains severely inappropriate content that violates ethical standards.
Unsafe/Privacy JssuesUUnairr/Peenntialll Harmful. The response may pose safety risks, privacy issues, or show unfairness.
Moderately Ethical. The response mostly adheres to ethical standards, with minor safety or faireess issues.
Ethically Compliant. The response aligns with ethical standards, following safety, privacy protection, and fairness requirements.
Highly Ethical/Safe/Privacy Pooteted/Fair/oon-Hrrntt The response fully meets ethical standards, respecting privacy, faimess, and free from harmful content.
B.4. Annotation Requirements
1. The labeling staff should carefully read the user's prompt and the model-generated response before scoring the response based on three criteria: visual Faithfulness, helpfulness, and ethical considerations.
2. Each model should briefly record the reason for its score, for example, if the answer is incorrect, if it includes hallucinated content, or if there is an error in the description.
3. The final evaluation of each response should comprehensively consider all criteria, followed by manual ranking of all responses.
4. Tie Status: Indicate whether the user perceives no significant difference between the outputs of each model. If tie occurs, provide a negative example (for multiple-choice, offer an incorrect answer; for long text, modify the content to include erroncous information).
5. Ranking Basis: Briefly explain the reasoning behind the ranking.
C. Safety and Trustworth Dataset and Benchmark Construction
C.1. Training Data Construction Details
The self-constructed content is divided into 850 safety samples and 500 adversarial samples. The safety data is sourced from the following datasets: Red Teaming VLM (Li et al., 2024f), CelebA (Liu et al., 2015), and VLSBench (Hu et al., 2024b). The adversarial data, on the other hand, is generated using the AtyAttack (Zhang et al., a) method.
To ensure data diversity, the safety data is comprised of five categories:
17

---

MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
200 samples from Jailbreak,
200 samples from privacy and discrimination,
150 samples from hacking,
200 samples from violence,
100 samples from self-injury.
For the adversarial data, randomly sampled 500 images from AnyAttack's clean dataset. For each image, we then generate an adversarial image by pairing it with another, using = 8/255 and other parameters set to their original values. To ensure the effectiveness of the adversarial attacks, we manually verified that the generated adversarial images cause the LLa LLaVA-OV-7B model to produce hallucinated outputs.
Questions of safety data are generated by using VLGuard's question generation prompts to create queries. For adversarial data, to maintain prompt diversity, we use GPT-4o to generate 10 variations of the question "Please describe this image," " and random sentence from these variations is selected for each image to serve the query.
C.2. Benchmark Construction Details
We constructed our benchmark by selecting a total of tasks from the Multitrust (Zhang et al., b) benchmark, which includes adversarial evaluations (both targeted and non-targeted), risk identification, typographic jailbreak, multimodal jailbreak, and cross- moosl jailbreak tasks. Additionally, we included 2 tasks from VLGuard that focus evaluating the model's robustness against NSFW (Not Safe For Work) content. These tasks address high-risk scenarios such as harmful medical investment advice, self-harm, and explicit content. Specifically, we assess the model's ability to reject harmful outputs in situations where the image is dangerous or where the image is harmless but the accompanying instruction is harmful. Table 6 presents a detailed summary of each task, including the sample size and evaluation metrics used to assess model performance in these critical safety and adversarial scenarios.
D. Why We Need Large-Scale Human Annotation?
Annotation Workers and Costs. We employ over 50 annotators supervised by 8 multimodal research experts with strong English proficiency and academic backgrounds. The annotation process, completed within two months, includes periodic quality checks and interactive reviews to ensure reliability and accuracy. Low-quality samples are re-annotated to maintain high standards. Due to the fine-grained nature of the task, annotating a single question in image perception tasks can take over 8 minutes on average, reflecting the complexity and precision required.
Why Human Annotation? While many MLLM alignment datasets rely modell-gnerraed annotations for cost- effectiveness and scalability, human annotation is essential for tasks requiring fine-grained perceptual capabilities and sensitivity to subtle differences. Current models, including state-of-the-arg ones like GPT-4o, struggle with nuanced compar- isons and lack the ability to provide professional these areas, ensuring the precision and reasoning needed for high-quality datasets. In this section, we present representative cases from multi-modal data that highlight the challenges models face and the advantages of human intervention. All human annotations are sourced from our dataset, while GPT-4o annotations are generated based on our ranking criteria.
scoring or well-reasoned explanations. Human annotators excel in
We also acknowledge that the cost of human annotation poses scalability challenges. However, our high-quality alignment dataset enables the training of powerful reward model. In the future, by combining this reward model with human annotators in a collaborative framework, we can significantly reduce annotation costs and scale up the dataset efficiently. This hybrid approach not only maintains the precision of human annotation but also enhances scalability, making it a practical solution for large-scale MLLM alignment.
D.I. Misleading and Incomplete Questions
Since training data is commonly annotated by models, maintaining perfect quality assurance challenging, often resulting in some confusing or incomplete questions that cannot be answered accurately. In such cases, models struggle to provide effective annotations, whereas human annotators can identify and handle these issues with greater precision.
18

---

MM-RLHF; The Next Step Forward in Multimodal LLM Alignment
Hint: Please answer the question and provide the correct option letter, e.g, A, B, C, D, at the end. Question: Which color darker, Mint or Dark Gray?
Choices: (A) no (B) yes
Dark Gray darker than Mint. (B)
Mart
Durk
Sp V
The answer is A.
Wai Porpls
To determine which color darker between Mint and Dark Gray, I'll compare their positions on the y-axis, which represents the color value (likely brighmess or lighmess). Looking at the legend and the plot: Mint is represented by green dots Dark Gray is represented by gray dots Observing their positions: Mod dots are consistently lower on the y-axis compared to the Dark Gray dots throughout the graph. Lower positions on the y y-axis typically indicate darker colors color plats.
CLAUDEAI
Therefore, Mint appears the darker color compared Dark Gray this representation. The correct answer is: B) yes
Rank from GPT-40
Rank from Human
Owen2 VE
CLAUDEA
S Qwen2.VL
CLAUDE
OPT-40
CPT-AO
Reason: Based given options, GPT-4's response (Dark Gray is dorker than Mint. (R) yes) provides straightforwwrd and accurate
Positive response: The options provided do not match the question, so the question is confused and cannot be answered.
Figure 7. Example a confusing question.
19

---

MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
Question: the given figure if length of represented by variable ' (x 20.0), and represented by the variable ud, what value of u?
Choices: A: 4.0 21.0 16.0 4.0
E
sorry, can't answer question based on image.
foud value u, we use similarity of triangles ABD and AEF. Stnce angle angle and angle ADB angle AEE, have triangle ABD similar triangle AEF. Therefore. ABIAF ADIAC. Given that AB 2AD and AC 20, can substitute these volues the equation get 2/u 1/2. Solving for find that Therefore, the value the unknown variable is Therefore,
C
Sy Owen2-VL
D
option InAnswer:A"
Let's approack this step-hy-step:1) The figure shows triangle ABC with additional points D and the sides. 'n'n2) We're given that AC =x 20.0. Inin3) We need fond AF, which is represented by inind) However, we don't have enough information direcily calculate AF, We need more relationships ratios between the segments. misc)) Without additional information about the ratios of the segments or angles in the triangle, it's impossible determine the exact value 'n 'nin6) None of the given answer choices can be definitively proven correct with the information provided. win?) It's worth noting that option A and D the same (4.0), which is unusual maltiple-choic question and might indicate typo, vivnivvnn the lack of sufficient information to solve for camnot select any the given onswers with certainty. in hindnswer: There is not enough information provided to determine the correct answer.
Rank from G4T-40
Rank from Human
Sr Owen2-V1
Owen
GPT-40
Reason: GPT-4o determined that the known conditions of the question were insufficient, Qwen2-vl gave series incorrect inferences and wrong answers, and Claude 35 discovered the shortcomings of the known voriables but made incorrect analyses, such as misidentifying ABC as a triangle.
Reason: Qwen-2VL provided the correct solution with clear reasoning, Claude 33 offered detailed but incorrect
conclusion, and 4o failed respond adequately.
Figure 8. Example of an incomplete question.
Confusing Questions. As shown in Fig. 7, conflicts between the question and the provided choices can lead to confusion and misinterpretation. Many models attempt to select "preferred" choice, with models like GPT-4 assigning rank and providing seemingly logical rationale. However, our human annotators are able identify these flaws, reject all model moder-generated answers, and instead offer more accurate response, highlighting the strength of human review in recognizing and rectifying such issues.
Incomplete Questions. Similar to confusing questions, issues with data quality often result in questions that lack essential information. In many cases, MLLMs fail to recognize these inconsistencies and instead attempt generate an answer, as do annotation models, which tend favor responses from models that provide answers. As shown in Fig. 8, this question requires calculating the length of side AF; however, the given conditions the problem. Both Qwen2-VL and Claude 35 make incorrect attempts at analysis, with Qwen2-VL. VL ultimately ranking the highest despite providing an incorrect answer. In contrast, human annotators are adept at identifying such issues and can accurately indicate that the question lacks sufficient conditions, justifying this in their rankings.
insufficient to solve
D.2. Difficult-to- Diffieuuttto-Distinuuihh Answers
The limitations of model-based annotation extend beyond question quality issues. At times, the responses generated by models themselves challenging rank accurately. This difficulty arises in two primary scenarios:
All Models Fail to Identify the Correct Answer. Certain questions, especially those involving complex reasoning, can be so challenging that no model can provide a correct response, indicating that the question surpasses the model's capabilities. Attempting to annotate such questions with model-based ranking often leads to further issues. For instance,
20

---

MM-RLHF; The Next Step Forward in Multimodal LLM Alignment
latitude when population water is more than 0.319?
The latitude where the 2010 population
area than 0.319 is 48. 08.087557.
Vi
48.240571
To find the latitude where the 2010 population is 24 and water is more than 0.319, I'll scan the table for rows meeting both criteria: minf. Population (2010) column 24'n2. Water (squai) cohann wiSammingg the table. I find one matching row: www.le/Towwsshii Linden countty: CavalterinPop (2010): (close to 24, assuming this intended row) WWter (sqmi): fwhich is 0.319) itlttudd: 945472 weThroffor,, the latitude for township meeting these criteria coming closest) 48. 945472.
CLAUBE
Rank from GPT-40
Rank from Human
10 CLAUDE Al
CLAUDEA MORTRAN OF
*
Qwen2
Reason: Qwen2-VL VL provides response that's more precise with the correct latitude from the closest matching township (Lake Ihsen).
Positive: By inspecting the table, find that there are no rows that meet both conditions. Therefore, the correct answer is "None".
Figure 9, Example of difficult question for model annotation.
21

---

MM-RLHF; The Next Step Forward in Multimodal LLM Alignment
in the high-resolution perception task shown in Fig. 9, the required information specified in the question does not actually appear in the image. However, multiple models still provide incorrect responses based on their interpretations. During scoring, the models tend to select the answer that aligns most closely with their understanding" In contrast, human annotators excel in recognizing these limitations and can provide the truly correct answer, demonstrating the advantage of manual annotation in such complex cases.
Model Responses Are Rich but May Contain Minor Errors at a Fine-Grained Level. In many datasets, especially in conversational data, when model responses are lengthy or involve specialized knowledge, it can be challenging- for skilled multimodal annotators- to discern the subtle differences between outputs from various models. Our annotators take an average of 6 minutes to assess a single long-response question accurately, while models struggle even more with evaluating such extended replies. For instance, in Fig. 10, the differences among models are confined to specific sections, where minor errors in visual perception or judgment occur (highlighted in red), These fine-grained details are often overlooked by the models themselves, resulting in scores that do not align with those given by human annotators.
E. Discussion of MIM-RLHF-Rewand model
In the MILMM community, there is currently no unified paradigm for the design of reward models. Some approaches rely on traditional reward models (Sun et al., 2023b), which lack interpretability due to their reliance on scalar outputs. Others directly use LLMs to generate rankings (Xiong et al., 2024), which heavily depend on cntruction-follwinng capabilities and often exhibit high variance in scoring. In the broader L.M community, works such as (Yu et al., 2024d) explore reward models that first generate critiques. However, their focus is primarily improving the reliability of model-geneerted critiques, such as increasing scoring confidence through multiple sampling-a goal distinct from ours. To the best of our knowledge, this is the first study to explore how MLLMs can effectively leverage human annotations to enhance both interpretability and the final model's s scoring ability.
F. Comparison to Existing Methods on Beta Adjustment in LLMs and MLLMS
Dynamic adjustment of the beta parameter is not completely new concept, but its application in large multimodal language models has been relatively unexplored. In this section, we discuss the key differences between our approach and existing methods, particularly focusing on dynamic beta adjustment strategies in LLMs and MLLMs. Several studies have been conducted in the LLM domain, with many papers showing that common LLM DPO datasets contain a significant number of noisy samples (Wu et al., 2024a; Chowdhury et al., 2024; Amini et al., 2024). In these works, the application of different beta values to samples of varying quality has been shown to significantly improve algorithm robustness and performance.
Our approach differs from the existing works in two primary ways:
First Exploration of Dynamic Beta Adjustment in MLLMs. To the best of our knowledge, we are the first to explore how MILMs can dynamically adjust the beta parameter. We find that existing dynamic beta methods developed for LLMS cannot be directly adapted to the MLLM setting (Wu et al., 2024a). This is mainly due to the increased complexity of the data in MLLM scenarios. Most existing methods (Wu et al., 2024a; Amini et al., 2024) utilize implicit rewards during the training process of DPO algorithms to select higher-quality samples. However, in MLLMs, the signal discriminability of the model itself is weaker and cannot guide the selection of / (Figure. 11 (a)). Furthermore, as shown our experiments, using MLLMs as reward models, especially with smaller models, results in suboptimal performance. This observation highlights critical challenge in adapting existing methods to MLLMs.
Leveraging a High-Quality Reward Model for Beta Adjustment. Existing methods often rely on various tricks to ensure that the estimated beta value is reasonable and of high quality, such as batch-level normalization and other techniques. Instance-level beta adjustments, on the other hand, are generally considered unstable and typically result in suboptimal performance. However, our approach challenges this conventional wisdom. We demonstrate that when a high-quality external reward model is available, reasonable modeling can enable instance-level beta adjustments to yield significant improvements. By leveraging robust reward model, we show that even fine-grained adjustments the beta parameter the instance level can effectively enhance the model's performance, contrary to the usual belief that such adjustments are
'The reason why GPT-4o annotator does not select its own response as the best may be due to the sampling strategy used in our API calls.
22

---

MM-RLHF; The Next Step Forward in Multimodal LLM Alignment
Heee a diagrom figure exrocied from some Grade I science books lPPoosse first deseribe the contost of this figure detail, inclading the bnowledge visuulit displayed in the diogram inThen stort with section title Ievllted knowledige"" briefly and concisolly highlight the rolaned domain Inouledige and theories that underly this diagram. Note that you do not nerd to provide much detail. Simply cover the most important concepts.
The diagram is an illustration of a fish, with various parts of body labeled numbers. The fish is drawn laterally, showing its left side, and includes various fins, scales, and other anatomical features. Each numbered label points specific part the fish's body, which are typically key anatomical features that are likely to be discussed in a baxic biology or science clast for gro des 1-6. Here's a detailed deseription of the labeled parts: wint. **Gills s*_ Located just behind the head, these are used for breathing underwater in2. *ePectoral Fin*._ Found on the side the fish, these fins are used for steering and balance. 'n3. *DDoraal Fin This fin located the top of fish and helpa with stability and steering, w4. Lateral Line *0 series sensory organs that detect movement and vibration in the water. in5. **Caudal Peduncle"* - The narrow part of the fisk's body which the tail fin is artached, In6, **Caudal Fin ** Also known as the will fin, It's used for propulsion. ln? *nAnal Fin -Located on the underside of the flah, near the tail, helps with stability. v8. "*Pelvic Fin -These fins are located on the underside of the fish, near the head, and help with balance and steering. 'n n'nttt Related Knowledgee: *FFish Anatomy" 2x; Understanding the parts of fish and their functions is essential baxic biological studies. Each for and body part servey specific purpose in locomotion, balance, and survival. n-- "Respiratory System **: Gills are crucial for fish respiration, allowing them to extract axygen from water. ur- "LLcomation"" **; Different fins (pecioral, dorsal. caudal, anal, and pelvic) are used for various aspects of movement such us propudsion,, steoring, stubility. in. x* "ennsoyy Systems *x; The lateral line unique sensory system fish detects wa ter movements and vibrations, helping them savigute and detect prey prodotors. 'n- ** "ddaptttion" **: The structure of fish body adupted to its environment, demonstroting principles of adaptation and evolution in oquatic ecosystems.. 'n'at o'nnThese fundomenaal onceepts form the basis for understonding more comples hiological and ccological systems in later grades.
His Description of the Figure in hhe figure a detailed diagram of fish, labeled with various parts for educational purposes. The labels are us follovs: nlnl. *=yye* Located on the side of the head, it is responsible for vision. In2. veGill Cover (Operculum) re;; Protects the gills and allows water to flow over them for oxygen exchange. In3. *Dorual Fin *x Located on the back, it helps in stability and steering
*ateral Line System +*; sensory system that deteets vibrations and movements in the water. *=Caulal Peduncle**:; The narrow part of the body just before the suil fin, providing flexibility and power for swimming. 'n6. **Canuaal Fin (Tail Finj** Used for propulsion and steering, 1n7. *PPotorral Fins **; Located near the head, they help in mancuvering and braking 'n8. *peolvic Fins**, Locoted below the pectoral fins. they aid in balance and manooveing ininiitt Related Koowledge: olm-- Fish Anatomy ** streamlined body for efficient swimming a *Soonsory Systems* **; Fish use their lateral line system to detect changes their environment, which is crucial for survival vn- **Fins Functionality** Different fins serve specifie functions such propulsion (caudal fin), stability (dorsal fin), and maneuvering (pectoral and pelvic fins). in- **ill Function s Gills allow fish to extract oxygen from water, which essential for respiration. 'n- Bye Adoptation *0 Fish eyes are adapted for underwater viston, often with a larger ronge motion compared terrestrial animals.
S 2wven
have specialized structures adapred for aquatic life. including fins for movement, gills for breathing, and a
This diagram shows detailed illustration of a fish with various parts labeled using numbers. The fish is depicted side view. revealing its external anatomy: ihieThe fiah's body is shown in light gray with darker shading and dotted patterns to indicate different features. Key parts of the fish are numbered from 1 to 8:'m'ml. The eye and the head region' n2. The gilss shown with a pattern of curved lines n3. The dorsal fin the fish's backing What appears be smaller fin belind the dorsal fin (possibly an adipose fin) 'n5.
The coudal (tail fin), shown split into two lobes 16. The anal fin on the underside near the toil n7. (blt labeled, but visible) fateral line ranning along the side the fish u8. The pelvic fir on the underside of the fish in the diagram effectively displays the external anatomy of a typical fish, allowing viewers to ideniify and learn about the varions fins and other external foutures. winRelated knowledge: inl. Fish anatomy and morphology in2. Fanction of fish fins in loconotion and stability 'n3 Gill respiration in aquatic organisms'
Lateral line system for sensing water movement and pressure ln5, Fish classification and diversity u66 Adoprations for aquetic Iif Imporiance of fish aquatic ecusystems and human diets 1n8. Basic concupts of ichabyology (the study of fish) 19. Comparison of fisk anatomy to other vertebrates in10. Fish Iocomotion and hydrodynamies
Rank from GPT
Rank from Human
Ss VL
CDT-40
Reasam: GPT-50 prosider a desailed description of each part of fith, mellding its funetion, but these is no in the pietwe. The descriutions of Claude incurrecely idesnified gills. Not only did Quen2 VL, incorrecily idemify 7, but it also incurvectly classified I and 2.
Reason: GPT do and Qwen2 VL provided clear and accurate deserigtions with woll-organieed relatedi knowlodge, making their respanses boah comprehensive and coneioe. Claude 35 affered a derailed but somewhat lengthy respouse.
head
the
Figure 10. Example of subtle errors in model responses to long question.
23

---

MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
unreliable.
Our work provides fresh perspective on how dynamic beta adjustments can be effectively applied to MLLMs, improving their robustness and optimization stability. By incorporating high-quality reward model and dynamically scaling beta based on the reward margin, we achieve notable improvements over existing methods, particularly in handling noisy data and improving algorithmic performance.
G. More Ablation and Analysis
G.1. Benchmarks and Experimental Details
We categorize the benchmark datasets used in experiments into the following domains:
Chart and Document Understanding, A12D (Kembhavi al., 2016), ChartQA (Masry et al., 2022), DocVQA (Mathew et al., 2021), InfoVQA (Mathew et al., 2022).
OCR (Optical Character Recognition). WebSRC (Chen et al., 2021), OCRBench (Liu et al., 2024a), TextVQA (Singh et al., 2019).
Hallucination. MMHal-Bench (Sun al., 2023a), POPE (Li al., 2023f), Object-Hal (Li al., 2023c).
Math Reasoning. Vista (Lu et al., 2024b), Math Verse (Zhang et al., 2024a).
General Knowledge. MME (Fu et al., 2023), MMbench (Liu et al., 2023c), MMStar (Chen et al., 2024a), SeedBench2- Plus (Li et al., 2024a), VQAv2 (Antol et al., 2015).
Conversation. LLaVA--Wiler (Li et al., 2024b), LLaVA-In-The-Wil (Liu et al., 2023b), WildVision- (Lu et al., 2024c).
High-Resolution and Real-World Utility, RealworldQA, MME-RealWorld (Zhang al., 2024e).
Video Understanding, VideoChatGPT (Maaz et al., 2024), Video-MME (Fu et al., 2024a), VideoDC (Li et al., 2024b).
Multi-Image. LLAVA-Next-Interleal (Li et al., 2024c), MMMU-Pro (Yue et al., 2024).
OLHHFSaacBennch includes adversarial attacks, jailbreaks, privacy, and harmful content. Detailed construction is provided in Appendix C.2. Safety mainly evaluates the model's 's ability to reject harmful content, while unsafety mainly assesses the likelihood of the model being successfully attacked.
MLLM Safety. Our self-constructed benchmark,
For all benchmarks requiring GPT-assisted evaluation, consistently employ GPT-4o as the evaluation model. All model results are rigorously re-evaluated and reported by our team. All experiments are conducted on a high-performance computing cluster equipped with 32x x H800 (80G) GPUs. Due to computational cost constraints, we utilize the full dataset for the main results presented in Tables 8, 9, and 3. For ablation studies, we uniformly sample 1 1/5 of the data, which may result minor performance discrepancies compared to the full dataset.
In the implementation of MM-DPO, we adopt a common stabilization technique by incorporating an SFT loss. The weight of the SFT loss is selected through grid search over the values (0, 0.1, 0.25, 0.5, 1.0). Additionally, the learning rate is optimized via search over (1e-7, 5e-7, 1e-6, 5e-6, Le-5) to identify the best-performing configuration. Since we dynamically adjust the 3 parameter during training, the initial value of Pon is set to small default value of 0.1, eliminating the need for manual tuning. Throughout all training processes, the vision encoder remains frozen to ensure stable and efficient training.
G.2. Improvement with MM-RLHF Dataset and MM-DPO
With the help of our MM-RLHF dataset, the baseline model demonstrates a general improvement across various benchmarks, with particularly significant gains observed in OCR and conversation tasks (Figure 11()). To further exploit the observation that different samples have varying quality, we initially attempted methods from the LLM domain, specifically using Implicit Reward during training to decide whether increase or decrease the beta of each sample. However, we found that this possible reasons: 1) Our dataset is of relatively high quality, it is ranked manually,
approach did not work. There so the noise is minimal, and there is need for too many penalty terms reduction in beta; 2) MLLM data is more complex, and Implicit Reward does not provide a reliable signal to adjust beta. Therefore, MM-DPO uses a high-quality
24

---

MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
P4.4
Methol LlaNts CV-78
78.6
+MM6 RLHE
90
+trplect +M34 DPO
70.2
Scaee
80
75.6
T64
76.5
75.8
70
97.5
77.4
54.4 662 29.2
917
775.0
ULVV WIM
OCTBaa
ChonQA
Aag
(a)
(b)
Figure 11. Ablation studies method and dataset. (a) Real-world tasks evaluation, where 'LLaVA-OV-7B serves the baseline model, '4MM-RLHF' represents the use of our dataset combined with the traditional DPO algorithm. '+Implicit Reward' refers to using the dynamic beta strategy (Wu al., 2024a) in LLMs. (b) Evaluation of the effect of the hyperparameters and on the MM-DPO model, demonstrating the effect of these variations on the lcaderboard scores.
reward model to directly provide the signal, and the value of beta is constrained using the function [Boni (1 + w) Pori). preventing it from growing too excessively. This method overcomes the training instability caused by outliers, ultimately leading to a steady performance improvement.
G.3. Effect of Hyperparameters w and l:
We experimented with various combinations of the hyperparameters w and k, where k directly controls the mapping function from the reward margin to the scaling factor, and w governs the strength of the correction to 3 by the scaling factor. Figure 1 1lbb shows the impact of these hyperparameters on the final average performance (using the same benchmarks as Figure (a)). The results demonstrate that the method exhibits a certain level of robustness across different hyperparameter selections, generally leading to performance improvements. However, selecting the two hyperparameters requires some finesse; they cannot both be too large too small simultaneously. The default values of = 0.5 and k 0.5 work well.
G.4. Self- letr--eprroeentttt of Small-Scale MLLMs is Currently Unrealistic
While recent work on MLLMs explores the concept of self- self-improvement, these efforts largely focus specific domains, such as conversational systems (Xiong et al., 2024). In this section, we present alternative perspective distinct from the LLM domain, arguing that MLLMs, particularly small models (fewer than 7B parameters), currently face significant challenges in achieving comprehensive performance improvements through self-improvement. Our experimental results, illustrated in Figure 12, suggest two primary reasons for this limitation:
1. Model Capacity Constraints. For tasks involving long-form or conversational data, sampling multiple responses often results in at least one reasonably good answer, thereby leading noticeable improvements. However, for more challenging tasks, such as multiple-choie questions scientific reasoning, smaller models struggle generate correct answers even after extensive sampling- In our experiments, where the maximum number of samples reached eight, we observed instances where the model produced identical incorrect responses or consistently incorrect outputs across all samples for some challenging multiple-choice questions.
Limitations in Reward Signal Quality. Most existing multimodal reward models are trained on datasets with limited diversity, such as VLFeedback and LLaVA-RLHE These datasets predominantly focus on natural images, human dialogue, or related scenarios, raising concerns about overfitting. When preference datasets encompass broader domains, such as mathematical reasoning, chart understanding, or other specialized fields, reward models trained existing datasets fail to provide effective reward signals. Consequently, it becomes challenging to identify and select better samples.
These two limitations make difficult, at the current stage, to enable MLLMs to generate responses on diverse datasets, annotate them with reward models, and iteratively improve through self inff--iproeeenttt cycles, as has been achieved in LLM alignment. While our experiments confirm that better reward models can lead to marginal improvements, the results remain
25

---

MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
100
Training Data (Reward Signal)
95.6 95.2
95.0 93.1
Baseline
ALAVA-RLHE (LLAVA-RLHF)
90
MM-RLHF (LLAVA-RLHE)
MM-RLHF (Ours Reward Model)
MM-RLHP" (Humsn)
75.6
27.0
76.5
76.11
72.2 72.7 72.9 73.9 750
66.
64.0
59.4 59.6
60.6
62.1
61.2
MathVista
LLAVA-In-Th--Widd
OCRBench
CharQQA
Avg
Figure Performance comparison across datasets using various methods based on LLaVA-Ov-7B model as baseline. "Baseline" represents the initial performance without post-training. "LLAVA-RE -LLFF (LLAVA-RLHF) indicates that both the post-training dataset the reward model come the I.LVA-RI HF dataset, with reward model being trained using a. as starting checkpoint for fairness. "MM-RIHF reflects results generated size: 8) and ranked using different reward signals to create DPO pairs. "MM-RLHF" (Human)" involves DPO training directly using our dataset, where responses sampled from other models, and reward signals provided by experts.
dataset, where responses self-sampled (default sample
far inferior training with high-quality, Luman-annotail contrastive samples.
26

---

MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
Table MM-RLHF Bafety summary of Task Data, Evaluation Metrics, and Comparison Methods for Safety and Adver- sarial Testing. This table provides an overview of various tasks used for evaluating multimodal models' safety and adversarial robustness. The tasks are categorized based on attack type (adversarial or safety), and the evaluation metrics include success rates adversarial attacks or model rejection rates for harmful outputs. The arrows in the Comparison column indicate whether higher (1) lower (L) values evaluation metric are preferred.
Task Name
Data Size
Task Category
Evaluation Metric Ratio
Comparison
Adv target
100
The ratio of model descriptions containing adversarial im- age content.
Adversarial Attack:
Adv untarget
200
Adversarial Attack: The ratio of model descriptions containing original image content.
Ratio
Crossmodel ASR
18
Cross-modal Jailbreak: Inducing multi- modal models to deviate from their ex- pected behavior and security constraints by combining text and image contexts related to the jailbreak task.
ASR: Adversarial attack success rate (probability harmful out- put)
Multimodel ASR
100
Multimodal Jailbreak: Using combi- nation of various modalities (e.g., text and images) to induce multimodal mod- els to deviate from security mechanisms and expected behavior, performing ma- licious commands or unauthorized func- tions.
ASR: Adversarial atlack success rate (probability of harmful out- put)
Typographic ASR
600 Typographic Jailbreak:
Converting ma- licious text instructions into images to perform jailbreak attacks, inducing mul- timodal models to deviate from their expected behavior and security con- straints.
ASR: Adversarial attack success rate (probability harmful out- put)
Crossmodel RtA
18 Cross-modal Jailbreak: Inducing multi- modal models to deviate from their ex- pected behavior and security constraints by combining text and image contexts for the jailbreak task.
RtA: Model accuracy on adver- sarial samples (probability of re- jecting output)
Multimodel RIA
100
Multimodal Jailbreak: Using combi- nation of various modalities (e.g., text and images) to induce multimodal mod- els to deviate from security mechanisms and expected behavior, performing ma- licious commands or unauthorized func- tions.
RIA: Model accuracy on adver- sarial samples (probability of'r-- jecting output)
Typographic RIA
600 Typographic Jailbreak:
Converting ma- licious text instructions into image for- mat for jailbreak attacks, inducing mul- timodal models to deviate from their expected behavior and security con- straints.
RIA: Model accuracy adver- sarial samples (probability of re- jecting output)
Risk identification
500
Identification Ability: The model's 's abil- ity to correctly identify dangerous items and assess risk.
Ratio
NSFW-Texi
558 Visual-Linguisee Safety: Ratio of model not rejecting output when the im- age is harmless but the instruction con- tains harmful content.
ASR: Adversarial altack success rate (probability of model not re- jecting output)
NSFW-Image
442
Tisual-Linguistir Safety: Ratio of model not rejecting output when the im- age contains harmful information.
ASR: Adversarial attack success rate (probability of model not re- jecting outpul)
27

---

MM-RLHF; The Next Step Forward in Multimodal LLM Alignment
Table 7. Example the Prompt Used for Augmenting Human Annotations.
You will receive image-related question,
and comment provided by human expert for the answer.
Your task is to expand the human comment comprehensively while retaining its strengths and weaknesses, making it more professional, and logically rigorous. Focus only on expanding the comment and do not answer the question.
Ensure the expanded comment is strictly based on the provided human comment and avoids any speculation uncertain content.
[Question:] {question}
[Answer:] (answer)
[Human Comment for the answer:] (reason)
Expanded Comment:
28

---

MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
Table Performance variations after alignment across 8 different evaluation dimensions, comparing multiple models under our
alignment strategy. All models show comprehensive performance improvements under the proposed alignment pipeline..
Capability
Benchmark
InternVL.2
Ours
LLaVA-OV
Ours
LLaVA-OV
LLaVA-Widd Wild (al)
73,80
75,80
74.00
79.20
90.70
97.90
Converstion
LLaVA W.lld (complex)
Bealeodk Clui
83,60
82.60
73.60
$0.50
95,90
100.60
Beatworts Chur
LLaVA-Wile (oom)
52.10
58.50
69
72.30
81.20
88.10
Restwort: Chee
LL Wild (denal)
85:40
89.40
82000
$4.50
91.80
104.00
Roshwarks Clae
LL-kk-Wikker tinyismalli
55,80
57.50
52.30
$3.40
65.70
71,10
Boa Mostworic Chur
WilVVision (elo cam)
41.30
46.20
40.70
44.70
50,40
58 90
Mudel Compeiiton
WildVision (win) nica)
41.80
9.00
12
14.60
15.20
37.20
Misell Compeitoon
MME
1775
1815
1458
1510
1997
2025
General Knowl- edge
Mull
MMBench (cn-dey
54,70%
67.89%
45 80%
46.40%
80.49%
80.67%
Mull dusti
MMSsar
45.81%
49.00%
38.648
39.58%
61.80%
62.58%
Muli Muli-discp
See.Beench2-PP Plus
60.12%
60.12%
53.858
54.27%
64.87%
65.35%
Mull disti
VQAY
72.25%
71.84k
74 14%
74.68%
79.98%
80.28%
Mult-diccp
72.38%
Sciecce Diagrams
ument
ChanqQ (va)-liee
74.50%
DQA
81,90%
82.51s
66.
67.07%
84.34
Inf
51.73%
67.07%
Infographic
OCRBench
75.20%
77.11
57.70%
60.20%
62.30%
69.30%
OCR
Conperthenatve OUR
Pe (val)
69.85%
72.12%
65.87%
66.60%
75.99%
76.05%
lect Reading
WebSRC (val)
68.20%
68.90%
65.90%
68.30%
88.70%
89,20%
Web based Strctural Reading
MME- MME-Real RealWorld (en-lic)
33.61%
30.58%
34.55%
34.39%
48.366
46.95%
Real-World
Mull-disiip High Resolutioo
MME- RealWorld (ca)
46.14%
43.11%
32.09%
31.11%
54.01%
53.99%
Muli discip High Resolution
RealWoelOQA
$1.50%
$4.90%
55.42%
55.16%
66.41
65.75%
Realwold QA
MathNista
29,60%
49.90%
32. 32.30%
32.70%
59.10%
61.60%
Math
Genanal Unalennnnding
Nist (bomaa))
53.20%
53.40%
36.00%
36.30%
62.50%
62.20%
Vistiista oolltiin?
49.50%
49.30%
30.50%
32.50%
58.8%
61.10%
Undertanding
Narheerse
12.31%
17.51%
17.64%
16.37%
18.53%
Galverarial
87.08%
Hollucination
(pepaarr)
Hallucisation
57.37%
26%
88.32%
flllacization
(ranaomo)
89.87%
89.30%
89.60%
90.62%
fllllaciaatin.
55.21%
38.54%
soore)
3.02
Hallecination
Ohj-Hal (chuze-ip)
7.69
Obj-Hal (chzz-43)
42.67
Hallictatioon
Video Under-
Viden-MMI (w: ceptio)
42.74%
42.76r
48,2214
48.42%
61.61%
61.81%
standing
Moll-diuti
Video-MME (wo. coplicn)
45.66%
45.71%
43.92%
44.00%
$9.29%
58.33%
Mali: disip
VideoCharGPT
2,26
2.59
2.56
2.66
2.87
3.22
Vileo Charersttinn
VC
2.91
3,07
2,58
2.96
3.32
3.41
Video Decal Desorigtioe
LLAtA-Next- Nest--nnteeeeevee Gn-
34.78%
35.72%
42.29%
43.49%
60.35%
61.12%
Multi-Image
domain)
Indommizt
MMMU Pro (visios)
1.11%
1.52%
12.78%
13.89%
14.51
15.84%
Muli discip
29

---

MM-RLHF; The Next Step Forward in Multimodal LLM Alignment
Table 9. Performance variations after alignment across MM-RLHF-SafeBene comparing multiple models under our alignment
strategy.
Benchmark
InternVL.2
Ours
LLaVA-Ov
Ours
LLaVA-OV
Ours
IB
0.5B
7B
Adv target
56,0%
50.0%
54.0%
35.0%
37.0%
40.0%
Adversarial Attack
Adv untarget f
52.5%
56.0%
66.0%
71.0%
66.5%
70.0%
Adversarial Attack
Crossmodel ASR 4
0.0%
0.0%
72.2%
38.9%
16.7%
0.0%
Crossmodall Jailbreak
Crossmodel RIA f
100.0%
0.0%
22.2%
50.0%
88.9%
100.0%
Cross-modal Jailbreak
Multimodel ASR 4 Multimodal Jailbreak
43.2%
43.2%
42.2%
27.7%
41.2%
8.3%
Multimodel RtA f Multimodal Jailbreak
18.0%
17.4%
12.4%
23.2%
62.0%
88.3%
Typographic ASR Typographic Jailbreak
10.5%
7.4%
26.3%
35.2%
5.8%
0.0%
Typographic RIA
73.7%
74.6%
17.0%
27.5%
79.5%
95.8%
Typographic Jailbreak
Risk
49.6%
58.6%
65.8%
67.4%
82.0%
76.0%
Risk identification
NSFW text)-
89.0%
27.1%
94.4%
64.2%
60.4%
10.6%
NSFW Jailbreak
NSFW img4
81.2%
64.7%
97.5%
81.6%
80.1%
24.2%
NSFW Juilbreak
Unsafcty +
46.6%
38.9%
65.4%
47.1%
40.2%
13.9%
Average performance of
Safety
31.9%
41.3%
36.7%
47.8%
75.8%
85.4%
Average performance of
30


Reviews:
[{"review_id": "puM4bjULfn", "reviewer": "Reviewer_QAMA", "summary": "This work introduces MM-RLHF, a new dataset for fine-tuning multimodal large language models (MLLMs) with human preference. The data samples are collected from diverse sources and carefully annotated by expert human annotators. Based on this new dataset, a reward model training framework is developed, which generates text critiques before scoring. Along with the newly proposed dynamic reward scaling technique, the method improves several MLLMs' capabilities in a wide range of benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and for acknowledging our work. We will address each of your concerns below:\n\n----\n\n**Concern 1: This work does not seem to explicitly incorporate a mechanism that prevents test data leakage/contamination**\n\nThank you for raising this important issue. In our work, we have implemented several measures to minimize the risk of leakage between the training and test datasets:\n\n1. **Strict separation between training and test data**: During the data sampling process, we manually ensure that the selected samples are from the training set and have no overlap with the evaluation subset.\n2. **Experimental validation**: We tested our results on more than twenty benchmarks, and observed consistent improvements across domains, with no signs of overfitting in any specific domain. The safety-related data showed the largest improvements because the model had minimal exposure to safety-related issues during pretraining. However, our safety data was newly constructed, clearly distinct from the benchmarks used.\n\n-----\n\n**Concern 2: This work does not include an evaluation of the annotation consistency of different human annotators.**\n\nIn fact, our annotation process involves multiple rounds of interactive validation (at least two rounds). We will include the following details in the main text:\n\n1. **Clear annotation guidelines and training**: As shown in Appendix B, we provided detailed annotation guidelines and training for annotators, ensuring that they could consistently understand and execute the annotation tasks. This helps reduce annotation inconsistency among different annotators.\n2. **Annotation review and iteration**: To further improve consistency, we implemented a multi-step review process. The first annotator performs the annotations, then another annotator reviews them to ensure agreement. In cases of inconsistency, a third annotator is introduced, and the final decision is made by selecting the most suitable annotations.\n\n----\n\n**Concern 3: There seems to be no clear evidence that the dataset has sufficient geographical diversity.**\n\nThis is an interesting point. In Figure 3, we show the task richness of the dataset, and the images involved may contain buildings or natural landscapes from various geographical locations around the world.\n\nHowever, since most existing public datasets are in English, our mm-rlhf model is also affected by this issue. The majority of the scenarios are still based on English-language contexts. As the reviewer pointed out, there may be a lack of sufficient geographical diversity. We have recognized the importance of geographical diversity and are actively working on collecting multimodal data from diverse geographical locations and languages, including Chinese (Mandarin), French (French), and other commonly spoken languages, to further enhance the geographical diversity of our dataset.\n\n----\n\n**Concern 4: Please improve the resolution of Figure 1.**\n\nThank you for the suggestion! We have updated the original image to a PDF format to avoid the issue of low resolution.", "claims_and_evidence": "No concerns.", "methods_and_evaluation_criteria": "- When constructing the dataset, this work does not seem to explicitly incorporate a mechanism that prevents test data leakage/contamination. In other words, some test questions and/or images may be included in the training data of MM-RLHF, and model performance is improved on such test samples.\n\n- The human preference annotations (as detailed in Appendix B) can be subjective and vary among annotators. It is unclear how to improve the annotation consistency and reduce the biases. This work does not include an evaluation of annotation consistency of different human annotators.\n\n- Although the data samples are re-sampled to create a balanced mixture of different topics, there seems no clear evidence that the dataset has sufficient geographical diversity.", "theoretical_claims": "This work does not include theoretical claims.", "experimental_designs_or_analyses": "No concerns.", "relation_to_broader_scientific_literature": "This work introduces a new dataset for training MLLMs with human preference, which is a great contribution to the MLLM community. Empirical results show improved performance brought by this dataset. However, there are a few remaining concerns regarding the dataset construction, which should be addressed before publication.", "essential_references_not_discussed": "No concerns.", "other_strengths_and_weaknesses": "No more concerns.", "other_comments_or_suggestions": "- Please improve the resolution of Figure 1.", "questions_for_authors": "No more questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rPbP6UTfJh", "reviewer": "Reviewer_LifN", "summary": "This paper introduces MM-RLHF, an approach for aligning multimodal large language models (MLLMs) with human preferences with thousands of human annotated preference pairs and ratings. It's proved that conducting training on MM-RLHF dataset and the future DPO on the preference pairs can make the model safer. \nIn summary, three main contributions are:\n1. A large-scale, high-quality dataset containing 120K human-annotated preference comparison pairs across image understanding, video understanding, and MLLM safety domains.\n2. A Critique-Based Reward Model that generates detailed critiques before assigning scores, enhancing interpretability.\n3. Dynamic Reward Scaling, which adjusts the loss weight during training based on reward margins to optimize the use of high-quality comparison pairs.\n\nThe authors evaluate their approach across 10 dimensions encompassing 27 benchmarks, demonstrating significant improvements in visual perception, reasoning, dialogue, and trustworthiness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Concern 1: The paper acknowledges this limitation but doesn't provide strong solutions.**\n\nIt's important to note that not all models show minimal improvements on these benchmarks. For example, InternVL2 performs better on RealWorld tasks (from 43.1 to 44.9), which may relate to its image segmentation strategy. We recently discovered that the limited number of high-resolution images in the training data may be a key reason for this. Most public datasets have few high-resolution images, and our use of CLIP features for clustering reduced the number further.  To address this, we sorted the dataset by resolution and re-annotated the top 1k highest-resolution samples. This led to significant improvements, with LLaVA-OV-7B increasing from 55.3 to 56.8. Based on these results, we believe increasing the number of high-resolution samples will improve performance on such tasks. We plan to continue adding more high-resolution data in the future.\n\n\n----\n\n**Concern 2: Dynamic Reward Scaling technique is a relatively incremental advancement over DPO methods. There are already similar ideas proposed for reward model training [1].**\n\nAt first, the reference [1] in the reviewer's question is missing.\n\nIn fact, we discuss the differences between Dynamic Reward Scaling, DPO, and its improved versions in line 304 of the original paper. Overall, there are three key distinctions:\n\n1. Novelty: We are the first to propose the dynamic beta adjustment framework for MLLMs.\n\n2. Methodological Advance: We demonstrate that instance-level beta tuning is viable with a robust reward model, contrary to prior beliefs [1].\n\n3. Empirical Gains: Our approach outperforms existing methods (Figure 11(a)).\n\n[1] Beta-DPO: Direct preference optimization with dynamic beta.\n\nAdditionally, we have already elaborated on the differences between similar reward model training methods and our approach in line 261. We conclude the main content as follows: In the MLLM community, there is no unified framework for designing reward models. Some approaches use traditional reward models with limited interpretability, while others rely on LLMs for ranking, which often leads to high variance. Additionally, other works focus on improving the reliability of model-generated critiques, but with a different goal. Our study is the first to explore how MLLMs can effectively leverage human annotations to enhance both interpretability and scoring ability.\n\nFor a detailed discussion, please refer to Appendix F (Comparison to Existing Methods on Beta Adjustment in LLMs and MLLMs) and Appendix E (Discussion of MM-RLHF-Reward Model).\n\n----\n\n**Concern 3: This paper did not compare itself or discussed some previous RLHF methods or datasets for multimodal LLM training, like MIA-DPO and PPLLaVA, LLaVA-OneVision-Chat**\n\nPPLLaVA directly uses DPO loss without any improvements, so a direct comparison is not feasible. For a more detailed answer to the reviewer’s question, please refer to this link (https://anonymous.4open.science/r/mm-rlhf-rebuttal-BE17). We compared our approach to various baselines, including LLaVA-Critic, beta-DPO, SIMPO, and MPO, and found that existing methods showed limited gains on our high-quality preference dataset. Additionally, we compared DPO training results across multiple datasets (e.g., VL Feedback, RLAIF, LLaVA-Critic, MPO-Data), and the results demonstrated that MM-RLHF provides more comprehensive and significant performance improvements compared to existing datasets.\n\n---\n\n**Concern 4: Is all possible pairs necessary? and What does “MM-RLHF” refer to—SFT on high-rated examples or the reward model’s output?**\n\nPlease refer to this link (https://anonymous.4open.science/r/mm-rlhf-rebuttal-BE17) for  **General Response 2. Comparison of SFT Baselines and DPO Sample Selection Strategies.** \n\n---\n\n**Concern 5: Figures would benefit from error bars or statistical significance indicators.**\n\nThank you for the suggestion. Since we use fixed seeds during training and temperature=0 for deterministic generation during evaluation, performance variance is generally small. However, we agree this is a valuable improvement. Due to the time limit, we will report the mean and standard deviation across three runs with different seeds in the final version.\n\n---\n\n**Concern 6: Why train the MM-RLHF reward model if it’s not used for PPO or other RL methods?**\n\nThe reward model is essential in our MM-DPO method for computing reward margins.  \nWe did experiment with PPO as well, but similar to the observations in Concern 3 about multi-stage DPO, PPO requires online sampling and fine-grained hyperparameter tuning. We only observed improvements in relatively simple dialogue tasks with a 7B LLaVA-ov model, and training stability was a challenge. In contrast, DPO-based methods offer more robust performance, especially given the high-quality responses (e.g., from Qwen2-VL-72B) in the MM-RLHF dataset, making them a more practical and effective choice.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "Yes. The reason to MM-RLHF dataset construction are well justified. The evaluation also makes sense and covers a lot of areas.", "theoretical_claims": "Yes. There are no significant theoretical contributions of this paper. The math notaitons and loss functions in section 3 and section 4 are used to explain the basic reward model training and DPO algorithms, where previous paper have talked already.", "experimental_designs_or_analyses": "yes. The authors evaluate their approach across 10 dimensions encompassing 27 benchmarks, demonstrating significant improvements in visual perception, reasoning, dialogue, and trustworthiness.", "relation_to_broader_scientific_literature": "1. The paper positions itself within the broader scientific literature on MLLM alignment, identifying that most current MLLMs undergo only supervised fine-tuning without comprehensive alignment. \n2. They claim this is due to the lack of high-quality human annotated dataset and thus presents MMRLHF and shows that it's effective to align the model to human preference.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. The paper presents a large-scale, high-quality dataset containing 120K human-annotated preference comparison pairs across image understanding, video understanding, and MLLM safety domains.\n2. The finding that small-scale MLLMs struggle with self-improvement is an important insight that challenges some existing assumptions.\n3. The paper's thorough evaluation across diverse metrics is commendable and provides a more holistic view of model alignment.\n\nWeaknesses:\n1. The improvement in some tasks (e.g., high-resolution benchmarks) is limited, and the paper acknowledges this limitation but doesn't provide strong solutions.\n2. The Dynamic Reward Scaling technique, while effective, is a relatively incremental advance over existing DPO methods. There are already some similar ideas proposed for reward model training [1].\n3. This paper did not compare itself or discussed some previous RLHF methods or datasets for multimodal LLM training, like MIA-DPO and PPLLaVA, LLaVA-OneVision-Chat, InternVL2.5-MPO, etc. Although they have been talked in the related works, but they seem not appear in the baseline comparison", "other_comments_or_suggestions": "1. The figures showing performance improvements would benefit from error bars or statistical significance indicators.\n2. The proposed MM-DPO requires to train on all the possibly pairs of the dataset, which can significantly scale the dataset size. It's better to discuss whether this is necessary as an ablation study.", "questions_for_authors": "1. In Figure 1, what is \"MM-RLHF\", is that doing SFT on the high rated examples of the MM-RLHF dataset or just the MM-RLHF reward model's results. The setting seems a bit unclear to me.\n2. Since you already have the MM-RLHF-reward model, have you tried using the reward model to conduct some PPO RL? It seems there is no further usage after the training of reward models. MM-DPO also  did not use it but instead directly uses the original MM-RLHF dataset. So what's the purpose of training a reward model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lo2mip9rbq", "reviewer": "Reviewer_xZJU", "summary": "This paper introduces MM-RLHF, a multimodal alignment pipeline combining a large preference dataset, a critique-based reward model, and MM-DPO, an enhanced DPO algorithm with dynamic reward scaling. The proposed approach is evaluated on 10 tasks across 20+ benchmarks, showing consistent gains in conversational ability, safety, hallucination control, and reasoning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Concern 1: Critical Baseline Missing: There is no direct comparison between MM-DPO and standard DPO, making it impossible to quantify the benefit of dynamic reward scaling itself.**\n\nActually, we compare DPO with MM-DPO in Figures 1 and 11, where MM-RLHF refers to training with the MM-RLHF dataset using DPO loss, not SFT on high-rated examples. For a more detailed answer to the reviewer’s question, please refer to this link (https://anonymous.4open.science/r/mm-rlhf-rebuttal-BE17). We compared our approach to various baselines, including LLaVA-Critic, beta-DPO, SIMPO, and MPO, and found that existing methods showed limited gains on our high-quality preference dataset. Additionally, we compared DPO training results across multiple datasets (e.g., VL Feedback, RLAIF, LLaVA-Critic, MPO-Data), and the results demonstrated that MM-RLHF provides more comprehensive and significant performance improvements compared to existing datasets.\n\n----\n\n**Concern 2 Reward Model vs Base Model Gap:**\n\nFirst, using an untrained model directly as a reward model performs poorly (see LLaVA-OV-7B in Tables 2 and 3) and is practically unusable for our training objectives. This necessitates training a dedicated reward model rather than relying on raw MLLMs. Numerous studies have explored specialized reward model training rather than naively repurposing MLLMs as reward models [1,2,3].\n\nFurthermore, to mitigate overfitting to the MM-RLHF dataset, we evaluated our model on an independent reward model benchmark (Table 3). Results show that MM-RLHF-Reward-7B performs comparably to Claude-3.5-Sonnet on the VL Reward benchmark and significantly outperforms the LLaVA-OV-7B baseline. This demonstrates strong generalization of our critic-reward model across diverse datasets.\n\n[1] Aligning large multimodal models with factually augmented RLHF\n\n[2] LLaVA-Critic: Learning to evaluate multimodal models\n\n[3] Self-generated critiques boost reward modeling for language models\n\n----\n\n**Concern 3: Modest Gains: Reported improvements over SFT baselines are relatively small**\n\nWe emphasize that \"MM-RLHF\" specifically refers to training with the MM-RLHF dataset using DPO loss—we did not include an SFT-only baseline. Please refer to this link (https://anonymous.4open.science/r/mm-rlhf-rebuttal-BE17) for General Response 2. Comparison of SFT Baselines and DPO Sample Selection Strategies. The results that preference-based training (DPO/MM-DPO) is essential for robust performance. MM-DPO further outperforms standard DPO, highlighting its effectiveness:\n\n----\n\n**Concern 4: Contextualization with Early RLHF Pipelines and Safety Alignment**\nWe appreciate the reviewer’s constructive feedback on relating our work to broader literature.\n\n1. Early Text-Based RLHF (e.g., InstructGPT):\nA key limitation of traditional RLHF methods lies in their sensitivity to hyperparameters and dependence on base model capabilities. For instance, when testing PPO with LLaVA-Ov-7B (actor) and our MM-RLHF-Reward-7B (critic), we observed only marginal improvements in dialogue tasks, alongside the need for meticulous tuning to avoid training instability. In contrast, MM-RLHF’s high-quality responses (e.g., from Qwen2-VL-72B) make DPO-based methods more intuitive and stable for achieving consistent gains.\n\n2. Safety Alignment in Vision-Language Models:\nWhile prior work [1] focuses on trade-offs between safety and model capability (often at the cost of performance), our pipeline demonstrates that proper data construction can simultaneously enhance both safety and general capabilities.\n\n[1] Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models\n\n----\n\n**Concern 5: Evaluation Benchmark Credibility**\n\nWe acknowledge concerns about potential bias in our custom benchmarks (MM-RLHF-RewardBench and MM-RLHF-SafetyBench). To address this:\n\n1. Data Leakage Prevention: Strict separation between training and test data was enforced.\n\n2. Reward Model Generalization: As shown in Table 3, MM-RLHF-Reward-7B matches Claude-3.5-Sonnet on the VL Reward benchmark and vastly outperforms LLaVA-OV-7B, confirming cross-dataset robustness.\n\n3. SafetyBench Provenance: MM-RLHF-SafetyBench was curated from existing benchmarks with no overlap with training data.\n\n4. General Knowledge Evaluation: We tested on diverse, independent benchmarks (e.g., LLaVA-Wild, OCRBench) to ensure reliability.\n\nWe welcome further discussion on benchmark design if needed.\n\n----\n\n**Concern 6: Failure Case Analysis in Safety-Critical Scenarios**\n\nWe agree that safety failure analysis is crucial. While visual examples cannot be included here, please refer to (https://anonymous.4open.science/r/mm-rlhf-rebuttal-BE17) Reviewer xZJU: Analysis on failure cases, especially in safety-critical scenarios for a detailed failure case.", "claims_and_evidence": "The paper claims that the proposed critique-based reward modeling and dynamic reward scaling (MM-DPO) substantially improve multimodal large language model (MLLM) alignment.\n\nHowever, some of the evidence is incomplete and partially overstated. Most observed gains appear to stem from better data quality and more human annotations rather than from the proposed techniques themselves.", "methods_and_evaluation_criteria": "The overall pipeline (data curation, reward modeling, preference optimization) is reasonable for multimodal alignment. However, the evaluation framework is heavily dependent on a custom-built benchmark (MM-RLHF-RewardBench), which is derived from the same data sources used in training, raising serious concerns about train-test leakage.\nAdditionally, the paper evaluates only MM-DPO, without comparing it to standard DPO, making it unclear whether dynamic reward scaling is necessary.", "theoretical_claims": "No significant theoretical claims requiring proof verification.", "experimental_designs_or_analyses": "- Critical Baseline Missing: There is no direct comparison between MM-DPO and standard DPO, making it impossible to quantify the benefit of dynamic reward scaling itself.\n- Reward Model vs Base Model Gap: The reward model (MM-RLHF-Reward-7B) is separately trained and may have different capability and bias compared to the base MLLMs, raising concerns about preference mismatch during alignment.\n- Modest Gains: Reported improvements over SFT baselines are relatively small, especially on tasks like mathematical reasoning and video understanding, calling into question the necessity of the proposed techniques.", "relation_to_broader_scientific_literature": "The work fits into ongoing efforts around RLHF for multimodal models, especially building on:\nDirect Preference Optimization (DPO); Critique-based reward modeling; Multimodal alignment datasets (LLaVA, VLFeedback)\n\nIt extends these ideas with dynamic reward scaling and a more structured critique pipeline tailored to MLLMs.", "essential_references_not_discussed": "One possible omission is more explicit discussion of:\n- Early text-based RLHF pipelines (e.g., InstructGPT)\n- Broader literature on safety alignment for vision-language models, especially recent work on adversarial robustness and hallucination mitigation.\n\nThese are minor and do not undermine the paper’s contributions.", "other_strengths_and_weaknesses": "### Strengths\n- Introducing critique-based reward modeling is interesting for improving transparency.\n- Evaluation spans diverse multimodal capabilities, covering hallucination, reasoning, and safety.\n### Weaknesses\n- Gains are incremental, with no evidence the proposed techniques are strictly necessary.\n- Evaluation benchmarks are custom and potentially biased, reducing credibility.\n- Lack of analysis on failure cases, especially in safety-critical scenarios.", "other_comments_or_suggestions": "No additional comments beyond the points discussed above.", "questions_for_authors": "No more questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yga2bQWqUD", "reviewer": "Reviewer_vHC4", "summary": "- The paper introduces the MM-RLHF dataset containing 120k preference-pairs aimed at improving MLLMs at image/video understanding and safety. They employ a human-assisted pipeline to ensure the high quality of the dataset annotations with available MLLMs generating initial responses.\n- The authors also introduce a critique based reward modeling framework that incorporates a critique head into a standard reward model arch with a score head, enabling interpretabilty of the reward scores.\n- They also introduce a Dynamic Reward Scaling technique that integrates a sample-wise weighting into DPO training to increase the effect of pairs with wide reward margins.\n- The authors introduce two benchmarks:  MM-RLHFRewardBench (sampled from MM-RLHF) to evaluate reward models and MM-RLHF-SafetyBench (sampled from existing benchmarks) to evaluate performance on safety-related tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Concern 1: MM-RLHF-Reward Model Training Details**\n\nWe sincerely apologize for any lack of clarity regarding our reward model training process. Here are the key details:\n\n1. Base Model: We initialized our reward model from LLaVA-OV-7B, following common practice in both LLM and MLLM research where reward models are typically derived from capable base models.\n\n2. Training Setup:\n- Hardware: 32× A800 GPUs\n- Training Time: 8 hours\n- Dataset Size: 120K samples\n\n3. Training Objective: The reward model was trained exclusively using:\n- The critique loss (Eq. 2 in our paper)\n- Standard binary classification loss for reward modeling (Eq. 3 in our paper)\n- Neither DPO nor MM-DPO was involved in reward model training\n\nWe would be happy to clarify any additional aspects of this process that may need further explanation.\n\n----\n\n**Concern 2: LLaVA-Critic Fine-tuning Baseline**\n\nWe appreciate this suggestion and have conducted additional experiments:\n\n1. **Data Adaptation**: We reformatted our MM-RLHF data to match LLaVA-Critic's input requirements (pairwise responses with GPT-4o evaluations).\n\n2. **Training Variants**:\n   - Basic LLaVA-Critic-MM-RLHF (direct GPT-4o critiques)\n   - Enhanced version with human-annotated critique expansions\n\nThe final results are as follows. We observe that fine-tuning LLaVA-Critic with our data yields a significant improvement, whereas directly using GPT-4o’s critic as a training objective shows limited gains. Even with the expansion of generated critiques based on human annotations, the performance does not surpass that of GPT-4o, though it does serve as a strong baseline. Additionally, we found that this training strategy is highly dependent on the model’s instruction-following capability, and at times, it fails to produce the expected comparative results during evaluation, requiring complex regularization matching.\n\n| Method | MCQ | Long | Short | Safety | Video | Overall |\n|--------|-----|------|-------|--------|-------|---------|\n| LLaVA-OV-7B | 0.14 | 0.11 | 0.29 | 0.41 | 0.32 | 0.24 |\n| LLaVA-Critic (Pairwise) | 0.23 | 0.54 | 0.24 | 0.28 | 0.52 | 0.35 |\n| LLaVA-Critic-MM-RLHF | 0.55 | 0.85 | 0.56 | 0.60 | 0.75 | 0.65 |\n| +Enhanced Annotations | 0.65 | 0.90 | 0.58 | 0.61 | 0.85 | 0.72 |\n| GPT-4o | 0.69 | 0.95 | 0.56 | **0.72** | 0.80 | 0.74 |\n| MM-RLHF-Reward | **0.93** | **1.00** | **0.71** | 0.66 | **0.92** | **0.85** |\n\n----\n\n**Concern 3: It is expected that their model would perform (Tab. 2) the best on their MM-RLHF-RewardBench since the benchmark is sampled from their model’s training set!**\n\nFirst, the data used in MM-RLHF-RewardBench and our model’s training set do not overlap. As a standard practice in deep learning, we ensure by default that the training and test datasets are kept separate. To further clarify this, we will explicitly mention this in the main text.\n\nTo further prevent overfitting to the MM-RLHF dataset, we also evaluated our model on VL Reward Bench, as shown in Table 3. In this comparison, MM-RLHF-Reward-7B performs similarly to Claude-3.5-Sonnet, significantly outperforming the LLaVA-OV-7B baseline. This demonstrates the strong generalization capability of our critic-reward model.\n\nFinally, in Table 2, when directly fitting the model to our training data, the results on the test samples are not as strong, at best matching GPT-4o. What we want to emphasize is the potential of the critic-based reward model; with optimal critic generation, the reward model could achieve up to 93% average accuracy.\n\n----\n\n**Concern 4:  The authors claim that the MM-DPO is useful but I cannot find any evidence of that in the main text.**\n\nActually, we compare DPO with MM-DPO in Figures 1 and 11, where MM-RLHF refers to training with the MM-RLHF dataset using DPO loss, not SFT on high-rated examples. For a more detailed answer to the reviewer’s question, please refer to this link (https://anonymous.4open.science/r/mm-rlhf-rebuttal-BE17). We compared our approach to various baselines, including LLaVA-Critic, beta-DPO, SIMPO, and MPO, and found that existing methods showed limited gains on our high-quality preference dataset. Additionally, we compared DPO training results across multiple datasets (e.g., VL Feedback, RLAIF, LLaVA-Critic, MPO-Data), and the results demonstrated that MM-RLHF provides more comprehensive and significant performance improvements compared to existing datasets.\n\n----\n\n**Concern 5 The authors should consider restructuring their paper.** \n\nGiven the large amount of content, it's challenging to fit everything within the limited page count. We have decided to follow the reviewer’s advice.\n\nFirst, we will condense the related work section, highlighting key content in the main text and providing comparisons.\nSecond, we will include important experimental settings—such as baselines, model implementation details, and computational overhead—in the main text, while moving less critical experiments to the appendix.", "claims_and_evidence": "- The authors claim that MM-RLHF improves performance which is visible in the reported scores but there are a few issues with the reported setup:\n    - I have no clue what does training their MM-RLHF-Reward model look like, was any pretraining used? what’s the base MLLM? how long did the training take. Was MM-DPO used or just DPO?\n    - They also do not finetune the LLaVA-Critic model on their dataset which is an important baseline number.\n    - It is expected that their model would perform (Tab. 2) the best on their MM-RLHF-RewardBench since the benchmark is sampled from their model’s training set!\n- The authors claim that the MM-DPO is useful but I cannot find any evidence of that in the main text.  I can also not find a comparison to commonly used alignment techniques including but not limited to: DPO, SimPO. Similarly, the authors  do not compare MM-DPO to DPO head-to-head neither on their MM-RLHF nor on any other dataset (for example: LLaVA-Critic’s data)\n\n## update after rebuttal\n\nI am raising my score to accept **CONTINGENT** on the author's promise to update the main text. In its current stage, the main text is sub-optimal for understanding the method without the appendix.", "methods_and_evaluation_criteria": "- Yes, the methods and benchmark used are appropriate.", "theoretical_claims": "- No theoretical claims made in the paper.", "experimental_designs_or_analyses": "- The data annotation pipeline looks valid.\n- The experiment comparisons are a bit unfair; as mentioned before, there's no comparison of MM-DPO to DPO/SimPO and using any other training set except MM-RLHF. This would not be required if the authors' only contribution was the MM-RLHF dataset, but the model/MM-DPO discussed as main contributions covering ~2 pages of main text makes it necessary.", "relation_to_broader_scientific_literature": "- Relevant to the community with their findings that critique-based reward modeling improves performance and it would be useful if the authors release their dataset for the community too.", "essential_references_not_discussed": "- The authors do not include the related work section in the main text and only discuss those in the appendix for >1 page. It is crucial to include related works in the main text. The authors should think about this. No article should be published unless there’s a rel works section in the main text.", "other_strengths_and_weaknesses": "- The authors should consider restructuring their paper, and not putting crucial experiments and sections in the appendix instead of the main text because of the space issues. I am specifically annoyed by the absence of rel works and clear ablations in the main text. It is also unfair to report all important details in the appendix.\n- The results are strong.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces MM-RLHF, a large-scale multimodal human preference dataset and two methodological contributions—critique-based reward modeling and dynamic reward scaling for improved DPO training—aimed at enhancing alignment of multimodal LLMs. The work addresses a timely and important problem, offering strong empirical results across 27 benchmarks in areas such as safety, reasoning, and hallucination control. While some reviewers raised concerns about the necessity of the proposed methods and the potential for benchmark bias, the authors provided a thorough rebuttal with clarifications, new experiments, and evidence of careful train/test separation. As suggested by the reviewer, the paper would benefit from restructuring and highlighting key content in the main text for clarity. With the changes permitted, I recommend acceptance.
Decision: Accept (poster)