The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models
Authors: Zichao Li
Xueru Wen
Jie Lou
Yuqiu Ji
Yaojie Lu
Xianpei Han
Debing Zhang
Le Sun
Venue: ICML 2025 poster (2025)
Keywords: Multimodal Reward Model
Generalization
Spurious Correlations
Primary area: deep_learning->robustness

Abstract: Multimodal Reward Models (MM-RMs) are crucial for aligning Large Language Models (LLMs) with human preferences, particularly as LLMs increasingly interact with multimodal data. However, we find that MM-RMs trained on existing datasets often struggle to generalize to out-of-distribution data due to their reliance on unimodal spurious correlations, primarily text-only shortcuts within the training distribution, which prevents them from leveraging true multimodal reward functions. To address this, we introduce a Shortcut-aware MM-RM learning algorithm that mitigates this issue by dynamically reweighting training samples, shifting the distribution toward better multimodal understanding, and reducing dependence on unimodal spurious correlations. Our experiments demonstrate significant improvements in generalization, downstream task performance, and scalability, establishing a more robust framework for multimodal reward modeling. Our source code is provided on https://github.com/alignrm/Generalizable-MM-RM.

The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models
Zichao Li Xueru Wen 12 Jie Lou Yuqiu Ji
Yaojie Lu
Xianpei Han Debing Zhang² Le Sun
Abstract
2023) and safety concerns (Liu et al., 2025), making it crucial step toward more reliable and aligned Al systems.
Multimodal Reward Models (MM-RMs) are cru- cial for aligning Large Language Models (LLMs) with human preferences, particularly as LLMs increasingly interact with multimodal data. How- ever, we find that MM-RMs trained on existing datasets often struggle to generalize to out-of- distribution data due to their reliance on unimodal spurious correlations, primarily text-only short- cuts within the training distribution, which pre- vents them from leveraging true multimodal re- ward functions. To address this, we introduce a Shortcut-aware MM-RM learning algorithm that mitigates this issue by dynamically reweighting training samples, shifting the distribution toward better multimodal understanding, and reducing dependence on unimodal spurious correlations. Our experiments demonstrate significant improve- ments in generalization, downstream task perfor- mance, and scalability, establishing more robust framework for multimodal reward modeling, Our source code is provided on https: //gi thub. MM-RM RM.
The effectiveness of Multimodal Reward Models (MM- RMs) hinges on their ability to generalize and adapt to diverse inputs, as generalization is critical for ensuring their reliability (Yang al., 2024; Dou al., 2024). an RM fails generalize to out-of-distribution (o.o.d.) during RL training, the policy model may prioritize maximizing re- wards at the expense of aligning with human intent (Gao et al., 2023), potentially leading to reward hacking (Amodei et al., 2016; Pan et al., 2022). Furthermore, poor general- ization undermines the trustworthiness of an R.M on unseen data, hindering test time scaling strategies (Snell et al., 2024; Luo et al., 2024) such as selecting optimal responses from candidate outputs. Therefore, understanding and improving the generalization capabilities of MM-RMs is essential for ensuring their robustness in real-world scenarios.
In this work, we identify a key limitation of Multimodal Reward Models (MM-RMs) trained on existing multimodal preference datasets: their inability to generalize effectively due unimodal spurious correlations. These correlations occur when models over-rely non-robust text-only pat- terns that hold within the training distribution but fail to generalize to distt--dist (o.o.d.) scenarios. To investigate this issue, we conduct cross-distribution transfer experiments across diverse multimodal preference datasets. Our results reveal a significant drop MM-RMs' accuracy on o.o.d. data, while their performance remains stable in i.i.d. settings. To further analyze the impact of text-only shortcuts generalization, we propose the Ahortcut-Failurt Degradation (SFD) metric, which measures the performance drop of MM-RMs when unimodal correlations fail to gen- cralize. Our results demonstrate a critical flaw: Even in multimodal training settings, MM-RMs often default to text-only shortcuts, limiting their ability to learn gener- alizable multimodal reward functions. This suggests that existing datasets and training strategies may not sufficiently encourage true multimodal understanding in MM-RMs.
com/ al wwwnnn///Geeeaaaaaaanee
1. Introduction
Reward Models (RMs) (Stiennon et al., 2020; Ouyang et al., 2022), serving as proxies for human preferences, are crucial in Large Language Models (LLMs) alignment (Achiam al., 2023; Touvron et al., 2023). As LLMs increasingly perceive the world in multimodal ways (Yin et al., 2023), multimodal reward modeling (Sun et al., 2023; Amirloo et al., 2024; Yan et al., 2025) has become essential for addressing challenges like visual hallucinations (Zhou et al.,
This work was conducted during Zichao Li and Xueru Wen's internship at Xiaohongshu. CChinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences Universsity of Chinese Academy of Sciences 'Xiaohongshu Inc. Correspondence to: Jic Lou <loujic0822 @ gmail,com>> Yaojie Lu <luyaojic @iscas.ac.cn..
To address this challenge, we propose a Shortcut-aware MM-RM learning algorithm, designed to mitigate the re- liance on unimodal spurious correlations the presence of biased training data. Building on the discovery of uni-
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models
modal shortcuts, our method introduces a text-only RM shorteut-proxy during training. This proxy explicitly iden- tifies instances where text-only shortcuts fail, enabling dynamic sample reweighting to shift the training distri- bution toward scenarios where multimodal understand- ing is essential. Experiments show that our algorithm signif- icantly improves generalization in cross-distribution transfer evaluations, reducing dependence on unimodal spurious cor- relations. Furthermore, it achieves stronger performance and scalability in downstream test-time tasks, demonstrating its robustness and practical applicability.
comparison accuracy as the RM performance metrie:
Acc
E
I(r(ve q;, (() > r(v; 96 a "]))
(Eten
(2)
2.2. Cross-Distribution Generalization
In the classical machine learning setting, both the training set Stroin and test set Steat are i.i.d. samples drawn from the same environment, i.e., there exists a distribution DD ~ P(X, Y) such that Strain i.id and Stest i.i.d D. To systematically explore the generalization behavior of MM- RMs on unseen but related environments (Arjovsky et al., 2019), we now introduce our cross-distribution framework.
To conclude, our primary contributions are three-fold:
We identify unimodal spurious correlations as a critical challenge in MM-RM development, demonstrating that they undermine MM-RM's generalization on unseen data, even in multimodal training settings.
Specifically, we consider a collection of distinct distribu- tions (D ~ (X", (X*,)")) Y obtained from a set of envi- ronments ε, where e is the the environment index. Despite varying in factors like annotation methods and sampling models, different environments share common objectives: the pursuit of faithful and helpful responses. More formally, each environment satisfies the Invariance Condition (Ahuja et al., 2020) assumption, i.e., there exists representation that is common to all environments, allowing for consistent label prediction across different probability distributions.
A Shortcut-aware MM-RM learning algorithm is pro- posed, which mitigates MM-RMs' reliance on uni- modal spurious correlations by dynamically reweight- ing samples to emphasize multimodal understanding.
Our algorithm achieves generalization improvements across various biased training sets, along with en- hanced downstream performance and scalability, set- ting a more robust paradigm for MM-RM construction.
We thus obtain distinct datasets (St ii.d. DD,, }, where each S can be divided into Strain and Stest- This allows us to study the generalization performance of multimodal reward models under both i.i.d. and o.o.d. scenarios: We restrict the training data to any single environment, denoted as St Train then for model M trained on this dataset, Steet serves as the i.i.d. test scenario, while any Si'st where e' * e constitutes o.o.d. test scenarios. By evaluating M performance variations across these scenarios, we gain insights into its robustness and generalization.
2. Preliminary
2.1. Multimodal Reward Modeling
Throughout our paper, we formalize the problem of mul- timodal reward modeling in a pairwise preference label- ing (Ouyang et al., 2022) paradigm. Consider a prefer- ence dataset S = ((.;, })) where x; can be denoted as x; (v;, q: a! 0, and Vi E [1. Here, v, de- notes an image, residing in the vision space V, and qi. at. -I correspond to query and two distinct answers, and a,
2.3. Unimodal Spurious Correlations
all within a shared language space T. We thereby define : = (4,, of, a, ). The preference label yi identifies the chosen answer index given the query q, and the image 0;-
Previous work attributes machines' poor generalization to spurious correlations, also known as shorteut learning (Si- mon, 1954; Geirhos et al., 2020). While machines excel at learning correlations between inputs and labels, some of these correlations act as shortcuts that hold only within the training distribution, failing to generalize to unseen data.
We consider multimodal reward model M as a pointwise predictor that maps the provided input (consisting of an im- age, a query and an answer) to numerical score, denoted as r: V.L + R. At the training stage, we follow the standard Bradley- Terry Model (Bradley & Terry, 1952) paradigm, optimizing the MM-RM by minimizing the empirical loss of r over the training set Strain as follows:
In this paper, we identify a specific type of spurious correla- tions in MM-RMs, termed unimodal spurious correlations. Following Cadene et al., we use "unimodal" to refer to "text- only". More specifically, unimodal spurious correlations refer to text-only shortcuts learned by MM-RMs on training data, which achieve unexpected success in i.i.d. scenarios but fail in o.o.d. settings (see Section 3.3), thereby hindering MM-RMs' generalization to unseen data (see Section 3.4).
C C=
E (
[og(o qs a.th r(us, q:: az ""))]
(1)
In the evaluation phase. given the test set Stest, we adhere to the typical setting (Lambert al., 2024), using pairwise
To systematically analyze and diagnose unimodal spurious
2

---

The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models
Multimodal RM
Text Text-only RMs
Shortcut-aware MM-RMs
VLF
87 1.
89.5
69.0
Training Set RLHF-V POVID VLF
85.4
48.0
65.0
Training Set REMFAV POVID VLF
86.0
93.3
72.0
RLHF-V Training Set POVID
-00
60.0
100.0
47.8
70
50.5
99.5
35.0
-10
72.4
98.8
63.0
70
-c0
-00
-00
50
67.5
75.0
87.0
58.0
37.0
85.5
77.3
93.3
85.8
VLF
POVID
RLHF-V
VLF
POVID
RLHF-V
VLF
POVID Set
RLHF-V
(a) Standard MM-RM
(b) Text-only RM (Shortcut proxy)
(c) Shortcut-awarg MM-RM (Ours)
Figure 1. Cross- distriiittiittion evaluations of three kinds of RM, where the diagonal elements represent ii.d. tests, while the off-diagonal elements represent o.o.d. tests. (a) Standard MM-RM has significant room for improvement in certain o.o.d. test scenarios. (b) Text-only shortcuts achieve high accuracy under L.d. tests but demonstrate poor generalization in o.o.d. scenarios. (c) Our algorithm demonstrates substantial improvements in generalization, with average accuracy across six o.o.d. scenarios increasing from 68.1 to 78.5.
correlations, we introduce two specialized text-only settings:
3.2. How Well Does MM-RM Generalize?
(1) Text-only Training, i.e., training the reward models after removing the visual modality from the preference datasets, modifying the loss function in Equation (1) as shown below:
We conduct the diotrdiistr tests and present the re- sults in the form of a generalization matrix (see Figure (a)), i... the (e, e') element in the matrix represents the perfor- mance of the model trained on Stain when tested on Si
L. E
(re(.ar)( - re re(gi: ))]]
(BL:B))EStraL
Insight 1: Standard MM-RMs exhibit a marked decline in accuracy on o.o.d. data compared to i.idd scenarios. We first analyze the average accuracy across three i.i.d. test scenarios and six o.o.d. test scenarios (91.4 4 vs 68.1) reveal- ing non-negligible performance gap of 23.2 (see Figure (a)).. Moreover, the training set has significant impact on the robustness of MM-RMs. The MM-RM trained on VLFeedback demonstrates relatively good generalization; however, there are still environments where the model fails to transfer successfully, such as a 18.1 accuracy drop on RLHF-V. POVID represents the least generalizable training data, which perfectly fits i.i.d. data (100.0) but results in below-random accuracy (47.8) in certain o.o.d. scenarios.
(3)
(2) Text-only Test, which involves removing the visual modality during the inference phase on the test set, thereby modifying the accuracy metric in Equation (2) as follows:
Acc, = (#L.
E
((re((saa")) > re(qi,a," 0, ")))
(4)
3. Generalization and Unimodal Spurious Correlations of Multimodal Reward Models
3.1. Setup
Data. We utilize three existing lingunel preference datasets: VI.Feedback (Li et al., 2024b), POVID (Zhou al., 2024), and RLHF-V (Yu et al., 2024), which have been used to construct MM-RMs in previous work (Amirloo et al., 2024). Each dataset is split into training and test sets to represent distribution drift across distinct environments. These datasets focus on vision-related tasks like VQA and image captioning, but differ in construction methods. Addi- tional details of data setup are shown in Appendix A.
We are interested in understanding the performance degra- dation of MM-RMs in o.o.d. scenarios. Although training data size is a consideration, we demonstrate through scaling experiments on VLFeedback that size not the decisive factor, as detailed Appendix In this work, we identify unimodal spurious correlations more subtle yet critical issue inherent in current MM-RMs' construction (see Sec- tion 3.3), which significantly hinders their generalization on unseen data (see Section 3.4).
Model. We primarily utilize Intern VL2-8B (Chen al., 2024b) as the basis for reward model construction. To adapt the Vision-Lanugage Model to MM-RM architecture, we remove the language modeling head i and add an RM head h, which is linear layer that converts the hidden states from the last layer of the LLM decoder into a scalar reward score. Additional details regarding model architecture and training hyperparameters are presented in Appendix A.2.
3.3. MM-RMs Learn to Exploit Text-only Shortcuts
We investigate text-only shortcuts as non-negligible form of spurious correlations in MM-RMs' construction and draw the following insights.
Insight 2: Existing multimodal preference datasets in- evitably exhibit text-only shortcuts, which only hold in
3

---

The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models
Shortcut-Success
100
SFD 14.2
SFD 57.5
55.1
SFD 25.7
(%)
SFD
SFD 45.2
VLFeedback
VLFeedback RLHF-V
VLFeedback
Figure 2. Accuracy and Shortcut-Failure Degradation of MM-RMs in various o.o.d. scenarios. S + S indicates that the MM-RM trained on Strai is tested on Siest. MM-RMs show consistently poor performance when text-only shortcuts become ineffective.
Table Accuracy multimodal reward models and text-only variants Li.d scenarios. Image removal at any stage results only marginal in-distribution accuracy degradation.
Text-only Reward Score
VLFoedback Spearman
Text-only
Dataset POVID
Avg. Drop
Train
Test
VLF
RLHF-V
x
87.1
100.0
87.0
V
85.4
99.5
85.5
1.2
Reward Score
x
86.2
99.8
84.0
1.4
x
82.5
98.8
84.3
2.8
Text-only RM Reward Score
RLHF-V POVID: Spearman Cor, 0.650
-2
their corresponding distributions. Based text-only training and testing settings, derive the generalization matrix of text-only RM (see Figure 1(b)). The text-only model achieves comparable accuracy to the standard MM- RM under i.i.d. conditions across all data distributions. Con- versely, text-only shortcuts fail in o.o.d. scenarios, which indicates that these shortcuts merely serve as spurious cor- relations valid only under specific distributions, rather than the genuine rewarding functions that the model should learn.
Figure 3. The correlation between MM-RM reward scores and text-only RM scores, using two o.o.d. test scenarios as examples.
3.4. Shorteuts Hinder MM-RMs Generalization
To systematically examine the impact of text-only short- cuts on MM-RMs' generalization, we propose the Shortcut- Failure Degradation (SFD) metric, which quantifies the performance drop of MM-RMs when unimodal spurious correlations fail to generalize o.o.d. data.
Insight 3: Even when trained multimodal preference environments, MM-RMs still learn to exploit unimodal spurious correlations. We alternate between multimodal and text-only modes during training and testing, examining reward model's s performance under i.i.d. conditions (see Table 1). Here, we find that models trained on multimodal preference data still achieve comparable in-distribution per- formance during text-only evaluation, indicating the pres- ence of text-only shortcuts in their learned correlations.
Definition 3.1. (Shortcut-Failure Degradation). Given standard MM-RM M trained on dataset Frain we employ the text-only RM M as a proxy for text-only shortcuts, which is initialized with the same model weights and trained on the identical dataset St This text-only RM is then used to split an o.o.d. test set Sy into two parts: one part is successfully classified by the text-only RM, termed the shortcut- success subset, while the other part, where the text-
These insights suggest that the poor generalization of MM- RMs may partially attributed to their acquisition of uni- modal spurious correlations during multimodal training.

---

The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models
Modality
Language Modality
Vision Modality
Language Modality
Image
Query q
Image
Query q
What color is the cat in the image?
What color is the cat in the image?
Chasen Response
Chosen Response
The cat in the imuge is grey and black.
The lying down on the dark surface grey and black.
Rejected Response a
Rejected Response
The cat perched on the window sill orange with white stripes.
The cat in the imuge is orange and white.
Multimodal RM
Text-only RM
Multimodal RM
Text-only
(v. or )= 7.16 (V..9 0=) -8.69
r, 8 ( =
r 01 (v. a-)
T. oz) 1122
8.88
(a) An example of POVID itd. tests. The models learn unimodal spurious correlations between query-irrelevant descriptions and bad responses, rather than developing genuine rewarding of "right wrong" that requires multimodal understanding.
(b) A simulated example of o.o.d. scenarios where we create stress best by slightly modifying two responses. Both higher scores to the chosen response, demonstrating lack of robustness.
text-only fail assign
Figure 4. An empirically observed text-only shortcuts in POVID data, which generates rejected responses by injecting hallucinations into standard answers, often introducing spurious correlations between query-imelevant descriptive elements responses.
Text-only RM (%6)
3.5.
Where Do The Unimodal Spurious Correlations of MM-RMs Arise From?
100
Full Test Set Length-balanced Test Set
Given our statistical findings that MM-RMs tend to learn text-only shortcuts, leading to poor generalization, we seek to deepen our insights of the following question: Where do the unimodal spurious correlations of MM-RMs arise from?
85.4
83.6
85.5
79.4
VLFoedback
RLHF-V
Datasets
3.5.1. SHORTCUT OPPORTUNITIES WITHIN DATASETS
Figure Text-only RMs achieve high accuracy on both full test set and a length-balanced subset under i.i.d. scenarios. Besides length bias, there still remain fine-grained unimodal spurious correlations that can be learned by models.
Data biases (Bahng et al., 2020; Jung et al., 2023), or high- frequency co-occurrences of irrelevant features with labels, create opportunities for spurious correlations (Agrawal et al., 2018; Cadene et al., 2019). reward modeling, length bias (Shen et al., 2023; Chen et al., 2024a) is the most intuitive example: In preference datasets, longer and well-structured responses are often preferred, leading learnable bias that diverges from human expectations.
only RM fails, is termed the shortcut-fail subset. Shortcut- Failure Degradation refers the difference in accuracy achieved by the standard MM-RM between the shortcut- success subset and the shortcut-fail subset.
We first analyze the length characteristics of each train- ing set, where the proportion of chosen responses being longer is 31.5% (POVID), 59.8% (VL8% (VLFeedback), and 67.8% (RLHF-V), respectively. Then, we downsample each test set achieve length parity between chosen and rejected re- sponses, and observe the performance changes of text-only RMs, which serve as proxy for unimodal shortcuts (see Figure 5). We observe slight decrease in the accuracy of text-only RMs, but they still maintain unreasonably high performance, indicating that there still remain fine-grained unimodal spurious correlations that can be learned.
Insight 4: The generalization capability of MM-RMs is heavily constrained by the unimodal spurious correla- tions. The SFD values of MM-RMs range from 14.2 to 57.5 across different average of 39.5 (see Figure 2). This reveals that MM-RMs' reward-scoring process is heavily dominated by text-only shortcuts. When these shortcuts fail to generalize to o.o.d. data, particularly in scenarios requiring true multimodal understanding, the models exhibit substantial performance degradation. We further validate this observation by analyz- ing the correlation between the reward scores of MM-RMs and text-only RMs (see Figure 3).
scenarios, with an
We further consider the fine-grained spurious correlations from case study perspective, using POVID data example (see Figure 4). As POVID generates rejected re- sponses by injecting hallucinations into standard answers,
'We further analyze the implications of SFD Appendix C.
5

---

The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models
Vision Modality
Reweighting sample loss with SFC valuee
Training Stage
X
Shorteut-aware Multimodal RM
Multimodal Loss ((xaa)
Inference Stage
Shorteut-Failure Coefficient (SPC) 4a (r.a)) / (L((x)) (xxa))
Shortcut-aware Multimodal RM
Language Modality
Y
A'
A
Shortcul-prxxy Text-only RM
Text-only Loss (x.y)
X
Figure 6. An overview of our Shorteut-aware MM-RM learning paradigm.
this process often introduces -eree-irrelevant descriptive elements. Consequently, the models capture unimodal spu- rious correlations between query-irrelevat descriptions and bad responses, rather than developing genuine multimodal understanding. Once there are slight perturbations in the response text, the multimodal reward model fails, reflecting its inability to adapt to dynamic real-worle environments.
4. Shortcut-Aware Multimodal RM
Previous sections demonstrate that MM-RMs tend to exploit unimodal spurious correlations during multimodal learning, which hinder their ability to perform generalizable multi- modal reward-scoring. A key insight emerges from our observation that MM-RMs consistently underperform on o.o.d. data where text-only shorteuts become ineffective. This observation indicates that samples where unimodal shortcuts fail could potentially be the cases that most require proper multimodal integration. Building upon this insight, we propose to leverage this relationship as a prior knowl- edge during MM-RM training: By explicitly identifying and focusing on cases where unimodal spurious corre- lations fail, we can encourage more robust multimodal reward modeling. This paradigm shift essentially trans- forms the curse of text-only shortcuts into an opportunity for improvement, forming the cornerstone of our methodology.
While one might consider eliminating such correlations in the data annotation process, we argue this is impractical in practice. Even large-scale, diverse datasets inherently contain numerous shortcuts that may be unobservable, even to humans (Calude& & Longo, 2017; Geirhos et al., 2020). Instead of explicitly modifying the datasets, our work im- plements implicit dynamic debiasing of the data, which applicable to any multimodal preference datasets. The detailed methodology will be presented in Section 4.
3.5.2. FEATURE ACQUISITION UNDER LIMITED REPRESENTATION CAPACITY
4.1. Training with Shortcut-Proxy
As datasets create opportunities for unimodal spurious cor- relations, how these correlations are learned also depends on the model's limited representation capacity (Yang et al., 2022). Under empirical risk minimization, models tend to prioritize learning easily acquired features (Vapnik, 1991: Hermann et al., 2023), such as text-only shortcuts in our con- text. This can be explained by the modality gap (Liang et al., 2022) in multimodal models: a fixed-size image contains vast fine- mne-graiied information (e.g., shapes, textures, and backgrounds), creating a significant information imbalance compared to textual content (Schrodi et al., 2024).
We introduce robust and generalizable algorithm for MM- RM learning. Our goal is to devise a method that can learn generalizable multimodal reward modeling capabilities even when trained on biased datasets, enabling the model to tran- scend dataset-specific unimodal spurious correlations.
The essence of our methodology is to identify and highlight scenarios where text-only shortcuts fall short. To accom- plish this, we propose dual-branch architecture during the training phase (see Figure 6). Each branch employs an identically initialized reward model but varies in its ap- proach to modality: The primary branch M is trained on standard multimodal preference data, acting as our Shortcut- aware MM-RM; the auxiliary branch M, works with pref- erence data with the image modality removed, serving as proxy for text-only shortcuts. To quantify and leverage the disparities between the two branches, we introduce the Chortcut-Failurr Coefficient (SFC) as follows:
To verify this, we conduct investigation experiments using visual patch numbers as proxies for feature acquisition dif- ficulty, with results presented and discussed in Appendix D. While results confirm their correlation with MM-RMs' generalization, we note that simply scaling the visual patch numbers cannot serve as a universal solution for all data see- narios, and there exists a notable performance bottleneck, highlighting the importance of a systematic algorithm to address unimodal spurious correlations.
Coefficient). This metric measures, from a sample-wise perspective, the proportion of loss contributed by the auxiliary branch (shortcut- proxy) to the total training objective, thus indicating the extent to
Definition 4.1. (Shortcut-Faite
6

---

The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models
which unimodal spurious correlations fail capture com- plete preference patterns. Formally, we define SFC as:
RLHF-V- POVID
RLHF-Y- VLFoodback
SFD from 55.1 to 18.2
(a Us) SFC(#; C( yi) + ( V/)
(5)
SFD from 45.2 to 18.5
RLHF-V
where
VLFeedback
POVID
SFD from 57.5 to 25.3
C(æs, wi) log(o (o gi, a.) - r(u;, qi, (k),a;1)) 1))
VLFeedback
SFD from 39.1 to 22.9
-RLHF-V
L, 91) (r) a.) rilgi a
VLFeedback POVIDD
SFD 14.2
Here, (I;; Wi) is sample from the training dataset Strain, [(æ;, Vi) and Lr(a;, mi) represent the sample-wise loss val- ues from the primary branch and text-only branch respec- tively, which are derived from Equations (1) and (3). r(-) and re(+) represent the reward functions the main branch M and shorteut-proxy M, respectively. In practice, detach the gradients involved in the SFC computation, suring that these values only serve as weighting coefficients without participating in backpropagation.
10
20
30
40
50
60
Shortcut-Fallure Degradation
Figure 7. Shorteut-Failure Degradation (SFD) the Shortcut- aware MM-RMs in various o.o.d. scenarios. Compared with standard MM-MMs, Shorteut-aware MM-RMs exhibit consis- tent robustness improvements.
After calculating the SFC values, we can reformulate the loss function of the primary branch into a "shortcut-aware" (sa) form as follows:
While our method explicitly targets text-only shorteuts, the framework's
design allows for broader applicability other forms of spurious correlations. We discuss this further in Appendix F.
Lea
E (SS
[SFC( v.) -( y:3l
Essentially, our shortcut-aware loss function utilizes the SFC value to dynamically reweight samples in training dis- tribution: A sample with higher SFC value indicates that the text-only branch struggles model preferences, suggest- ing that multimodal fusion is crucial for robust learning, thus receiving an increased weight; conversely, a sample with a lower SFC value indicates that the text-only branch can easily discriminate it, leading to a reduced weight. We con- sider this reweighting mechanism as an adaptive method to shift the training data distribution towards environ- ments where multimodal understanding is crucial. This approach helps prevent the MM-RM from becoming overly dependent on text-only shortcuts, which have been shown to significantly hinder the model's generalization. We fur- ther explore the theoretical underpinnings of our proposed method in Appendix E.
5. Experiments
5.1. Cross-Distribution Evaluation
This section demonstrates the generalization improvement of our Shortcut-awars MM-RM algorithm under cross- distribution transfer tests. The main settings remain identical to those introduced in Section 3.1, except that utilize the Shorteut-aware algorithm proposed in Section 4.
As shown in Figure 1(c), compared to standard multimodal RMs, our Shortcut-aware MM-RMs achieve substantial improvements in generalization performance, with av- erage accuracy across six o.o.d. scenarios increasing from 68.1 to 78.5. We observe that this comes with slight trade-off in i.i.d. performance, where the average accuracy marginally decreases from 91.4 to 90.2.
We further analyze the changes in the Shortcut-Failure Degradation (SFD) metric, which measures the accuracy drop of MM-RMs when text-only features fail in o.o.d. scenarios. As shown in Figure 7, our proposed Shortcut- aware algorithm demonstrates robust performance improve- ments across all o.o.d. scenarios, exhibiting substantial and consistent reductions in SFD values compared to standard MM-RMs. This indicates that Shortcut-aware MM-RMs rely less on text-only shortcuts for reward-scoring and achieve more accurate judgment in scenarios where uni- modal spurious correlations fail to generalize.
4.2. Shortcut- Direettt-Dsenttagledd Inference
After completing the model training, we can simply remove the auxiliary branch from the architecture, as it only serves as a proxy for text-only shortcuts during training. During inference, only need to deploy the primary branch, which means the inference process is identical to that of a standard MM-RM, with no additional overhead. However, because the model is trained in a shortcut-aware environment, we demonstrate that it has significantly better generalization and downstream performance (see Section 5).

---

The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models
84
82
48
3.6
MMHalA
46
3.4
76
Shortcut-aware
Shortcut-aware
Standard MM-RR
74
Standard MM4-RM
Standard MM-RM
64
Solutions
Number Solutions
of Solutions
(a) MM-Vet
(b) Llava-bench
(c) MMHal-V
Figure 8. Best-of-N downstream evaluation with N ranging from I to 64, using VLFeedback as the training set. Shortcut-aware MM-RM demonstrates better scalability, while standard MM-RM shows some degree of reward overoptimization (Gao al., 2023). For any N less than 64, we use the unbiased estimator (Nakano al., 2021) to calculate its average score, as detailed in Appendix H.
Model Scale Analysis. To further validate the robustness of our method across different model capacities, we extend experiments to 2B and 4B variants of VL2 As shown in Appendix G, the Shortcut-aware algorithm consistently improves generalization on o.o.d. scenarios across all model sizes, with average accuracy gains of +6.4 (2B) and +6.8 (4B) compared to standard MM-RMs.
Table 2. Best-of-66 64 evaluation across downstream tasks, with GPT- 4o serving the universal judge. The first line refers to the results of IntemnVL2-8B without Best-of-N selection.
Method
MM-Vet
Llava-bench
MMHal-V
43.4
73.3
3.26
MM-RM trained on VLFeedback
Standard
49.0
80.5
3.70
5.2. Downstream Test-time Performance
Shortcut-aware
50.2
84.7
3.74
MM-RM trained POVID
We conduct evaluations on downstream tasks using a Best- of-N (BoN) test-time selection strategy. The process in- volves generating 64 candidate responses from Intern VL.2- VL.2 8B for each image-query pair. MM-RMs are then employed to score these candidates, and the response receiving the highest reward score is selected for downstream evaluation.
Standard
46.7
69.6
3.43
Shortcut-awarg
47.3
73.8
3.71
MM-RM trained on RLHF-V
Standard
39.9
73.1
3.55
Shortcut-aware
44.5
79.6
3.61
To facilitate Best-of-N evaluation, we select three natural QA benchmarks that utilize free-form responses: (1) MM- Vet (Yu et al., 2023b), which comprehensively evaluates VLMs across areas such visual recognition, OCR, knowl- edge integration, and other critical capabilities; (2) LLaVA- Bench-in-the- Wild (Liu et al., 2024), which examines the helpfulness of VLMS in aspects like instruction following and complex reasoning; and (3) MMHal -Bnlch (Sun et al., 2023), which gauges the faithfulness of VLMs by identify- ing hallucinations in their outputs. To ensure consistency and rigor in the assessment process, we employ GPTT40 (Achiam et al., 2023) as the uniform judge across all three benchmarks. Furthermore, following Amirloo et al., we in- corporate images into the MMHal-Bendi evaluation process, creating MMHal-V, to enhance assessment accuracy.
overoptimization (Gao al., 2023), shown in Figure
6. Related Work
Reward Modeling. The development of Large Language Models (Achiam et al., 2023; Bai et al., 2023) heavily relies on Reward Models (Bai et al., 2022), which align mod- els with human values by providing rewards model re- sponses. An RM is first trained on preference data annotated by humans (Ouyang et al., 2022; Touvron et al., 2023) or AI (Lee et al., 2023; Cui et al., 2024), and can then be incorporated into RLHF (Kaufmann et al., 2023) or rejec- tion sampling (Dubey et al., 2024) during training, or used for response selection (Zhang et al., 2024) and decoding guidance (Khanov al., 2024) during inference. Recently, with the rise of Multimodal LLMs (Liu et al., 2024; Team et al., 2024), Multimodal RMs (Sun et al., 2023) have gained increasing attention, with works focusing on both the con- struction (Sun et al., 2023; Wang et al., 2024; Zang et al., 2025) and evaluation (Li et al., 2024a) of MM-RMs.
The experimental results are presented in Table 2. Notably, MM-RMs trained with our Shorteut-aware algorithm demonstrate superior Best-of-64 improvements across all benchmarks. highlighting the algorithm's 's strong gener- alization and real-world application value. We also discover that Shortcut-aware MM-RMs demonstrate better scal- ability and exhibit stronger robustness against reward
Spurious Correlations. Another line of related work fo-
8

---

The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models
cuses on spurious correlations (Simon, 1954; Jackson & Somers, 1991; Calude & Longo, 2017), fundamental challenge that hinders the generalization of intelligent ma- chines. Spurious correlations, also known as shortcut learn- ing (Geirhos et al., 2020; Hermann et al., 2023), occur when certain input features in the training data environments (Ar- jovsky et al., 2019) are highly correlated with task-specific labels but do not represent the intended task objective. For example, in a cow-camel classification task, if cows are only seen grassland and camels in deserts during train- ing, the model may rely on background features, failing when animals appear in new contexts (Asgari et al., 2022). Among studies on spurious correlations, the most relevant to our work is unimodal bias (Cadene et al., 2019; Zhang et al., 2023), which examines how multimodal models learn correlations that depend solely on single modality.
direct negative ethical or social implications.
Acknowledgements
We sincerely thank the reviewers for their insightful com- ments and valuable suggestions. This work was sup- ported by Beijing Natural Science Foundation (L243306), Beijing Municipal Science and Technology Project (Nos. 7231 100010333002), the Natural Science Foundation of China (No. 62306303).
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L.., Almeida, D., Altenschmidr, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
7. Conclusion
Agrawal, A., Batra, D., Parikh, D., and Kembhavi, A. Don't just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. -971-4988 2018.
In this paper, we address a critical challenge of MM-RMs: unimodal spurious correlations that limit their ability to gen- eralize. Our cross-distribution experiments reveal a signifi- cant performance disparity of MM-RMs between i.i.d. and o.o.d. scenarios. Moreover, we find that MM-RMs are able to exploit text-only shorteuts present in multimodal prefer- ence datasets, even when trained in multimodal context, which negatively impacts their generalization. To overcome this limitation, we introduce a Shortcut-aware learning al- gorithm for MM-RMs, which dynamically identifies and emphasizes samples where text-only shortcuts fail, greatly enhancing their generalization and real-world effectiveness.
Ahuja, K., Wang. J., Dhurandhar, A., Shanmugam, K., and Varshney, K. R. Empirical or invariant risk minimiza- tion? a sample complexity perspective. arXiv preprint MiXiv:2010.16411 2020.
Amirloo, E., Fauconnier, J.-P., Roesmann, C., Kerl, C., Boney, R., Qian, Y., Wang, Z., Dehghan, A., Yang, Y., Gan, 7.,, et al. Understanding alignment in mul- timodal Ilms: A comprehensive study. arXiv preprint arXiv:2407.00 024477777777. 2024.
Impact Statement
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schul- man, J., and Mané, D. Concrete problems in ai safety. arXiv preprint arXiv: 1606.0666006666655 2016.
As Multimodal Large Language Models (MLLMs) gain in- creasingly diverse means of perceiving the world, aligning them with human values from a multimodal perspective becomes crucial. Multimodal Reward Models (MM-RMs) play a pivotal role in acting as proxies for human prefer- ences. However, research on the generalization capabili- ties of MM-RMs has been lacking, which poses significant risks. If RMs fail to generalize to out-of-distribution, un- seen data, it could lead to MLLMs producing high-reward outputs that do not align with actual human expectations, a phenomenon known as reward hacking, thereby raising con- cerns about their reliability and safety. By designing a more generalizable MM-RM learning algorithm, our work signifi- cantly enhances the generalization capabilities of MM-RMs and their robustness against unimodal spurious correlations. This advancement promotes the reliability of MM-RMs in real-world scenarios, ensuring that MLLMS better adhere to human values, with increased faithfulness and safety, thus having positive impact on society. Moreover, our proposed method serves as general algorithm and does not pose any
Arjovsky, M., Bottou, L., Gulrajani, I. and Lopez- Paz, D. Invariant risk minimization. arXiv preprint arXiv: arXiv:1907.00 1907.02893, 2019.
Asgari, S., Khani, A., Khani, F., Gholami, A., Tran, L., Mahdavi Amiri, A., and Hamarneh, G. Masktune: Mit- igating spurious correlations by forcing to explore. Ad- vances in Neural Information Processing Systems, 35: 2022.
Bahng, H., Chun, S., Yun. S., Choo, J., and Oh, S. J. Learn- ing de-biased representations with biased representations. In International conference on machine learning, 528- 539. PMLR, 2020.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. Qwen technical report, arXiv preprint ar Xrvv:22309 16609, 2023.

---

The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint orXiv:2204.0586. 2022.
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Bren- del, W., Bethge, M., and Wichmann, F. A. Shorteut learn- ing in deep neural networks. Nature Machine Intelligence, 2(1 1::65-6733 2020.
Hermann, K. L., Mobahi, H., Fel, T., and Mozer, M. C. On the foundations of shortcut learning. arXiv preprint arXiv:2310.16222 2023.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 394344):32--345, 1952.
Jackson, D. A. and Somers, K. M. The spectre of 'spurious' correlations. Occologia, 86: 147-151, 1991.
Cadene, R., Dancette, C.,, Cord, M., Parikh, D., et al. Rubi: Reducing unimodal biases for visual question answering. Advances in neural information processing systems, 32, 2019.
Jung. Y., Shim, H., Yang, J. Y., and Yang. E. Fighting fire with fire: contrastive debiasing without bias-free data via generative bias-transformation. In International Confer- ence on Machine Learning, pp. 15435-15450, PMLR, 2023.
Calude, C. S. and Longo, The deluge of spurious corre- lations in big data. Foundations of science, 22:595-612, 2017.
Kaufmann, T., Weng. P., Bengs, V., and Hüllermeier, E. survey of reinforcement learning from human feedback. arXiv preprint arXiv:2312. 14925, 2023.
Chaudhuri, K., Ahuja, K., Arjovsky, M., and Lopez-Paz, D. Why does throwing away data improve worst-group error? In International Conference on Machine Learning, pp. 4144-4188. PMLR, 2023.
Khanov, M., Burapacheep. J., and Li, Y. Args: Alignment as reward-guided search. arXiv preprint arxiv:2402.01694, 2024.
Chen, L., Zhu, C., Soselia, D., Chen, J., Zhou, T., Goldstein, T., Huang, H., Shoeybi, M., and Catanzaro, B. Odin: Disentangled reward mitigates hacking in rlhf. arXiv preprint vrXiv:242..03199 2024a.
Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403. 13787, 2024.
Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., et al. Expanding per- formance boundaries of open-source multimodal models with model, data, and test-time scaling, arXiv preprint vrxiv:2412.0527/. 2024b.
Lee, H., Phatale, S., Mansoor, H., Lu, K. R., Mesnard, T., Ferret, J., Bishop, C., Hall, E., Carbune, V., and Rastogi, A. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. 2023.
Coste, T., Anwar, U., Kirk, R., and Krueger, D. Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:2310.02743 2023.
Li, L., Wei, Y., Xie, Z., Yang, X., Song, Y., Wang, P., An, Liu, T., Li, S., Lin, B. Y., et al. Vlrewardbench: A challenging benchmark for vision-language generative reward models. arXiv preprint arXiv:2411.17433 2024a.
Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W. Ni, Y., Xie, G., Xie, R., Lin, Y., al. Ultrafeedback: Boosting language models with scaled ai feedback. In Forty-first International Conference Machine Learning, 2024.
Li, L.. Xic, Z., Li, M., Chen, S., Wang, P., Chen, L., Yang, Y., Wang. B., Kong, L.,, and Liu, Q. Vlfeedback: A large- scale ai feedback dataset for large vision-language models alignment. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp- 2227-6246 2024b.
Dou, S., Liu, Y., Zhou, E., Li, T., Jia, H., Xiong, L.,, Zhao, X., Ye, J., Zheng, R., Gui, T., et al. Metarm: Shifted distributions alignment via meta-learning. arXiv preprint arXiv:2405.00438, 2024.
Liang, V. W., Zhang, Y., Kwon, Y., Yeung, S., and Zou, J. Y. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Ad- vances Neural Information Processing Systems, 35: 17612-17625, 2022.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A.. et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. International Conference on Machine Learning, pp. 10835-10866 PMLR, 2023.
Liu, H., Li, C., Wu, Q., and Lee, Y.J. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024.
10

---

The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models
Liu, X., Zhu, Y., Gu, J., Lan, Y., Yang, C., and Qiao, Y. Mm-safetyhench: A benchmark for safety evaluation of multimodal large language models. In European Confer- ence on Computer Vision, pp. 386-403. Springer, 2025.
P. F. Learning summarize with human feedback. Ad- vances in Neural Information Processing Systems, 33: 3008-3021, 2020.
Sun, Z., Shen, S., Cao, S., Liu, H., Li, C., Shen, Y., Gan, C., Gui, L.-Y., Wang, Y.-X., Yang, Y., et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.1452 2023.
Luo, L.,, Liu, Y., Liu, R., Phatale, S., Lara, H., Li, Y., Shu, L., Zhu, Y. Meng, L., Sun, J., ct al. Improve mathemati- cal reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.00 2465:22006.6692,, 2024.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang. S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.05533 2024.
Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C. Jain, S., Kosaraju, V. Saunders, W., et al. Webgpt: Browser-assisted question-ansswrring with hu- man feedback. arXiv preprint arXiv:2112.09332 2021.
Nam, J., Kim, J., Lee, J., and Shin, J. Spread spurious attribute: Improving worst-group accuracy with spurious attribute estimation. arXiv preprint arXiv:2204.02070, 2022.
Touvron, H., Martin, I.., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Ouyang. L., Wu, J.. Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
Vapnik, V. Principles of risk minimization for learning the- ory. Advances neural information processing systems, 4, 1991.
Pan, A., Bhatia, K., and Steinhardt, J. The effects of reward misspecification: Mapping and mitigating misaligned models. arXiv preprint arXiv:2201.0335 2022.
Wang. C., Gan, Y., Huo, Y., Mu, Y., Yang, M., He, Q. Xiao, T., Zhang, C., Liu, T., Du, Q., et al. Rovrm: A ro- bust visual reward model optimized via auxiliary textual preference data. arXiv preprint arXiv:2408 12109, 2024.
Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case gener- alization. arXiv preprint arXiv::9111. 0.0731, 2019.
Yan, S., Bai, M., Chen, W., Zhou, X., Huang, Q. and Li, L. E. Vigor: Improving visual grounding of large vision language models with fine-grained reward modeling. In European Conference on Computer Vision, pp. 37-53, Springer, 2025.
Schrodi, S., Hoffmann, D. T., Argus, M., Fischer, V., and Brox. T. Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive languaee representation learning. arXiv preprint arXiv:2404.000 009933, 2024.
Yang, R., Ding, R., Lin, Y., Zhang, H., and Zhang, T. Reg- ularizing hidden states enables learning generalizable reward model for 1lms. arXiv preprint arXiv:2406. 102116 2024.
Shen, W., Zheng, R., Zhan, W., Zhao, J., Dou, S., Gui, T., Zhang, Q., and Huang, X. Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback. arXiv preprint arXiv:2310.05799 2023.
Yang, W., Kirichenko, P., Goldblum, M., and Wilson, A. G. Chroma-vae: Mitigating shortcut learning with generative classifiers. Advances Neural Information Processing Systems, 35:20351-20365, 2022.
Simon, H. A. Spurious correlation: A causal interpretation. Journal of the American statistical Association, 49(267); 467-479, 1954.
Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., and Chen, E. A survey on multimodal large language models. arXiv preprint arXiv:2306. 13549, 2023.
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling Ilm test- time compute optimally can be more effective than scal- ing model parameters. arXiv preprint arXiv:2408.0333 03314, 2024.
Yu, T., Hu, J., Yao, Y., Zhang, H., Zhao, Y., Wang, C., Wang, S., Pan, Y.. Xue, J., Li, D., et al. Reformulat- ing vision-language foundation models and datasets to- wards universal multimodal assistants. arXiv preprint arXiv:2310.0066 2023a.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano,

---

The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models
Yu, T., Yao, Y., Zhang, H., He, T., Han, Y., Cui, G., Hu, J., Liu, Z.,, Zheng, H.-T., Sun, M., et al. Rlhf-v: To- wards trustworthy milms via behavior alignment from fine- fine-graine grained correctional human feedback. In Proceed- ings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 13807-13816, 2024.
Yu, W., Yang, Z., Li. L., Wang, J., Lin, K., Liu, Z., Wang, X., and Wang, L. Mm-vet: Evaluating large multi- modal models for integrated capabilities. arXiv preprint arXiv:2338.0244 02490, 2023b.
Zang. Y., Dong, X., Zhang, P., Cao, Y., Liu, Z., Ding. S., Wu, S., Ma, Y., Duan, H., Zhang, W., et al. Internim- xcomposer2.5-rewaadde A simple yet effective multi- modal reward model. arXiv preprint 133.2,2501.12368, 2025.
Zhang, L.,, Hosseini, A., Bansal, H., Kazemi, M., Ku- mar, A., and Agarwal, R. Generative verifiers: Re- ward modeling as next-token prediction. arXiv preprint arXiv:2408. 15240, 2024.
Zhang, Y., Latham, P. E., and Saxe, A. A theory of unimodal bias in multimodal learning, arXiv preprint arXiv:23/2.00935, 2023.
Zhao, B., Wu, B., He, M., and Huang, T. Svit: Scaling up vi- sual instruction tuning, arXiv preprint arXiv:2307.04087, 2023.
Zhou, Y., Cui, C.. Yoon, J., Zhang. L., Deng. Z., Finn, C., Bansal, M., and Yao, H. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.00754 2023.
Zhou, Y., Cui, C., Rafailov, R., Finn, C., and Yao, H. Align- ing modalities in vision large language models via prefer- ence fine-tuning. arXiv preprint arXiv:2402.11411 2024.
12

---

The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models
A. Details of Cross-Distribution Setup
This section mainly supplies detailed setup of the cross-distribution experiments.
A.1. Data Setup
In this paper, we instantiate the cross-distribution framework proposed in Section 2.2 as a practical experimental setup through the selection three existing multimodal preference datasets: VLFeedback (Li et al., 2024b), POVID (Zhou et al., 2024), and RLHF-V (Yu et al., 2024). The datasets are carefully partitioned into training and held-out test sets, with strict measures taken to prevent any image overlap between the splits. For the larger VLFeedback dataset, we allocate 1,000 instructions the test set, while POVID and RLHF-V each receive 400 test instructions.
While each preference dataset focuses common Vision-Language tasks, such VQA and image captioning, they derive their instructions from different sources (Yu et al., 2023a; Zhao et al., 2023; Liu et al.. 2024). Moreover, they differ in their data annotation and construction paradigms: VLFeedback utilizes Al-generated preference annotations, POVID is created by deliberately introducing hallucinations into standard responses, and RLHI-V employs human labeling to rectify model response errors. This diversity data sources enhances the representativeness of our experimental findings.
Table 3 summarizes the key characteristics of the three multimodal preference datasets, including their instruction sources, size statistics (instruction and response counts), and annotation methods.
Table Three multimodal preference datasets used in cross distribution experiments.
Dataset Name
Instruction
Instructions Training
Responses
Annotation Paradigm
Source
Test
per Instruction
VLFeedback
SVIT, LLaVA, etc.
79,258
1,000
GPT-4V annotation
POVID
16,336
400
2
Hallucination Injection
RLHF-V
UniMM-Chat
5,265
400
Human refinement
A.2. Model Setup
We primarily utilize Intern VL.2-BB (Chen et al., 2024b) the basis for reward model construction. To adapt the Vision- Lanugage Model an MM-RM architecture, remove the language modeling head I and add an RM head h, which is linear layer that converts the hidden states from the last layer of the LLM decoder into scalar reward score. We implement a dynamic patch mechanism that segments images into up to 6 sub-images for encoding. During the training phase, we froze the vision encoder while only fine-tuning three components: the linear projection module that connects the vision encoder and LLM decoder, the LLM decoder, and the RM head. We train each model variant with its corresponding loss function: standard MM-RM with Equation (1), text-only RM with Equation (3), and our shorteut-aware MM-RM with Equation (7).
We also detail the training hyper-parament which remain constant across our experiments, shown in Table 4.
Table 4. Hyper-parameters used in our RM training.
Hyper-parameter
Value
Learning Rate
le-5
Global Batch Size
256 for VI.Feebaack and POVID, 128 for RLHF-V
Training Epoch
1
Max Input Length
4096
Optimizer
AdamW
Weight Decay
0.05
Scheduler Type
Cosine
Warmup Ratio
0.1
13

---

The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models
B. Investigating the Limited Role of Training Data Size in Generalization
In Section 3.2, we present the generalization performance of standard MM-RMs and observe varying generalization capabilities across models trained on different datasets. In this section, we further discuss the role of training dataset size, demonstrating that it is not the decisive factor in MM-RMs' generalization.
Since VLFeedbacc has the best generalization ability (which also has the largest data scale), we randomly downsample the VLFeedback training data to eliminate its scale advantage in instruction data size compared to the other two training datasets. However, we find that MM-RMs built on the downsampled VLFeedback training data still exhibit better generalization abilities compared to models built on other datasets (see Table 5). This suggests that the size of the training data plays only a limited role the model's generalization ability and is not decisive factor.
We believe that the varying degrees of unimodal spurious correlations in different datasets are more critical reason that prevents MM-RMs from generalizing effectively. This is of the key motivations for introducing the Shortcut-awand algorithm in our paper, which is effective training data of different sizes and with varied unimodal spurious correlations, thereby comprehensively enhancing the generalization ability of MM-RMs.
Table Impact of VLFeedback training data size on the standard MM-RM performance.
Training Set
Test Set
Performance Comparison
Dataset
Instructions
VLFeedback
POVID
RLHF-V
in-distribution
out-of-distributioo
3k
86.3
90.3
67.3
86.3
78.8
VLFeedback
5k
86.4
88.5
68.3
86.4
78.4 ...4
79k
87.1
89.5
69.0
87.1
79.3
POVID
16k
60.0
100.0
47.8
100.0
53.9
RLHF-V
5k
67.5
75.0
87.0
87.0
71.3
C. Further Discussion on Shortcut-Failure Degradation
The Shortcut-Failtue Degradation (SFD) metric serves as a specialized diagnostic tool for analyzing unimodal spurious correlations MM-RMs. By measuring the accuracy gap between thortcut-succes, and shorteut-fail subsets, SFD directly quantifies model's dependence on text-only shorteuts. This makes it particularly valuable for identifying unimodal bias, as it isolates this specific failure mode from other potential robustness issues that might be captured by more general metrics.
Conceptually, SFD aligns with established robustness evaluation paradigms like worst-group accuracy (Sagawa et al., 2019; Nam et al., 2022; Chaudhuri et al., 2023), where the shortcut-fail subset represents our "worst group" in multimodal generalization. Both approaches focus on model performance in challenging scenarios where biased correlations break down. Just as worst-group accuracy reveals models that exploit majority- features, SFD exposes models that over-rely on text shortcuts that may dominate training but fail during deployment.
From practical standpoint, SFD's significance becomes particularly clear in real-world applications. For instance, an MM-RM might learn to favor "long, descriptive captions" as a text shorteut during training, but would fail dramatically when deployed to domains where response quality critically depends visual understanding (e.g., medical imaging or technical diagram interpretation). SFD precisely measures this vulnerability by evaluating performance when text shortcuts become unreliable, making it an essential metric for assessing MM-RM robustness in practical deployment scenarios.
D. Investigating the Impact of Feature Acquisition Difficulty on Generalization
In this section, conduct initial research to explore whether the limited representation capacity causes MM-RMs to prioritize the learning of unimodal spurious correlations under empirical risk minimization (Vapnik, 1991; Yang et al., 2022; Hermann et al., 2023). We refer to the text-only shortcuts "easily acquired features," which stem from the modality gap (Liang et al., 2022): A fixed-size image can encompass an abundance of fine-grained details, including shapes, textures, and backgrounds, leading to a significant information imbalance when compared to the textual content (Schrodi et al., 2024).
Specifically, we conduct scaling experiments by using the maximum number visual patches as an indicator of feature
14

---

The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models
acquisition difficulty, which represents the upper limit of sub-images derived from image segmentation (see Figure 9), A larger number of visual patches enables the reward model to capture fine-grained visual features.
0.8
0.65
Accuracy
Accuracy
0.8
VLFeedback L.Ldd
0.6
VLFeedback o.o.d.
POVID
Shortcut: ThortcutFFaiurr cut-cuu-uuurrre
0.45
VLPeedback -oo.d.
POVID - a.o.d.
0.7
0.5
0.35
RLHFV +00.d.
1+0
1+6 Max Number
1+2
1+4
1+12
1+0
1+2
1+4
1+6
1+12
1+0
1+2
1+4
1+6
1+12
Patch Number
Visual Patch Number
(a) i.idd Accuracy
(b) Average 0.O.d. Accuracy
(c) Shortcut- Faiture Degradation
Figure Impact of number of visual patches standard MM-RM performance.
We observe that: in i.id. test scenarios, the model's performance remains stable regardless of the number of visual patches. In contrast, under o.o.d. test scenarios, we find that providing fine-grained visual features enhances the generalization capability of MM-RMs. Except for models trained on VLFeedback, which already demonstrate relatively strong generalization, we observe an overall increase in accuracy and decrease in SFD values. This suggests that adjusting the accessibility of visual features can help reduce the learning of text-only shortcuts.
However, current results indicate that simply increasing the number of visual patches cannot systematically address the model's robustness issues, the improvement in generalization is not substantial, and the SFD values barely decrease when scaling the number of visual patches from 6 to 12. While future multimodal architectures that provide more accessible visual features could be a promising direction for mitigating unimodal shortcuts, we propose in this paper an effective MM-RM algorithm that applicable to all architectures, as introduced in Section 4.
shortcut-ing
invariant shortcut-invariant
E. Further Discussion on the Proposed Method
In this section, we further discuss how Shortcut-aware MM-RM learning algorithm provides both an information- theoretic justification for dynamic sample weighting and practical connection to robust optimization, ensuring that MM-RMs prioritize learning truly multimodal reward functions.
Information-Thooee Interpretation of SFC. The Shortcut- Failuur Coefficient (SFC) can be understood information- theoretically as quantifying the insufficiency of text-only shortcuts for prediction. Assuming optimized losses approximate conditional entropy terms, we can relate SFC to the underlying information structure:
H(Y TT)
SFC
(8)
C.+ Lm
H(Y + M(YIT,V)
(T) as the normalized contribution of visual information beyond text, derive that SFC 2يح This shows that SFC monotonically increases with p. meaning samples requiring stronger visual grounding receive higher weights. Thus, the Shorteut-Failure Coefficient provides principled foundation for reweighting based on the unique predictive contribution of multimodal information.
Defining p I(VYTT)
Connection to Distributionally Robust Optimization. Our method also shares conceptual connection with Group Distributionally Robust Optimization (Sagawa al., 2019). with each sample representing group without explicit labels. We hypothesize that worst-case group probability depends the degree which samples cannot be predicted using text information alone, aligning with our observations of Shortcut-Failn Failure Degradation.
By setting sample weights w(x,, y1)
SFC(#.
effectively shift the training distribution from P to Q
E x.) SFC [36]
dC where: dP (z,, Ut) x w(x,, y:). These weights positively correlate with the optimal reweighting for minimizing worst-case
15

---

The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models
risk, as they upweight samples where unimodal shortcuts inadequate. This also aligns with empirical observations in Section 5.1, where SFC-based reweighting significantly reduces generalization gaps.
F. Adaption to Other Spurious Correlations
While our method explicitly targets text-only shortcuts, the framework's modality-agnostic design allows for broader applicability to other forms of spurious correlations. To adapt the algorithm, two key modifications are required: proxy adaptation and dynamic reweighting. First, the auxiliary branch can be replaced with a problem-specifie proxy (e.g., an image-only branch) to isolate the target spurious feature. Second, analogous to the Shorteut-Failuu Coefficient (SFC) in Equation 5, failure signals from the new proxy be computed reweight training samples where the proxy struggles, thereby prioritizing cases requiring robust learning.
This flexibility stems from the core principle of our approach: dynamically shifting focus toward samples where spurious correlations fail, regardless their modality. For instance, background biases dominate dataset (e.g., visual QA), training a background-only proxy and reweighting based on failure rate would similarly encourage reliance on complementary information. The reweighting mechanism remains unchanged, ensuring the framework's scalability while preserving its effectiveness in mitigating diverse spurious correlations.
G. Model Size Effects on Cross-Distribution Performance
To investigate the impact of model capacity on generalization, we extend our experiments to 2B, 4B of the base architecture InternVL, as shown in Figure 10. While increasing model scale improves overall performance- absolute accuracy than smaller models we observe that scaling alone does fully resolve unimodal shorteut reliance. For example, in challenging out-of-distribution scenarios (e.g.. POVID RLHF-V), all model scales exhibit poor performance, demonstrating that merely increasing model parameter scale is insufficient completely address generalization issues.
8B achieving higher
Critically, our shortcut-aweed awaree algorithm consistently improves generalization across all model sizes, while demonstrating considerable scalability (see Figure 11). In -fittridditiittt scenarios, the 2B, 4B, and 8B models achieve average accuracy gains of +6.4, +6.8, and + 10.4 respectively, compared to standard MM-RMs.
This demonstrates that our dynamic reweighting method addresses fundamental limitations multimodal align- complecompen (rather than being superseded by) increased capacity. While scaling plays certain role, explicit shortcut mitigation remains essential for robust reward modeling.
Multimodal RM
Multimodal RM
Multimodal RM
100
VLF
VLF
Training Set POVID VLF
Training Set RUHF-V POVID
Training Set RLHF-V POVID
70
100.0
47.8
70
RLHF-V v
50
-50
550
75.0
VLF
POVID Test Set
RLHF-V
VLF
POVID
RLHF-V
VLF
POVID
RLHF-V
Test Set
Test Set
(a) InterVLL2-2B VL2-2B
(b) VL2-4B
(c) Intern VL2-8B
Figure 10. Cross-distribution evaluation of standard MM-RM across different model scales.
H. Best-of-N Unbiased Estimator
Following previous work (Nakano et al., 2021; Coste et al., 2023), we use an unbiased estimator to calculate the average score the Best-of-N policy for any N < 64, where 64 represents the total number of sampled responses. This approach enables more reliable evaluation. Specifically, the estimator computation is as follows:
16

---

The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models
Shortcut-aware MM-RRMs
Scutt-w-aa MM-RMS
Shortcut-aware MM-RMs
VLF
84.5
90.8
71.5
Training Sec RLHF-V FOVID VLF
85.6
93.5
69.3
Training REHF-V POVID VLF
86.0
93.3
72.0
Training Set POVID
-0
57.1
99.8
50.3
70
65.6
99.5
55.3
-00
72.4
98.8
63.0
70
-00
-00
RLHF-V
50
65.1
56.8
84.0
68.3
64.5
79.3
77.3
93.3
85.8
VLF
POVID
RLHF-V
VLF
POVID
RLHF-V
VLF
POVID
RLHF-V
(a) InterVLL2-22 VL2-2B
(b) UnterVL2-45
(c) Intern VL2-8B
Pigure 11, Cross-distrihuttin evaluation of our Shoricut-aware MM-RM across different model scales.
SBest (w, (v,g))
Σ
SGPT-4
arg max r(v, q, a)
64 (N)
15h C. S.
SiN S61
("
Here, r(-) represents the reward function of the evaluated MM-RM; (v, q) denotes vision-related query from downstream benchmark; a represents an answer sampled from the policy model (Intern VI.2-8B (Chen et al., 2024b)); and SGPT-to indicates the score answer a evaluated by the judge GPT-4o (Achiam et al., 2023).
17


Reviews:
[{"review_id": "QplnSxGMh2", "reviewer": "Reviewer_Dzcf", "summary": "The paper proposes to improve the generalization of Multimodal Reward Models (MM-RMs) by addressing the issue of unimodal spurious correlations. It introduces a Shortcut-aware MM-RM learning algorithm that dynamically reweights training samples to emphasize multimodal understanding, reducing reliance on text-only shortcuts. Experiments show the effectiveness in generalization, downstream task performance, and scalability.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of the strengths of our work. We address your concerns as follows.\n\n**C1: It would be more demonstrative if the authors could conduct evaluations on more benchmarks like MMMU and MMStar.**\n\nA1: Thank you for your concern about benchmark evaluation. We clarify that our assessment focuses on MM-RMs' ability to select highest-quality responses generated by a VLM. Unlike traditional multiple-choice benchmarks, we deliberately selected free-response benchmarks (MM-Vet, Llava-bench, and MMHal-V) for a more appropriate evaluation.\n\nNevertheless, your suggestion is insightful. We conducted additional evaluations on VL-RewardBench, a benchmark designed to assess MM-RMs.\n|Method|Hallucination|Reason|General|Overall|\n|-|-|-|-|-|\n|MM-RMs (VLFeedback)|||||\n|Standard|56.9|55.7|41.0|54.2|\n|Shortcut-aware|65.0|61.0|44.8|**61.0**|\n|MM-RMs (POVID)|||||\n|Standard|84.6|56.6|63.9|74.5|\n|Shortcut-aware|85.7|57.9|62.3|**75.2**|\n|MM-RMs (RLHF-V)|||||\n|Standard|65.3|57.2|33.3|58.6|\n|Shortcut-aware|75.8|58.8|48.1|**67.4**|\n\n**C2: For the results on Table 2, have the authors tried to combine these three training datasets to train the MM-RM?**\n\nA2: Thank you for this insightful suggestion. We conducted an additional experiment combining equal samples from the three datasets (15K total) for MM-RM training. The evaluation results demonstrated that training with the mixed dataset yielded performance that fell between the individual datasets: \n|Training Set|MM-Vet|Llava-bench|MMHal-V|\n|-|-|-|-|\n|VLFeedback|49.0|80.5|3.70|\n|POVID|46.7|69.6|3.43|\n|RLHF-V|39.9|73.1|3.55|\n|Mixture|47.3|76.5|3.53|\n\nThis finding suggests that while combined preference datasets provide diverse training signals, they may partially interfere with each other due to differences in methodologies and preference distributions.\n\n**C3: Can the Shortcut-aware learning algorithm be extended to address other forms of spurious correlations beyond text-only shortcuts? If so, what modifications would be required?**\n\nA3: Thank you for raising an important point about the generalizability of our approach. While our experiments focused on text-only shortcuts due to their prevalence in MM-RMs, the framework's design is intentionally modality-agnostic and can be extended to other forms of spurious correlations with two key modifications:\n1. Proxy Adaptation: Replace the text-only proxy with a problem-specific proxy (e.g., image-only) to isolate the target spurious feature.\n2. Dynamic Reweighting: Compute failure signals analogously to Eq. 5 (SFC) for the new proxy, then reweight samples where the proxy struggles. The core mechanism (shifting focus to \"hard\" multimodal cases) remains unchanged.\n\nWe will explicitly discuss this generality in our revisions to highlight the framework's broader applicability.\n\n**C4: It would be helpful to include a comparison with other state-of-the-art methods for improving generalization in multimodal reward models.**\n\nThank you for your valuable suggestion. We clarify that while text-only RM generalization is well-studied, MM-RMs face a critical research gap despite growing importance. Due to space limitations, we follow ICML’s guidelines and recommend that you refer to our response to Reviewer Rc2L (C5).\n\n**C5: How does the method handle scenarios where multimodal preference datasets are limited or unavailable? Are there any alternative strategies to ensure robust generalization in such cases?**\n\nA5: Thank you for raising this practical consideration. We'd like to clarify that: \n* Preference datasets are fundamentally necessary for RM development, not a limitation specific to our method but rather an inherent requirement of the existing paradigm itself. Any approach to developing MM-RMs will require some form of preference data to learn what constitutes high-quality responses.\n* A key strength of our approach is that it improves bias mitigation without requiring additional data beyond what existing methods already use. Without these innovations, addressing generalization issues would demand significantly more labeled data.\n\n**C6: Could we potentially avoid phenomena like text-only shortcuts by better constructing the Preference dataset? Or could this improve the generalization performance of the MM-RMs trained with it?**\n\nA6: Thank you for highlighting dataset quality as another path to improving generalization. While more balanced datasets can help mitigate shortcuts, achieving truly unbiased data demands extensive human verification to eliminate subtle shortcut patterns—an inherently challenging task.\n\nOur solution addresses this without requiring additional data collection or expensive curation. It works with existing datasets to reduce shortcut learning, offering immediate benefits while minimizing costs. Rather than competing with dataset improvements, our method complements them, strengthening any preference dataset—including those refined in the future.\n\nPlease let us know if you have any additional questions.", "claims_and_evidence": "The authors demonstrate through cross-distribution transfer experiments that standard MM-RMs struggle with generalization due to unimodal spurious correlations. They introduce the Shortcut-Failure Degradation (SFD) metric to quantify this issue and show that their proposed algorithm reduces SFD values, indicating better generalization. However, it would be more demonstrative if the authors could conduct evaluations on more benchmarks like MMMU and MMStar.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "The paper does not present any formal theoretical proofs. All equations and definitions are checked.", "experimental_designs_or_analyses": "All experiments are checked. For the results on Table 2, have the authors tried to combine these three training datasets to train the MM-RM? Would combining preference datasets obtained through different construction methods help, or would they interfere with each other?", "relation_to_broader_scientific_literature": "This paper attempts to addresses a critical limitation of existing MM-RMs by proposing a novel learning algorithm that enhances generalization. A more-robust MM-RMs can enhance the alignment of MLLMs.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "- Strengths:  The introduction of the SFD metric provides a new way to diagnose and measure the impact of unimodal spurious correlations.\n- Weaknesses: The paper focuses on a specific type of spurious correlation (text-only shortcuts) and may not address other forms of spurious correlations that could exist in multimodal data. Additionally, the method relies on the availability of multimodal preference datasets, which may not always be practical in real-world applications.", "other_comments_or_suggestions": "It would be helpful to include a comparison with other state-of-the-art methods for improving generalization in multimodal reward models.", "questions_for_authors": "- Can the Shortcut-aware learning algorithm be extended to address other forms of spurious correlations beyond text-only shortcuts? If so, what modifications would be required?\n- How does the method handle scenarios where multimodal preference datasets are limited or unavailable? Are there any alternative strategies to ensure robust generalization in such cases?\n- Could we potentially avoid phenomena like text-only shortcuts by better constructing the Preference dataset? Or could this improve the generalization performance of the MM-RMs trained with it?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yNpnopOVI0", "reviewer": "Reviewer_6qLS", "summary": "This paper reveals that multimodal reward models (MM-RMs) are struggling to address out-of-distribution (OOD) input queries and identify the unimodal spurious correlation (text-only reliance behavior of MM-RMs) as the main cause of this issue. The authors provide some hypotheses behind this issue and propose to leverage this bias to dynamically reweight the per-sample loss based on the expertise of text-only and multimodal reward models. This shortcut-aware multimodal reward modeling induces better OOD generalization of the reward model and improves downstream open-ended question-answering tasks through best-of-N inference protocol.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of the strengths of our work. We address your concerns as follows.\n\n**C1: Although the authors provide rich analysis to strengthen the motivation and rationale behind the method development, one concern is that they only validate their approach with InternVL2-8B. It is not sure that the proposed method can be applied to other model architectures / model sizes.**\n\nA1: We appreciate the reviewer's concern about the broader applicability of our method. To address this point, we have conducted additional experiments on other base models:\n\n1. InternVL2-2B (Model Backbone: InternLM2-Chat-1.8B)\n|Training Set|IID Accuracy ($\\Delta$)| OOD Accuracy ($\\Delta$)|\n|-|-|-|\n|VLFeedback|84.5 (-1.9)|81.1 (+3.5)|\n|POVID|99.8 (-0.3)|53.7 (+8.4)|\n|RLHF-V|84.0 (-1.8)|60.9 (+7.3)|\n\n2. InternVL2-4B (Model Backbone: Phi-3-Mini-128K-Instruct)\n|Training Set|IID Accuracy $\\Delta$| OOD Accuracy $\\Delta$|\n|-|-|-|\n|VLFeedback|85.6 (-1.5)|81.4 (+1.5)|\n|POVID|99.5 (-0.5)|60.4 (+10.1)|\n|RLHF-V|79.3 (-0.6)|66.4 (+8.7)|\n\n$\\Delta$ measures the accuracy differential between Shortcut-aware MM-RM and Standard MM-RM. The results demonstrate that our method generalizes well across different model architectures and parameter scales, confirming that the effectiveness of our approach extends beyond just InternVL2-8B.\n\n**C2: The proposed method hosts an additional large language model branch to model the text-only shortcut bias. This will significantly increase the amount of resources required during reward modeling, and the authors do not provide cost comparison between standard reward modeling and their proposal.**\n\nA2: Thank you for raising this important concern about computational resources. We would like to clarify two key points:\n\n1. The additional text-only branch is only present during the training phase. During inference, this branch is completely removed, meaning our approach has identical computational requirements to standard multimodal reward modeling when deployed in practice.\n2. The additional computational overhead is comparable to what's already widely accepted in language model alignment research (similar to DPO [1] algorithms that also use a reference model). In the context of the overall model development pipeline (especially compared with pre-training stage), this marginal increase is quite modest.\n\n**C3: I would recommend the author to remove the negative sign from equation (6) and equation (7).**\n\nA3: Thank you for your thorough review of our paper—your attention to detail is valuable for improving our work! We apologize for the typo error you identified. We will correct this mistake in future versions of the paper to ensure mathematical consistency with our textual statement.\n\n**C4: It would be great if the authors could further provide some justifications for the proposed metric Shortcut-Failure Degradation (SFD). Is it correlated with other robustness metrics? or is it just for measuring the degree of text-only shortcut bias inside of MM-RMs?**\n\nA4: Thank you for raising this important question about the justification of our SFD metric. We would like to clarify that:\n\n- SFD addresses the specific challenge of unimodal spurious correlations in MM-RMs. By calculating the accuracy difference between shortcut-success samples and shortcut-fail samples, SFD directly quantifies how much a multimodal model relies on text-only shortcuts. This makes it a tailored metric for diagnosing unimodal bias, unlike general robustness metrics that might conflate multiple failure modes.\n- While SFD isn't directly equivalent to existing metrics, it shares conceptual grounding with worst-group accuracy [2][3][4], where the shortcut-fail subset represents our \"worst group\" in the multimodal generalization context. Both metrics prioritize performance on challenging subgroups where biased correlations lead to failures. Just as worst-group accuracy penalizes models exploiting majority-group features, SFD penalizes over-reliance on text shortcuts that may dominate training but fail during test phase.\n\nWe will elaborate on these connections in our revision.\n\n**C5: There have been many attempts to leverage biases to mitigate biases from the main branch network [Bahng et al. 2020; Jung et al. 2023].**\n\nA5: Thank you for pointing out the pioneering papers related to our research direction! We will cite them in the future version of our paper.\n\nPlease let us know if you have any additional questions.\n\n[1] Rafailov, Rafael, et al. \"Direct preference optimization: Your language model is secretly a reward model.\" NeurIPS, 2023.\n\n[2] Sagawa, Shiori, et al. \"Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization.\" ICLR, 2020.\n\n[3] Nam J, Kim J, Lee J, et al. \"Spread Spurious Attribute: Improving Worst-group Accuracy with Spurious Attribute Estimation.\" ICLR, 2022.\n\n[4] Chaudhuri, Kamalika, et al. \"Why does throwing away data improve worst-group error?.\" ICML, 2023.", "claims_and_evidence": "The main claim is that the poor OOD generalization capability of MM-RMs is due to the text-only shortcut, and this claim was validated through systemic empirical evaluation.", "methods_and_evaluation_criteria": "The proposed method and evaluation protocol are both reasonable enough to make a persuasive empirical validation.", "theoretical_claims": "There are no theorems.", "experimental_designs_or_analyses": "The experiments are intuitively designed and valid to investigate the authors' hypotheses, but there is a slight concern about the volume of the experiment (which will be covered in the strengths and weaknesses section).", "relation_to_broader_scientific_literature": "There have been many attempts to leverage biases (by modeling them with an auxiliary model) to mitigate biases from the main branch network [Bahng et al. 2020; Jung et al. 2023]. This work presents a simple yet effective application of this kind of approaches in the multimodal reward modeling regime.\n\n\n- [Bahng et al. 2020] Learning De-biased Representations with Biased Representations\n- [Jung et al. 2023] Fighting Fire with Fire: Contrastive Debiasing without Bias-free Data via Generative Bias-transformation", "essential_references_not_discussed": "The authors properly cited essential references.", "other_strengths_and_weaknesses": "## Strengths\n* The presentation quality is excellent.\n* The main hypothesis -- MM-RMs' poor OOD robustness is due to text-only shortcuts -- is well-supported by extensive intermediate experiments.\n* The authors try to further dive into the reason behind the phenomenon (text-only shortcuts) by providing some conjectures (information imbalance between modalities and so on).\n\n## Weaknesses\n* Although the authors provide rich analysis to strengthen the motivation and rationale behind the method development, one concern is that they only validate their approach with InternVL2-8B. It is not sure that the proposed method can be applied to other model architectures / model sizes.\n* The proposed method hosts an additional large language model branch to model the text-only shortcut bias. This will significantly increase the amount of resources required during reward modeling, and the authors do not provide cost comparison between standard reward modeling and their proposal.", "other_comments_or_suggestions": "* I would recommend the author to remove the negative sign from equation (6) and equation (7). In its current form, the statement \"A sample with a higher SFC value indicates that the text-only branch struggles to model preferences, suggesting that multimodal fusion is crucial for robust learning, thus receiving an increased weight ...\" is wrong because the higher value of SFC means text-only branch produces large preference loss (the logarithm of sigmoid is range from -inf to 0). If the negative signs in Eq (6) and (7) are removed, the above explanation will be correct.\n* It would be great if the authors could further provide some justifications for the proposed metric Shortcut-Failure Degradation (SFD). Is it correlated with other robustness metrics? or is it just for measuring the degree of text-only shortcut bias inside of MM-RMs?\n\n# Post-rebuttal\n> The authors' rebuttal is quite professional and addresses some of my concerns successfully. I will be thinking about editing my initial review and rating after carefully going through other rebuttal contents to other reviewers (but will not require any additional details or raise questions from/to authors). -> After looking through the other reviews and rebuttals, I want to adhere to my current rating -- Weak Accept -- by echoing the concerns raised by `hs7o ` while acknowledging some unique insights provided by this work.", "questions_for_authors": "See the above reviews.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MksQ3kEgka", "reviewer": "Reviewer_hs7o", "summary": "This paper highlights how unimodal spurious correlations reduce generalization in multimodal reward models. In cross-distribution tests, MM-RMs trained on large, seemingly robust datasets still fail to generalize in unseen environments, primarily because they exploit textual cues rather than genuinely integrating vision and language. The paper proposes a “Shortcut-aware” algorithm that dynamically weights training examples where text-only reasoning fails, thereby forcing the model to rely more on multimodal inputs. Empirical results show that this method improves out-of-distribution accuracy while modestly sacrificing in-distribution performance, leading to better overall robustness and downstream task performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of the strengths of our work. We address your concerns as follows.\n\n**C1: While the paper presents a creative fix, it does not introduce new core ML techniques or theory.**\n\n**C2: From a rigorous ML research perspective, it may be viewed more as an engineering improvement rather than breaking fresh ground in ML methodology.**\n\nA1-2: We sincerely thank you for recognizing our method’s creativity and for the thoughtful critique. We appreciate the chance to clarify our contributions.\n\n- We systematically reveal unimodal spurious correlations as a critical bottleneck in MM-RM generalization - a novel insight in multimodal alignment. While prior works address single-modality biases (e.g., text-length bias [1]), we uniquely demonstrate how modality gaps fundamentally drive models toward text-only shortcuts, even in multimodal training paradigms. This discovery exposes a critical blind spot in existing approaches and establishes a new research direction for multimodal robustness.\n- We propose a principled framework anchored in the Shortcut-Failure Coefficient (SFC), which dynamically quantifies modality reliance through dual-branch interaction. Unlike heuristic reweighting methods, SFC leverages intrinsic modality conflicts to identify samples requiring multimodal integration. Our cross-distribution evaluation protocol and Shortcut-Failure Degradation metric further provide standardized tools for diagnosing generalization failures, enabling systematic comparisons in future work.\n- We conduct rigorous experiments to analyze the impact of text-only shortcuts on generalization and demonstrate the real-world effectiveness of our approach. Our method not only achieves significant generalization gains (e.g., +10.4 average OOD accuracy) but also improves performance on real-world downstream tasks (e.g., +6.5 on LLaVA-Bench). These results underscore that our approach is not just a heuristic but a principled framework toward more robust multimodal systems.\n\nWhile building on existing ML components, our work establishes a new paradigm for analyzing and mitigating modality-specific biases - a fundamental challenge as multimodal systems scale. We believe these contributions meaningfully advance the development of robust multimodal alignment.\n\n**C3: There is little formal treatment of how or why the method shifts model capacity away from text-only cues toward genuinely multimodal representations. This lack of theoretical grounding might limit the paper’s appeal.**\n\nA3: Thank you for your interest in the theoretical analysis of our approach. Due to space limitations, we follow ICML’s guidelines and recommend that you refer to our response to Reviewer Rc2L, where we provide a theoretical analysis of how our SFC measure quantifies the insufficiency of text-only shortcuts for prediction from an information-theoretic perspective. Additionally, we analyze how sample reweighting can shift the training distribution to optimize worst-case OOD risk. This theoretical foundation illustrates how our method reallocates model capacity away from text-only cues toward genuinely multimodal representations.\n\n**C4: The problem is demonstrated in the context of MM-RMs for reward modeling. Extensions to tasks like sequence generation, multi-turn conversation, or reinforcement learning from (truly) diverse signals remain unexplored.**\n\nA4: We appreciate the reviewer's concern about the scope of our work. We would like to clarify that while our experiments focus on MM-RMs as the implementation context, the implications and applications of our findings are much broader.\n\nReward models serve as preference modeling tools that guide AI responses across various paradigms. As essential components shaping AI behavior in both training and inference stages, they directly connect to tasks including the sequence generation, conversation, and RL applications the reviewer mentioned:\n1. Our experiments demonstrate how MM-RMs improve model outputs through Best-of-N selection at test time across diverse benchmarks including MM-Vet, Llava-Bench, and MMHal-V. These benchmarks specifically evaluate capabilities in visual recognition, OCR, instruction following, and complex reasoning - all essential components of effective sequence generation and conversational AI.\n2. We have supplemented our initial experiments with additional results showing how our improved MM-RMs enhance policy models when used in RL training loops. The policy models trained with our shortcut-aware MM-RMs demonstrate significant performance improvements across the same benchmarks.\n||MM-Vet|Llava-bench|MMHal-V|\n|-|-|-|-|\n|Policy (InternVL2-8B)|43.4|73.3|3.26|\n|Policy + RL (w/ Standard MM-RM)|43.5|76.3|3.46|\n|Policy + RL (w/ Shortcut-aware MM-RM)|**44.1**|**77.8**|**3.58**|\n\nPlease let us know if you have any additional questions.\n\n[1] Chen, Lichang, et al. \"Odin: Disentangled reward mitigates hacking in rlhf.\" ICML 2024.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n1. The paper identifies a very practical gap in multimodal reward modeling: many so-called multimodal models mainly leverage textual signals, limiting their reliability on unseen data.\n\n2. The proposed dynamic reweighting mechanism, which highlights cases where text-only responses fail, is conceptually straightforward but shows notable gains in generalization, a good example of tackling “shortcut learning” with minimal changes to the overall architecture.\n\nWeaknesses:\n1. While the paper presents a creative fix, it does not introduce new core ML techniques or theory. The reweighting approach, although usefu, mostly builds on known dynamic data debiasing heuristics.\n\n2. There is little formal treatment of how or why the method shifts model capacity away from text-only cues toward genuinely multimodal representations. This lack of theoretical grounding might limit the paper’s appeal.\n\n3. The problem is demonstrated in the context of MM-RMs for reward modeling. Extensions to tasks like sequence generation, multi-turn conversation, or reinforcement learning from (truly) diverse signals remain unexplored.\n\n4. Although the empirical results are valuable, the paper’s novelty is mainly spotting a subtle problem in multimodal RMs and offering a practical fix. From a rigorous ML research perspective, it may be viewed more as an engineering improvement rather than breaking fresh ground in ML methodology.", "other_comments_or_suggestions": "NA", "questions_for_authors": "See Weaknesses Above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "1QjPVOA1lk", "reviewer": "Reviewer_Rc2L", "summary": "The paper addresses the challenge of unimodal spurious correlations in Multimodal Reward Models (MM-RMs), particularly how these models fail to generalize to out-of-distribution (o.o.d.) data.  These spurious correlations occur when models over-rely on text-only features, which hold in the training distribution but fail in o.o.d. settings.  The authors introduce a “Shortcut-aware” MM-RM learning algorithm that dynamically identifies and reweights samples where unimodal shortcuts fail, thus promoting a stronger multimodal understanding.  Experimental results show improvements in generalization and performance across various tasks, suggesting that this approach is effective in reducing the impact of unimodal biases.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of the strengths of our work. We address your concerns as follows.\n\n**C1: More detailed analysis on the theoretical aspects of why the proposed approach outperforms other methods.**\n\n**C2: Justify the use of SFC as a meaningful metric for measuring the failure of unimodal shortcuts.**\n\nA1-2: Thank you for your interest in our theoretical analysis. Below is a more detailed justification:\n- Information-theoretically, SFC quantifies text-only shortcut insufficiency for prediction. SFC = $\\frac{L_t}{L_t + L_m}$ estimates $\\frac{H(Y|T)}{H(Y|T) + H(Y|T,V)}$ assuming optimized losses approximate entropy terms. Defining $\\rho = \\frac{I(V,Y|T)}{H(Y|T)}$ as visual modality's normalized information contribution, we derive: $SFC \\approx \\frac{1}{2-\\rho}$. This establishes SFC as monotonically increasing with $\\rho$, providing principled foundation for reweighting based on visual information's unique contribution.\n- Our method connects with group DRO [1], with each sample representing a group without explicit labels. We hypothesize that worst-case group probability depends on the degree to which samples cannot be predicted using text information alone, aligning with our observations of Shortcut-Failure Degradation. By defining weights as $w(x, y) = \\frac{SFC(x, y)}{\\mathbb{E}{(x, y)} [SFC(x, y)]}$, we shift from $P_{train}$ to $Q$ where $\\frac{dQ}{dP_{train}}(x, y) \\propto w(x, y)$. These weights positively correlate with optimal weights minimizing worst-case risk, reducing generalization gaps while maintaining computation efficiency.\n\nWe will expand this analysis in the future revision.\n\n**C3: While the evaluation metrics like SFD are useful, they might need further justification in terms of how they specifically relate to the real-world applicability of MM-RMs.**\n\nA3: Thank you for this important point about SFD's real-world relevance. SFD quantifies MM-RM performance when text shortcuts fail - a scenario critical for practical applications. Consider a concrete example: an MM-RM might learn to prefer \"long, descriptive captions\" as a text shortcut, but would fail when deployed to a domain where caption quality depends on visual grounding (e.g., medical imaging). SFD measures how severely the MM-RM fails in such scenarios. We'll add more discussion about SFD's practical implications in our next revision.\n\n**C4: The evaluation could also benefit from more rigorous analysis of the trade-offs between in-distribution and out-of-distribution performance.**\n\nA4: We appreciate your insightful comments. Our experiments reveal a nuanced balance: while the proposed algorithm slightly reduces IID accuracy (91.4→90.2), it achieves an absolute gain in OOD scenarios (68.1→78.5)—a critical robustness improvement. This trade-off aligns with our expectations: suppressing text-only shortcuts weakens overfitting to dataset-specific biases, which marginally impacts IID performance but significantly enhances generalization. We will provide more detailed discussion in our revision.\n\n**C5: The paper could be clearer in its discussion of how the proposed method compares quantitatively to other established methods for multimodal model generalization.**\n\nA5: Thank you for your valuable suggestion. We would like to clarify that the research landscape shows an imbalance: text-only RM generalization are well-studied [2] while MM-RMs remain under-explored despite their growing importance. Our work addresses this gap by identifying and mitigating unimodal spurious correlations that significantly impair MM-RM generalization, establishing new understanding in this emerging area.\n\n**C6: Would other debiasing techniques, such as invariant risk minimization, perform similarly in your setup?**\n\nA6: While IRM and related work inspire us, we believe that direct application of IRM would be infeasible in our setup:\n1) IRM requires multiple training environments with explicit distribution shifts. In contrast, real-world RMs are typically trained on a single environment due to practical constraints in data collection [3]. Therefore, our setup limits the training data to a single environment, while our method circumvents this by implicitly defining \"environments\" per sample via SFC.\n2) IRM and similar algorithms involve bi-level optimization with second-order gradients, challenging to optimize for large models like InternVL2-8B due to computational overhead and optimization instability. Our approach instead builds on ERM with sample-wise risk reweighting, ensuring compatibility with existing training frameworks and efficient backpropagation.\n\n[1] Sagawa, Shiori, et al. \"Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization.\" ICLR, 2020.\n\n[2] Yang, Rui, et al. \"Regularizing hidden states enables learning generalizable reward model for llms.\" NeurIPS 2024.\n\n[3] Ouyang, Long, et al. \"Training language models to follow instructions with human feedback.\" NeurIPS 2022.", "claims_and_evidence": "The claims about unimodal spurious correlations limiting MM-RMs’ generalization to o.o.d. data are supported by experimental evidence, particularly the cross-distribution tests.  However, the discussion could benefit from more detailed analysis on the theoretical aspects of why the proposed approach outperforms other methods, especially in terms of sample reweighting.  The evidence is generally convincing, but the connection between the empirical findings and the theoretical foundations could be made clearer.", "methods_and_evaluation_criteria": "The proposed methods, specifically the Shortcut-aware learning algorithm and the use of the Shortcut-Failure Coefficient (SFC), are appropriate for the problem of improving generalization in multimodal models.   The experimental design, including cross-distribution and downstream performance evaluations, is well-suited to test the algorithm’s robustness.   However, while the evaluation metrics like SFD (Shortcut-Failure Degradation) are useful, they might need further justification in terms of how they specifically relate to the real-world applicability of MM-RMs.", "theoretical_claims": "The paper introduces some theoretical concepts like the Shortcut-Failure Coefficient (SFC), but does not provide a full theoretical proof of how this approach leads to better generalization.  This is an area that could use more depth, particularly regarding how the shortcut-aware approach compares to standard training regimes in a formal theoretical context.", "experimental_designs_or_analyses": "The experimental setup appears solid, with a clear focus on cross-distribution and real-world applicability.  The use of multiple datasets (VLFeedback, POVID, RLHF-V) strengthens the claims about generalization.", "relation_to_broader_scientific_literature": "The work is well-grounded in the current literature on reward models and multimodal learning.   The authors effectively position their work within the context of previous research on unimodal biases and generalization in MM-RMs.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "A key strength of the paper is its novelty in tackling unimodal spurious correlations in multimodal models.  The Shortcut-aware algorithm is a promising contribution that addresses a clear gap in the current research.  However, the paper could be clearer in its discussion of how the proposed method compares quantitatively to other established methods for multimodal model generalization.  The evaluation could also benefit from more rigorous analysis of the trade-offs between in-distribution and out-of-distribution performance.", "other_comments_or_suggestions": "The paper is well-written but could improve in terms of theoretical clarity, especially concerning the Shortcut-aware algorithm’s formal underpinnings.  It might also be helpful to include more experimental results on how the algorithm performs across a broader range of real-world scenarios, particularly in domains outside of vision-language models.", "questions_for_authors": "1.  How do you justify the use of the Shortcut-Failure Coefficient (SFC) as a meaningful metric for measuring the failure of unimodal shortcuts?\n2.  Could you provide a more detailed theoretical analysis of how the Shortcut-aware algorithm ensures better generalization in multimodal settings?  Would other debiasing techniques, such as invariant risk minimization, perform similarly in your setup?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Quoting reviewer Dzcf:

"The paper proposes to improve the generalization of Multimodal Reward Models (MM-RMs) by addressing the issue of unimodal spurious correlations. It introduces a Shortcut-aware MM-RM learning algorithm that dynamically reweights training samples to emphasize multimodal understanding, reducing reliance on text-only shortcuts. Experiments show the effectiveness in generalization, downstream task performance, and scalability."

All reviewers are supportive of accepting the paper.
Decision: Accept (poster)