Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approach
Authors: Changdae Oh
Zhen Fang
Shawn Im
Xuefeng Du
Yixuan Li
Venue: ICML 2025 poster (2025)
Keywords: multimodal large language models
distribution shifts
robustness
trustworthy AI
Primary area: deep_learning->large_language_models
TL;DR: We propose the first formal framework for characterizing the performance gap of multimodal LLMs under distribution shifts using information-theoretic metrics.

Abstract: Multimodal large language models (MLLMs) have shown promising capabilities but struggle under distribution shifts, where evaluation data differ from instruction tuning distributions. Although previous works have provided empirical evaluations, we argue that establishing a formal framework that can characterize and quantify the risk of MLLMs is necessary to ensure the safe and reliable application of MLLMs in the real world. By taking an information-theoretic perspective, we propose the first theoretical framework that enables the quantification of the maximum risk of MLLMs under distribution shifts. Central to our framework is the introduction of Effective Mutual Information (EMI), a principled metric that quantifies the relevance between input queries and model responses. We derive an upper bound for the EMI difference between in-distribution (ID) and out-of-distribution (OOD) data, connecting it to visual and textual distributional discrepancies.
Extensive experiments on real benchmark datasets, spanning 61 shift scenarios, empirically validate our theoretical insights.

Understanding Multimodal LLMs Under Distribution Shifts:
An
Approach
Changdae Oh
Zhen Fang²
Shawn Im
Xuefeng Du Yixuan Li
Abstract
robustly in the wild, where distribution shifts occur-that is, when the evaluation data deviates from the instruction tuning data, whether due to changes in visual inputs (e.g, domain- specific images), textual inputs (c.g., linguistic variations), or the combination thereof. However, negative reports on MLLM failures in edge cases have steadily emerged, raising concerns about their reliability.
Multimodal large language models (MLLMs) have shown promising capabilities but struggle under distribution shifts, where evaluation data differ from instruction tuning distributions. Al- though previous works have provided empirical evaluations, we argue that establishing formal framework that can characterize and quantify the risk of MILMS is necessary to ensure the safe and reliable application of MLLMs in the real world. By taking an information -hhooretict perspective, we propose the first theoretical framework that en- ables the characterization of the maximum risk of MLLMs under distribution shifts. Central to our framework is the introduction of Effective Mu- tual Information (EMI), principled metric that quantifies the relevance between input queries and model responses. We then derive an upper bound for the EMI difference between in-distribution (ID) and out-of-distribution (OOD) data, connect- ing it to visual and textual distributional discrep- ancies. Extensive experiments on real benchmark datasets, spanning 61 shift scenarios, empirically validate our theoretical insights.
For example, MLLMs struggle with queries in specialized domains such as medicine and chemistry (Zhang et al., 2024a; Han et al., 2024; Zhou et al., 2024), perform poorly on simple image classification tasks compared to open- ended question answering (Zhang et al., 2024b; Zhai et al., 2024), and exhibit hallucinations biased queries (Li et al., 2023b; Ye-Bin al., 2025). Given the increasing impact of MLLMs, it is crucial to understand their failure modes under distribution shifts. Despite the significance of the problem, existing works often lack a fine-grained diagnosis for various factors of shifts. More importantly, the absence of a formal framework to explain the underlying principle further hinders the systematic understanding of MLLM's behavior. This motivates us raise the research question:
Could we derive a theoretical framework framewwo to characterize MILM's behavior under distribution shifts?
Code: ?
To address this, we propose an information-thhorrticc frame- work that characterizes MLLM performance under distribu- tion shifts with theoretical rigor and practical interpretabil- ity. Our framework well-suited for analyzing instruction- tuned models, wherein the purpose of learning is naturally connected to the mutual information between the input query and response. In this framework, we introduce effective mu- tual information (EMI) as a principled measure to access the relevance between an input query and model response. Intuitively, EMI is expected to be higher when a test input query originates from the i-distribution (ID) data, similar to those used during instruction tuning, compared to when the input comes from out-of-distribution (OOD). To quantify this performance gap, we compute the difference in EMI between ID and OOD and derive an upper bound for this dif- ference, expressed in terms of distributional discrepancies in input and output spaces (see Theorem 4.5 and 4.6). By grounding the measure in information theory, we provide
1. Introduction
Multimodal large language models (MLLMs) have demon- strated remarkable capabilities in handling complex tasks that require reasoning over both visual and textual modal- ities. By leveraging visual instruction tuning (Liu et al., 2023; Dai al., 2023; Zhu al., 2024), MLLMS have shown promise in answering open-ended questions and gen- erating contextually relevant captions. As critical aspect for real-world deployment, MI.LMs are expected to operate
'Department of Computer Sciences, University of Wisconsin- Madison, WI, USA Facculty of Engineering Information Tech- nology, University of Technology Sydney, Sydney, Australia. Cor- respondence to: Yixuan Li <charonli@@ cs. wis..
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Understanding Multimodal LLMs Under Distribution Shifts: An Thforoition-Therrrtiic Approach
the first theoretical framework to analyze and understand the impact of distribution shifts on MILLM performance.
MLLM and visual instruction tuning. MLLM usually consists of three components: (1) visual encoder, (2) vision-to-language projector, and (3) an LLM backbone that processes a multimodal input sequence to generate a valid textual output y in response to an input query x. An MLLM can be regarded as modeling a conditional distribu- tion Po (z((x), where 0 is the model parameters. To attain the multimodal conversation capability, MLLMs commonly undergo phase, so-called visual instruction funing (Liu et al., 2023), with a conditional language modeling loss:
Beyond the theoretical rigor, we further show that our frame- work holds practical value. In particular, we demonstrate that EMI is closely related to a widely used empirical metric for MLLMs, relative preference score under the LLM-as- a-judge paradigm (Zheng et al., 2023). The relative pref- erence score commonly relies on an external judge model, e.g., GPT-4o (Hurst et al., 2024), thus it lacks mathematical guarantees due to the black-box nature of these models. In contrast, EMI provides an grounded measurement of the relevance between input queries and output responses of the MLLM being evaluated, offering a fundamental basis for assessing the MI.MMss performance gap under shifts.
arg min Ex,y PXY EEB
log Po(ui HeU],
(I)
where L is a sequence length and = (,1,,..,y11)) After being trained by Eq. (1), MLLM produces a response given query of any possible tasks represented by text.
Finally, we conduct comprehensive validation for the theo- retical framework and show that our theorems empirically hold in real-world benchmarks. Our experiments compre- hensively examine 34 synthetic and 27 natural distribution shift scenarios, resulting in a total number of 61 ID-OOD ovaluations for each MLLM. Results confirm strong correla- tions between empirical estimates of EMI and relative pref- erence, as well as correlations between the EMI difference and its upper bounds, demonstrating the effectiveness of our framework in capturing performance gaps under diverse shifts. Our contributions can be summarized as follows:
Evaluation of open-ended generations. (M)LLM--sS-a
judge method (Ouyang al., 2022; Zheng et al., 2023; Kim et al., 2024) is commonly adopted to evaluate open- ended generation. In this paradigm, a judge model produces preference scores or rankings for the responses given a query, model responses, and a scoring rubric. Among the evaluation metrics, the relative preference score (RP score; Eq. (2)) is one of the most representative ones.
Definition 2.1 (Relative Preference Score). Given re- ward function r X x y - R. the relative preference (RP) score of model Po w.r.t. Pxx are defined as follows:
We propose new framework, effective mutual infor- mation (EMI), to analyze MLLM under distribution shift, and justify the use of EMI by showing the theo- retical connection between EMI and the LLM-judge- driven relative preference score.
RP(Pxy: Pa) = Expyxxx (
[r(x, (0/r0))r/x)))) v)l-
(2)
Here, the reward function can be any possible MLLMs, such as GPT-4o (Hurst et al., 2024) or discriminative lan- guage models (Lambert et al., 2024), and we often take an output from another MLLM (usually more powerful than Pb) as reference answer y.
We derive theoretical upper bounds of MILM's perfor- mance gap, which can be characterized by shifts over multimodal input queries and output discrepancies.
We empirically verify our theoretical statements on 61 real-world distribution shift scenarios of open-ended question-answering answering benchmarks with six MLLMs.
3. Motivation
A systematic understanding of MILMM under distribu- tional shifts. While instruction-following MLLMs are designed to handle a diverse range of tasks, they often strug- gle with specialized domains (Zhang et al., 2024a; Zhou et al., 2024), perform poorly on simple image classification tasks (Zhai et al., 2024; Zhang et al., 2024b), and hallu- cinate to biased queries (Li et al., 2023b; Ye-Bin al., 2025). We argue that the fundamental cause of these failure modes in MLLMs can be traced back to distribution shifts. Specifically, poor performance on classification tasks and specialized distributions can be attributed to shifts between instruction tuning distribution Pxy and evaluation distribu- tion Qxy This work comprehensively analyzes three types of distributional shifts that can arise in MILMM as follow:
2. Preliminary
Random variable and distribution. Let X' = X, x Xx denote the input space, where X. and X, correspond to the visual and textual feature spaces, respectively. Similarly, let y denote the response space. We define the random variables X (X,, E X' and Y E y, where X is the sequence of tokens that combine visual and text input queries, and Y represents the associated response tokens. The joint distribution is denoted by Pxy, with marginals Px. Py, and the conditional distribution Pylx- For sub- sequent sections, Pxy refers to the instruction tuning data distribution, which we consider as in-distribution (ID).
2

---

Understanding Multimodal LLMs Under Distribution Shifts: An Information Approach
Visual Shift (COCO - Wild)
Text Shift (COCO Eng Non-Eng)
Joint Shift (COCO Eng - Wild Non-Eng)
Relative Preference
LLOVA v1.5 78
Rclative Prefeercnce
LaNA v1.5 78
Relative Preference
LLAGA v1.5
LLOVA v1.5 133
LONk v2.5 138
LLDSA 138
Basy Dop
Nammal Deap
Hewl Dop
German
Chines
1800
OF
Visual Shift (COCO- Wild)
Shift Eng Non Non-Eng)
Joint Shift (COCO Eng
Mon-Eng)
Reletive Preference 0
Reletive Preference
LL
LLvi
LLNV
138
LLaia
Nummal 5330
Arabic
Hiod
Ean
farm
tar
Num
how Meno--
Figure 1. Performance variation against varying distribution shifts. We evaluated LLL v1.5 (top) and I. NeXT (bottom) models on 27 out-of-distribution (OOD) variants of the a COCO (ID). Here, the z-axis is sorted by the severity of shifts between ID and OOD. There is a consistent trend, increased degrees of distribution shifts result in performance degradations of MLLM.
In-dismibution
Visual shift
Test shin
Joint shift
Specifically, we adopt LLa (Liu et al., 2023) and LLaVA-NeXT (Liu et al., 2024a) in 7B and 13B sizes as our target MILM,, with LLa COCO (Liu al., 2023) serving as the ID dataset, which is distributionally similar to the instruction tuning data. We adopt LLaVA-Bench Wild (Liu et al., 2023) to vary visual input semantics, and we apply language translation with GPT-4, e.g., from English to { (Chinese, German, Chinese, Korean, Hindi, Arabic, and Greek], to realize shifts in text query. We vary the severity of shifts by controlling the magnitude of perturbations in synthetic shift setup and partitioning dataset based on the mean embedding distance from ID samples in natural shift setup. Following Liu et al. (2023), we evaluate the performance using RP score (Eq. (2)) with GPT-4 judge.
MONKAT
NEKERT
X
3ST. MONBEZ
Ceozzibe the foll
Plance explast the
InTrIEWa TET
Figure 2. Types of distribution shifts between train and evalu- ation of MLLMs. We simulate visual, text, and joint shifts by controlling the shift of each input modality.
1. Visual shift: the marginal distribution of visual query undergoes shift ((Px. [Qx.) >> 0, while that of text query remains largely unchanged D(Px, (x,) x O.
2. Text shift: the marginal distribution of text query dergoes shift D(px, llQxxx >> O, while that of visual query remains largely unchanged D 1(xx. x O.
Figure/ shows the performance variations of MLLMs under different types and magnitudes of distribution shifts, where the x-axis is sorted by the severity of shifts (more results from different types of shifts can be founded in Appendix C). Across all models, consistent trend emerges: as the severity of the shift increases, the performance degradation becomes more significant. This trend robustly holds for both visual and text shifts. Joint shifts result in greater per- formance degradation, suggesting complementary effect of shifts across modalities. These consistent observations suggest that there might exist underlying principle ex- plaining the relationship between performance variation and distributional discrepancy, which motivates us to inves- tigate the theoretical model behind these empirical results.
3. Joint shift: both visual and text queries suffer shifts simultaneously, and the relationship between visual and text queries may also shift D(Px (xx) > O,
where D denotes a divergence that measures the discrepancy between distributions and Q. For M (P + Q2, one can measure the Kullback-Leibler (KL) divergence and
(JS) divergence as below:
DRL(P = Pex-p[log ((z)/Q(z)), Dis( (II[)) [Dkl(P]|M) (1
Pilot study. We hypothesize that: (1) performance degra- dation in MLLMs becomes more severe Qxy deviates further from the Pxy: (2) the amount of total performance degradation can be factored into visual query shift and text query shift. To test these hypotheses, we design three types of viiftal shift, text shift, and joint illustraat in Figure 2, and evaluate MI.LMMs under these shifts.
Our position. Although there have been similar observa- tions the performance degradation of MLLM under distri- bution shifts (Achiam et al., 2023; Zhang et al., 2024a; Zhou et al., 2024; Zhang et al., 2024b), all of them present only the coarse empirical evaluation results without finer analysis on the underlying factor of those performance degradations. To the best of our knowledge, there no formal framework
3

---

Understanding Multimodal LLMs Under Distribution Shifts: An Inforrition-Theorrii Approach
explain the performance variations of MILMs in terms of distribution shifts- despite its crucial importance for ensur- ing reliable applications of MLLMs. To bridge this gap, we propose the first theoretical framework that characterizes MLLM performance variations under distribution shifts from an information-theore perspective.
Going from instruction tuning to test-time MI. While I(xx) measures the MI between the input query and ground truth response from Pxy, one may be interested in measuring MI between the query and model response on inference-time distributions. We use a tensor product Px & Po to present a joint distribution between the input distribution Px and model oulpul distribution Po(y)x):
4. Inforettion-theoreiic Analysis on MLLM Performance Gap
Px Po Px (o v) E xXxx (5)
Accordingly, the mutual information w.r.t. the joint distribu- tion Px @ Pe can be written as:
In this section, we start with introducing the mutual infor- mation (MT) and its limitation as a metric in Sec, 4.1, and present a new metric on MLLM evaluation (Sec. 4.2). Then, we derive theorems based it to characterize the MLLM performance gap under distribution shifts (Sec. 4.3).
T(Px @ Po)
H(E 1x (Po(-)))) Ex Px (Po((-))
(6)
Limitation of test-time MI under distribution shifts. Al- though one could directly use [[xx @ Po), the mutual in- formation between the input query and the model response, as metric, the vanilla MI may not be suitable for scenar- ios involving distribution shifts. For example, consider the distribution P'x (e.g., general domain), and the distribution Qx (e.g., medical domain). Suppose the MI w.r.t. model Pe on Px, i.e., (Pxx @ Po) is 2.0, while on the Qx, is [[Qx @ Po) 1.0. Does this imply that model P& performs twice as poorly on Qx? ? The answer is unclear.
4.1. Mutual Information for MILM
A fundamental capability of MILMMs is their instruction- following property (Ouyang et al., 2022)-a direct outcome of instruction-tuninn where the model is trained to generate responses that are aligned with the intent of a given input query or instruction. To evaluate instruction-followini ca- pability, we first consider the mutual information (Shannon, 1948) to measure the shared information between the query and the corresponding model response.
The challenge lies in the inherent variability of MI scales across data domains. Recall the formulation of I [(xx @ Po) in Eq. (6). the first term H (Exx~x& [Po(-Ix)) represents the upper bound of MI and varies with the data domain; and the second term Ex (H<()() reflects the input-output dependency and depends on the true data-generating pro- cess. For instance, given fixed vocabulary, the responses from MLLM could contain more diverse words in gen- eral domain, whereas a narrower subset of words could be expected in the specialized domains, such as medical. Then, (Exx [Po (-)))) and Ex~Px (Po ( (x)) would be larger in a general domain. Therefore, we argue that a desired evaluation metric should disentangle the pure input- response relevance from the intrinsic characteristies of the dataset.
Definition 4.1 (Mutual Information (MI)). For a joint distribution Pxy over X V, the mutual information with respect to Pxy is defined as,
I 1(xx)) = Exyx [log Pxx (x. v) P
(3)
Pr(x)
MI is deeply related to the entropy, which is defined as H == -E Ex [log Px (x)). It is easy to check that T(xx)) = (Pv)) Ex (Pyix==))] Intuitively, MI captures how much the response tells us about the query and vice versa. One reason for considering MI for model evalu- ation is that the conditional language modeling objective in Eq. (1) is closely connected to MI as shown in Eq. (4).
4.2. Effective Mutual Information for Reliable MILM Evaluation
cntemmattion-hhooretee interpretation of instruction tun- ing. In Eq. (4), we show that the objective for instruction tuning (Eq. 1) forms a lower bound of the MI between X and Y subtracted by entropy over Y, given sufficiently large representation capacity of the model, i.e., small As we are interested in measuring the input-output dependency, we fo- cus on the 7 (Pxx) term as metric to effectively gauge the upper bound of visual instruction tuning objective, Eq. (1).
To remove the influence of the Nomain-dependent scale, we propose effective mutual information (EMI) as a remedy. Definition 4.2 (Effective Mutual Information (EMI).
Given a joint distribution Pxy and an MILM Po parame- terized with O, the effective mutual information between the input and model response is defined as below,
T(Pxy) - H(Py) =Ex, Pxy [log Pa x + 5
EMI(PxY; Po) = T(Px 2 Po) T(Pxy).
(7)
Ex [log Po(y))),,
(4)
Compared to the standard MI, EMI Po) measures the "effective" relevance between the query x and the model
where 5 Exxix (Pr|X=x] (o(lxxxPPooo))) Po(-
4

---

Understanding Multimodal LLMs Under Distribution Shifts: An Information- Theorettc Approach
response ij by subtracting a ground truth MI T(Px)) ) from T(xx & Po). Refer to Figure 5 in Appendix A, for intuitive example: by accounting for baseline level of MI, EMI quantifies the extent to which the model captures the effective relevance between the input and output. The use of EMI as an evaluation metric for MLLMs can be further supported by (1) its analogy to the excess risk and effective robustness; (2) its connection to an LLM-judge score.
Therefore, (M Pe) and RM(Pxr; Po) exhibit mu- tual equivalence, i.e., increase in PM(Pxy; Po) corre- sponds to the increase in [M Po), and vice versa. In Lemma 4.3, we establish an upper bound for the abso- lute difference between EMI and PM( (M Po), thereby demonstrating their closeness and ultimately highlighting a connection between EMI and LLM-judge score, e.g., Eq (2). Lemma 4.3, Given distribution P'x and MLLM Pe- if Ex [DKL (Po- Pr|x S 6, and let the reward function r r(x, y) be log Py|x-x(U), then
Analogy to the excess risk and effective robustness. The minimum achievable error varies depending on the data- generating process. To enable reliable model selection that is agnostic to data distributions, excess risk-defined as the difference between a model's risk and the minimum possible risk-has been extensively studied (Castro & Nowak, 2008; Koltchinskii, 2010; Mohri, 2018). More recently, Taori al. (2020) introduced the concept of effective robustness to quantify the "effective" OOD generalization accuracy of classification models by subtracting their ID accuracy. The motivation behind EMI aligns with these concepts, i.e., mitigating the influence of external confounding effects that hinder the accurate measure of model performance. EMI ensures that the metrie focuses on the model's ' effective ability to capture input-output relevance, independent of confounding effects from the data domain.
[EM; Po) PM Po)1 S 8 4.48%
Intuitively, Lemma 4.3 shows that if MLLM Po can approx- imate the given distribution PxY with approximate error 8, the difference between EMI and PM can be bounded by small term w.r.t. the approximate error 8. Furthermore, assuming that the model class (Po : V0 E }} has sufficient expressive power (i.e., Eq. (8)),
derive additional bound for the case of the optimal solution of autoregressive objective (i.e., Eq. (1)), as shown below.
Theorem 4.4. Given a distribution Pxy with Pxx > e> for some constant or Py|x > c, if the e-representation capacity assumption holds, i.e.,
min Ex-Px (PyIx sc. (8)
Connection to LLM-judge score. also show that EMI is closely related to an LLM-judge-basee based metric, i.e., RP score in Eq. (2), a common metric used to assess MLLM outputs. Conceptually, EMI quantifies the effective rele- vance between the imput query and the model's response by accounting for the baseline mutual information, whereas the RP score measures the relative preference of the model's re- sponses over a reference response. More formally, their con- nection can be mathematically established through the lens of logit Bradley-Tery preference model (PM) formulation (Bradley & Terry, 1952; Hunter, 2004), logitP(Ú > y/x), an alternative formulation of relative preference. To be spe- eific, we commonly use log Piğ > y/ ( train reward model (RM) which is adopted to compute M-jdee score, such Eq. (2). We compare both terms in the following.
and let the reward function r r(x, v) be log Py Py(x=x(U), then
[EMI(Pxy; Pτ) - PM PT)1 S 5 4.40
where 8 is the optimal solution of Eq. (1) over PxY, and 8 8= 4.4c logg ev/2c
Theorem 4.4 shows that with sufficiently expressive model class, EMI exhibits a stronger alignment with PM when the optimal MLLM parameter 8 is obtained through the autoregressive objective. This alignment underscores the validity of using EMI as a reliable metric for evaluating MLLM and quantifying the relative preference of responses.
confine our analysis to [(xx)) and EMI(Pxy; Po). the chain rule of MI, i.e., I(Pxy) = I(xx,v) + ((Px,yIx.) allows us to further factorize the query-respones relevance into two input modalities, which is suitable for multimodal LLM's -ne-grained evaluation.
Although
PM PMxxx Po) Ex.xxxxy (logit P(ù y|x) (-
= Ex.yxIey [r(x, () r(x, v)], (-Po((
4.3. Characterizing MLLM Performance Gap via Effective Mutual Information Difference
RM(Pxr; Po): Exxxxxxy (log P(ù > y/x) (o(1))
= Exxxxxx (r 1) - r(x, v))], Po(-1M)
Now, based on EMI, we are ready to establish formal guaran- tees on the performance gap of MLLM via effective mutual information difference (EMID). EMID is defined the dif- ference between the EMI on the ID distribution Pxy and the OOD distribution QXy, as follows:
where r(:-) is the latent score function so-called reward model that generates preference for (x, y). It is clear that
PM Pa) RM(Xxy; Pa) - log(1 1 e (MM(Xxy: PP)), RM(Xxy: Pa) = PM( Po) - log(1 + PMC PM(Pxr:PP)).
EMID(PxY, Qxv: Po) = EMI(Pxy: Pa) EMI(QxY; Po).
(9)
5

---

Understanding Multimodal LLMs Under Distribution Shifts: An Informattio-Theorelle Approach
To elucidate the key insight and provide a clear foundation, we begin by analyzing simple scenario where the condi- tional variables remain consistent across both ID and OOD distributions. In this case, we can derive an upper bound for EMID, as stated in Theorem 4.5. This bound enables us to characterize the maximum performance gap of MLLM over two distributions by measuring the severity of the marginal distribution shift over visual and language modalities.
in practice and analyze how such shifts collectively influ- ence the performance gap of MLLMs. The formal upper bound is provided in Theorem 4.6.
Theorem 4.6 (General Scenario), Given PxY and Qxy distributions and an MLLM Po. if there exist some constants 8p and bi such that
(Ps (1pp) S Ep,, Dis(QrolQxy) A = =ôptdQ
Theorem 4.5 (Simplified Scenario). Given an MLLM Pe and distributions Pxy, QXy which have consistent con- ditional distributions over variables X.)X,, Xe|X, vs and Y|X, if there exist some constants öp and óq such that
and denote Pr. = Epx [Po(- and Qy = Bex(P((((x)) then EMID( Qxx: Po) upper bounded by
A (Px. + (Px.ll1)
Das (1l.11lPP))))) [l) S 8r. Das(Qx. S 8q. A 0d
(D (Xx,Ix + (xxxxxxXXxx
and denote Py, = Epx [P)(- and Qy = Exx [P(-)x)], then EMID(Pxx. Qxv; Po) is upper bounded by
PxEx~ Djs (P Qxxx=x + +8A
where f = maxxe [H (QX + ((Po(-)x)) and
[(DiS(xx.11xx)) DJs (Px, (1x))) + + (10)
Dis(Pxxx(1QXIIx) = =E x' DDss( (Px Xxxxx=XXXXXXxxxxxxxx +Ex~ x (Dss (Px |x [x
where A = maxxxxx ( (Q + H(P6(-)))
Implication. Theorem 4.5 implies that in the simpli- fied scenario, EMID depends on two main factors: (1) the divergence between the marginal distribu- tions of the visual and textual inputs; (2) the diver- gence between the model's predictions and the true output distributions, encapsulated by 8p and ōq.
Implication. Compared to Theorem 4.5, Theorem 4.6 indicates that, in the general case, EMID is also influenced by divergences in conditional distri- butions. Specifically, EMID upper bounded by marginal distribution shifts in visual and textual in- puts (X, and X,): divergence between marginal out- put and model response distributions; shifts in condi- tional dependencies (X,, |X, and X, Xx);; and shift between conditional output distributions (Y X)..
Theorem 4.5 naturally captures special cases such as visual- only or text-only imput shifts. For visual-only shift, where (is(Px, [xxx) = 0, the EMID upper bound depends pri- marily on the divergence between visual input distributions, and output discrepancy terms. Similarly, for text-only shift, the bound reflects the divergence in textual input dis- tributions, and output discrepancy terms. These cases not only underscore the flexibility of Theorem 4.5 in isolating the impact of modality-speeiiff distribution shifts on model performance but also highlight the importance of visual and text input shifts on it. In Appendix D, we provide looser yet better interpretable version of this upper bound (Corol- lary 1.1 by replacing A with discrepancy terms between model output and ground truth conditional distributions.
Theorem 4.6 holds for broader cases, whereas Theorem 4.5 is much simpler to analyze. Thus, we focus on the validation of Theorem 4.5 in the following section. If we have some knowledge of the data-generating process of Pxy and QXY, we can choose the one that is suitable for given distributions. Although the bounds in both theorems are not such tight, they help us understand the source of the MILMM performance gap. That is, both Theorem 4.5 and 4.6 provide an analytic tool to characterize the performance gap of MLLM, representing the first formal framework for evaluating MI..M under distribution shifts.
General scenario. Moving beyond the simplified sce- nario, we now consider the general scenario in which no assumptions are made about the consistency of conditional distributions across ID and OOD settings. This more realis- tic scenario accommodates shifts not only in the marginal distributions of visual and textual inputs but also in their con- ditional dependencies and the relationships between inputs and outputs. By relaxing these constraints, we aim to cap- ture the full complexity of distributional shifts encountered
5. Empirical Validation on Real Benchmark
Setup. As done in the pilot experiments (Figure I and 6), we mainly used LLaVA v1.5 (Liu et al., 2024a) and LLaVA NeXT (Liu et al., 2024b) in 7B and 13B sizes and evaluated them on the LLaVA-Bench COCO and LLaVA- Bench Wild (Liu et al., 2023) datasets to assess open-ended generation quality. We also considered two advanced MILMss: QWe-2,5-VL--7B (Bai et al., 2025)
6

---

Understanding Multimodal LLMs Under Distribution Shifts: An Cinommation-Thooree Approach
Synthetic Shift
Natural Shift
Synthetic Shift
Natural Shift
Slope
LLaRA 5 78, 0.067 LLaGA vt.5 135: 0.004
145
320
0.124
220-
EMID
EMID
EMID
Stope Laak v2.5 73:00.20 0.200
ULNA va.3 135 0.250
LLakk MexT 78: 0.240
-366
LLaWA NoxT 1201 0.231
EMID Upper Bound
EMID Upper Bound
21 EMID Upper Bound
EMID Upper Bound
Figure 3. Scatter plot with regression line between empirical estimates of EMID and its upper bound. Over the 34 synthetic and 27 natural distribution shift scenarios, we evaluate four MLLMs and get 136 cases and 108 cases of synthetic shifts and natural shifts, respectively, for visualizing EMID and its scale-adjusted upper bound estimates (See Appendix B for details). The two panels the left show results for all four models, whereas the right ones distinguish them per model with fitted linear regression coefficients (Slope).
Table I. Summary of distribution shift scenarios.
Table 2. Spearman rank correlation and Kendall's tau between relative preference (RP) score and EMI. We conduct correlation analysis between the RP score (Eq. (2)) and EMI (Eq. (7)) on 34 synthetic and 27 natural distribution shifts across four MLLMs.
Type
Strategy (# of category)
Synthetic visual shift
Perturbation (2): Defocus blur, frost
(COCO Images)
Severity (3): Weak, Normal, Strong
Spearman
Kendall
Synthetic text shift
Perturbation (2): Typo, Word Replacement
Model
p-val
p-val
Severity (2): Weak, Strong
Synthetic
LLaVA v1.5 7B
0.794
<0.001
0.604
<0.001
Natural visual shift (Wild Images)
LLa -enhhh Wild or LLa Split (3); Easy, Normal, Hard
LLaVA v1.5 13B
0.652
<0.001
0.483
<0.001
LLaVA NeXT 7B
0.738
<0.001
0.564
<0,001
Natural text shift
Translation (6): GE, CH, KO, EL, AR, HI
LLaVA NeXT 13B
0.726
<0.001
0,527
<0.001
LLaVA v1.5 7B
0.610
0.001
0.450
0.001
Natural
LLaVA v1.5 13B
0.720
<0.001
0.575
<0.001
and InternVL2.55 (Chen et al., 2024), and a domain- specialized
LLaVA NeXT 7B
0.593
0.001
0.435
0.001
question answering dataset, LLaVA- Med (Li et al., 2024), to explore the broad applicability of our framework. For a comprehensive examination on diverse types of shifts, we further simulate synthetic dis- tribution shifts as well as natural distribution shifts. For synthetic shifts, we consider 7 visual scenarios (1 ID case + 2 synthetic perturbation types at 3 severity levels), and 5 text scenarios (1 ID case + 2 synthetic perturbation types at 2 severity levels), resulting in 7 x where scenario ID and the other 34 are OOD cases. For natural shifts, we use visual scenarios (1 ID + OOD diffi culty levels) and 7 text scenarios (1 ID-English + 6 different languages), yielding 4 x 7 = 28 natural scenarios. This comprehensive design covers 34 synthetic and 27 natural shifts spanning 61 shift scenarios in total. A summary of the OOD construction strategies is in Table 1.
LLaVA NeXT 13B
0.457
0.014
0.321
0.017
23 alternative implementations with other estimators derived consistent conclusions (Please refer to Table 7 and 8).
Correlation between RP score and EMI. We first con- duet the Spearman correlation analysis and Kendall's tau analysis between the RP score and our empirical estimates of EMI In Table 2, we can see that EMI estimates exhibit a strong correlation with RP score, both in terms of absolute coefficient and p-value, across all models. This empirical evidence validates the theoretical connection between EMI and RP score discussed in Theorem 4.4. Therefore, our EMI can be used as a reliable and cost-effective alternative to RP for MILM evaluation with theoretical grounding.
35 synthetic scenarios,
Table 3. Pearson correlation analysis between EMID and its upper bound. We provide Pearson r and p-value between the empirical estimates of EMID (Eq. (9) and its upper bound (Eq. (10)) on 34 synthetic and 27 natural shift scenarios per model.
Estimation of MI and JSD. For the empirical realization of our theoretical statements, we adopt a popular neural estimator for MI, CLUB (Cheng et al., 2020) compute empirical EMI and EMID, and a JS divergence estimator, RISD (Hoyos-Oss Osorio & Sanchez-Giraldee 2023), to com- pute empirical estimates of EMID upper bound on top of embeddings of X, Y, and Y, extracted from CLIP-VIT- B/32 (Radford et al., 2021) and XLM-RRBERTaabba (Con- neau, 2019) (see Appendix B for details). Experiments with
Symhhtic
Natural
Model
Pearson
p-val
Pearson
p-val
LLaVA v1.5 7B
0.755
<0.001
0.553
0.003
LLAVA v1.5 13B
0.785
<0.001
0.638
<00.00
LI-VA NeXT 7B
0.742
<0.001
0.594
0.001
LI NeXT 13B
0.807
<0.001
0.550
0.003
models
0.746
<0.001
<0.001

---

Understanding Multimodal LLMs Under Distribution Shifts: An Intormation-Therree Approach
Verification of bound. We now validate our main The- orem. Figure 3 (left two) shows the scatter plots compar- ing EMID with its upper bound across four models, each evaluated over 34 and 27 synthetic and natural distribution shifts. We see a clear trend between HMID and its upper bound in the synthetic shift where we could directly control the severity of shifts. While the result of the natural shift case is noisier, the overall trend is similar. Meanwhile, our bounds depend on the distributional discrepancy between the model's response !/ and the ground truth response y. Thus, they naturally induce different bounds per MI..M. The right panel of Figure 3 presents model-wise plots with linear regression coefficients, where we observe that each model has a different degree of sensitivity against shifts. Pearson correlation analysis results in Table further con- firm statistically significant correlations between EMID and its upper bound, supporting the validity of our theorems.
Table 4. Verification of EMI and EMID bound with advanced MLLMs. We conduct correlation analysis between EMI and RP. and between EMID and its upper bound (UB) with Qwen2.5-VLL VL- 7B (Bai et al., 2025) and Intern AiternVL2.5577 (Chen et al., 2024), and
observe statistically significant correlations.
Model
EMI RP
EMID UB
Spearman (p-val)
Kendall (p-val)
Pearson (p-val)
Qwen2.5-VL VL2.
0.767 (<00.001)
0.571 (<00.01)
0.672 (<00.01) 0.810 (<0.000)))
0.375 (0.029)
0.273 (0.023)
score are somewhat weak in the case of Intern VL2.5, those are still non -on-negliiiele correlations (Schober et al., 2018) with the significance level of 0.05, implying the generality of our framework across different models.
EM analysis on a specialized domain. Since the pro- posed information-theorriic measures rely on embeddings from external models, e.g., CLIP and RoBERTa, to easily compute empirical estimates of MI and JSD, one may won- der whether our framework could be applied to a kind of specialized domain datasets such as medical visual ques- tion answering. To investigate this, we adopt LLaVA-Med instruction dataset (Li et al., 2024) as domain-specific open- open-ended benchmark on the medical images and corre- sponding questions. We demonstrate, in Table 5, that EMI and EMID upper bound show strong correlation with RP score and EMID, respectively, indicating the broad applica- bility of our method in a specialized domain as well.
Partial bound analysis. is common that we can not access the ground truth response Y from our training and evaluation datasets. Then, one may want to use part of EMID upper bound (e.g.. JSD terms on visual and textual input only) as an estimator of EMID, by neglecting the output-related term A. In Figure 4, we investigate whether the summation of two divergence terms can still be predic- tive for EMID. Although the trends become loose compared to the full bound due to the non-optimality of MLLM pa- rameters, the partial upper bound still has moderately high correlations (denoted by Pearson r) with EMID.
Table S. Verification of EMI and EMID bound on LLdVA-Med dataset. We conduct correlation analysis between EMI and RP. and between EMID and its upper bound (UB) with LL v1.5 (7B) on open-ended medical domain QA tasks.
Synthetic Shift
Natural Shift
<1>
<0.555
EMI <> RP
EMID <> UB
EMID
EMID
Spearman P (p-val) Kendall 7 (p-val)
Pearson r (p-val)
0.718 (<0.001)
0.572 (<0.001)
0.930 (<0.001)
Application: EMID upper bound as a regularization. Although our framework mainly stands for the performance analysis of MLLMs under distribution shifts at inference time, one can also leverage EMI and/or EMID upper bound at training time. We give a simple example that instantiates the EMID upper bound as an additional regularization term during visual instruction tuning as below,
JSD Visual+ JSD
JSD JSD
Figure 4. Scatter plot with regression line between empirical estimates of EMID and partial components of its upper bound. We remove the A term of bound (Eq. (10)) and only use the estimates of JSD terms over visual and text inputs.
Validation with advanced MILMs. So far, our eval- uation has focused on LLaVA series MLLMs. Now we validate EMI and EMID upper bound with two state-of- the-art MLLMs, Qwen2.5 VL-7BB (Bai et al., 2025) and Intern VL.2.5-BB (Chen et al., 2024), under synthetic shift scenarios. In Table 4, we see that Qwen2.5-VL. model shows strong correlations between EMI and RP score, and between EMID and its upper bound with remarkable statistical sig- nificance, Although the correlations between EMI and RP
II2) E|H (D) +
DJs((22 (11)
where Z = -Zv, Zz) denotes an intermediate representa- tion of MLLM given X and N is an isotropic Gaussian distribution that has the same dimensionality as Z,, and 2, Here, we utilize a non-informative prior distribution / as an alternative to the target distribution Q (which is usually inaccessible during training) to implicitly penalize repre- sentation discrepancy between and Q. The above term
8

---

Understanding Multimodal LLMs Under Distribution Shifts: An Innemmation-hhorrele Approach
2022), and rationale evaluation (Chen al., 2023), along-
Table Visual instruction tuning with EMID upper bound. We use EMID upper bound as a regularization term (Eq. 11) during instruction tuning LL v1.5 (7B) on a 10% subset LLaVA- mix-665K w/ and w/o R. and report relative preference scores.
side learning objectives (Alemi et al., 2016; Chen al.,
2016; Tschannen et al., 2020; Kong al., 2020; Wang et al., 2021), but relatively unexplored for MLLM yet. We also approaches to analyze models under distribution shifts (Federici et al., 2021; Shui et al., 2022). Although they focused on classifi- cation tasks with discriminative models, we established new theorems for MLLM analysis based on new metric, EMI.
Method
ID
V Shift
T Shift
Shift
note some works adopting information-
Baseline
72.7
65.8
68.0
59.6
Baseline w/ R
72.7
66.3
68.3
60.8
encourages intermediate representations of visual and text inputs to be shrunk into zero-centered Gaussian where the penalizing strength is scaled by an averaged model output entropy across batch samples. Table 6 presents evaluation results of ILLvV v1.5 7B model on LLAVA-Bench COCO (ID) and its synthetic shift variants (V, T, and Shift) after being instruction tuned on LLaVA-mix- -65k subset, where we see our EMID-based regularization effectively improves robustness to shifts while maintaining the ID performance.
7. Discussion
This work urged the development of a formal framework to understand MLLMs under distribution shifts which is unexplored yet crucial for reliable artificial intelligence in the wild. As a first step for this, we devised effective mu- tual information (EMI) and showed its theoretical connection to an existing stan- dard metric, relative preference score. Then, we provide a theoretical upper bound for an MLLM's EMI difference between ID and OOD that consists of JS divergences for marginal and conditional distributions of input/output vari- ables. Through experiments on benchmarks spanning 61 distribution shifts, we show the correlation between relative preference and EMI, and further show correlations between EMI difference and its upper bound, thereby empirically verifying our theoretical claims across various models.
metric for MLLM evaluation
6. Related Works
Fine-tuned foundation models under distribution shifts. Recent findings imply that fine-tuning amount of ID datasets hurts the OOD generalization capa- bility of foundation models (Kumar et al., 2022; Wortsman et al., 2022). Although lots of follow-up studies (Goyal et al., 2023; Tian et al., 2023; Oh et al., 2025) including theory-inspired methods (Kumar et al., 2022; Ju et al., 2022: Oh et al., 2024) have been proposed, almost all of them focused on discriminative model such as CLIP (Radford et al., 2021) for the image classification task. Given the rapidly growing popularity of MLLMs, it is necessary to in- vestigate the reliability of MLLMs under distribution shifts with a tangible formulation. We lay a cornerstone for this.
relatively small
Practical implication. As shown in Table 2, EMI strongly correlates with RP. Compared to RP, the MI estimator can be computed more efficiently without relying on the compu- tationally expensive judge LLM (Achiam et al., 2023) (see Appendix C.4). Thus, FMI can be used as a theoretically grounded, cost-effective evaluation metric that measures the effective relevance between multimodal queries and open- ended responses. Besides, the upper bound of EMID can be adopted as a regularizer during model training, as we showed in Table 6, or test-time adaptation of MLLM to im- prove its robustness to distribution shifts (Li et al., 2023a), as well as a robustness measure of MILMM.
Performance analysis of MILM. There have been merous reports on MLLMs' corner-case behaviors. Zhang et al. (2024a), Zhou et al. (2024), and Verma et al. (2024) observed that MLLMs poorly perform under specialized do- mains or synthetic perturbation, while Zhai et al. (2024) and Zhang et al. (2024b) showed that MLLMS are bad at some simple image classification tasks. Besides, Li et al. (2023b) and Ye-Bin et al. (2025) focused on the object hallucination of MLLM under spurious correlation. However, they all lacked formal framework to explain such degradation of MLLMs. We recast the degeneration of MLLMs via robust- ness under distribution shifts between instruction- -utnin and evaluation data (Liang et al., 2025), and devise the first theoretical framework to analyze MILMSs
Limitation and future work. Although input-output rele- vance measured by EMI is one of the most important prop- erties for instruction -ostuuttionfoolooinng models, other crucial quality attributes are not captured by the form of the relevance term. Extending the theory to support evaluation across multiple facets of MLLM will be worth exploring. In addition, we only simulated some intuitive types of distribution shifts with a simplified assumption for the data structure, leaving the analysis on some complex shifts driven by spurious cor- relation (Simon, 1954) that may be covered by Thm. 4.6. Meanwhile, despite its consistent correlation to EMID, our upper bound is not quite tight in theory, and we did not discuss a lower bound of EMID. Pursuing a tighter bound or exploring the lower limit of EMID can also be worthwhile.
ceermattiontthhrree approach for model evaluation. The information-thhone view has been steadily adopted to establish evaluation criteria for language model probing (Hewitt et al., 2021), prompt engineering (Sorensen al.,

---

Understanding Multimodal LLMs Under Distribution Shifts: An information-Theorciic Approach
Acknowledgement
Castro, R. M. and Nowak, R. D. Minimax bounds for active learning. IEEE Transactions on Information Theory, 54 (5):2339-2353 2008.
We gratefully appreciate the ICML anonymous reviewers for their valuable feedback, and also appreciate construc- tive comments from Max Khanov, Min-Hsuan Ych, Dongk- wan Kim, and Kyungeun Lee. Changdae Oh, Shawn Im, Xuefeng Du, and Yixuan Li are supported by the AFOSR Young Investigator Program under award number FA9550- 23,1-0184, National Science Foundation (NSF) Award No. IIS-2337037 and IIS-2331 2331669, Office of Naval Research under grant number
Chen, H., Brahman, F,, Ren, X., Ji, Y., Choi, Y., and Swayamdipta, S. Rev: informationn-heoretii evaluation free-text rationales. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2007-2030, 2023.
Schmidt Sciences Foundation, and Alfred Sloan Fellowship. Also, Zhen Fang funded by the Australian Government through the Australian Research Council (ARC) under grant number DE250100363.
Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., and Abbeel, P. Infogan: Interpretable representation learning by information maximizing generative adver- sarial nets. Advances in neural information processing systems, 29, 2016.
Chen, Z., Wang. W., Cao. Y., Liu, Y., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., et al. Expanding per- formance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05277 2024.
Impact Statement
This work lays the first theoretical foundation for quantify- ing the reliability of MILMs. The theoretical statements we made shine light on amalyzing the MLLM performance gap under distribution shifts, which commonly emerged in real-world upplications. Our framework can help aware of the potential risk, i.e., performance variation, of ML.Ms, and guide practitioners to devise a method towards robust adaptation for chat assistants, thereby ensuring the trustwor- thiness of artificial intelligence solutions in crucial social applications such as finance and healthcare.
Cheng, P., Hao, W., Dai, S., Liu, J., Gan, Z., and Carin, L. Club: contrastive log-ratio upper bound of mutual information. In International conference on machine learning, pp. 1779-1785 PMLR, 2020.
Cheng, P., Hao, W., Yuan, S., Si, S., and Carin, L. Fair- fil: Contrastive neural debiasing method for pretrained text encoders. In International Conference on Learning Representations, 2021.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I,, Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Conneau, A. Unsupervised cross-lingual representation 2019.
learning scale. arXiv preprint
Cover, T. M. Elements of information theory. John Wiley & Sons, 1999.
Alemi, A. A., Fischer, I., Dillon, J. V., and Murphy, K. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410. 2016.
Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. Instructblip: towards purpose lisgun-lanuuage models with instruction tuning. In Proceedings of the 37th International Con- ference Neural Information Processing Systems, pp. 49250-49267, 2023.
Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song. S., Dang, K., Wang, P., Wang, S., Tang, J., et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Ben- gio, Y., Courville, A., and Hjelm, D. Mutual information neural estimation. In Proceedings of the 35th Interna- tional Conference on Machine Learning, pp. 531-540. PMLR, 2018
Federici, M., Tomioka, R., and Forré, P. An information- theoretic approach to distribution shifts. Advances in Neu- ral Information Processing Systems, 34:17628-17641, 2021.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, (9(3/)):324-33)) 344- 1952.
Fraser, A. M. and Swinney, H. L. Independent coordinates for strange attractors from mutual information. Physical review A, 33(2):1 1134, 1986.
Bu, Y., Zou, S., Liang, Y., and Veeravalli, V. V. Estimation of kl divergence: Optimal minimax rate. JEEE Transactions on Information Theory, 64(4):2648-267) 2674, 2018.
Furuya, T., de Hoop, M. V., and Peyré, G. Transform- ers are universal in-context learners. arXiv preprint arXiv:2408.0/367, 2024.
10

---

Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theorris Approach
Goyal, S., Kumar, A., Garg, S., Kolter, Z., and Raghunathan, A. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19338-19347 2023.
Kim, S., Shin, J., Cho, Y., Jang, J., Longpre, S., Lee, H., Yun, S., Shin, S., Kim, S., Thome, J., et al. Prometheus: Induc- ing fine-grained evaluation capability in language models. In The Tweifth International Conference on Learning Representations, 2024.
Han, Z., Zhou, G., He, R., Wang. J., Wu, T., Yin, Y., Khan, S., Yao, I.., Liu, T., and Zhang, K. How well does gpt-4v (ision) adapt to distribution shifts? a preliminary inves- tigation. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024.
Koltchinskii, V. Rademacher complexities and bounding the excess risk in active learning. The Journal of Machine Learning Research, 1 11:2457-2485, 2010.
Kong, L.,, de Masson d' Autume, C., Yu, L., Ling, W., Dai, Z., and Yogatama, D. A mutual information maximiza- tion perspective of language representation learning. In International Conference on Learning Representations, 2020.
Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., and Choi, Y. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2021.
Kraskov, A., Stögbauer, H., and Grassberger, P. Estimat- ing mutual information. Physical Review E-Statistical, Nonlinear, and Soft Matter Physics, 69(6):066138, 2004.
Hewitt, J., Ethayarajh, K., Liang. P., and Manning, C. D. Conditional probing: measuring usable information be- yond baseline. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1626-1639, 2021.
Kumar, A., Raghunathan, A., Jones, R., Ma. T., and Liang, P. Fine-tuning can distort pretrained features and under- perform out-of-distribution. In International Conference on Learning Representations, 2022.
Hoyos. J. K. and Giraldo, L. G. S. A kernel two-sample test with the representation jensen-shannon divergence. In Latinx in Al @ NeurlPS 2024, 2024. URL https: //openreviiieee
Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint 200:2443..3787, 2024.
Hoyos-Osorio. J. K. and Sanchez-Giraldo, L. G. The rep- resentation censen-shanoor divergence. arXiv preprint arXiv:2305.16446 2023.
Li, C., Wong, C.,, Zhang, S., Usuyama, N., Liu, H., Yang. J., Naumann, T., Poon, H., and Gao, J. Llava-med: Training assistant for biomedicine in
Hunter, D. R. Mm algorithms for generalized bradley-terry models. The annals of statistics, 32(1))384 406, 2004.
a large hinguage-and-viste one day. Advances in Neural Information Processing Systems, 36, 2024.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Li, M., Wang, W., Feng, F., Cao, Y., Zhang, J., and Chua, T.- S. Robust prompt optimization for large language models against distribution shifts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.
Jiang, T., Song, M., Zhang, Z., Huang, H., Deng, W., Sun, F., Zhang, Q. Wang, D., and Zhuang, E5-v: Univer- sal embeddings with multimodal large language models. arXiv preprini arXiv:2407. 000224477./5580 12580, 2024.
2023a.
Li, Y. and Turner, R. E. Rényi divergence variational infer- ence. Advances in neural information processing systems, 29, 2016.
Ju, H., Li, D., and Zhang, H. R. Robust fine-tunin fine-tuning of deep neural networks with hessian-based generalization guar- antees. In Chaudhuri, K., Jegelka, S., Song, L., Szepes- vari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, vol- ume 162 of Proceedings of Machine Learning Research, pp. 10431-10461. PMLR, 17-23 Jul 2022.
Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, X., and Wen, J.-R. Evaluating object hallucination in large vision-language models. In Proceedings the 2023 Conference on Empir- ical Methods in Natural Language Processing, pp. 292- 305, 2023b. URL https: / aaclanhhhlogyy...oo 2023. emnlp main.20.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint 20Xv:20001.0866. 2020.
Liang, Y., Zheng, T., Du, X., Zhang, G., Qu, X., Yue, X., Zheng, C., Liu, J., Ma, L., Chen, W., et al. Align- ing instruction tuning with pre-training. arXiv preprint arXiv:2501.0930 2025.

---

Understanding Multimodal LLMs Under Distribution Shifts: An Information- Approach
Liu, F., Xu, W., Lu, J., Zhang, G., Gretton, A., and Suther- land, D. J. Learning deep kernels for non-parametric two-sample tests. In International conference on machine learning, pp. 6316-666 6326. PMLR, 2020.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Liu, H., Li, C., Wu, Q. and Lee, J. Visual instruction tuning. Conference on Neural Information Processing Systems (NeurIPS), 36, 2023.
Schober, P., Boer, C.,, and Schwarte, L. A. Correlation coefficients: Appropriate use and interpretation. Anes- thesia & Analgesia, 12665):1733-1768, May 2018. doi: 10. 10.113//AEE0000000000002333
Liu, H., Li, C., Li, Y., and Lee, J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296-26306, 2024a.
Shannon, C. E. A mathematical theory of communication. The Bell system technical journal, 27(3):379-423, 1948.
Liu, H., Li, Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024b.
Shui, C., Chen, Q. Wen, J., Zhou, F., Gagné, C., and Wang, B. A novel domain adaptation theory with jensen- shannon divergence. Knowledge-Bord Based Systems, 257: 109808, 2022.
Luo, S., Li, S., Zheng, S., Liu, T.-Y., Wang, L., and He, D. Your transformer may not be as powerful you expect. Advances in Neural Information Processing Systems, 35; 4301-4315, 2022.
Shwartz-Ziv, R. and Tishby, N. Opening the black box of deep neural networks via information. arXiv preprint arXiv: / 703.00810, 2017.
Mohri, M. Foundations of machine learning, 2018
Simon, H. A. Spurious correlation: A causal interpretation.
Nguyen, X., Wainwright, M. J., and Jordan, M. I. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(11):5847-1 1):55)7--88811 5861, 2010.
Journal of the American Statistical Association, 49(267): 467-479, 1954.
Sinn, M. and Rawat, A. Non-parametric estimation of jensen-shannon divergence in generative adversarial net- work training. In International Conference on Artificial Intelligence and Statistics, pp. 642-651. PMLR, 2018.
Oh, C., Kim, M., Lim, H., Park, J., Jeong, F.,, Cheng, Z.-Q.. and Song, K. Towards calibrated robust fine-tuning of vision-language models. Advances in Neural Information Processing Systems, 37, 2024.
Sorensen, T., Robinson, J., Rytting, C., Shaw, A., Rogers, K., Delorey, A., Khalil, M., Fulda, N., and Wingate, D. An information-theune without ground truth labels. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 819-862, 2022.
Oh, C., Li, Y., Song, K., Yun, S., and Han, D. Dawin: Training-free dynamic weight interpolation for robust adaptation. In The Thirteenth International Conference on Learning Representations, 2025.
approach to prompt engineering
Oord, A. v. d., Li, Y., and Vinyals, O. Representation learn- ing with contrastive predictive coding. arXiv preprint arXiv: Kkiv:/807.000 1807.03748 2018
Sreekumar, S. and Goldfeld, Neural estimation of statis- tical divergences. Journal of machine learning research, 23 (3(126):1-75 2022.
Ouyang. L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
Sriperumbudur, B. K., Fukumizu, K., Gretton, A.. Schölkopf, B., and Lanckriet, G. R. G. On the empir- ical estimation of integral probability metrics. Electronic Journal of Statistics, 6: - 1599,
Paninski, L. Estimation of entropy and mutual information. Neural computation, 191-1133, 2003.
Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., and Schmidt, L. Measuring robustness natural distribu- tion shifts in image classification. Advances in Neural Information Processing Systems, 33: 33:18583-18599, 2020.
Pinsker, M. S. Information and information stability of random variables and processes. Holden-Day, 1964.
Poole, B., Ozair, S., Van Den Oord, A., Alemi, A., and Tucker, G. On variational bounds of mutual information. In International Conference on Machine Learning, pp. 5171-5180. PMLR, 2019.
Thekumparampil. K. K., Khetan, A., Lin, Z.. and Oh, S. Robustness of conditional gans to noisy labels. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa- Bianchi N., and Garnett,
12

---

Understanding Multimodal LLMs Under Distribution Shifts: An Information Approach
(eds.), Advances in Neural Information Process- ing Systems, volume 31. Curran Associates, Inc., 2018. URL https://proceedingg neurips. cc htapee/_iillee/ppper//////////// 1eeeaea4413333222eeeeee3333333 pdf.
Zhai, Y., Tong, S., Li, X., Cai, M., Qu, Q., Lee, Y. J., and Ma, Y. Investigating the catastrophic forgetting in multi- modal large language model fine-tuning. In Conference on Parsimony and Learning (Proceedings Track), 2024.
Papper
Paper.
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. Bertscore: Evaluating text generation with bert. In International Conference Learning Representations, 2020.
Tian, J., He, Z., Dai, X., Ma, C.-Y., Liu, Y.-C., and Kira, Z. Trainable projected gradient method for robust fine- tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7836- 7845, 2023.
Zhang, X., Li, J., Chu, W., Hai, J., Xu, R., Yang, Y., Guan, S., Xu, J., and Cui, P. On the out-of-distribution gener- alization of multimodal large language models. arXiv preprint arXiv:240.006399 2024a.
Tschannen, M., Djolonga, J., Rubenstein, P. K., Gelly, S., and Lucic, M. On mutual information maximization for representation learning. In International Conference on Learning Representations, 2020.
Zhang, Y., Unell, A., Wang. X., Ghosh, D., Su, Y.. Schmidt, L., and Yeung-Levy, S. Why are visually-grounded lan- guage models bad at image classification? Conference on Neural Information Processing Systems (NeurIPS), 2024b.
Verma, A. A., Saeidi, A., Hegde, S., Therala, A., Bardoliya, F. D., Machavarapu, N., Ravindhiran, S. A. K., Malyala, S., Chatterjee, A., Yang, Y., et al. Evaluating multimodal large language models across distribution shifts and aug- mentations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5314- 5324, 2024.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang. Y., Lin, Z., Li, Z., Li, D., Xing, E., al. Judging Ilm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595-46615 2023.
Zhou, G., Han, Z., Chen, S., Huang, B., Zhu, L., Khan, S., Gao, X., and Yao, L. Adapting large multimodal models to distribution shifts: The role of in-context learning. arXiv preprint arXiv:2405. 12211,2222222 2024.
Wainwright, M. J. High-dimensional statistics: A non- asymptotic viewpoint, volume 48. Cambridge university press, 2019.
Wang, B., Wang, S., Cheng, Y., Gan, Z., Jia, R., Li, B., and Liu, J. Info[bert): Improving robustness of language models from an information theoretie perspective. In International Conference on Learning Representations, 2021.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. MiniGPT-4: Enhancing vision-languaa understanding with advanced large language models. In The Twelfth International Conference on Learning Representations, 2024.
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7959-7971, 2022.
Yang, and Barron, A. Information-theoree determina- tion of minimax rates of convergence. Annals of Statistics, pp. 1564-1599, 1999.
Ye-Bin, M., Hyeon-Woo N., Choi, W., and Oh, T.-H. Beaf: Observing before-after changes to evaluate hallucination in vasion-language models. In European Conference on Computer Vision, pp. 232-248.. Springer, 2025.
Yuan, W., Neubig, G., and Liu, P. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34:27263-27277 2021.
13

---

Understanding Multimodal LLMs Under Distribution Shifts: An information-Theorriic Theoretic Approach
Appendix
Contents
Detailed Description for Effective Mutual Information
14
Implementation Details
14
Extended Empirical Validation and Discussion
17
C.1 Additional Result from Pilot Study
17
C.2 Different Design Choices of MI and JSD) Estimation
17
C.3 Hyperparameter Sensitivity
18
C.4 Runtime Analysis
19
C.5 Notes on Practical Usage
20
D Extended Theoretical Analysis with Full Proof
21
D.1 Proof for the relationship between EMI and preference model
21
D.2 Proof for EMID upper bound
24
D.3 Discussion
28
A. Detailed Description for Effective Mutual Information
We propose effective mutual information (EMI) as an alternative to vanilla mutual information (MI) to evaluate a model- generated output response given an input query. As explained in Section 4.2, MI (e.g., I(Px @ P&)) can not take into account the intrinsic characteristics of data distribution. See Figure for an intuitive example. The amount of information represented by entropy H (+) and conditional entropy (-I-)) can vary depending on the data-generating process of each dataset. For example, if the task we interested in is closer to solving narrow problem in some specific domain (e.g.. O0D1: LLaVA-Med; Li et al. (2024)), the cardinality of the desired output response space may be significantly smaller than that of general problem-solvine task in a general domain (e.g., OOD2: -enBhhchh Wild; Liu et al. (2023)), and the ground truth MI can differ depending on the domain. By considering these baseline amounts of information, EMI can measure how much our model captures effective relevance between input and output.
In Section 4.2, we provide some justifications for using EMI as an evaluation metric of MILMs by revealing analogies to excess risk and effective robustness and presenting its theoretical connection relative preference score. While LLM-assa- Judge enables flexible evaluation for open-ended generation tasks with multiple user-defined criteria, EMI confines the facet of evaluation query-response relevance. However, compromise in the flexibility of evaluation enables us to build solid theoretical statements that are necessary for understanding MLLMs to shifts and improving them in principled way.
Meanwhile, as we adopt neural network models for empirical estimation of EMI, is somewhat similar to the model-based heuristic metrics, such as BERTscore (Zhang al., 2020), SART (Yuan al., 2021), and CLIPscore (Hessel et al., 2021), that map input(s) to a scalar score through single forward evaluation of the model. However, we take step further beyond the simple working- method and lay theoretical foundation with EMI.
B. Implementation Details
In this paper, we proposed EMI for a reliable evaluation of multimodal large language models (MLLMs) with theoretical ground. Based on EMI, to analyze the MLLM performance gap under distribution shift, we provided the upper bound for EMI difference between ID and OOD data. this section, we describe the procedures for estimating EMI and its upper
14

---

Understanding Multimodal LLMs Under Distribution Shifts: An information-Theorrii Approach
Model
((Px08%)) 2.0
1(Pa)
ROTIO FO:: Y BOrA
0001
HXX,Y)
0.6 1.5
MI difference:
1CP:00%)) (QQ:9F))) K0, 1000-PP))
(P) 0.5
((8++)
O0D
SMI(Ps:: EMI (Gy, EMI(P EMI (M, 1.0
2
2
((08BPs)
1(4gy)
Figure 5. Information diagram and motivation of effective mutual information. The difference between vanilla MI terms does not consider the domain-deppnnent intrinsic scale and mutual information, thereby failing to fairly measure the relevance between input query and model prediction Meanwhile, EMI ablates the domain--ependent characteristic to focus measuring effective relevance between a and ý.
bound in detail.
Overview. To estimate EMI and its upper bound, we first need to define estimators for and Jensen-Shannon divergence (ISD). Those estimators commonly adopt neural network encoders project the raw data such as text and image into embedding space of neural networks reduce problem complexity (Oord et al., 2018; Liu al., 2020), and then, MI estimator commonly optimizes simple critic function (Poole al., 2019) top of the embeddings of data. After training of MI estimator, we evaluate empirical MI over different data distributions. For JSD) estimation, given the embedding spaces of pre-trained models, additional training is not necessary. Therefore, the procedures can be divided into two phases: (1) neural MI estimator training, and (2) inference of MI and JSD.
MI estimation. Estimating MI with finite samples from unknown population distribution is a non-trivial problem, and has been actively studied (Fraser & Swinney, 1986; Paninski, 2003; Kraskov et al., 2004; Nguyen et al., 2010; Shwartz-Ziv & Tishby, 2017; Belghazi et al., 2018; Poole et al., 2019; Cheng et al., 2020). We adopted the contrastive log-ratio upper bound (CLUB; Cheng et al. (2020)) as our default MI estimator similar to (Cheng et al., 2021). We first extract embeddings for visual input query Z, enc,,XX, and text input query Z₂ enc, (X,) from visual and text encoder models and take the mean of them provide input query embedding Zx tu+ Specifically, we adopt the most representative embedding (Conneau, 2019) visual and text encoders, respectively by default. We also obtain the embedding vectors for the model response Zy enc, (Y) and reference response Zy = enc, with text encoder model. Then, we train the MI estimator (-.-) with parameter & via gradient descent. To be specific, CLUB formulates the unbiased estimation for MI below,
models for each modality, i.e., CLIP pre-trained' VIT-B/32 and XLM- ALM-RoBERTa--Bin
N
N
N
(PzxZy)
log go (Zy, (Zx,)
log go (zx,),
(12)
N
N
1=
where go (-1-) denotes variational approximation ground truth probability density function p(-l.).
Following (Cheng et al., 2020; 2021), we parameterize the qu multi-variate Gaussian distribution and estimate the mean and variance parameters of Gaussian with separated two-layer MLPs with 250 hidden dimension size. During mini-batch training, those MLPs consume the concatenated input and response embeddings {[>x:> zy J produce scalar estimate of MI, and they are simultaneously optimized by Adam optimizer with learning rate 0.001 and batch size 1,024 for 5,000 iterations. However, if we have to train an estimator for every ID-OOD data pair, it may not be practical when the number
'https://gt thub. com http:::://huggingffee.e http://///aaaaaaaa...ococceccoo/
wwt.c/cwa/owpocccttootcoocoa.aaa openai /CLIP.
15

---

Understanding Multimodal LLMs Under Distribution Shifts: An Thhorri Approach
of data pairs to be evaluated is large. Therefore, we constructed a dataset that integrates all ID-OOD data subsets for MI training (integration of all variants of LLL datasets reach roughly 5,000 samples for natural shift, and 9,000 samples for synthetic shift), trains it only once, and then infers all ID-OOD scenarios (27 for natural shift, 34 for synthetic shift) using these common MI estimators. This not only significantly reduces the time required to evaluate the model's s robustness against multiple distribution shift scenarios, but also stabilizes the training process by increasing the size of the data set used in the training process.
JS divergence estimation. Estimation of distribution divergences from finite samples has been also a central topic of research (Y (Yang & Barron, 1999; Sriperumbudur et al., 2012; Li & Turner, 2016; Bu et al., 2018; Sinn & Rawat, 2018 8;; Sreekumar& & Goldfeld, 2022: Hoyos- Hoyos-Osorio & Sanchez-Giraldo, 2023). We adopt the most recent one, representation (2023); Hoyos Giraldo (2024)), which proves its effectiveness on real benchmark datasets as our JSD estimator. The formula is as follows:
Jensen-Shannon divergence (RJSD; Hoyos-Osone Osorio &
DIUDD ( (P,Q) (.Q)=5(( Cp + Co 1 (S(CP)) +
(13)
2
2
where Cr- = (xx) @ (()) and S(Cp) = - Trace log Cr). Similar to MI, we compute DRISD (P, Q) in the embedding space of the same frozen pre-trained models, i.c., leverage neural network embedding space as a kernel < ((x), o(x') >. In contrast to the case of MI, RJSD with frozen neural embedding model does not require additional training. Still, one might consider learning the embedding model from scratch if necessary.
Scale-adjusted EMID upper bound construction. By leveraging the estimator described above, one can compute all JSD terms in the proposed EMID upper bound (Eq. 10): Prs|lxxl|Qxx Das (xx, (1x,), Das (Py,llPy),, ), and (Or, [1Qx)) )). Because the exact computation of the entropy scaler term H in Eq. (10) is intractable, we relax it with an estimate on batch samples, and replace the inaccessible true conditional distribution Qyix with Pg. For a case of sentence output Y ,y., 3 with L tokens, the length-normaliee batch entropy estimate over N samples indexed through i e I {1, [l, N) can be formulated as below,
L
A = max- RE
log Po(yn.lb...,....
(14)
In pilot experiments, we observed that the values of H are centered on 2.0 in the datasets that we considered in this work. Therefore, = maxxex [ [Qy]x=x) (P)(())) can approximately have a value 4.0. As the main implications of
EMID upper bound are providing the characterization of MILMM performance gaps and serving as a practical measure of robustness, we compute the scale-adjusted upper bound estimates for EMID upper bound (UB) as below,
EMID UB = (0x
4PD] Djs (Qr. (rr [1Qr))
= ((D 11xx) + (Px | Qx + D (Py, PP) + D (
Scale-adjusted UB = D (Px., Qx.)
D RISD (Px, Qx.) + D (Py, Py) D (Qya Qy).
See Section D for the detailed derivation of EMID UB. Note that applying the linear transformation to the target variables of Pearson correlation analysis does not affect the correlation coefficient value, so that the correlation coefficients between the EMID and its scale-adjusted UB has the same values as between the EMID and its original UB, even though it is not the exact estimate of EMID UB.
MILM judge and relative preference (RP) score. For evaluation of open-ended generation tasks, (M)LLM-as-a-Judee has been adopted current de facto standard. Following (Liu al., 2023; 2024a), we use GPT-43 with text-only inference mode (with plain-text form visual cue such as ground truth caption for image) as a judge model and also use the output of the same model as reference answer for each query. We leverage the prompts provided by the source code of LLa, and compute the RP score of a model of interest by comparing its output with that of the reference answer.
"gpt-4 4gpt-4-turbe with 2024-08-01-pprr "htip://githuh... "http///gihuu..cm//haniaaaaa/ hwtphh/ww....aaaaiiiiillaaaaaaa.
ew API version was mainly adopted
16

---

Understanding Multimodal LLMs Under Distribution Shifts: An Cnnommation-Thoreete Approach
(Defocus
SNR (Post)
Replacement)
(Typo)
Join:
Freference
Hann yi ei
AARA veb IN
MAAA VEA LL
Delanne Steleernce
LANNE.U2 IN
Eawe seo3
Nane 1el
Eayy Geb
Harr rood
Wi.al Shitt (Defocus Blur
Wouel Shift iFrest
Text Shift (Word Restacement)
Toxt Shih (yppo)
Joint Shift
Lunch
PolNti Fredooro 10
20
RoMthee Reederence
LLAK NOT 138
Usdn N457 334
LLAA NEST
Sevelop
Nomay
Figure Performance variation against varying degrees of distribution shifts.
We evaluated LLaVA v1.5 and LLaVA NeXT models
on 34 dittrf--iittiuution (OOD) variants induced by image and text perturbations of the LLaVA- Bench COCO dataset (ID). Here, the r-axis sorted by the severity shifts between ID and OOD. There consistent trend increased degrees distribution shifts result in performance degradations of MLLM.
C. Extended Empirical Validation and Discussion
C.1. Additional Result from Pilot Study
In Section 3, we conduct an experiment to validate our hypotheses on the relation between MLLM performance degradation and the severity of natural distribution shift. In Figure 6, we provide additional results from another type of distribution shift that occurred by image and text perturbations. For image perturbation, we consider defocus blur and frost with three different magnitudes, and for text perturbation, we consider keyboard typo error and word synonym replacement with two different magnitudes (We adopt the source code of https: // / github. com/ JielinnQiuu Baaeaa/MMMMuuuuutSaaaaae Robustness to generate perturbed datasets). We observe the consistent trend in the relation between MLLM performance degradation and the severity of distribution shifts for the case of visual-only, text-only, and joint shift, likewise the case of natural shifts in Figure 1. That is, the increased magnitude of distribution shifts induces more severe MLLM performance degradation, and the degree of performance degradation can attribute to shifts in two modalities.
C.2. Different Design Choices of MI and JSD Estimation
Note that the results of all theorem (Lemma 4.3, Theorem 4.4, Theorem 4.5, and Theorem 4.6) are not limited to specific class of MI and JSD estimators. To investigate whether our empirical verification of theorems robustly holds in an astimator-agnostic manner (if the estimator valid), we provide an ablation study for the MI estimator, JSD estimator, and embedding space that the estimators are built on.
Specifically, we consider four MI estimators [NWJ (Nguyen al., 2010), MINE (Belghazi et al., 2018), InfoNCE (Oord al., 2018), CLUB (Cheng et al., 2020)), three embedding spaces (individual models (CLIP ViT and XI.M-RoBERTa) ES-V joint (Jiang et al., 2024), E5-V disjoint (Jiang et al., 2024)}. and two JSD estimators (MMD (Liu et al., 2020), RJSD(Hoyos-Osort) & Fanchez-Giraldo 2023)}. E5-V (Jiang et al., 2024) is recently proposed embedding extraction method that leverages an MILM with carefully designed prompt. We used the default prompt "Summary above sentence / image in one word: to separately extract embeddings (E5-V disjoint) for images and sentences, and design an ensemble of four custom prompts,
1. "Summary of the image <image>, and sentence <sent> in one word:
2. "Summary of the visual content "<image>" with an associated text query "<sent>" in one word:
3. "Civen <image>, Summary of the sentence "<sent>" in one word:
4. "Given visual content "<image>", Summary of the text query "<sent>" in one word: "
17

---

Understanding Multimodal LLMs Under Distribution Shifts: An Intormation-Thoreele Approach
extract multimodal joint query embedding (E5-V joint) by averaging four embedding vectors per (image, sentence) pair. In Table 7, we conduct Spearman correlation analysis over the 12 (4 x 3) cases of MI estimator and embedding
Table 7. Ablation study for MI estimator and embedding space. We evaluate four MI estimators with three different embedding space choices in terms of Spearman correlation coefficient/ p between RP score and EMI. We can see that EMI and RP score are robustly correlated to variations in the embedding space and the MI estimator, but CLUB shows the most stable correlation.
Configuration
LLaVA v1.5 7B
LLaVA v1.5 13B
LLaVA NeXT
LLaVA NeXT 13B
MI estimator
Embedding
p-val
p
p-val
p-val
p
p-val
CLUB
E5-V disjoint
0.695
0.000
0.726
0.000
0.581
0.001
0.579
0,001
CLUB
E5-V joint Individual models
0.910
0.000
0.846
0.000
0.817
0.000
0.902
0.000
CLUB
0.606
0.001
0.720
0,000
0,594
0.001
0.457
0.014
InfoNCE
ES-V disjoint
0.670
0.000
0.708
0.000
0.638
0.000
0.590
0,001
InfoNCE
ES-V joint
0.800
0.000
0.717
0.000
0.636
0.000
0.609
0,001
InfoNCE
Individual models
0.519
0.005
0.421
0.026
0.410
0.030
0.275
0.157
MINE
ES-V disjoint
0.664
0.000
0.605
0.001
0.269
0.167
0.278
0.153
MINE
E5-V joint
0.632
0.000
0.559
0.002
0.610
0.001
0,308
0,111
MINE
Individual models
0.632
0.000
0.562
0.002
0.632
0.000
0.613
0,001
NWJ
ES-V disjoint
0.583
0.001
0.552
0.002
0,513
0.005
0,429
0,023
NWJ
ES-V joint
0.502
0.005
0.519
0.005
0.492
0.008
0.480
0.010
NWJ
Individual models
0.510
0.006
0.717
0.000
0,488
0.008
0.322
0.095
space ablation. We can clearly see that EMI is consistently correlated with the RP score which demonstrates the robust effectiveness of our theorem in practice. Meanwhile, among the candidate MI estimators and embedding space, CLUB and two E5-V joint embeddings show outstanding results. However, E5-V embedding extraction require a forward pass of MLLM in contrast to the case of leveraging relatively small individual models (VIT base and BERT-base). To strike the balance between effectiveness and efficiency, we adopt CLIP ViT-B/32 and XLM-RoBERTa- Base embedding spaces by default.
Next, we present the Pearson correlation analysis result in Table 8 by ablating the ISD estimator with the MI estimator and embedding space choices. Although there are some variations in the exact values, we also observe consistently significant correlations between EMID and its upper bound (that of Theorem 10). Therefore, the upper bound of EMID we derived robustly holds in practice across diverse estimator configurations.
C.3. Hyperparameter Sensitivity
We provide the hyperparameter configuration for MI estimator training (Table 9) and further provide sensitivity analysis for varying hyperparameters (Figure 7). We see that the CLUB estimator is quite robust to varying hyperparameters. ie.. batch size, learning rate, and hidden dimension, which implics the effectiveness of EMI estimation without intensive hyperparameter tuning.
0.63
0.63
0.63
CLUB
CLUB
CLUB
0.62
Average Spearman
0.62
Average Spearman
0.67
Average Spearman
0.61
0.63
0.61
0.60
0.60
0.00
0.59
0.59
0.59
058
0.58
0.58
0.57
057
0.0011
0.0005
t.01
0,085
0.57
250
500
1000
2010
10
Batch Size (2*)
Learning Rate
Hidden dimension
Figure 7. Sensitivity analysis for batch size, learning rate, and hidden dimension during MI estimator training. For the considered hyperparameter searching grid, MI estimates derived by the CLUB estimator robustly achieve high Spearman correlation with RP score (the largest deviation is less than 0.02 absolute value).
18

---

Understanding Multimodal LLMs Under Distribution Shifts: An IetormattionThhoreeti Approach
Table Ablation study for MI estimator, JSD estimator, and embedding space. We evaluate four MI estimator candidates and two JSD estimator candidates, with three different embedding space choices in terms of Pearson correlation coefficient between EMID and its upper bound. In the considered variations, EMID and the upper bound of EMID (i.e., the simplified version Theorem 4.5) show strong correlations, implying that our theorem robustly holds in practice.
Configuration
Pcarson
JSD estimator
MI estimator
Embedding
p-val
RISD
CLUB
E5-V disjoint
0.618
0.000
RJSD
CLUB
ES-V joint Individual models
0.659
0.000
RISD
CLUB
0.565
0.000
RJSD
InfoNCE
ES-V disjoint
0.618
0.000
RJSD
InfoNCE
ES-V joint
0.617
0.000
RISD
InfoNCE
Individual models
0.295
0.002
RISD
MINE
ES-V disjoint
0.602
0.000
RISD
MINE
E5-V joint
0.534
0.000
RJSD
MINE
Individual models
0.630
0.000
RISD
NWT
ES-V disjoint
0.611
0.000
RISD
NWJ
ES-V joint
0.413
0.000
RISD
NWJ
Individual models
0.468
0.000
MMD
CLUB
ES-V disjoint
0.618
0.000
MMD
CLUB
ES-V joint
0.659
0.000
MMD
CLUB
Individual models
0.478
0.000
MMD
InfoNCE
ES-V disjoint
0.432
0.000
MMD
InfoNCE
ES-V joint
0.617
0.000
MMD
InfoNCE
Individual models
0.295
0.002
MMD
MINE
ES- V disjoint
0.602
0.000
MMD
MINE
ES-V joint
0.623
0.000
MMD
MINE
Individual models
0.630
0.000
MMD
NWJ
ES-V disjoint
0.611
0.000
MMD
NWJ
ES-V joint
0,273
0.004
MMD
NWJ
Individual models
0.468
0.000
Table 9. Hyperparameter tuning grid and selected value for MI estimator training. The hyperparameters are selected based on the variance of last 10 iterations during training.
Parameter
Selected
Sweep
learning rate
0,001
{0.005, 0.001, 0,0005, 0.0001 (64, 128, 256, 512, 1024, 2048) {250, 500, 1000, 2000}
batch size
1024
hidden dimension
100/500
C.4. Runtime Analysis
In addition to the advantage of allowing rigorous theoretical statements, EMI also has practical advantages over the RP score derived by the LLM judge. Specifically, while both RP score and EMI are model-dependent, the former relies on models with tens to hundreds of billions of parameters, while the latter enables meaningful evaluation even with relatively small embedding models with millions parameters. To quantitatively argue this, we compare the inference time per instance and for the entire LLaVA-Bende COCO dataset in Table 10. As shown in the table, EMI can shorten the time by 138 times compared the RP score. Even including the time required for MI training, EMIIbased evaluation is still 3 times faster than MILM judgeent-based evaluation. Since MI training is performed only once and then transferred to all ID-OOD scenarios, the efficiency of EMI-based evaluation over MLLM judgment becomes more evident as the number of datasets to be evaluated increases. In addition, in the LLM judge paradigm, although open-source LLM judges (Kim et al., 2024) have been actively studied recently, proprietary LLMs are still dominant in practice, so one must pay per instance query, whereas EMI can make meaningful inferences with publicly available open-source feature extractors without paying per query.
19

---

Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoree Approach
Table Runtime comparison on LLaVA-BBuch COCO 90 samples. We compare the actual wall-clock time (second) of EMI estimation and MLLM judgment protocols by evaluating model- generated response and reference response given input query from the LLaVA-Benn --ABencc COCO dataset (Liu et al., 2023). EMI estimation protocol on top of CLIP ViT-B/32 and XLM-RoBERTa: Base embedding
achieves 138 times boosting from MILM judgment protocol with GPT-4o.
LLaVA v1.5
LLaVA v1.5 13B
LLaVA NeXT 7B
LLaVA NeXT 13B per instance
Total runtime
per instance
dataset
per insiance
dataset
per instance
dataset
dataset
MI estimator training
663.45
663.45
EMI estimation
0.0388 5.49
3,5884 493.75
0.0392 5.48
3,5652 493.59
0,0412 5.52
3.7107 496.66
0.0411
3.7039 524.45
14:56 (x 138 boosting)
MLLM judgment (GPT-4o API)
2008.45
C.5. Notes on Practical Usage
Although the empirical estimates of EMI and EMID upper bound show consistent correlations with the relative preference score and EMID, the absolute value of MI estimates varies depending on the estimator types and their configurations. Therefore, one has used the empirical EMI estimates (computed by a two-layer MLP-bbsed CLUB estimator on CLIP-VIT VIT and RoBERTa emhedding spaces) to assess the qualities of responses from MILMMs, the same estimator and the same embedding models should be used across targeted models and datasets for fair comparison.
20

---

Understanding Multimodal LLMs Under Distribution Shifts: An Eetormation-Theoreeie Approach
D. Extended Theoretical Analysis with Full Proof
In this section, we provide proof theorems (Lemma 4.3, Theorem 4.4, Theorem 4.5, and Theorem 4.6) our manuscript, and introduce an additional theoretical result (Corollary D.12).
D.I. Proof for the relationship between EMI and preference model
First, we provide proof of the closeness between the effective mutual information (EMI) and the preference model. Lemma D.1 (Restatment of Lemma 4.3). Given a distribution Pxy and an MLLM Po, let the reward model fanction r(x, y)
be log (v)) If Ex DKL (Po (-))) Py xr(x=x) S 6, then,
EMr( Pa) PM(pxr; Pa)l S 4.487.
Proof. Let Py, = Exxx Po(-(x), and note the expression for EMI below,
EMI Po) I(Px @ Po) I(Pxy)
=H(Py, - ((oo(.()) H(y)) H(Prix)
(H(())) (Pv)))
+ ExxPx (1())) log (o(x) - Ex- Px BB Pr x-x log Py|x xx)x=x(U)]
Next, given r(x. v) log Pyix=x (v), logit Bradley- Terry preference model (PM) (Hunter, 2004) can be expressed as,
PM(Pxx: Po) = Ex Pxx Po(tx) log Xv|x=x Prxxx log Py|x=x(1))
Therefore,
EMI(Pxy: Po) PM((Pr::P)) (H(Pv)) H(P)) + Ex (og- log (o
Pyxx=x
< (Pv))) - Px
< 4.48 +
Here, we adopted Lemma D.4 to replace H(Pr, - H(P)) into its upper bound 4D inequality (Pinsker, 1964).
(Pv, (Py,l and used Pinsker's
We provide a proof for the extended theorem from the Lemma D.2 by considering the optimal model parameter as below. Theorem D.2 (Restatment of Lemma 4.4). Given a distribution Pxy and an MILMM Po, and assume Pxy > c 0 constant e, if the e-representation capacity assumption holds, i.e.,
min Ex~ Px (PrI Se, 0EB
(15)
and let the reward function r(x, y) be log Py PyIx=x(U), then
IEMI(PxY:8") PM((xr:;')) S 8 4.45%
where 8* is the optimal solution of Eq. (1) over Pxy, and 4.4ch log c/2c.
Proof. Recall the formulation of mutual information as below,
I(Pxy)
Ex, Pxy [log Pr|x== (x)==(y)]= ((Pr)
Ex.yxPxy [log (y()x)) Exyxxxx [log Po(y(x) + Ex [log Prix ( + H(y))
Pxx [log (o(y) xx] + Exxoxx (x + ((y))
21

---

Understanding Multimodal LLMs Under Distribution Shifts: An IntommationTThorrdee Approach
So, I(Px)) Pxyy~Por log Po(y|x)] H(P)) Ex~ Px (Xx== Po(-(x))) P Therefore, for the optimally learned parameter 0", we know that f* arg mingeo Ex Ix DRL Py xxxxx (-)x) which implies below,
Ex~Px [DKL Pa- ( [x]] s.
Meanwhile, have the below upper bound by leveraging Lemma D.7,
E Ex-rx [D (Po(t))))) x)] S 4.4c log ov
By denoting 8 == 4.4ca - loog 2e and plugging the Lemma D.I, we complete the proof.
Note that the assumption Pxy > c > 0 is reasonable in practice given the following two statements. First, we can only observe the samples that Pxy (x, y) > 0. Therefore, investigating on the case x, such that Pxx > 0 solely does not affect the practical implication of our analysis. Second, for the space X' x V, is obvious that | x 3 < +00. Therefore, X' x y a compact space, and (x, y) > over compact space, there exists a constant > 0 such that Pxy y) > c.
Lemma D.3. Given two distributions Px and Qx defined over X. let X [0,c], then we have the below,
Px (x)) Qx [f(x)]] c. Drv (Px, Qx).
where Drv (Px, Exex Px(x)) Qx(x)) is the total variation distance between two distributions.
Proof.
Ex-Py [f(x)] - Ex- [f(x]]
Σ Px(x))(() Σ (x(x)((=)
ZEX
ZEN
1 (Px(x) - -xx(x))f(=)
ZEX
1 (Px(x) - -xx()))(((z)) - cc) +c( Px(x) (x(x))
ZEX
#EA
Σ Px(x)) - Qx(x)) (5(x)- -c
REX
Se. ||xx - Qxllı
=c Drv(Px,Qx)
Lemma D.4. Given random variable X. and two distributions Pxy = PvxxPx and Qxx = QvixQxx, we have the bounds for the difference between Entropy H(-) over two distributions as below:
[H(Px) H(Qx)) < 4 H(Pr) - H(Qx) < (PvI(x),,
(P H(QYIx=x) -H(0Xxx)))(((11xxx)))
(Prxx=x1|11xxxx PP
where Das(ss)) is the Jensen-Shannon divergence between two distributions.
Proof. Let Mx = (Px + Qx)/2, Das(Px Qx) and Drv (Px.Qx) be the Gensen-Shanme divergence and total variation
22

---

Understanding Multimodal LLMs Under Distribution Shifts: An information-Theorriicc Approach
distance between Px and Qx, respectively.
H(Px) H(Qx)]
Pxxxx log (x) 4x log Qx (x)
Px log (x)
Px log Mx (x)
Ex--Qx log Qx + Ex-oxx log Mx(x) + Px log Mx(x) Ex-ox log Mx(x)]
Ex Bxxxx Px log Px(x)
Qx log Qx(x)
Px log Mx (x) Ex-ex log Mx (x)]
Mx (x)
Mx (x)
(x(
Rx log Qx
Bx log
Px log Mx(x) Ex-ex log Mx (x)]
Mx (x)
Mx (x)
(x(
Qx(x)
(2Dxs(Px(1xx) +
log Mx Mx(x)) (a))
2
(x
@x(x)
(x
Qx (1r)
=2x >
log
2
2
<2Xxs(Px|I(x) + + (x)
Qx(x)
(x(x)
Qx(x)
log
2
2
(x(x)
Qx(x)
(PxIx( + Σ
2
(xxll)x + 2 Σ [x(xx) - Qx(x))
-2Dxs(Px|1QX 22Drv(Px, Qx)
(Px 2D (Px (x)
In above inequalities, we have used v + 2log 0 for x E (0, 1). Holder's inequality, and Prv(Px Qx) S 22 (xxllxx proved in Lemma Thekumparampil al. (2018). We can prove below with the same strategy,
H(Pr) -H(Qx) < 4D (PyllQy),
(P -HQee))))HHxxx---- ))1 s <DD ( x11xxxx
Corollary D.5. For data distribution Pxy Prixl'x. MLLM Po(-(x), and Kullback-Leible divergence DKL. if Ex-Px DKL (yy Po S efor a constant then
Px (H(P(()))) H(Py x=x)] S $..6
Proof.
Ex- (H(o( - (XX(xx)= S Ex Exxxx H( HP(y))
< 4Ex Px is (P (
< 4 28 Ex Px DRL (PY1x=xlIPo((x))
< 4.4c3.
We started from Lemma D.4 and use Pinsker's inequality (Pinsker, 1964) leverage D.s(+I-) & Drv(s) S 22Drt.
Lemma D.6. For a data distribution Pxy = PvixPxx MLLM P=(+)x), let DKL and Drv be Ballback-Leible divergence and total variation distance, respectively. Denote Py, Ex (P (: x)), we have below inequality.
Drv(Py, Pv,) / 2xxx DKL
23

---

Understanding Multimodal LLMs Under Distribution Shifts: An Infornition-Therrtiicc Approach
Proof.
Pt,)
-
Exx Py(x=x(1) Ex Po(y)
vEY
Σ
Px Py Py(x=x(U) Po(x1x)]
yEY
1x
Px DTV xx Po(+(x))
S Px DKL
Lemma D.7. For a data distribution PxY PyixPx. MILM Po(:(x), and Kullback- divergence DEL if Ex DKL D (xx((xx=xx1PP(( & and Pxx > >>> for constant and c, then
Ex DKL. Pr)x=x) S 4.4c logev2e. 2c.
Proof. Note that
Ex (H(P((1x)) -((Yx)x=x)) =Bx Px P log Py Pyxx=x(v) Py log Pr|x=x(U)
+ExxPx Px DKL Prxxxx)
Therefore,
Ex-Px (
S Ex (ByxP)()1x) log Py Py|Xxx(1) (1) Evxee log Pr Pr|xxx(y))
+ 1x [H - H(Py x=x]
Given Lemma D.3, it is easy to check that
Ex~ Px [ Po(txx log Pr Pr(x=((1) Bym-Pr log Py xxxxxx()))]
< log Ex- (rv(Px())), Pr(x=x)
< c2e
Then, with Corollary D.5, complete this proof.
D.2. Proof for EMID upper bound
Now, we give the proof for the upper bound of the EMI Difference (EMID) below.
Theorem D.8 (General scenario). Given Pxy and Qxy distributions and an MLLM Ps. if there exist some constants 6p and &Q such that
Das( (rr Pr & 8p, Das(Qr, 1Qr) < 5q.
where Pv; = ExxPx (o(+)x and Qy. = Ex-ox Pol (- (x), then EMID Qxy; Po) is upper bounded by
Ā (P. DJ
+ (D (P 111xIx)) + D3 (1
+4Ex- [D Py P Q @xxx==)) + 8A
where A = 6p dq, Ā maxxex H (Q (Q + H (P&( and
DIs(PxIx/1Qxxxx)) == Ex~Py. Das ( -x 1apxv=xxxxxxxxx [xxxx==x) + Ex~ Das xx
24

---

Understanding Multimodal LLMs Under Distribution Shifts: An Information Theoretic Approach
Proof. Let Py, Ex Po(-|x) and Qy Ex-Qx Po(+Ix), EMID be expressed with entropy and conditional entropy
terms below.
EMID(Pxy, Qxv: Po)
EMI(P Po) - EXI( Po)
(Hr) - (H (()])))))))))
((H(Qy)) 9x - H(QYI)))
& (H((Y)) - (QQx)x) + (Ex-Qx -BoPs)((())))
+ +HH(Pr,) H(Pr) + H(Qy) H(v)))
S (H(P)))) (QQ1)))) + ( Px
+ (H))) H((r)) (Q())) - (Qrr.)
$ (H(P))x) (Qrix)) + (Ex-Qx (Po (-xx)) - Ex-
4 Pr) + Dss(O10))) Qy)) (16)
$ (H(P)))) (1xx)) (A)
(Po (x
where A 6p 5q. Now, will derive the upper-bound for the terms (A) and (B), independently. First, by adopting Lemma D.4, we can express the term (() as below.
H(P)x))
=
- H [x)x)xx)))+++++++x+>++0>))>>>+=+ B (H
$ EEIxl II ((yx(x=x) -
(
S 41xxx 1
( 111x== X=xxx
4E 1D Py|xx 11 2r E (Q - Eox [H (QrIxx=)) Eox [(QIxx=)))
Then, the term (A) of ineq. (16), i.e. H(Pvix) - H (Q is represented as below:
H(PrIx)= H(Qr(x)= S 4E (Dis Qy|x=x) + Erx [M(Q)) EQx (QQIix=)
To replace Exx [=( Eqx [H (Qv1x=x)) ) into a more interpretable term, and we start from the restatement of Lemma 1 of Shui et al. (2022).
Lemma D.9 (restatement of Lemma 1 of Shui et al. (2022)). Let Z E Z be the real-valued integrable random variable, and denoting two distributions
common space Z by P and Q such that Q is absolutely continuous w.r.t. P. for any function f and X E TR such that Ep (x)))(((([(=)) - Er((()))))] < co, then we have:
X(Eof(z) - Exf(x)) < DKL (QIPP) + log Ep Ep(exp(A(((=) - Ep(f(=))))
Now, let X and Y denote observable variables from joint distribution DXY (Pxr, Qxx and we denote f(x) H (QQ]x==x) 2 0 as loss function of our interest, e.g., conditional entropy of y given Then, f has finite value of Ep[exp(A(f(x) - (x))))) and is bounded within interval [o, A (Qy1)) where Á (Qv1x)= == maxxex H (QQ x=x).
We next define a mixture distribution Mxy = (Pxv + +xx)) where the support of Mxy covers that of Pxx and QXY. Then, we get the incquality below by setting P = Mxx and Q = QXY for all À > 0 according to the Lemma D.9:
Eox [[Qrix==x] EM (QrIx=x)
(17)
(log EMX (cxp(X(f(x) - EMx (f(x)) + DKL (Qx|[xx)))
Also, we get similar inequality by setting = Mxx and Q = PxY for all < according to the Lemma D.9 as below:
Ex [H(Qvxx=x) - EMy [H (QrIx=x)
(18)
(log EMX (exp(A(f(x) - EMy (f(x)) DKL (Px (M))).
25

---

Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theerrii Approach
Meanwhile, give that /(x is bounded within interval (x))x, the 2019) with the scale parameter (Q at most. Then, we can leverage the property of sub-Gaussian for the log moment generating function,
becomes sub-Gaussian (Wainwright,
X²o2
(Q
log (exp(AA( -Exx (f(x))) & (ex(
(19)
2
By plugging the ineq. (19) into ineq. (17) and ineq. (18), can derive the following new inequalities:
10
1 Dxxx(2x\1MMx)
Eex (Qxxxx Exx [H (
+
8
Emx (H(Q)Ix=x))) - Epx (x 10
Dxx(Px|1Mxx)
Au
where λo À stands for À > 0 in the ineq. (17) and λo = - for X < 0 in the ineq. (18).
By adding both inequalities above, and setting the Àu = H((rix) v ( Mx) + DKL (Qx D
results in:
Eox (M(9 E, (HQY x=x) & 2
(20)
Next, a decomposition of KL. divergence and the definition of IS divergence leads to the following inequality,
2D 1Dss(Pxxx,11xxxx
XIl + DKL. (Qx, (1
1 (DK) ( Bxx, Dic.. (Px, /X, Xx, |X, (x,=)
2
(DKL(Qx, (M..) ) + Eex. DRL. (Qx,) =x M N No=
(Px, [1 Exx, DKL (Px |x,=x, x (x==)))
(QxL BQxx DEL (0x, (MX
= (Ps(Px, x1Qx....) + Das Px, | (xx))
+ (EPx. (P M -x. + Eox. DRI (Qx, x.=r, x
+ (Bpx, DKL (P. 1x M ) + Box, DKL
(x.x)))))
Das Px. 11Qx.) DassssPPpppp 1xx) + Das (xx, 11 x.) Das [xxxxx
where Das (orrpPyllQyii = Ex-Px Das (
Ex-ox Das (Pxxx==ll@@
Based on the above decomposition, we can modify the bound as below,
-xx Epp
< (Is 11xx)) Qx) Dis(Px, x. 11x,xx,), + Dis
11Qx.ix.)
Therefore, we get an upper-bound for the (A) term in ineq. (16) as below,
26

---

Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theorriicc Approach
H(Pyix) - H(Qrix)
$ sH(Qr) Dast (xx, 11Qx.x 1xx(2x110xx)) Das xxx(Pxxx,,1 + Das( Px,1x, 1x.1x.))
+ +4 (PXx
< (x Dis (Px. + Djs (Px,
+ xx [ xx (x.xx.) Djs (Px (1xxxxxx)
+ +4
x=x)]
Then, deriving a bound for the remaining (B) term in ineq. (16), ie.. Ex~Qx (H((o(()))) - Ex~Xx (Poo x))] is very similar to the procedure of deriving the upper-bound for the term (A) by switching Qyix to Po(-(X) and set the f for Lemma D.9 fxx : ((Pa thereby having the interval (0, HPP) where H(P)) maxxex (()o--)x). This induces an upper-bound as below,
Ex~qx (o.2x(((P(+(x))] Ex [H(P(()xx) 2H
(21)
- (D + Dis 1Qxx) + DJS (Px, [xxxx.xxxxxxx D (xIxx,xxx
Finally, we complete the proof by adding the ineq. (D.2) and ineq. (21) induces the upper bound of EMID(Pxy, Qxv; Po).
SH - H(Qrx))) + (Bx~Qx [H(Po(-Ix)) B= + + < (Px.g) + (Px,
(D] (xx,Ix 11xx (X.x)xX,11Xxxxx
4x D (Py x xx1x=x]] SA
where Ā maxxxx (Q ((Po(-)))) and A 8p + EQ-
Then, we introduce an assumption over the consistency between conditional distributions as below.
Assumption D.10 (Consistency of conditional distributions). For the distributions Pxy and Qxy over X x y, conditional distributions of X given X., X, given X,, and Y given X = (X., Xt) are consistent between Pxy and Qxy. That is,
Px. Qx,xx and Px,Ix. Qx.Ixe
PY = QY
Finally, we present the simplified version of EMID upper bound by leveraging the Assumption D.10.
Theorem D.11 (Simplified scenario). Given an MILM Py and distributions Pxv, Qxy which have consistent conditional distributions over variables X, X, and Y|X, if there exist some constants 8p and 8q such that
Djs Py, 1 Py) S 6p, Dıs(Qv, 1Qx) S åq.
where Py, = Ex Po(-(x) and Qy. = Ex- ( (x). then EMID(PxY Qxv; Po) is upper bounded by
A (Px. + D (xx. (x.))xx.)) 8
(22)
where A maxxex [H(Qix==x) ((Po(-1x))) and A 8r + 8Q-
27

---

Understanding Multimodal LLMs Under Distribution Shifts: An anformation-Theorele Approach
Proof. Given Theorem D.8, Assumption D.10 zeros out the terms (Djs (Xx,xxx Djg 1xxx and
Ex [Djs Xy which induces Eq. (22), accordingly.
Corollary D.12. Given MILMM Po and distributions Pxv, QXy which have consistent conditional distributions over variables X Xz and Y X, EMID( Qxx; P6) is upper bounded by
(D Px. [Qx)) Djs ( (Pxx))(x)))
(23)
+ 8(Ex Drv (Pyjx-xx Po(+(x)) + Ex~Qx Drv Dxx
where :)) is the total variation distance, and H = maxxex (Q ((P()))))))(((((())))
Proof. Let Py, = Ex Po(-1x) and Qy = Ex Po (-)xx), then Dis(+1) S Drv(:;) allow us to induce below,
Das( Pv, Py) = Ex-rx D Po |x) S Ex Drv (Pr x=x Po(+xx))
(24)
Das(Qroll4 1|2)) = 1x Qx D (P (1 S Ex ( Pa)-
Noting that a +b4 < 2(a + b) for a, b 2 0, plugging the above inequality into the Theorem D.11 complete the proof.
Although this alternative upper bound looser than Theorem D.11, Corollary D.12 is more interpretable in the sense that it directly represents the model-specific discrepancy terms via distance between model output distribution and true conditional distributions, rather than expresses through marginal distribution terms in Das (Py,, PY) and Das Dus(Qra) QQ).. Therefore, model becomes more accurate modeling the ground truth conditional distribution of Y given X, the EMID mainly depends the divergence between the marginal distributions of visual and text imputs.
D.3. Discussion
We made the c-representation capacity assumption to derive Lemma 4.3 and Theorem 4.4 that claim the theoretical justification for EMI by showing its connection to a classic preference model. The assumption captures minimum achievable discrepancy between the truth distribution Py|x and the model's distribution Po(- As the models become more expressive-c.g, by increasing model size (Kaplan et al., 2020) and leveraging advanced positional encoding (Luo et al., 2022) approaches the universal approximator of seuuuencet--seqeeeee mapping (Luo et al., 2022; Furuya et al., 2024), as a result, the minimum expected discrepancy tends to decrease, leading to smaller
Meanwhile, EMI and EMID trigger flexible potential use cases given their generality. For example, they can be used as metrics to evaluate pure LLM well multimodal LLM, depending the types of targeted problems. Moreover, although we confined our analysis on T(Pxy), the chain rule of the mutual information (Cover, 1999) allows us to conduct partial modality EMI analysis through I(Pxv) 0.5l((x..)) 0.5I( + 0.5T we can decompose the aggregated EMI into the modality-specific EMI terms and modality interaction (conditional) EMI terms.
+ 0.57 where
28


Reviews:
[{"review_id": "c8W7YHGc3L", "reviewer": "Reviewer_o1Fb", "summary": "The paper proposes an information-theoretic framework to analyze the performance of multimodal large language models (MLLMs) under distribution shifts. It introduces Effective Mutual Information (EMI) to quantify the relevance between input queries and model responses. The authors also derive an upper bound for the EMI difference between in-distribution (ID) and out-of-distribution (OOD) data, connecting it to visual and textual distributional discrepancies. Extensive experiments on real benchmark datasets validate the theoretical insights.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> _A1. Applicability on SOTA MLLMs._\n\nThanks for the suggestion! Following your comment, **we additionally conduct the full evaluation with`Qwen2.5-VL-7B-Instruct` and `InternVL2.5-7B`**.\n\nSpecifically, we first evaluate the official release of `Qwen2.5-VL-7B-Instruct` and `InternVL2.5-7B` models on 35 synthetic shifts, then compute EMI estimates over the model responses. We perform a correlation analysis between estimates of EMI difference (EMID) and its upper bound. **Consistent with our existing finding, we observe a strong correlation between EMID and its theoretical upper bound in both models**.\n\n|Model|Pearson $r$|$p$-val|\n|-|-|-|\n|InternVL2.5-7B|0.67|0.00|\n|Qwen2.5-VL-7B|0.81|0.00|\n\n> _A2. How do assumptions impact the applicability of proposals to more complex cases?_\nThis is an insightful question.\nFirst, **to claim the closeness between EMI and win rate (Thm 4.4), we assumed $\\epsilon$-representation capacity of the MLLM**. \n  * $\\epsilon$-representation capacity essentially reflects the model’s ability to approximate the target task’s conditional distribution, meaning that the model can approximate this distribution with a KLD no greater than $\\epsilon$. \n  * The assumption mainly argues the strong expressive and approximation capabilities of MLLMs. **Given the strong expressive and approximation capabilities of recent large-scale MLLMs, this assumption is generally reasonable in practice.**\n  * Moreover, numerous efforts improve the diversity of an instruction tuning dataset and robustness of the visual encoder of MLLM [1,2], which makes the learned distribution robustly approximate conditional distributions encountered during evaluation.\n  * As we continually pursue enriching dataset construction and enhancing visual recognition of the encoder, $\\epsilon$-representation capacity assumption becomes reasonable in more complex cases.\n\nSecond, **to claim the relationship between EMID and its upper bound in a simple case (Thm 4.5), we assumed consistent conditional distributions** over $X_v|X_t$, $X_t|X_v$, and $Y|X$.\n  * This assumption zeros out the discrepancy between conditional distributions. If the conditional distributions are quite different between ID and OOD in some real-world scenarios, this makes our upper bound underestimate the performance gap, e.g., EMID.\n  * However, we highlight that the strong correlations between EMID and this upper bound have been observed through 61 distribution shifts, implying the validity of our upper bound to quantify EMID.\n  * **Meanwhile, we also provide a bound for general cases to address non-consistent conditional distributions in Thm 4.6**. This general-case bound can also be empirically estimated using a procedure similar to that of Thm 4.5.\n  * Therefore, as mentioned in our manuscript (L302-303), we recommend choosing a proper bound based on the knowledge of the data-generating process for datasets.\n\n> _A3. Details for practical implications of EMID upper bound_.\n\nWhile we confined the scope of this project to _presenting the first theoretical framework to quantify MLLM's performance gap_, we further provide **a potential application of EMID upper bound, instruction tuning with regularization, for this rebuttal**.\n\nWithout loss of generality, we can assume the input sequence $X=(X_v,X_t)$ as a sequence of intermediate representation vectors of MLLM, i.e., $Z=(Z_v,Z_t)$, and can also assume that $P_{\\theta}(.|.)$ maps this representation to responses, i.e., $P_{\\theta}:Z \\rightarrow Y$. This induces a modified bound with representation variable $Z$ rather than raw data input $X$.\n\nWe instantiate this modified EMID bound in two distinct setups below, where we set the 24th layer's hidden states as $Z$, and adopt RJSD [3] as a differentiable estimator for JSD and an average of empirical model output entropy. We provide evaluation results with LLaVA-v1.5-7B on in-distribution (ID) and visual (V), text (T), and joint (J) synthetic shifts.\n\n`Regularization term for instruction tuning`: $\\mathbb{E}[H(P_{\\theta}(\\cdot|z))] \\cdot ((D_{JS}^{0.5}(P_{Z_v}||N(0,I))+(D_{JS}^{0.5}(P_{Z_t}||N(0,I)))$\n  * One can not access $Q_X$ during the training phase, so we alternatively enforce the distribution of the intermediate representation to be close to the standard Gaussian.\n  * We sampled 10% of the instruction dataset from LLaVA-v1.5, and trained the entire LLM and modality connector parameters of LLaVA-v1.5 with and without the regularization.\n  * As shown in the table, we confirm that the EMID can be leveraged as a regularizer during instruction tuning to pursue better robustness to distribution shifts.\n|Method|ID|V Shift|T Shift|J Shift|\n|-|-|-|-|-|\n|Baseline|72.7|65.8|68.0|59.6|\n|Baseline + Ours|72.7|**66.3**|**68.3**|**60.8**|\n\n> Reference\n1. Qwen2.5-VL Technical Report, Alibaba Group 2025\n2. Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders, Shi et al. 2025\n3. The Representation Jensen-Shannon Divergence, Hoyos-Osorio et al. 2024", "claims_and_evidence": "The claims made in the paper are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "The correctness of the theoretical claims, including Lemma 4.3, Theorem 4.4, Theorem 4.5, and Theorem 4.6, is supported by detailed proofs in the supplementary material. The derivations appear to be sound.", "experimental_designs_or_analyses": "All checked", "relation_to_broader_scientific_literature": "The proposed information-theoretic framework for analyzing MLLMs under distribution shifts provided insights for better understanding and alleviating that issue.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "- Strengths:  The connection between EMI and win rate provides a practical and efficient alternative for model evaluation.\n- Weaknesses: Evaluations are only made on LLaVA v1.5 and LLaVA NeXT. It could benefit from involving the SOTA and representative MLLMs like Qwen2.5-VL and InternVL2.5.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- How do the assumptions made in the theoretical framework impact the applicability of the results to more complex real-world scenarios? Can these assumptions be relaxed in future work?\n- The paper mentions the potential use of the upper bound of EMID as a regularizer during post-training or test-time adaptation. Can you provide more details on how this could be implemented and its potential impact on model robustness?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BW2o1ifUKf", "reviewer": "Reviewer_sTSP", "summary": "I struggled to understand the paper so take this with a grain of salt:\n\nThe authors suggest there is a risk involved in using multimodal models (language models conditioned on visual input) and suggest a way to measure this is using a difference of mutual information between a model and the distribution of the training data (is this right?) – though it remains unclear to me how that distribution is gotten.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate sTSP's effort to read our paper and provide comments. Here is a notation table and our responses.\n\n|Var.|Def.|\n|-|-|\n|$X_t=(X_{t,1},...,X_{t,L_t})$ where $X_{t,i}\\in V$|a random variable (r.v.) of a text input sequence with length $L_t$ of tokens in vocabulary $V$|\n|$X_v=(X_{v,1},...,X_{v,L_v})$ where $X_{v,i}\\in \\mathbb{R}^{D_v}$| a r.v. of a $D_v$-dimensional image embedding sequence with length $L_v$ of tokens produced by a frozen vision encoder|\n|$\\mathbf{X}=(X_v,X_t)$| a joint r.v. of an input query constructed with a tuple of $X_v$ and $X_t$|\n|$Y=(Y_1,...,Y_L)$ where $Y_i\\in V$|a r.v. of a text response with length $L$ of tokens|\n|$P_{\\mathbf{X}}=P(X_{v,1},...,X_{v,L_v},X_{t,1},...,X_{t,L_t})$|a probability distribution (p.d.) of an input|\n|$P_{X_v}=P(X_{v,1},...,X_{v,L_v})$|a p.d. of a visual input|\n|$P_{X_t}=P(X_{t,1},...,X_{t,L_t})$|a p.d. of a text input|\n|$P_{Y}=P(Y_1,...,Y_L)$| a p.d. of a text response|\n|$P_{Y \\|\\mathbf{X}}=P(Y_1,...,Y_L\\|X_{v,1},...,X_{v,L_v},X_{t,1},...,X_{t,L_t})$|a conditional p.d. of a text response given input|\n|$P_{\\mathbf{X}Y}=P(X_{v,1},...,X_{v,L_v},X_{t,1},...,X_{t,L_t},Y_1,...,Y_L)$|a joint p.d. of input and response|\n|$P_{\\theta}(Y\\|\\mathbf{X})=P_{\\theta}(Y_1,...,Y_L\\|X_{v,1},...,X_{v,L_v},X_{t,1},...,X_{t,L_t})$|model's prediction p.d. for a response given input|\n\n> What problem is being solved? What is the risk?\n\n* As we noted in the introduction, MLLMs suffer from performance degradation when they encounter distribution shifts. We denote risk as this performance degradation.\n* `Problem focus`: As noted in the abstract, introduction, and motivation sections, **our goal is to quantify the performance degradation of MLLMs under distribution shifts by presenting an information-theoretic framework**.\n\n> It seems this is about (bi-modal) vision and language models with vision only in the input. \n\n* As we noted in `L107-109`, MLLMs take multimodal input (both visual and text) to produce text output, not \"vision only in the input\".\n* The term MLLM is commonly used in the literature to denote LLMs that receive a visual input as well as text [1,2], so we took this term by following the convention.\n\n> Definition of $P_X,P_Y,P_{XY},P_{\\theta}$.\n\n* In `L92-103`, we put the definition of random variables and distributions. $P_X,P_Y,P_{XY}$ denote the probability distributions of the input $\\mathbf{X}$, target response $Y$, and their joint $\\mathbf{X}Y$, respectively.\n* The $P_\\theta$ is a model being trained.\n\n> Definitions of $D(P_{X_v}||Q_{X_v})$ and $D(P_{X_t}||Q_{X_t})$. How are these defined in terms of next token probabilities? What is $P_{X_v}$ and $P_{X_t}$?\n\nAs noted in `L96-97`, $X_v$ and $X_t$ are the sequences of visual and text input tokens, so the $P_{X_v}$ and $P_{X_t}$ are the corresponding distributions. $D(P_{X_v}||Q_{X_v})$ and $D(P_{X_t}||Q_{X_t})$ are defined at the input, which should not be confused with the next token probabilities (at the output level).\n\n> Put the definitions of variables before shift examples. Domains of the variables in L93? How is (X_v,X_t) combined into a sequence?\n\n* We put the definition of all variables in `L92-103` before the motivation section. \n* We concatenate the visual input tokens $X_v$ and textual input tokens $X_t$ into a single sequence, where the visual tokens are obtained by encoding an image using a vision encoder (CLIP-ViT), and then projecting them into the language embedding space. This follows the standard practice in MLLMs, where visual tokens are prepended to the text tokens to form a unified input sequence.\n\n> Definition of MI and EMI\n\n* Our main interest is to express the model performance gap across different distributions. Thus, instead of defining MI with individual variables, we define MI as their joint distribution, which is mathematically equivalent to random variable-based MI, as can be seen in Eq 3. \n* The definition of EMI was explicitly introduced in Eq 6. Please refer to `L168-169`.\n\n>  L101: instruction tuning has not been introduced. L99: joint probability? Eq 1, should this not be argmin? Eq 2. Should this be P_{\\theta} instead of \\theta\n\n* Instruction tuning was introduced from `L104`.\n* In statistics, the population (distribution) [3] is used to denote a distribution of the entire collection of objects in contrast to a sampled distribution.\n* In Eq 1, both min and argmin can be valid, where the former aims at the objective-centric whereas the latter stands for the parameter-centric perspective.\n* In Eq 2, we use the first argument to denote the data distribution that the metric is computed on, and use the second argument to denote the model parameter to be evaluated. **We will use $P_{\\theta}$ in the revised version.**\n\n> Reference\n\n1. A Fully Open, Vision-Centric Exploration of Multimodal LLMs, Tong et al. 2024\n2. Exploring The Design Space for Multimodal LLMs with Mixture of Encoders, Shi et al. 2025\n3. Sampling of Populations: Methods and Applications, Levy and Lemeshow 2013", "claims_and_evidence": "The abstract talks about the risk of MLLMs and safe and reliable applications, and the proposed framework about quantifying risk under distribution shifts. It’s not clear what this risk is. Would it make sense to just call it the performance? \n\nYou talk about multimodal language models in general in the introduction, but it seems this is specifically about (bi-modal) vision and language models with vision only in the input. It could also be made clearer that the paper is only about textual output.\n\nI also don’t see the reason for focusing on vision in the input and instruction tuning. It seems to me the part of the ideas I followed apply to sequence models in general.  As such the question is to what degree a conditional distribution (given some prefix) differs from the non-conditional distribution.", "methods_and_evaluation_criteria": "I don’t understand how the various probability distributions are defined. I don't understand the definitions of the visual and textual shifts, e.g. the relation between $D(P_{X_v}|| Q_{X_v})$ and $D(P_{X_t}|| Q_{X_t})$. How are these defined exactly in terms of next token probabilities? What is $P_{X_v}$ and $P_{X_t}$? You only define KL with some arbitrary distribution P.\n\nHow are $P_X$ and $P_Y$ defined? Are these trained models? Are $P_X,P_Y,P_{\\theta},P_{XY}$ all the same trained model differing only in prefixes? E.g. as used in equation 3. For all of these, I would expect to see exact definitions of the distributions in terms of the exact model configuration you say your method works for.", "theoretical_claims": "I struggled to follow the definitions of the authors and so did not go through the proofs.", "experimental_designs_or_analyses": "No, since I struggled to understand what is being done", "relation_to_broader_scientific_literature": "Unclear.", "essential_references_not_discussed": "Unclear", "other_strengths_and_weaknesses": "See other answers", "other_comments_or_suggestions": "Some notes:\n\nIn the motivation with visual shifts, I would put the definitions of the variables before the shift examples, it was confusing.\n\nRandom variables: Could you define the domains of the random variables in line 93? How is (X_v, X_t) combined into a single sequence?\n\nLine 101: instruction tuning has not been introduced.\nLine 99: joint population → joint probability?\nEquation 1, should this not be argmin?\nEquation 2. Should this be P_{\\theta} instead of \\theta", "questions_for_authors": "I think the biggest problem with the paper is the writing and underspecified mathematics. The text is unclear throughout the abstract, introduction, and the motivation of what is being done. I don't find it clear what problem is being solved, nor what the exact configuration in which your method applies is.\n\nMore concretely, I don’t find the distributions and the distribution shifts well-defined making it very hard to follow any derivations or reasoning. Even if these (in your mind) are somewhat standard conditional factorizations, I want to see it spelled out, for every such $P_X, P_Y P_{XY}, P_{\\theta}$, otherwise it’s very hard to be sure that what is written is correct. It was also unclear what is trained and what is not trained.\n\nThe authors define mutual information as a function of a single distribution with an implicit factorization instead of over two random variables. This did not make it easier to follow the writing. For instance, in equation 7 of the EMI (a main contribution), what is the definition of $I(P_X\\otimes P_{\\theta})$, the definition of $I$ is hardcoded for a distribution $P_{XY}$ based on the factorization given. If there is some marginalization used to define it this needs to be defined, or maybe stick to the mutual information between two random variables.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cu32NhIVXf", "reviewer": "Reviewer_cWre", "summary": "This paper introduces Effective Mutual Information (EMI), a novel metric for quantifying the relevance between input queries and model responses in MLLMs. Unlike standard Mutual Information, EMI removes domain-dependent components, making it a more generalizable measure especially in OOD settings. The authors establish theoretical connections between EMI and win rate, providing an intuitive explanation of EMI’s effectiveness. Additionally, they derive an upper bound for the performance gap in OOD scenarios using EMID. Extensive experiments validate EMI’s utility, confirming its theoretical predictions and demonstrating its effectiveness across diverse distribution shift scenarios.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> _A1. Effect of modality fusion/interaction on generalization_.\n\n* MLLMs commonly undergo a modality alignment phase during training, which may affect generalization, and it is known that modality fusion can reduce the sample complexity to improve generalization [1]!\n* As noted in `line 250-254` in our paper, **$I(P_{\\mathbf{X}Y})$ can be factorized into $0.5 I(P_{X_{v}Y}) + 0.5 I(P_{X_{t}Y}) + 0.5 I(P_{X_{t}Y|X_v}) + 0.5 I(P_{X_{v}Y|X_t})$ where the conditional MI terms $I(P_{X_{t}Y|X_v})$ and $I(P_{X_{v}Y|X_t})$ encapsulate modality interaction.**\n* Based on this factorization, we can define per-modality EMI based on $I(P_{X_{v}Y})$ and $I(P_{X_{t}Y})$, and then derive a new upper bound that is constructed with $I(P_{X_{t}Y|X_v})$ and $I(P_{X_{v}Y|X_t})$ terms to capture the effect of modality interaction. We leave the explicit derivation for future work.\n\n> _A2. Interdependencies between $\\epsilon$-representation capacity and the model size, number of training samples, and input dimension_.\n\nThe $\\epsilon$-representation capacity assumption captures the minimum achievable discrepancy between truth distribution $P_{Y|X}$ and the model's distribution $P_{\\theta}(\\cdot|X)$. Due to the expectation and $\\min$ operator, it does not depend on the training sample size but is mainly influenced by model capability.\n\nSpecifically, as the models become more expressive—e.g., by increasing model size [5] and leveraging advanced positional encoding [6]—MLLM approaches the universal approximator of sequence-to-sequence mapping [6,7], as a result, the minimum expected discrepancy tends to decrease, leading to a smaller $\\epsilon$.\n\nWe will elucidate this in the next version, thanks!\n\n> _A3. How to leverage EMID upper bound to guide model optimization?_\n\nWhile our primary focus is on presenting the theoretical framework to quantify the performance gap of MLLMs, we also showcase an application of the EMID upper bound in this rebuttal, a **regularization term for visual instruction tuning**.\n\nDue to the space limit, we have included the setups and results in the `rebuttal to reviewer o1Fb, response A3`. **Please refer to that thread!** As shown in the tables, our instantiation of Theorem 4.5 can indeed be used to optimize model to improve robustness under shifts.\n\n> _A4. Eq. (4) can be an inaccurate estimate of MI lower bound due to potentially large $H(P_Y)$_.\n* In the Eq. (4): $I(P_{XY})\\geq\\mathbb{E}[\\log P_{\\theta}(y|x)]+H(P_Y)$, maximizing $\\mathbb{E}[\\log P_{\\theta}(y|x)]$ through instruction tuning can be interpreted to learn a parameter $\\theta$ that maximizes $I(P_{XY})-H(P_Y)$ rather than solely $I(P_{XY})$.\n* **We do not claim that the log-likelihood term is a tight lower bound of MI but rather suggest that maximizing $\\mathbb{E}[\\log P_{\\theta}(y|x)]$ can implicitly maximize the MI between input and model's response**. We will revise the paper to make this clear.\n* To validate this, we reproduce the visual instruction tuning of LLaVA-v1.5-7B on a 10% subset of data, and show how the empirical estimate [2] of MI evolves during training.\n    |Step|$\\hat{I}$|\n    |-|-|\n    |1|0.166|\n    |5|0.172|\n    |20|0.182|\n    |100|0.187|\n    |200|0.194|\n    |500|0.197|\n* As shown above, visual instruction tuning can effectively maximize MI between input and model response.\n\n> _A5. Encoder sensitivity analysis to domain shifts for EMI estimation._\n\n* In **Table 4 and 5 of Appendix**, we already discussed two alternative choices of the encoders, e.g., [3], and showed that _our theoretical claims hold in practice consistently across varying encoders with statistical significance_.\n* We further conduct encoder sensitivity analysis under domain shifts by replicating experiments on medical domains with a CLIP-ViT-B32 and XLM-RoBERTa encoders.\n* Specifically, we use 200 samples of LLaVA-Med [4], get three splits of them based on embedding distance with COCO images, and translate English queries into six different languages used in the paper by using GPT-4o to induce 28 subsets of shifts to conduct correlation analysis for EMID and its upper bound.\n  |Model|Pearson $r$|$p$-val|\n  |-|-|-|\n  |LLaVA-v1.5.7B |0.93|0.00|\n* We see the correlation between EMID and its upper bound estimates is very strong, even though the medical image and text are relatively minority instances compared with general object and text, implying that our theorem robustly holds even on the special domains that encoders may not excel at.\n\n> Reference\n1. A Theory of Multimodal Learning, Lu 2023\n2. A Contrastive Log-ratio Upper Bound of Mutual Information, Cheng et al. 2020\n3. Universal Embeddings with Multimodal Large Language Models, Jiang et al. 2024\n4. LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day, Li et al. 2023\n5. Scaling Laws for Neural Language Models, Kaplan et al. 2020\n6. Your Transformer May Not be as Powerful as You Expect, Luo et al. 2022\n7. Transformers are Universal In-context Learners, Furuya et al. 2024", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "above average", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Strengths:\n\n1. This paper provides a theoretical framework to analyze and understand the impact of distribution shifts on MLLM performance via EMI.\n2. The intuition of this new metric EMI is explained via analogy with excess risk, effective robustness and win rate.\n3. The empirical findings strongly support most of the theoretical conclusions.\n\nWeaknesses:\n\n1. I appreciate the theoretical part. However,  the current theories do not address how modality fusion and modality interaction affect generalization, despite these techniques being commonly employed in  (MLLMs).\n2. In Theorem 4.4, the paper assume that the $\\epsilon$-representation capacity  holds, where itself requires to be quantified, e.g. it it closed related to the model size, the number of training samples  and the input dimension.  Given that the assumption fundementally underpins the theoretical contributions presented, I suggest expanding the discourse to elucidate the quantitative interdependencies between   $\\epsilon$-representation capacity and these factors.\n3. The paper does not demonstrate how to leverage EMI or the EMID upper bound to guide model optimization (e.g., designing robust training objectives or adaptation strategies). It only uses EMI as a post-hoc evaluation tool.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "1. The paper states, “In Eq. (4), we show that the autoregressive objective for instruction tuning (Eq. (1)) effectively estimates the lower bound of MI when the model’s representation capacity is sufficiently high.” However, while the term 𝛿 can be omitted under this assumption, I suppose that 𝐻(𝑃𝑌) could be significantly large, making Eq. (4) an inaccurate estimate of the MI lower bound.\n\n2. The computation of EMI relies on pre-trained encoders (e.g., CLIP and XLM-R) for feature extraction, but the paper does not discuss the sensitivity of these encoders to domain shifts. For instance, CLIP may underperform on medical images, leading to distorted EMI estimates.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tTUvcHSEPV", "reviewer": "Reviewer_vm1D", "summary": "This paper introduces an information-theoretic framework to analyze and understand the performance of Multimodal Large Language Models (MLLMs) under distribution shifts, which occur when the evaluation data differs from the instruction tuning distribution. The authors propose the concept of Effective Mutual Information (EMI), a metric that quantifies the relevance between input queries and model responses. EMI provides a more robust and theoretically grounded alternative to existing empirical metrics like the win rate. The paper derives upper bounds for the EMI difference between in-distribution (ID) and out-of-distribution (OOD) data, linking it to visual and textual distributional discrepancies. Through extensive experiments across 61 real-world distribution shift scenarios, the authors validate their framework, demonstrating strong correlations between EMI and model performance. The results suggest that EMI can be used for more reliable and cost-effective evaluation of MLLMs, particularly in situations where traditional evaluation methods may be computationally expensive or less transparent.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> _A1. Legend issue in Fig. 1. and non-monotonic performance trend in text shifts_.\n\nThank you for pointing out the visualization issue—we will revise the legend in Fig. 1 to improve clarity and avoid confusion!\n\n* Regarding the non-monotonic trend in win rate under text shifts, this behavior arises in part from the inherent stochasticity of win rate computation based on GPT-4 API evaluations, making it fundamentally difficult to observe strictly monotonic trends in practice [1]. Additionally, the x-axis in Fig. 1 is sorted by embedding space distances—computed using CLIP ViT for visual shifts and XLM-RoBERTa for text shifts—which may not always reflect the true degree of distributional shift.\n* That said, we still observe a meaningful overall relationship between embedding distance and performance degradation, both in the 27 natural shifts presented in Fig. 1 and in the 34 synthetic shifts shown in Fig. 6 of the Appendix. By taking inspiration from this empirical analysis, we derive a much more rigorous framework, i.e., EMID upper bound (Theorem 4.5), to quantify the performance gap that consistently shows statistical significance across diverse settings.\n\n> _A2. Clarification for the meaning of Spearman correlation and Kendall’s tau_.\n\n* Spearman's $\\rho$ and Kendall’s $\\tau$ are both representative measures for monotonic relationships between two variables, where the former is preferred for detecting weak correlation and the latter is preferred to capture strong correlation in small sample sizes and is more robust to outliers with large sample sizes. Both of them are standard approaches to measure the correlation between LLM Judge score and other metrics [2], and are good for measuring the relation of two variables, even if they have different data types, e.g., discrete (win rate) versus continuous (EMI).\n* Both correlation coefficients range from -1.0 (negative correlation) to 1.0 (positive correlation), where 0.0 indicates there is no monotonic relationship between two variables. For Spearman's $\\rho$, a 0.2-0.4 range of values denotes weak correlation, 0.4-0.6 denotes moderate correlation, and 0.6-0.8 and 0.8-1.0 denote strong and very strong correlations, respectively, and Kendall's $\\tau$ can be similarly interpreted by multiplying 1.5, i.e., $\\rho=1.5 \\tau$, to compensate its relatively smaller scale in practice.\n* Our analysis in the paper (Table 2) indicates that the EMI consistently shows moderate or strong correlation with the LLM-judge evaluation metric, win rate, across different types of shifts and model architectures.\n\nWe will add this description in our future version of the manuscript. Thank you for the suggestion.\n\n> _A3. Intricate visualization of Figure 3 and the tightness of EMID upper bound_.\n* On the left two panels in Figure 3, we presented the overall relationship between EMID and its upper bound, whereas **we distinguished four different models on the right two panels to show the model-dependent difference over the relationship**. We did so to show that Theorem 4.5 actually differentiates the model itself through the output entropy of each model $H(P_{\\theta}(\\cdot|x))$, i.e., LLaVA NeXT shows higher sensitivity to the shifts compared with LLaVA v1.5 implied by the larger slope.\n* In this work, we do not claim the tightness of the derived upper bound. Moreover, verifying the tightness of the proposed bound can be affected by the choice of estimators for the MI and JSD terms during empirical validation. However, we would like to emphasize that **the bound shows consistent correlation with statistical significance across 61 cases of distribution shift over four different models, which means that our analytic bound of EMID effectively stands for performance degradation of MLLM.** We appreciate your valuable concern, and we will explore devising a much tighter bound in future work.\n\n> Reference\n1. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Zheng et al. 2023\n2. PROMETHEUS: INDUCING FINE-GRAINED EVALUATION CAPABILITY IN LANGUAGE MODELS, Kim et al. 2024", "claims_and_evidence": "Based on the results in Figure 1, the authors claim that as the severity of the shift increases, the performance degradation becomes more significant. First, part of the result is covered by the legend in text shift LlaVA v1.5, which requires some alteration. Second, the result in text shift does not clearly verify the statement. According to the context, the x-axis is sorted by severity of shifts, but the win rate in text shift isn’t decreasing monotonously. Also, LlaVA NeXT 7B performs better in Korean than in Arbaic, yet LlaVA NeXT 13B shows opposite performance respectively.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "The results in terms of Spearman correlation analysis and Kendall’s tau analysis needs further explanation, it is beneficial to specify the meaning of the value for better understanding.\nIn Figure 3, all four model results are shown in one graph, making it difficult to understand the authors’ implications. Also, from the figure, it seems that EMID is a lot smaller than EMID upper bound, for example, in synthetic shift graph, EMID ranges from -0.02 to 0.10, but the upper bound ranges from 1.1 to 1.8. If not misunderstood, this shows that the upper bound is very loose and has little constraint on EMID.", "relation_to_broader_scientific_literature": "This paper introduces Effective Mutual Information (EMI) to evaluate MLLMs under distribution shifts, extending information-theoretic frameworks to multimodal models. It provides a formal method to quantify performance degradation, addressing gaps in out-of-distribution generalization and model robustness in real-world applications.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths\n- Understanding MLLMs under distribution shifts seems a critical research problem.\n- The paper is well constructed and easy to read.\n\nWeaknesses\n- In Figure 1, the authors claim that as shift severity increases, performance degradation worsens. However, part of the result in the text shift is obscured by the legend for LLaVA v1.5, requiring adjustment. Additionally, the text shift results do not fully support this claim, as the win rate does not decrease monotonically with shift severity. For instance, LLaVA NeXT 7B performs better in Korean than Arabic, while LLaVA NeXT 13B shows the opposite trend.\n- The results from the Spearman correlation analysis and Kendall’s tau analysis require further clarification. It would be helpful to specify the meaning of the correlation values to aid in better understanding their implications.\n- In Figure 3, combining all four models in one graph makes it hard to grasp the authors’ implications. Additionally, the EMID appears much smaller than its upper bound—e.g., in the synthetic shift graph, EMID ranges from -0.02 to 0.10, while the upper bound spans from 1.1 to 1.8. If not misunderstood, this suggests the upper bound is very loose and offers little constraint on EMID.", "other_comments_or_suggestions": "No.", "questions_for_authors": "Please refer to the weakness part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes Effective Mutual Information (EMI), a new metric to measure query-response relevance in MLLMs. Unlike standard Mutual Information, EMI eliminates domain-specific elements, enhancing generalizability in out-of-distribution (OOD) settings. With multiple LLaVA variants and benchmarks, experiments confirm EMI’s effectiveness, aligning with theoretical predictions across various distribution shift scenarios.
Decision: Accept (poster)