Visual Instruction Bottleneck Tuning
Authors: Changdae Oh
Jiatong Li
Shawn Im
Sharon Li
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Large Language Models
Vision-Language Models
Instruction Tuning
Distribution Shifts
Robustness
Information Bottleneck
Primary area: deep_learning
TL;DR: We propose Vittle, a new visual instruction tuning framework that improves robustnessof MLLMs to data distribution shifts by pursuing the minimal sufficient representation.

Abstract: Despite widespread adoption, multimodal large language models (MLLMs) suffer performance degradation when encountering unfamiliar queries under distribution shifts. Existing methods to improve MLLM generalization typically require either more instruction data or larger advanced model architectures, both of which incur non-trivial human labor or computational costs. In this work, we take an alternative approach to enhance the generalization and robustness of MLLMs under distribution shifts, from a representation learning perspective. Inspired by information bottleneck (IB) principle, we derive a variational lower bound of the IB for MLLMs and devise a practical implementation, Visual Instruction Bottleneck Tuning (Vittle). We then provide a theoretical justification of Vittle by revealing its connection to an information-theoretic robustness metric of MLLM. Empirical validation of multiple MLLMs on open-ended and closed-form question answering and object hallucination detection tasks over 45 datasets, including 30 shift scenarios, demonstrates that Vittle consistently improves the MLLM's robustness under shifts by pursuing the learning of a minimal sufficient representation.

Visual Instruction Bottleneck Tuning
Shawn Im Department of Computer Sciences, University of -isconnin-Madison {changdae, sharonli)@cs wisc.edu
Changdae Oh
Jiatong Li
Sharon Li
Abstract
Despite widespread adoption, multimodal large language models (MLLMs) suffer performance degradation when encountering unfamiliar queries under distribution shifts. Existing methods to improve MLLM generalization typically require either more instruction data larger advanced model architectures, both of which ineur non-trivial human labor or computational costs. In this work, we take an alternative approach enhance the generalization and robustness of MLLMs under distri- bution shifts, from representation learning perspective. Inspired by information bottleneck (IB) principle, we derive a variational lower bound of the IB for MILMS and devise a practical implementation, Visual Instruction Bottleneck Tuning (Vittle). We then provide theoretical justification Vittle by revealing its connection to information-tee thhooretic robustness metric of MILM. Empirical validation of multiple MILMS on open-ended and closed-fore fordmffommmmmoo question answering and object ballucination detection tasks over 45 datasets, including 30 shift see- narios, demonstrates that Vittle consistently improves the MLLM's robustness under shifts by pursuing the learning of minimal sufficient representation. Code: https: //g thub. com/ www.lleannnnnwiscviiiieeee
Introduction
In intensive races on the track of frontier-level AI models, we have observed unprecedented achieve- ments through the form of a general-purpose chat assistant known as multimodal large language models (MLLMs) [1, 2, 3, 4] that combine visual encoder with large language model. Their universal yet flexible question guestion-answering answering interface enables MLLMs to easily permeate our lives from general problem- soloonnooiinggg [5, 6] to practical applications [7, 8, 9, 10]. While these models may achieve human-like or even surpass human-level performance on certain tasks, a critical gap remains in their Abbusness-ppariicuaaat particularly in handling input variations that humans process effortlessly.
Human intelligence thrives on the ability to distill a large amount of sensory and cognitive inputs into concise abstract representations, process akin conceptual compression [11, 12]. By priori- tizing sparse salient features while discarding redundancy, humans can shape a robust prototypical representation of complex data instances that captures proper level of invariance to low-level superficial features for generalization, yet maintains sensitivity to high-level abstract features for discrimination [13, 14, 15]. Unfortunately, there are consistent reports implying that the current MLLMs still lag far behind this desired trade-off between invariance and sensitivity [16, 17, 18, 19].
Specifically, MLLMs fail to produce relevant responses under query distribution shifts. That is, they are vulnerable to processing subtly perturbed samples and long-tailed samples [19]. This limitation partially stems from the difficulty of acquiring diverse high-quality multimodal instruction data at scale. When trained using standard maximum likelihood estimation on this relatively limited amount of instruction data, MLLM tends to fit data-specific patterns and results in brittle solution [20, 21, 22]. To enhance generalization, existing efforts typically fall into two categories (1) data-centric approaches, which collect more instruction data [23, 24, 25] and processes input in finer granularity [26, 27], and (2) model-centric approaches, which scale up the underlying model
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Distribution Shift
Open ended QA under Perturbations
Instruction Tuning
Evaluation
77-
Performance
74
-4.42
-5.53
72.
Whst cotor Aa ran leage?
what La 17M Fil tagge?
WRAY cotar
-15.05
the Fag
Clean
Pert.
Pert.
Pert.
(a) MLLM encounters distribution shifts
(b) Distribution shifts result in MLLM performance drops
Figure 1: Illustration of distribution shifts for an MLLM (a) and performance degeneration and embedding shiftss of the MLLM (b). An MLLM (LLaVA-v1.52) that might be visually and/or textually perturbed by unexpected noise. These distribution shifts result in performance drops, shown in the middle bar plot. A visualization of intermediate layer representations of the MLLM on LLaVA-Bal VA-Bench-COOR and its variants indicates that MLLM fails to learn proper level of invariance to generalize multimodal queries in the representation space.
receives arbitrary queries
using more expressive or specialized backbones [28, 29, 30, 31J. However, both data scaling and model scaling are resourceinttensiv--eeqee
significant annotation or computational cost.
In this work, we propose a new approach from representation--cnnre view to improve the ro- bustness of MLLMs under distribution shifts. Rather than scaling data or model, we introduce a lightweight, theoretically grounded module that enhances the internal representations of MLLMs via the information bottleneck (IB) principle. While the IB framework has been explored in small-scale or classification settings [32, 33, 34, 35, 36], integrating to autoregressive multimodal instruction tuning poses unique challenges due to the complexity of modeling mutual information across high- dimensional, sequential, and heterogeneous modalities. We overcome these barriers by formulating novel variational lower bound of the IB objective specifically tailored to the multimodal and sequential nature of MILMs. We further instantiate this formulation as a modular and scalable imple- mentation-Visaae Instruction Bottleneck Tuning (Vittle), which inserts one simple bottleneck layer within the LLM backbone. Vittle pursues minimal sufficient representations [37] that try to preserve only response-relevan information while discarding con-essential residual features. To our knowledge, this is the first work to investigate the IB framework for end-to-end instruction tuning of multimodal LLMs, offering model-agnostic pathway toward building more robust AI systems.
We conduct an extensive evaluation of Vittle across a wide spectrum of multimodal benchmarks to assess its robustness and generalization under distribution shift. Our experiments span 30 distribution shifts covering diverse forms of perturbation (in both vision and language) and long-tail distributions. Through these evaluations, we demonstrate that Vittle consistently improves robustness over standard instruction tnnigg baselines, without sacrificing performance on standard benchmarks and canonical tasks. Notably, we find that the bottlenecked representations induced by Vittle lead to enhanced invariance in the latent space, aligning semantically similar inputs more closely- closely-even even under input shifts whitl reducing overfitting to modality artifacts. We also show that Vittle is compatible with different MLLMs, offering robustness gains while maintaining similar inference- cost. These results underscore the practical benefit and theoretical promise of rrguuaation-regullaiiedd representation learning for robust multimodal instruction tuning.
(1) We propose a new representation-centti framework for improving the robustness of MLLMs under distribution shifts, grounded the information bottleneck principle. (2) We explore the IB-based end-to-end learning objective of an MLLM for the first time by inducing new variational lower bound of IB for MILM and devising a practical instantiation, Vittle, supported by theoretical analysis. (3) Through experiments on 30 diverse types of distribution shifts, we thoroughly validate the robustness of MLLMs on ppen-ended/elosed form QA and object hallucination detection tasks and show advantages of compressive representation induced by pursuing the IB principle.
Contributions:
2 Background, Related Work, and Motivation
Multimodal large language models (MLLMs). Recent advances MILMS integrate a pre-trained
language model with a vision encoder through visual instruction funing [38, 39]. To be specific, let X (X,, X2) denote a multimodal input query consisting of visual and textual input, e.g., an image and corresponding instruction question given that image, and Y denote desired

---

response given the input query. An MLLM fo with parameter 0 is trained to produce the desired response given an input query with a conditional autoregressive language modeling objective, i.e. arg ming Ex.y |Σm= log fY Lx Y=m] m] for a sequence of M Mleength responses, where the visual imput X, go through visual encoder and projector modules to be converted as sequence of tokens that have the same dimension as text embeddings and can be processed by an LLM backbone' After being trained, these models process wide array of multimodal instructions to solve arbitrary visual question answering tasks [40].
Robustness problem in MLLMs. Despite their impressive performance on standard benchmarks and their growing deployment in real-world applications [8, 41, 42], MLLMs remain vulnerable to input perturbations [433, 44, 45]. For example, MLLMS undergo systematic performance drop [19] when they encounter samples of superficial perturbations (e.g., varying brightness of image and typo in text) illustrated in Figure 1 (a). As shown in the bar plot of Figure (b), LLaVA-vv.5-777 model undergoes severe performance degradation on perturbations from visual input, textual input, and their joint (V, T, and Pert.).
(LB-COCO; [38]) under the
We posit that these vulnerabilities arise from the way MLLMs structure their internal representation space. In particular, inputs affected by perturbations are often embedded far from their intact (clean) counterparts, reflecting a distribution shift in the representation space that leads to poor generalization from
ceeematttoo--hoorttti perspective [19]. The right side of Figure (b) illustrates this phenomenon: using LLaVA-v we visualize representations of LB-COCO alongside its challenging variant, where the image and text inputs are perturbed. In this setting, semantically equivalent examples are mapped to distinct and distant regions in the latent space, suggesting lack of invariance superficial inpul variations, which is crucial for robustness distribution shifts.
Motivated by this, our work aims to enhance the robustness of MLLMs by explicitly regularizing their internal representations, encouraging them to retain task-relevant information while discarding input-specific noise-thereby finding a good balance between invariance to low-level superficial features and sensitivity to high-level abstract features for better generalization.
Information bottleneck principle. The information bottleneck framework provides principled approach to measure the quality of representations that are maximally predictive of target variable while compressing redundant information from an input variable [46, 47]. Numerous works have explored the use of IB training objective [32], across computer vision [48, 49], natural language pro- cessing [35, 36), graph learning [34, 50], and time-series modeling [51]. These efforts are supported
by theoretical insights suggesting that optimizing for the IB objective can reduce generalization error [33, 52]. However, most prior work focused on classification settings [35, 36] and/or relatively small-scale scale models [32, 53, 35J. Although recent study explored IB for MLLMs [54], the authors adopted IB training on a lightweight projector module while keeping the LLM backbone frozen. In contrast, our work is the first to investigate the IB framework for end-to-end training of large-scale autoregressive multimodal language models. Beyond shallow adaptations, we directly modify the internal structure of the LLM to promote IB IB-consistent behavior throughout the training process. We focus specifically on instruction tuning for MLLMs-whin have become increasingly central to modern Al ecosystems but remain largely unexplored from the perspective of IB-based learning.
3 Method
3.1
Preliminary: Information Bottleneck As Learning Objective
Let X be a multimodal input query (e.g., image-text pair), Y the desired output, and Z = (() an intermediate representation extracted by the MLLM encoder f(-). The Information Bottleneck principle aims to learn representations that are maximally informative about the output Y while being minimally informative about the input X. Formally, this expressed as the optimization objective:
max IB (X,
I(2
3
X2,
(I)
acquiring information from desired output
compressing input-spocific rodundant information
where I(---) :) denotes mutual information and 39 is the trade-off coefficient. Minimizing I (2,X) encourages removing redundant or input-specific variations, while maximizing I(Z, Y) ensures that the representation retains task-relevant signals necessary to predict the desired output.
For simplicity, will omit the visual encoder and projector in our learning objective at following sections.
3

---

In other words, the IB objective promotes representations that discard non essential features tied the input domain, while preserving those critical for solving the task. This property desirable for robust instruction tuning, where user queries that have the same latent goal must be mapped to consistent responses under varied conditions (e.g., visual and textual perturhations). Despite its appeal. integrating the IB objective into MILM training is highly non-trivial due to the intractability of mutual information estimation and the complexity of autoregressive and multimodal architectures.
3.2 Variational Inference for Information Bottleneck in MILMS
Directly optimizing the IB objective is generally intractable, as it involves mutual information terms over unknown data distributions. In this work, we introduce tractable variational bound on the IB objective, specifically tailored to the autoregressive and multimodal structure of MLLMs. We outline the key steps below and provide full derivations in the Appendix D.
We begin with the mutual information term T(Z,X) Given the sequential nature of MLLMs, we decompose both the input X (X.,, X2) and the latent representation Z (Zu, Ze) into visual and textual components, We can then derive the following upper bound for ((2
T(Z, x) = Ex, flog p(2)z) p(z)
- Ez, log p(2|z) = E E flog
pl 2 (r. xe )p(ppp((()p (2 (((2)(z)))
r(z)
r ((ze)r(2)) (2()
= E Ex E my E Dlog plzu (1))
111 + E Ez,.,, E2, [a flog p (e R4)
r(2v)
r(())
=E, (p(x xx) [r)))(()))) + Ex..
(2)
where the first incquality holds given the non-negativity of Kullback-Leible divergence (KLD), (ra( and p(z, To Ie) = p(2u (r.)) due to causal mask MLLM. Here, we introduce r(2) r(e,, 2) r (z2))(2z) factorizable variational approximation of the true prior p(2).
Next, for the output-relevant term I(Z, Y), we have the lower bound as the same as Alemi et al. [32]:
p(y|2)] p(3)
T(Z,Y) Ey. log
> Er,y. [log {(y|=)] log 2 Exr,y [Eyj [lg [vyl
where we replace the true posterior p(y|z) with variational approximation ((y|2) that will be parameterized by model component (will be elucidated in Section 3.3).
Finally, combining the lower bound of I(Z, Y) and the upper bound of I(,, X) yields variational lower bound for the IB objective as follows,
IB Y) 2 Ery 2/ [log [[vl=)]]
3 (E.,, (pLL [()))((()))) Ez.,,, [DKL (p(2)/= (1)](()
(4)
In the next section, elaborate how we can implement this variational lower bound for MLLM instruction tuning in practice.
3.3 Vittle: A Practical Implementation of Visual Instruction Bottleneck Tuning
By using Monte Carlo approximation of expectations over data, Eq. (4) can be written as follows,
C3
Σ B of (v)'()) 3 (Dkr.
+ (ww. (r(z)))
(5)
N
where, x² (2), a denotes the i-th sample query from an instruction tuning dataset. To compute this empirical estimate of the IB lower bound, we need to model the posterior distributions, p(=e| and p22// 22), and prior distributions r(z,) and r(2x), of the MLLM's inner representation Z.
Although in principle these distributions can take arbitrary forms, multivariate Gaussian distributions with simplified covariance matrices have been widely adopted variational inference and proba- bilistic embedding literature [55, 56, 57, 32, 58, 59, 60, 61] due to their mathematical tractability and empirical effectiveness. By following this common standard, we set the posteriors and priors as Gaussian with diagonal covariance, and will clucidate how exactly they are defined below.

---

Vittle
white
puppy
LLM - head fat-
(1 e)e, +all & a)2 az,
LLM-head gir
Bottleneck Layer
LLM- fa'
(2 M4..
Mf of
Bottleneck 96.
Bottleneck g
Projector
Embodding
Vision Encoder
Text Toieniner
Descrtbe
ASSISTANT:
LLM - stem
fat
USER:
this
widd
Xp
*
Figure 2: Vittle architecture. We insert a learnable bottleneck layer 9s {gow Id. on top of / blocks of LLM backbone (i.e., LLM-stem fai to estimate posterior distributions of token embeddings. After obtaining sample per token {Z,, ] from posteriors, we interpolate it with pre-bottlenecked token representation (Ze, and pass it through the remaining L.LM blocks (i.e., LLM-head fot+)
Posterior distributions. As illustrated in Figure 2, parameterize the posteriors p(ze (z.) and blocks. Specifically, they are non-linear mappings, Js. I + 2d implemented by multi-layer perceptron (MLP) for each modalities, which map each d-dimensional token embedding the posterior Gaussian parameter vectors μ Rd and の² R Given an intermediate 1-th layer output representation (Ze, 2+) = for (Fe, 21), we define:
p(z, 2r) 2.) using simple
p(2u|zo)= == (26i Mv: of I), p(2e x1) == ((2::
I),
where (y,, [ = 9o. (Jo (2.,)) and o = Do. (So (angs kz)), with the mean and variance parame- ters are bipartited along the output dimensions of the MLP. These MLPs are applied position-wise in the same manner as Transformer's feed-forward layers [62], producing token-wiss variational posteri- ors. Now, we can sample from the posterior distributions of MLLM representation by & p(Zulaz) and & p( ze [2,; 2.). Then, to strike balance between invariance and sensitivity, we interpolate the original representation
ppre-bottleneck, with the post-bottl bottleetteeeck counterpart غ as 2 = (1 (1-a)++a23 These representations are fed into the remaining layers to compute the predictive distribution over outputs, i.e., ((v[=)= := ført (y ,,, 2.). While direct sampling introduces non-differentiabiliiy we can enable the gradient flow using the reparameterization trick [56] to sample via = 1 + a O e with (0, 7) where μ and are the outputs of the bottleneck MLP module given input
Prior distributions. We consider two instantiations of the prior distribution for both Z, and Ze: (1) fixed standard Gaussian N(O, 7). which is input-independent and enforces strong isotropy, and (2) a learnable Gaussian N (jug a. T), where μı and o². are two learnable vectors shared across samples. Each prior affects the formation of representations differently-the fixed prior imposes stronger regularization and robustness, while the learnable prior introduces additional flexibility by allowing the model adapt to the instruction tuning distribution. We name the former Vittle (F) and the latter Vittle (L), and validate them altogether for all the evaluations in Section 4.
Overall objective and implementation. The first term of La (Eq. (5)) be easily computed through the standard cross-entropy, and our Gaussian instantiation of posteriors and priors allows us derive closed-fome expressions of KLD terms that can be computed from simple arithmetic between fa and o² parameters (See Appendix A.2). We set 8 = 0.1 where is the hidden dimension of the MILMM, normalize the KI. regularization terms relative the size of the latent dimension. The interpolation coefficient a in & = (1 a)= + a² increases progressively following cosine schedule up to 0.5. During inference, we consistently use an averaged representation with deterministic posterior representation ī = fu rather than using random sample. The target layer to apply the bottleneck module can differ between visual and textual tokens, but we = 24 for both modalities among 32 layers in a 7B-size LLM, i.e., top 25% layer, by default for simplicity (See Appendix B. for the ablation study). Figure 2 depicts the architecture overview.
d
(2 + E)/2
"This interpolation yields an asymmetric posterior specification (2, ) and / (2, Y) (Appendix D). An alternative is to use the representation statistics (mean and variance) from a pre instruction-tuup tune model host informative priors similar mixout [63], which may relax the need for interpolation (1 a)s + az.
5

---

3.4 Theoretical Justification
The learning objective of Vittle has an attractive theoretical interpretation that can support the improvement in robustness of Vittle. In this section, we first introduce recently proposed irformatiin-theoreeti measure of MLLM's robustness under distribution shifts, effective mutual information difference, EMID [19], and show how Vittle can contribute to reducing EMID.
Definition 3.1 (EMID), Let Pe X + y be an MLLM with parameters θ that produces an oulput response Ye given an input instruction X. For joint distributions Pxy and QXY, effective mutual information difference of Pe over distributions P and Q is defined below,
EMID(PxY, Qxx; Pe) -I(Pxx)) - [(Px)]] - [1(2xxs) [(Qx))]
(6)
where denotes the mutual information that measures the relevance between input query and response. A higher value of EMID indicates that MLLM Pe undergoes responees relavance degeneration in the distribution Q (e.g., test-time) compared to P (e.g., train-time), so we want to achieve a lower value of it to ensure robustness. We now derive upper bound for the EMID (See Appendix E).
Proposition 3.2 (EMID upper bound). Let Pe be an MLLM that maps X {X.: X,) } to Z {Z., 2 and then subsequently maps Z to Ye. Given joint distributions Pxx = Px Pyix and Qxy = Qx Qrix (resp. Pzy and Qzy). by assuming consistent conditionals over Zul Ze. ZoIZo. and Y Y|X between P and Q, we have upper bound for EMID( Qxy: PP) as below,
f [ (1Q..) DJs (Pz. + VAxiz) +IT(P...) HIPy)] ( HQQ))), (7)
where H and D indicate the entropy and square root of Jensen-Shannon divergence (JSD), respec- tively, Axiz == Errr x || + PKc.. DD..(Q... (Qx M|+)] with a mixture distribution M= 2+2 and A == max being optimized, the terms HPy)), H (Qy), and max H (Qv1x). can be ignored from Eq. 7 which are fixed across model parameters. We can also ignore /Axiz term because it cannot directly affect Ye given the conditional independence X ZZ. That is, the upper bound of EMID is boiled down to the multiplication and summation between the output entropy and the representational divergence.
H((v))) )}. As we are interested in the terms related to O
2-
Implication. Vittle maximizes the variational lower bound of IB, which consists of (1)
minimizing standard negative log-likelihood term representing an expected risk, and (2) minimizing KLD terms to enforce posterior distributions close to prior distributions. By (1),
MLLM Pe seeks a solution that minimizes the expected risk and reduces its output entropy (Py,)) ) and probably H (QQy)) as well [64, 65, 66]. By (2), it may reduce JSD between representation distributions Pz and Qz by promoting all posterior samples to be laid near the pre-defined priors [67, 68, 69]. In summary, reduced entropy and JSD terms induce lower EMID, which means that Vittle may achieve better robustness to distribution shifts than the standard training method that neglects the divergence in representation space.
We show that Vittle indeed reduces empirical JSD and EMID under distribution shifts in Table 4, and demonstrate in Section 4.2 that Vittle's 's nice theoretical property is translated into consistent robustness gains under 30 distribution shift scenarios while maintaining in-distribution performance.
4 Experiment
4.1
Setup
Model and implementation detail. We adopt LLaVA-v1.5 [70] as our main baseline MLLM, where we set CLIP ViT-L/14-336px [71] as a vision encoder, Vicuna-v1.5-7B [72] as an LLM, and two-layer MLP as projector. We follow the standard two-stage training of LLaVA [38], and replicate stage-1 for image-text alignment with the same configuration and dataset (LLaVA-prr) wttAA-pperrrin----n of LLaVA-v1.5 1.5 [70]. Then, on the LLaVA-mix-666l 665k, we apply Vittle method by inserting the posterior MLP blocks and training the whole model with our IB objective. To validate the scalability and broad applicability, we also consider V.L. 1.5-13BBBBBBBB. Prism-7B [73], LLaVA-Mini- Vicuna- 7B [74], and V.L.VA-Llaa -------aaa--------es Appendix B for the results on LLaVA-v1.5-133 13B and Prism-7B, respectively.
[75]. Refer to Appendix for additional details and

---

Object Hallucination Detection Task Under Visual Perturbations
+1.1%
+0.7%
+0.7%
+1.0%
+1.4%
+0.855
296
Acturncy
+1.6%
Gevaoite Nomid
Notso
Spect Notel
500
1992
Briptteni
Contrect
Part.
Figure 3: Object hallucination detection performance POPE variants. We enumerate the hallucination detection accuracy of each method on nine versions of perturbed samples, and observe consistent gains by Vittle (highlighted by green numbers of relative improvement from baseline).
Open-ended QA Task Under Textual Perturbations
Pelative Preference
2
+1.6%
+4.1%
+3.2%
+1.3%
+2.6%
2
+3.5%
22
+1.9%
+2.3%
Word 27
Coor 1yo0
Polate
clar Inbert
Gresk
Anbiic
Avi. Part
Wort
Werd
Open-enned QA Task Under Joint Perturbations
Relottve Preforence
Bisekine
Vifee 4J
VIDe 29
+5.0%
7.3%
+4.3%
7%
+1.5%%
-2.8%
+0.7%
+9.0%
+0.8%
Aodic
Hort Draek
Zoom Bur Anaiic
Zoom Bteniion
Zoom Bucdfind
bay Nolee Analic
Canesion in Greet
caussion Boise
Aeg Par..
Figure 4: Open-ended QA performance on LB-COCO variants. We enumerate relative preference score of responses from each model on 18 version of perturbed samples, and observe consistent gains by Vittle (especially for the Vittle (F)) on most of the textual (top), and joint (bottom) perturbations (results on visual perturbations deferred to Appendix B).
Task, metric, and datasets. We evaluate instruction-tunel MILMs with three representative tasks: (I) open-ended question answering, (2) object hallucination detection, and (3) closed-form question answering. All are formatted question answering (QA) with single image input, where use the average relative preference score measured by GPT-4o LLM judge [76] with three repeated runs for open-ended QA, while using exact matching accuracy for hallucination detection and closed-form QA. For open-ended QA tasks, we adopt four datasets: LB-COCO [38] as a clean and rypical dataset, and LLaVA-Benn -enchh in-the-wild A-ll), LLaVA-Bench- Wilder (LB-W- B-lddr, and Vildonsion- Bench (WV-Bench) as long-tail datasets. Then, we apply 27 types of image and text perturbations on LB-COCO samples* yield 28 variants of perturbed LB-COCO (one of clean and nine of visual, textual, and joint perturbations, respectively). For object hallucination detection tasks, we adopt POPE [77] clean and rypical dataset. Then, generate nine variants of perturbed POPE with visual perturbations. Here, we consider the LB-COCO and POPE as in-distribution (ID) datasets because they generated from MS-COCO samples that construct majorities of the instruction tuning set of modern MLLMs, including LLaVA VA. For closed-form QA, we adopt four representative datasets: ScienceQA [78], MMMU [79], MME [5], and MMStar [80]. summary, we experiment with 45 datasets (31 of open-ended, 10 of object ballucination detection, and 4 of closed-fome foosed-fomm tasks).
4.2 Results
Vittle improves robustness under input perturbations. We first evaluate Vittle on ob- ject hallucination detection tasks with nine variants of POPE perturbed by visual corruptions Figure Although MILMS trained with a standard objective and Vittle similarly suf- fer from perturbations, two instantiations of Vittle consistently outperform the standard ob- jective. Interestingly, Vittle outperforms the baseline even in clean POPE (See Appendix B).
"See Appendix A.3 for comprehensive summary of all perturbations and their generation processes.

---

We speculate that Vittle's information control prevents the reliance on a partial feature of single modality [81], e.g., language prior [82], which is common source of hallucination.
Next, we present the validation on the open-ended QA task
with 18 types of input perturbations, which are applied to visual and textual input independently or simultaneously in
Severity
Severity
Pelntww Praforncc
Daned
Vittl IPI
Figure 4, As we can see, Vittle greatly enhances perfor- mance in various perturbation datasets highlighted by green numbers that indicate the relative improvements of Vittle (F) compared the baseline. Among the two variants of Vittle, Vittle (F) showcases better generalization under perturbations than Vittle (L), suggesting the benefits of a conservative zero-centered isotropic prior distribution to address variety of subtle input perturbations. Next, further explore Vittle's 's robustness by evaluating varying generate perturba-
5400816
Vatte 33 Vide
spoc
Spp
Figure 5: Evaluation under vary- ing perturbation severity. Vittle achieves better performance, espe- cially on severe perturbations.
perturbation severity. To be specific, tions on three different degrees that determine how significantly the image or text would be changed. In Figure 5, see that Vittle achieves better performance in general, where the margin becomes larger under severe perturbations. In summary, we observe a consistent gain by Vittle on the perturbed input setting across two tasks, which indicates that Vittle enhances the robustmess to distribution shifts by pursuing the minimal sufficiency of data representation.
Vittle improves generalization long-tail distributions. Not only subtle perturbations on input, but long-tail tail samples are also commonly encountered in many MLLM applications. In Table 1, we validate Vittle on three long-tail QA tasks constructed with real-world user queries. We see that Vittle also excels in generalizing long-tailed samples compared to the baseline. Interestingly. Vittle learnable prior-exhibe
better performance compared with Vittle (F). We speculate that a learnable prior IB guides the model to learn a better sensitivity for high-level abstractions as well as an invariance to low-level noise by allowing additional flexibility to shape data-driven priors, yielding superior performance on tasks that require in-depth understanding of irregular queries.
Table 1: Performance comparison on long-tail open-ended QA tasks those contain queries that are quite different from typical training samples in terms of visual content and textual semantics.
Table 2: Performance comparison on general benchmark datasets. These four multi-choice QA datasets require a higher level of multimodal understanding across multiple domains.
Method
LB Wild
Li-Wilder
WV-Beenhh
Method
SciQA
MMMU
MME
MMStar
Avg.
Baseline
51.6
156.9
60.0
Baseline
64.6
35.6
69.7
33.7
50.9
Vittle (L)
54.6
168.8
60.4
Vittle (L)
64.7
35.3
70.5
33.7
51.1
Vittle (F)
52.2
166.1
59.7
Vittle (F)
65.4
34.5
70.1
33.5
50.9
Vittle preserves competitive performance on general benchmarks. Although the main focus of Vittle is to improve the model's robustness under distribution shifts, securing the rich multimodal understanding capability and knowledge to diverse disciplines also crucial as an essence MLLM. To validate this, we evaluate each method on four representative closed-form knowledge-intensive QA benchmark datasets covering various fields. In Table 2, we observe that Vittle shows competitive performance with the standard approach, which implies that Vittle can also be used general- purpose learning objective.
Table 3: Comparison with weight-space com- pression methods. We compare Vittle with the LoRA and weight decay (WD) methods on LB-COCO and its perturbed variants.
POPE Shits
LB-COCD Shifts
545
Method
Clean
V Pert.
Pert.
J Pert.
Baseline
77.8
73.4
72.2
62.3
9/5
LoRA
73.4
70.4
62.7
39.7
Vonte
WD
74.1
72.1
73.0
59.5
Figure 6: Comparison with other objective functions. We report the average performance for all perturbations in POPE and LB-COCO.
Vittle (L)
76.7
73.9
73.0
62.7
Vittle (F)
76.1
74.2
74.1
64.4
8

---

Visual Perturbation
LLaVA-v1.5
Vittle
Q) focus of the inage?
The mein focus of the image is giriife's heed, with the graafle looking at the carnern.
focus image close-up of grrafle head, with its eyes open and looking direcity
Q) What focus of the inage?
The main focus of the image is a gäraffe standing front of
The main focus the image a closo-up of giraffe's head, with looking directly the camer...
Textual Perturbation
Q) What is the nan doing the inage?
The men the image is sitting at table, talking on his cell phone.
The man in the image is aitting at table, talking his call phone.
2) Quat doinl in the inage?
nan
The mei the imagee is dozing off while sitiigg table.
The man the Image giving thumbs- sign while talking his cell phone.
Figure 7: Case study LB-COCO under perturbations. Although LLaVA-v1.5 produces a reasonable response for clean samples, the response and its quality vary under perturbations. Meanwhile, Vittle maintains the consistency for the responses.
Comparison with alternative learning approaches. Note that the regularization forced by Vittle works on the representation space penalize the amount of information encoded in the data rep- resentations. One of the natural alternatives is to regularize the model weight directly. In Table 3, we compare Vittle with LoRA [83] and the weight decay method (WD) as instantiations of weight-space regularization, and the results suggest that explicit regularization on weight-space does not ensure a good balance between adaptability on in-distribution and robustness to distribution shifts. The other line of alternatives is information maximization during visual instruction tuning [84, 85], which is the exact opposite of Vittle's design principle. We compare two recent methods on this line, ROSS [84] and LIT [85], with Vittle on LB-COCO and POPE under perturbations. As shown Figure 6, although these approaches are effective in improving object hallucination detection performances, they fail to achieve competitive performance on the open-ended QA task (See Appendix B for more results). This implies the non-trivial challenge of devising a general learning objective for MILMs that can consistently improve robustness across diverse tasks, where we can see the promise of Vittle towards broadly applicable robust instruction tuning.
Qualitative analysis. Figure shows responses of clean queries and their visually or textually perturbed counterparts. Although the query before and after each perturbation conveys the same meaning and intention, LLaVA-v1.5 reveals volatility in its responses, whereas Vittle shows stable behavior by providing consistent responses, i.e., generating exactly the same response in the case of visual perturbation while keep focusing on the same object in the case of textual perturbation.
Representation analysis. We next see how Vittle shapes the representation space and how affects robustness. In Table 4, we measure the average value of empirical JSD and EMID discussed in Section 3.4 over 27 perturbed variants of LB-COCO. Both JSD and EMID are computed between two distributions, clean and one of its perturbed versions, and then averaged over 27 clean- clean- perturbed pairs (See Appendix A.4 for details). As our hypothesis, Vittle reduces distri- butional gaps, e.g., achieving smaller JSDs, between clean and perturbed samples in its representation space, thereby achieving smaller EMID value that indicates better robust- ness. In Figure 8 (top), we further show PCA visualizations (in the same axis scale) for representations of LLaVA and Vittle on clean and image-text perturbed LB-COCO. We that Vittle embeds the clean and semantically equiva- lent perturbed samples more closely. Moreover, the bottom panel shows that Vittle induces smaller cosine distances between clean and perturbed pairs in terms of the histogram and the average value in parentheses. These results indicate that our learning objective is indeed effective in structuring a better representation space that drives robustness.
Table 4: JSD and EMID evaluation on 27 LB-COCO variants.
Method
JSD (L)
EMID (U)
Baseline
0.068
0.026
Vittle (L)
0.048
0.021
Vittle (F)
0.047
0.025
Baseline
vittle (F)
Baseline (0.197)
Count
15 5:
(0.152)
0.10
615
0.00
0.23
0.50
8:35
0.40
Courne Distance Paired Samples
Figure 8: PCA and pair-wise cosine distance of representations.

---

Cost analysis. Vittle introduces a lightweight bottleneck layer inside of LLM that slightly increases the total number of trainable parameters (by 1.5%). One may thus wonder how Vittle's 's training and inference time is compared with bottleneck-free baseline. In Table 5, we show the wall- clock training (per iter and total), and inference time per sample. Although Vittle increases the training time up 20% compared with baseline, its inference time is almost identical to the original model, which is a reasonable amount of cost overhead given significant gains in terms of robustness. We also compare the peak memory in Table 14, showing a negligible amount of memory overhead.
Table 5: Runtime comparison.
Tr.fit. (8)
Tt
Te
Method
Baseline
7.363
11.06
0.1048
Vittle (F)
9.482
13.36
0.1072
Varying MLLM specification. As Vittle a model-agnostic learning framework with- out archifecture-specfiic constraint, we now explore Vittle training on two different MLLM specifications in Table 6: (1) LLaVA- Mini that has the same vision encoder and LLM as LLaVA-v1.5 but has different archi- tectural design by incorporating visual token compressor and pre-m modality fusion layer; and (2) LLaVA-Llaam3--8 --B-a-----BBBB 8e---B--IInna Instructt (denoted in LLa++++) that we just replace the LLM backbone with Llama3-88 SB-Instruct. As we can see, Vittle consis- tently outperforms the standard LLaVA training in various MLLM specifications, demonstrating its generality across different model architectures (See Appendix B.5 for more results).
Table 6: Vittle with different model backbones.
Backbone
Method
POPE
POPE Shifts Avg.
Baseline
79.37
77.39
LLaVA-Mini [74]
Vittle (F)
81.07
78.32
Bascline
84,60
80.54
LLaVA+ [75]
Vittle (F)
85.87
84.08
5
Conclusion
This work provided the first investigation on the promise of information bottleneck in the context of MLLM instruction tuning to ensure the robustness of MILM under distribution shifts. We proposed a new mheoretically-grounde
visual instruction tuning method, Vittle. It injects bottleneck layer inside the LLM to induce posterior samples of internal representations that encode useful information to produce valid responses while discarding other residual information from input queries. With negligible additional cost, Vittle easily optimized with a variational lower bound IB and shows consistent gains in robustness in 30 types of distribution shifts while also achieving competitive performance on standard benchmarks, indicating that Vittle promotes a good balance between invariance and sensitivity during representation learning.
Limitation and future work. One possible concern with Vittle is its reliance on the quality of Y, i.e., a gold response to given instruction, which usually generated by another LLM. As disclosed by Yeh et al. [86], existing datasets for supervised fine-tuning are quite noisy, and we cannot ensure the advantage of IB on this noisy annotation setup. Moreover, IB alone does not guarantee the generalization of multi-modal counterfactual samples with conflicting language prior [87, 82] or samples from completely different domains [88, 89, 90], and may require additional annotations. Besides, we observed that Vittle slightly hurts the optical character recognition capability of MLLM, indicating the importance of further exploration to preserve the fine-grained recognition capability while pursuing robustness to distribution shifts. Investigating the potential of noisy annotation, counterfactual/domnin generalization, and fine-gratd mne-grained recognition setups can be interesting future research problems. Mcanwhile, the noise-robust representation space achieved by IB training be helpful for representation steering methods [91, 92] that are also worth exploring for future work.
Acknowledgments and Disclosure of Funding
The authors thank Seongheon Park and Sean Xuefeng Du for their valuable suggestions and dis- cussions that shaped the draft, and thank all the NeurIPS 2025 PCs. SACs, ACs, and anonymous reviewers. In addition, Changdae Oh particularly thanks Sanghyuk Chun for his sharp feedback on the manuscript. Research is supported in part by the AFOSR Young Investigator Program under award number 2495550-2311-11 84, National Science Foundation (NSF) Award No. IIS-223703" and 1IS-2331660 1669, Office of Naval Research under grant number 100014-23-1-2643 Open Philanthropy, Schmidt Sciences Foundation, and Alfred P. Sloan Fellowship.
10

---

References
[1] OpenAL. Gpt-5 system card. Technical report, OpenAI, August 2025.
[2] Anthropic. Claude sonnet 4.5 system card. Technical report, Anthropic, September 2025.
[3] Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang. Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
[4] xAI. Grok 4 model card. Technical report, XAI, August 2025.
[5] Zijing Liang. Yanjie Xu, Yifan Hong. Penghui Shang. Qi Wang. Qiang Fu, and Ke Liu. A survey of multimodel large language models. In Proceedings of the 3rd International Conference on Computer. Artificial Intelligence and Control Engineering, pages 40S-409, 2024.
[6] Hao Yang. Yanyan Zhao, Yang Wu, Shilong Wang. Tian Zheng, Hongbo Zhang, Zongyang Ma. Wanxiang Che, and Bing Qin. Large language models meet text-centric multimodal sentiment analysis: A survey. arXiv preprint urXiv:2406.08065 2024.
[7] Rawan AlSaad, Alaa Abd-Alrazaq, Sabri Boughorbel, Arfan Ahmed, Max-Antoine Renault, Rafat Damseh, and Javaid Sheikh. Multimodal large language models in health care: Applications, challenges, and future outlook. J. Med, Internet Res.. 26xe59505, 2024.
[8] Yiwei Li, Huaqin Zhao, Hanqi Jiang, Yi Pan, Zhengliang Liu, Zihao Wu, Peng Shu, Jie Tian, Tianze Yang, Shaochen Xu, et al. Large language models for manufacturing, arXiv preprint arXiv:2410.214: 2024.
[9] Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Marcella Cornia, and Rita Cucchiara. The revolution of multimodal large language models: A survey. In Findings of the Asxociation for Compulational Linguistics: ACL 2024, pages 13590-13618, 2024.
[10] Can Cui, Yunsheng Ma, Xu Cao, Wengian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang. Kuei-Da Da Liao, Tianren Gao, Erlong Li, Kun Tang, Zhipeng Cao, Tong Zhou, Ao Liu, Xinrui Yan, Shuqi Mei, Jianguo Cao, Ziran Wang, and Chao Zheng. survey on multimodal large language models for autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops, pages 958-979, January 2024.
[11] Mark Turner. The art of compression. The artful mind: Cognitive science and the riddle of human creativity, pages 93 114, 2006.
[12] Eddie Gray and David Tall. Abstraction as natural process of mental compression. Mathematics Education Research Journal, 19(2):23-40) 20.23,00,220 2007.
[13] George A Miller. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review, 63(2):1, 1956.
[14] Eleanor Rosch. Cognitive representations of semantic categories. Journal of experimental psychology: General, 104,3):192, 1975.
[15] Li Zhaoping. Vision: looking and seeing through our brain's information bottleneck. arXiv preprint a8880:2503.18804 2025.
[16] Xingxuan Zhang, Jiansheng Li, Wenjing Chu, Junjia Hai, Renzhe Xu, Yuqing Yang, Shikai Guan, Jiazheng Xu, and Peng Cui. On the -uitt--distiiuutionn generalization of multimodal large language models, arXiv preprint X4vv22402.00554 2024.
[17] Zhongyi Han, Guanglin Zhou, Rundong He, Jindong Wang, Tailin Wu, Yilong Yin, Salman Khan, Lina Yao. Tongliang Liu, and Kun Zhang. How well does gpt-4v (ision) adapt distribution shifts? preliminary investigation. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024.
[18] Moon Ye- fe-Bin, Nam Hyeon- Woo. Wonseok Choi, and Tae- Hyun Oh. Beaf: Observing before-after changes to evaluate hallucination in vision-language models. In European Conference on Computer Vision, pages 232-248. Springer, 2025.
[19] Changdae Oh, Zhen Fang, Shawn Im, Xuefeng Du, and Yixuan Li. Understanding multimodal 1lms under distribution shifts: An information-thhoreticc approach. In International Conference on Machine Learning. 2025.
11

---

[20] Robert Geirhos. Jömn-Herrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks, Nature Machine Intelligence, 2(1 ((11))6656673 2020.
[21] Wenqian Ye, Guangtao Zheng. Yunsheng Ma, Cao, Bolin Lai, James Matthew Rehg, and Aidong 7hang. MM-spuhench; Towards better understanding of spurious biases in multimodal LLMs. In Workshop on Responsibly Building the Next Generation of Multimodal Foundational Models, 2024.
[22] Yiming Liang, Tianyu Zheng, Xinrun Du, Ge Zhang, Jiaheng Liu, Xingwei Qu, Wenqiang Zu, Xingrun Xing, Chujie Zheng, Lei Ma, al. Aligning instruction tuning with pre-training arXiv preprint arXiv:2501.099 09368,, 2025.
[23] Bo Zhao, Boya Wu, Muyang He, and Tiejun Huang. Svit: Scaling up visual instruction tuning. arXiv preprint arXiv: 040877 2023.
[24] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 2024.
[25] Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing. Liangdong Wang, Zhou Cao, Jintao Jia, Zhuoyi Zhang. Yixuan Wang, et al. infinity-mm: Scaling multimodal performance with large-scale and high-quality instruction data. arXiv preprint arXiv:2410. 18558, 2024.
[26] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li. Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[27] Yunhang Shen, Chaoyou Fu, Shaoqi Dong. Xiong Wang, Peixian Chen, Mengdan Zhang, Haoyu Cao, Ke Li. Xiawu Zheng, Yan Zhang, al. Long-vita Scaling large multi-modal models to million tokens with leading short-context accuray. arXiv preprint 27Xi:22502.05177, 2025.
[28] Zhe Chen, Weiyun Wang. Yue Cao. Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu. et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint 2rXiv:2412.05271, 2024.
[29] Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang. Jihan Yang. Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal Ilms. Advances in Neural Information Processing Systems, 2024.
[30] Min Shi, Fuxiao Liu. Shihao Wang. Shijia Liao, Subhashree Radhakrishnan, Yilin Zhao, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, Bryan Catanzaro, Andrew Tao, Jan Kautz, Zhiding Yu, and Guilin Liu. Eagle: Exploring the design space for multimodal LLMs with mixture of encoders. In The Thirteenth International Conference on Learning Representations, 2025.
[31] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-v1 technical report. arXiv preprint arXiv:2502.13922: 2025.
[32] Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Learning Representations, 2017.
[33] Matias Vera, Pablo Piantanida, and Leonardo Rey Vega. The role of the information bottleneck in representation learning. In 2018 IEEE International Symposium on Information Theory (ISIT), pages 1580-1584, 2018.
[34] Tailin Wu, Hongyu Ren, Pan Li, and Jure Leskovee. Graph information bottleneck. Advances in Neural Information Processing Systems, 25:20437720448, 2020.
[35] Rabech Karimi Mahabadi, Yonatan Belinkov, and James Henderson. Variational information bottleneck for effective low-resource tinettniig. arXiv preprint #rXiv:2106.05663 2021.
[36] Yawei Li, David Rügamer. Bernd Bischl, and Mina Rezaei. Calibrating LLMs with information-theoree evidential deep learning. In The Thirteenth International Conference on Learning Representations, 2025.
[37] Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
[38] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-349) 2023.
12

---

[39] Wenliang Dai, Junnan Li, DONGXU LI. Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li. Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Prucessing Systems, 36:49250-49266 2023.
[40] Tony Lee, Haoqin Tu, Chi Heem Wong, Wenhao Zheng, Yiyang Zhou, Yifan Mai, Josselin Roberts, Michihiro Yasunaga, Huaxiu Yao, Cihang Xie, et al. Vhelm: A holistic evaluation of vision language models, Advances in Neural Information Processing Systems, 37: 1706642-146666, 2024.
[41] Xiang Li, Congcong Wen, Yuan Hu, Zhenghang Yuan, and Xiao Xiang Zhu, Vision-language models in remote sensing: Current progress and future trends. IEEE Geoscience and Remote Sensing Magazine, 2024.
[42] Mubashar Raza, Zarmina Jahangir, Muhammad Bilal Riaz, Muhammad Jasim Saced, and Muham- mad Awais Sattar. Industrial applications of large language models. Scientific Reports, 15(1):13755, 2025.
[43] Jielin Qiu, Yi Zhu, Xingjian Shi, Florian Wenzel, Zhiqiang Tang. Ding Zhao, Bo Li, and Mu Li. Benchmarking robustness of multimodal image-text models under distribution shift. Journal of Data- centric Machine Learning Research, 2024.
[44] Xuamming Cui, Alejandro Aparcedo, Young Kyun Jang, and Ser-Nam Lim. the robustness of large multimodal models against image adversarial attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 24625-24634, 2024.
[45] Aayush Atul Verma, Amir Saeidi, Shamanthak Hegde, Ajay Therala, Fenil Denish Bardoliya, Nagaraju Machavarapu, Shri Ajay Kumar Ravindhiran, Srija Malyala, Agneet Chatterjee, Yezhou Yang, et al. Evaluating multimodal large language models across distribution shifts and augmentations. Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition. pages 5314-5322 2024.
[46] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057. 2000.
[47] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pages 1-5. Ieee, 2015.
[48] Yawei Luo, Ping Liu, Tao Guan, Junqing Yu, and Yi Yang. Significance-aware information bottleneck for domain adaptive semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6778-6787, 2019.
[49] Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust represen- tations via multi-view information bottleneck. In International Conference on Leurning Representations, 2020.
[50] Siqi Miao, Mía Liu, and Pan Li. Interpretable and generalizahle graph learning via stochastic attention mechanism. International Conference on Machine Learning, pages 15524-15543. PMIRR 2022.
[51] Zichuan Liu, Tianchun Wang, Jimeng Shi, Xu Zheng, Zhuomin Chen, Lei Song, Wenqian Dong, Jayantha Obeysekera, Farhad Shirani, and Dongsheng Luo. Timex++; Learning time-scries explanations with information bottleneck. In International Conference on Machine Learning, pages 32062-32082. PMLR, 2024.
[52] Kenji Kawaguchi, Zhun Deng, Xu Ji, and Jiaoyang Huang. How does information bottleneck help deep learning? International Conference on Machine Learning, pages 16049-16096. PMI.R, 2023.
[53] Zifeng Wang, Tong Jian, Aria Masoomi, Stratis loannidis, and Jennifer Dy. Revisiting hilbert-schmide information bottleneck for adversarial robustness. Advances in Neural Information Processing Systems, 34:586-597, 2021.
[54] Jiaqi Bai, Hongcheng Guo, Zhongyuan Peng. Jian Yang. Zhoujun Li, Mohan Li, and Zhihong Tian. Mitigating hallucinations in large vision-language models by adaptively constraining information flow. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23442-23450, 2025.
[55] Alex Graves. Practical variational inference for neural networks. Advances in neural information processing systems, 24, 2011.
[56] Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2014.
13

---

[57] David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe and. Variational inference: A review for statisticians. Journal of the American Statistical Association, 1225118)859-877.22 2017.
[58] Seong Joon Oh, Andrew C. Gallagher, Kevin P. Murphy, Florian Schroff, Jiyan Pan, and Joseph Roth. Modeling uncertainty with hedged instance embeddings. In International Conference on Learning Representations, 2019.
[59] Sanghyuk Chun, Scong Joon Oh, Rafacl Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Probabilistic embeddings for cross-modal retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8415-8424, 2021.
[60] Sanghyuk Chun. Improved probabilistic image-text representations. In The Twelfth International Conference on Learning Representations, 2024.
[61] Sanghyuk Chun, Wonjae Kim, Song Park, and Sangdoo Yun. Probabilistic language-mmage pre-training. The Thirteenth International Conference on Learning Representations, 2025.
[62] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, Advances in neural information processing systems, 30, 2017.
[63] Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective regularization to finetune large-scale pretrained language models. In International Conference on Learning Representations, 2020.
[64] Binghing Wen, Chenjun Xu, Robert Wolfe, Lucy Lu Wang. Bill Howe, al. Mitigating overeonfidence large language models: A behavioral lens on confidence estimation and calibration. In NeurlPS 2024 Workshop on Behavioral Machine Learning, 2024.
[65] Haoyan Yang. Yixuan Wang. Xingyin Xu, Hanyuan Zhang, and Yirong Bian. Can we trust ilms? mitigate overconfidence bias in iims through knowledge transfer. arXiv preprint arXiv:2405. 16856,, 2024.
[66] Tobias Groot and Matias Valdenegro- Toro. Overconfidence is key: Verbalized uncertainty evaluation in large language and vision-language models. In Proceedings of the 4th Workshop on Trustworthy Natural Language Processing (TrustNLP 2024), pages 145-171, 2024.
[67] Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy, Fixing a broken elbo. In International conference on machine learning, pages 159-168. PMLR, 2018.
[68] Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin. Cyclical annealing schedule: A simple approach to mitigating kl vanishing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pages 140-250,,500 2019.
[69] Yixin Wang, David Blei, and John P Cunningham. Posterior collapse and latent variable non-identifiability. Advances in neural information processing systems, 34:5443 5455, 2021.
[70] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved basclines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 26296626306, 2024.
[71] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry. Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8766 8763, PmLR. 2021.
[72] Wei-Lin Chiang, Zhuohan Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
[73] Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vims: Investigating the design space of visuall--conditione language models. In Forty-first International Conference on Muchine Learning, 2024.
[74] Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. LLaVA-mini: Efficient image and video large multimodal models with one vision token. In The Thirteenth International Conference on Learning Representations, 2025.
[75] Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad S. Khan. Llava++: Extending visual capabilities with Ilama-3 and phi-3, 2024.
14

---

[76] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Erie Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595 2023.
[77] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision- languugg models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 292-305, 2023.
[78] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu. Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 01-2207725221 2022.
[79] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi- dustti-discipiiee multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556-9567, 2024.
[80] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang. Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? In The eighth Annual Conference on Neural Information Processing Systems, 2024.
[81] Ponyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Tacsiri, and Anh Totti Nguyen. Vision language models are blind. In Proceedings of the Asian Conference on Computer Vision, pages 18-34, 2024.
[82] Lin Long, Changdae Oh, Seongheon Park, and Sharon Li. Understanding language prior of Ivlms by contrasting -ebieddig- arXiv preprint 2rXivv2509.23050 2025.
[83] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[84] Haochen Wang, Anlin Zheng, Yucheng Zhao, Tiancai Wang, Zheng Ge. Xiangyu Zhang, and Zhaoxiang Zhang. Reconstructive visual instruction tuning. In The Thirleenth International Conference on Learning Representations, 2025.
[85] Zhihan Zhou, Feng Hong, Jiaan Luo, Jiangehao Yao, Dongsheng Li, Bo Han, Ya Zhang, and Yanfeng Wang. Learning to instruct for visual instruction uning. arXiv preprint arXiv:2503.222 22215, 2025.
[86] Min-Hsuan Yeh, Jeffrey Wang, Xuefeng Du, Seongheon Park, Leitian Tao, Shawn Im, and Yixuan Li. Position: Challenges and future directions of data-centric Al alignment. In Forty-second International Conference on Machine Learning Position Paper Track, 2025.
[87] Kang-il Lec, Minbeom Kim, Seunghyun Yoon, Minsung Kim, Dongrycol Lec, Hyukhun Koh, and Kyomin Jung. Vlind-bench: Measuring language priors in large lision-languag models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4129 4144, 2025.
[88] Yingjun Du, Jun Xu, Huan Xiong, Qiang Qiu, Xiantong Zhen, Cees GM Snock, and Ling Shao. Learning learn with variational information bottleneck for domain generalization. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X 16. pages 200-216. Springer, 2020.
[89] Bo Li, Yifei Shen, Yezhen Wang, Wenzhen Zhu, Dongsheng Li, Kurt Keutzer, and Han 7hao. Invariant information bottleneck for domain generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7399-7407, 2022.
[90] Jiao Zhang, Xu-Yao Zhang, Chuang Wang, and Cheng-Lin Liu. Deep representation learning for domain generalization with information bottleneck principle. Pattern Recognition, 143: 2023.
[91] Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Kathrit Dombrowski, et al. Representation engincering: A top-down approach ai transparency, arXiv preprint arXiv:2310.01 2023.
[92] Sheng Liu, Haotian Ye, and James Zou. Reducing hallucinations in large vision-language models via latent space steering. In The Thirleenth International Conference on Learning Representations, 2025.
[93] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m. Open dataset of elip-filtered 400 million image-text text pairs. arXiv preprint arXiv:2111.02114,, 2021.
15

---

[94] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed. image alt-text dataset for automatic image captioning. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556, Melbourne, Australia, July 2018. Association for Computational Linguisties.
[95] Vicente Ordonez. Girish Kulkarni, and Tamara Berg. 1m2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems, 24, 2011.
[96] Jumnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision llsion-language understanding and generation. In International conference on machine learning, pages 12888 12900. PMLR, 2022.
197] Kegin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal I1m's referential dialogue magic. arXiv preprint vrXiv:2306.15199 2023.
[98] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang. Anwen Hu, Pengcheng Shi, Yaya Shi, al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2.004. /4178, 2023.
[99] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCum, and Saining Xic. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568-9578, 2024.
[100] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-trainin In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975-11986, 2023.
[101] Maxime Oquab, Timothée Darcet. Théo Moutakanni. Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa. Alaaeldin El-Nouby. Mido Assran, Nicolas Ballas, Wojeiech Galuba. Russell Howes, Po- Yao Huang. Shang- Wen Li, Ishan Misra. Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. Featured Certification.
[102] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
[103] Tsung- Yi Lin, Michael Maire, Serge Belongic, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6- 2014, proceedings, part v 13, pages 740-755. Springer, 2014.
[104] Bo Li, Kaichen Zhang, Hao Zhang. Dong Guo, Renrui Zhang. Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger Iims supercharge multimodal capabilities in the wild, May 2024.
[105] Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, and Bill Yuchen Lin. Wildvision: Evaluating vision- language models in the wild with human preferences. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[106] Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club: A contrastive log-ratio upper bound of mutual information. In International conference on machine learning, pages 1779-1788. PMLR, 2020.
[107] A Conneau. Unsupervised cross-lingual representation learning at scale, arXiv preprint 191 2019.
[108] Jhoan K Hoyos-Osorio and Luis G Sanchez-Giralde The representation jensen- shannon divergence. arXiv preprint 1.XXv:2305516444: 2023.
[109] Oscar Skean, Md Rifat Arefin, Dan Zhao. Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations language models. arXiv preprint arXiv:2502.02013, 2025.
[110] Evan Hernandez and Jacob Andreas. The low low-dimensional linear geometry of contextualized word representations. In Proceedings of the 25th Conference on Computational Natural Language Learning. pages 82-93, 2021.
16

---

[111] Zhuo-Yare Song. Zeyu Li, Qing-Hong Cao. Ming-xing Luo, and Hua Xing Zhu. Bridging the dimensional chasm: Uncover layer-wise dimensional reduction in transformers through token correlation. arXiv preprint (oX.ix::503.255:: 2025.
[112] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan, A diversity-promooing objective function for neural conversation models. arXiv preprint 15000055 2015.
[113] Chantal Shaib, Joc Barrow, Jiuding Sun, Alexa F Siu, Byron C Wallace, and Ani Nenkova. Standard- izing the mcasurement of text diversity; A tool and comparative analysis of scores. arXiv preprint arXiv:2403.000 00555,, 2024.
[114] Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds deep nets via a compression approach. In International conference on machine learning, pages 254-263. PMLR, 2018.
[115] Nick Littlestone and Manfred Warmuth. Relating data compression and Icarnability. Unpublished notes, 1986.
[116] Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K. Warmuth. Occam's razor. Information Processing Letters, 24(6):377-388 1987.
[117] J. Rissanen. Modeling by shortest data description. Automatica, 14(5):465 1978.
[118] Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pages 5-13, 1993.
[119] Peter Grünwald. Minimum description length tutorial. Advances in minimum description length: Theory and applications, 5:1-80, 2005.
[120] Milad Sefidgaran, Abdellatif Zaidi, and Piotr Krasnowski. Minimum description length and generalization guarantees for representation learning, Advances in Neural Information Processing Systems, 36:1489 1525, 2023.
[121] Horace B Barlow et al. Possible principles underlying the transformation of sensory messages. Sensory communication, 1(01):217-233 233,, 1961.
[122] Andrew Gordon Wilson. Deep learning not mysterious different. arXiv preprint 2311032503.00113. 2025.
[123] Mare Finzi, Gregory Benton, and Andrew G Wilson, Residual pathway priors soft equivariance constraints. Advances in Neural Information Processing Systems, 34:30037-30049, 2021.
[124] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations, 2022.
[125] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 7959-7971, 2022.
[126] Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsca Finn. Surgical fine-tuning improves adaptation to distribution shifts, In The Eleventh International Conference on Learning Representations, 2023.
[127] Sachin Goyal, Ananya Kumar, Sankalp Garg. Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 1 19338-19347, 2023.
[128] Junjiao Tian, Zecheng He, Xiaoliang Dai, Chih-Yao Ma, Yen- Yen-Cheng Liu, and Zsolt Kira. Trainable projected gradient method for robust fine-tuning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7836-7845, 2023.
[129] Changdae Oh, Mijoo Kim, Hyesu Lim, Junhyeok Park, Euiseog Jeong, Zhi Qi Cheng, and Kyungwoo Song. Towards calibrated robust fine-tuning of vision vision-language language models. Advances in Neural Information Processing Systems, 37, 2024.
17

---

[130] Jaedong Hwang, Brian Cheung, Zhang- Wei Hong, Akhilan Boopathy, Pulkit Agrawal, and Ila Fiete. Imagenet-rib benchmark: Large pre-training datasets don't guarantee robustness after fine-tuning, arXiv preprint urXiv:2410.21555 2024.
[131] Changdae Oh, Yixuan Li. Kyungwoo Song. Sangdoo Yun, and Dongyoon Han. Dawin: Training free dynamic weight interpolation for robust adaptation. The Thirteenth International Conference on Learning Representations, 2025.
[132] Fuxiao Liu,, Kevin Lin, Linjie Jianfeng Wang, Yaser Yacoob, Lijuan Wang. Mitigating hallucination large multi-modal models via robust instruction tuning. The Twelfth International Conference on Learning Representations, 2024.
[133] Wei Han, Hui Chen, and Soujanya Poria. Towards robust instruction tuning on multimodal large language models. arXiv preprint arXiv:2402. / 14492, 2024.
[134] Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2897-22 2905, 2018.
[135] Changjian Shui, Qi Chen, Jun Wen, Fan Zhou, Christian Gagné, and Boyu Wang. A novel domain adaptation theory with jensen shannon divergence. Knowledge-Basee Systems, 257:109808, 2022.
18

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We explicitly mentioned our research problem with scope and contribution in the abstract and introduction sections. These consistently stated through the remaining sections (Sec and 3), and supported by empirical evidence 4.2.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification: We included the limitations this work at Appendix due to space constraint of the main paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [Yes]
Justification:
We clearly elaborated the full set of assumptions and proof in Appendix E.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In Section 4.1 and Section A, we provided the full experimental details for easy reproduction, and we also provide our code through the GitHub repository link in the abstract.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
20

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: provide the entire code for the full reproduction (from training to evaluation) with an appropriate level of instructions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips. wwtpccccpppppppeecccccccccccccec teliic/guiieee gww.g//iie/eeeeeeeo Codeeubmii sisiissiinns ssisnPolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provided all the training and test details, including data splits, hyperparam- eters, how they were chosen, type of optimizer in Appendix Section A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Given the fact that this a work on full fine-tuning of large-scale (2 7B) models, learning multiple training runs claim statistical significance computationally intractable for us, but we conduct multiple runs for the GPT-4o-Judge-based evaluation to provide stable conclusion and mentioned correctly.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We included the computation resource-related details in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttts:///eurippt cc wwtpppcccc/eeeeeeccceccceeeeeeee httppulc/ttthsuhliiiiplpe htt./hhihiiiieenneeee..ee
Answer: [Yes]
Justification:
The authors acknowledged the NeurlPS Code of Ethics, and this work conforms
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We provide an impact statement section (Section F) to introduce potential societal impacts of this work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
22

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: As we propose a learning objective for (M)LLM, it inherently has the risk of misuse of LLM, and we discuss this in Section F.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: In the implementation details section (Section A) and the instruction file of our code, we clearly mentioned the source of all assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
23

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented, and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide our code through the anonymous repository link, and the code supports the generation of new datasets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: There is crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There is no crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: We only used L.M for the purpose paper editing, logo generation, and model evaluation for the open-ended question-answer task following the academic benchmarking standard.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: http///eeurr.ppeee cc/Conferences///00 /LLM) for what should or should not be described.
25

---

Appendix
Contents
Extended Experiment Setup and Implementation Detail
26
A.I Model and Training
26
A.2 Vittle Implementation Details
28
A.3 Downstream Task Benchmark Construction
30
A.4 Evaluation Details
Additional Results
32
B.1 Ablation Study
32
................................
B.2
Full Results of Pair-wise Cosine Distance Comparison
................................
B.3
Full Results with LLaVA-v1.5-7B and LLaVA-v1.5-PB
33
B.4
Further Investigation on Vittle
B.5
Applicability to Other MLLMs
36
C Extended Literature Review
37
Derivation of Variational Bound for IB in MLLM
38
E Missing Proof
39
E.1 Preliminary
39
E,2 A New Upper Bound for Effective Mutual Information Difference
39
Impact Statement
A Extended Experiment Setup and Implementation Detail
A.1 Model and Training
In this work, we consider LLaVA-v1.5 [70] target multimodal large language model (MLLM) with CLIP ViT-L/14-336px [71] and Vicuna-vI.5-7B [72] as visual encoder and LLM backbone, respectively, and two-layer MI. projector (modality connector that maps features of the visual encoder into the text embedding space). Although all the results presented in the main body of the paper were produced with LLAVA-v1.5-7B. we also experimented with (with Vicua-v1.5-15
LLa LLLVA--1.55-11 method, and consider Prism-7B [73] as an additional MLLM architecture to validate the broad applicability of Vittle. For fair comparison, all models are trained on the LLaVA-pretrail wwt.-ppreettn-aee- -55k and LLaVA -ixa-666xx datasets, consisting of a mixture LAION [93], CC [94], SBU [95] datasets with BLIP captions [96] and mixture of (LaVA-instrus
as the LLM backbone) to validate the scalability of our
158K and academic-tane tasicc-tt--onoo faik-trieetedd (V)QA datasets, respectively. Training configurations such as optimizer, learning rate, and batch size are summarized in Table and Table 8. All training runs are conducted with eight A A100-80GB GPUs with DeepSpeed ZeRO library. The shortest run takes roughly hours, whereas the longest run takes about 14 hours. Now, claborate the overall workflow of LLaVA and Prism below.
26

---

Table 8: Hyperparameter list of Prism- Vittle training. We adopt exactly the same configurations with Prism-DINOSigLII Controlled-79 [73] single stage training.
Table 7: Hyperparameter list of Vittle train- ing. We adopt exactly the same configurations with LLaVA-v1.5 [70] for Stage and 2.
Config
Stage1
Stage2
Config
Value
Global batch size
256
128
Batch size per GPU
32
16
Global hatch size
Learning rate
le.3
2e-5
Batch size per GPU
Learning rate schodule
Cosine docay w/ lincar warmup
Learning rate
2e-5
Warmup ratio
0,03
Learning schodule
cosine docay w/ linear-wamup 0.03
Weight decay
0.0
Warmup ratio
Epoch
Weight decay
Optimizer
AdamW
Epoch Optimizer
Procision
b
Adam
Precision
bf16
LLaVA is built with pre-trained visual encoder that takes visual inputs, a pre- pre-trained LLM backbone that takes text instructions, and a lightweight projector that maps features produced by the visual encoder into the text embedding space of LLM backbone that the risually-grounde instruction input query can be processed by the L.M backbone. LLaVA undergoes a two-stage training: (1) The first stage takes into account modality alignment, where the projector is trained on image and corresponding instruction or caption with a conditional language modeling loss implemented by aggregating cross-entropy losses across response tokens, while the visual encoder and LLM backbone are frozen. (2) The second stage stands for the instruction tuning, where the projector and LLM backbone are jointly trained on multimodal instruction samples with the same conditional language modeling loss while the visual encoder is still frozen. This two-stage training has been considered a standard approach for developing MLLMs and is widely adopted [97. 98, 29].
multimodal
Prism has a model architecture similar to LLaVA, but provides some valuable insight into the design of the MILM training recipe, and we note two remarkable design choices of Prism that distinguish it from VAA (1) incorporating multiple visual encoders rather than hosting a single visual encoder, and (2) reducing the two-stage biignment-then-instteeee tuning. Note that different self-supervised visual representation learning induces features that have different strengths, and several works reveal the benefits of ensembling multiple different visual encoders to leverage complementary advantages [99, 73, 30]. Prism incorporates SigLIP [100] and DINOv2 [101] to enjoy both a robust global feature and fine-grained local feature. Meanwhile, Karamcheti et al. [73] showed that the simplified single-stage training strategy can be a cost-effective alternative to the standard two-stage training.
tuning into single-stage instruction
To train these MILLMs, we consider five baseline approaches: (1) the standard full LLM fine- tuning with conditional language modeling loss, (2) parameter-effiiient LoRA [83] fine-tuning with the conditional language modeling loss, (3) conditional language modeling loss with weight decay regularization, (4) reconstructive visual instruction tuning (ROSS) [84], and (5) learning to instruct (LIT) [85]. For the LoRA-based training configuration, we use the same one provided by the official LLaVA-v1.5 repository", and for the weight decay regularization, we select the regularization magnitude parameter among {0.1, 0.01, 0.001} based on the POPE evaluation result. We now elucidate two competitive baseline methods, ROSS and LIT, in the following paragraphs. It is worth noting that these methods are designed to encode more (visual) information into the representation space, which is opposite to our Vittle's design motivation that pursues a minimal sufficient representation for improving robustness to distribution shifts.
Reconstructive visual instruction tuning (ROSS) follows the two-stage training of LLaVA, but tries to reconstruct the visual inputs from the LLM backbone by adopting a regression or denoising learning objective in addition to language modeling loss during its second stage. By doing so, ROSS guides the MLLM to learn a much richer visual understanding, which is usually lacking in modern MLLMS 199, 81]. The reconstruction target can be a raw RGB pixel value or the latent representation from an external visual encoder such as VQGAN [102] or VAE [56], and ROSS requires an additional trainable module to reconstruct visual content, which is discarded during inference. We follow the training recipe from the official code repository to replicate -DOS----B with the same visual encoder and L.M backbone to L... For a fair comparison with I.L. VA and Vittle, we
http::///githuu..om//eoeeeee http://laocceen-vaaan0/////ee Wang409/ross
27

---

train the ROSS with the same dataset (that of LLaVA-v1.5) for both training stages, while the original ROSS model was trained on slightly larger dataset the second stage.
Learning to Instruct (L.IT) also focuses on the visual shortcomings of current MILM and tries to improve the visual understanding capabilitt of MILLM by incorporating an additional loss term that incentivizes the encoding of additional visual information. To be specific, while the cross-entropy loss in LLaVA's conditional language modeling objective aggregated through the response tokens only, LIT introduces an extra cross-entropy loss term, which is aggregated over the instruction (question) tokens only, thereby enforcing MLLM learn to predict proper textual instruction given an image. As LIT uses the same visual and language backbone model and training dataset as LLaVA-v1.5, we use the pre-trained checkpoint of LIT from Hugging Face' for evaluation.
In Figure 6, we observe that while ROSS and LIT are somewhat effective in improving performance on object hallucination detection tasks with the aid of enhanced visual understanding capability, they significantly underperform Vittle and even the original LLaVA on the open-ended QA task under distribution shifts. This implies that pursuing more information encoding during visual instruction tuning may not result in better robustness to distribution shifts, but aiming to learn a minimal sufficient representation via Vittle can be a promising solution for this (See Table 12 for details).
A.2 Vittle Implementation Details
This section provides additional details on implementing Vittle through Python-style pseudo code in Figure 9 and text below. Following the standard two-stage LLaVA training recipe, freeze the visual encoder and LLM backbone during the first stage and only train the projector module. In the second stage, Vittle inserts bottleneck layer Iq, consisting of two of the two- layer MLPs {0o. Jo. for visual and textual modalities, inside the LLM backbone to estimate the distributional parameters (mean and diagonal covariance) of the posterior Gaussian distributions for each visual and textual token. Each bottleneck module is constructed with Lanner (d,d), nn. GELU(), nn. Linear (d, 2*d)} } where d denotes the hidden dimension of the LLM backbone, and this results in a slightly increased number of model parameters (up to 1.5% from the baseline). We use these estimated distribution parameters to sample a representation from this posterior via E JA + a @ where ~ NO. 7). Then, for given bottleneck layer index and for the maximum length of visual M, and textual input tokens M,, the bottleneck layer Js takes sequence of token representations 2 = [2e,1, ..... M.. 2t,1, 2t,M, } produced from the layer 1 to build information- penalized representations 8 = (2 ...: M, ,,.,,.... M,,M, }. where 2 = (1 - a)= + agg(z). Here, we use an interpolated representation between the original pre-bottleneck representation and the post-bottleneck representation (=())) with an interpolation coefficient a that progressively grows from 0 to 0.5 by a cosine schedule during training. We observe that solely using the post-bottleneck representation induces a diverging language modeling loss at the later steps of training, and speculate that it is hard to generate a valid response with the information information-peenaized representation only".
Then, we jointly train the LLM backbone, the projector, and this bottleneck layer together during the second stage of training with the objective function 5. As we assume a diagonal covariance Gaussian for the prior and posterior distributions, Kullback-Leible divergence (KLD) between the prior p and posterior can be easily expressed as below.
1 1+ (ppl) M2DD) obDi
L Of
DKL p)
2 log obDi
(8)
where μ. and a. denote d-dimensional distributional parameter vectors and [j] indicates j-th element from the vectors. Vittle has two important hyperparameters: (1) target layer index / for bottleneck application, and (2) posterior KLD regularization strength parameter J. After tuning across e
7. 4 /) where d denotes the latent dimension of the LLM backbone, we set / = 24 and 0.1 /d based on the average performance of POPE and LB-COCO clean datasets. The interpolation coefficient can also be tuned, but we found that increasing beyond 0.5 hinders stable training and observing increased language modeling loss at the later parts of training progress. Figure 12 and Table 11 present the results of hyperparameter ablation study.
(24, 28, 31} and 8 E 00 0.05 0.1 0.2
7 htttps://hugiinggacc www.co///iii.
hh/ www.hhhhhhhhuuuha LIT- LLaVA-L A- -1. VicuVa-aaaa-- 7B/ "treeeemaai
'We suspect that this is because the initial parameter of MLLM's instruction tuning are far from random standard Gaussian zero therefore induces unstable learning signal from the KLD term
28

---

def reparam (mu, logvar):
std (logvar 2). exp()
batch_size, seq_len, hidden_diz mu shape
torch. t randn (batch____ssieee seq_len, hidden_dim) return mu std *
def forward(self. input_ len a, **kuargs):
hidden_states input_embede for 1_idx, 11m_layer in enumerate (self. (self 11m. layers):
layer_outputs alm_layer(hidden.staa states,, **kwaggs)
hidden_states layer ouyer_outputs
if 1_idx =- wwt..boteecc..ayerr..... posterior inference
v-params self selifege(hiddenstttteee
:issi_s. :])
t_params
htt.::/hhhcceeetttttteeeeeeeeeee ;]]
v_mean [.params(i..,:self.bbl
v_logvar ".prrass(::::sfff...ddd::::
t_mean i.paras[i....:efff..bd.
t_logvar s.parans(:....eff......
v_post reparam(v_mean) v_logvar)
t_post reparam(t_mean, t_logvar)
z_post = torch ((v_pos, t_post))
interpolation between original and boftlenecked hidden_ states (1-a) hidden_states
z_post
Listing 1: Forward pass of Vittle
def normalized_kld (mu, logvar, modality=None):
if modality None:
vittle (F) fized prior (0)
kl_loss -0.5
exp
exp ()). mean ()
else:
vittle (L) learnable prior
mu_pr, logvar_pr self.l.prior [modality]
logvar_d logvar
scaled_mu_d = (mu-mu_pr)
(20//ooo exp
var_ratio logvar w/wp://r__...eee//e. (xp
kl_loss -0.5 d
cerl-vvrrrattti waattoooooo mean ()
return kl_loss
def loss (self, logita, labels, v_mean, v_logvar, t_mean, t_logvar):
lm_loss self wow.looos_funtiiii functionnlogiiaa, (logits, labels) if self prirrra.ee.eepooor
flag_v, flag_t "y", else:
flag_v, flag-t None, None kld_v self nirmalized_kid (id kld_t self normalized_kle (t (t_mean, t_logvar, flag-t)
v_logvar, flag_v)
return lm_loss + self beta (kld_v kld_t)
Listing 2: Training objective of Vittle
Figure 9: PyTorch-style pseudo code for the forward pass and training objective of Vittle
29

---

A.3 Downstream Task Benchmark Construction
Open-ended QA task. One of the most representative applications of an MLLM is the generation of free-form responses given multimodal instruction queries. We consider LLaVA ---Becchhhh COCO (LB-COCO; [38]) as a typical in-distribution (ID) open-ended QA dataset, which is constructed from MS-COCO images [103] with GPT-gennrraed text queries that have 90 pairs of image and text. We then generate 27 variants of this LB-COCO by applying nine types of image perturbations, nine types of text perturbations, and nine types of image-text joint perturbations, to benchmark MLLMs' robustness under various distribution shifts (which will be claborated at the end of this section). Meanwhile, we also consider three datasets LLaVA-Benne in-the-wild (LB-Wild; [38]), LLaVA-Bench Wilder L.B-Wilder; [104]), and WildVision Bench (WV-Bench; [105]) constructed by real-world web users' image-ted text paired queries of 60, 128, and 500 samples, respectively, to validate models' capability to address long- tailed queries in practice. This results in 31 different open-ended QA datasets in total: clean and 27 perturbed LB-COCO variants, and three long- tai-tail datasets.
Object hallucination detection task. Mcanwhile, one of the most crucial evaluation aspects of an MLLM is the degree of hallucination of its output. A representative benchmark for this is the POPE dataset [77]. where the model is tasked to answer in binary (Yes, No} form given a question about the object's 's existence given an image. The POPE dataset was also created from the MS-COCO source images with 9,000 corresponding questions, and we consider this dataset as an ID dataset. As we did for the LB-COCO dataset, we generated 9 variants of POPE by applying nine types of visual perturbations to images. Textual perturbations were not considered here because the text query of this dataset is relatively short, so perturbing the core object word token can distort the desired semantics of the question. In summary, we conducted validation on 10 different POPE variants.
Closed-form QA task. There are numerous closed-form QA datasets that assess the internal knowledge of MLLMs from various perspectives. In this paper, consider four representative datasets: ScienceQA [78], MMMU [79]. MME [5], and MMStar [80]. which are designed to validate multimodal knowledge and understanding capability across various domains.
Distribution shift simulation. The goal of this study is to improve the robustness of MLLM under distribution shifts. We mainly foeus on subtle perturbations on image and text, which is worth-noting problem given the fact that current MI.MMs undergo systematic performance degradation under perturbations. We consider nine visual perturbations listed in Table 9, nine textual perturbations listed in Table 10, and nine image-text joint perturbations: {zoom blur, frost, gaussian [ x (arabic, greek, hindi]. The translations for Arabic, Greek, and Hindi languages from English are conducted by OpenAI GPT-4o with prompt: "Please translate a {SOURCE) sentence provided by the user into [TARGET).** and all the remaining perturbations are generated MMRobustness source code?. The actual examples of each visual textual perturbation are presented in Figure 1 and Figure 10.
Table 9: List of visual perturbations. We con- sider nine visual perturbations from four cate- gories: (1) Blur, (2) Digital, (3) Weather, and (4) Noise, to validate the robustness of MLLMs under diverse types of visual perturbations.
Table 10: List of textual perturbations. We consider nine textual perturbations from three categories: (1) character-level, (2) word-level, and (3) sentence-level, to validate the robustness of MILMss under diverse types of textual pertur- bations.
Name
Category
Name
Category
Defocus Blur Zoom Blur
Blur Blur
Char Typo Char Delete
Character-level Perturbation
Character-level Perturbation
Contrast
Digital
Char Insert
Character-level Perturbation
Brightness
Weather
Word Swap
Word-level Perturbation
Fog
Weather
Word Delete
Word-level Perturbation
Frost
Weather
Word Insert
Word-level Perturbation
Gaussian Noise
Noise
Arabic Translation
Sentence-level Perturbation
Shot Noise
Noise
Greek Translation
Sentence-level Perturbation
Speckle Noise
Noise
Hindi Translation
Sentence- Perturbation
'nttps://github wtt.tthhb..omm/illnia.. http:/iiiiiii////ooouuuueeeeee
30

---

Defocus Blur
Zoom Blur
Contrast
Original
Brightness
Fog
Frost
Gaussian Noise
Shot Noise
Speckle Noise
Figure 10: Examples of visual perturbations.
Char Typo
Char Delete
Char Insert
thaH 1s the main facus of the knagR?
Wh is the main fos the image?
DWwhat is the un(ain focus of the image?
Original
Word Swap main
Word Delete
Word Insert
what is the nain focus inage?
the nain focus image
what exist is the nain focus of the inage
facus of what inage
Arabic
Greek
Hindi
Nloo sivii sugio eucóvag
bia सा 18s कोमय साा
Figure 11: Examples textual perturbations.
A.4 Evaluation Details
Open-ended QA task. Compared to multi-choice closed-form QA tasks that have unique ground- truth answer per question, open-ended free-form generation-style QA tasks do not provide a single ground-truth answer. We follow the current standard evaluation paradigm, (M)LLM-as---Judee uses an external (usually more powerful) MILM to gauge the quality of our target MLLM of interest via prompting. To be specific, for given input query reference answer y. MLLM X - >, and the judge model r X'
that
Z+, relative preference score is defined as, Eyy. (t))) r(a.y)
For all of our open-ended QA evaluations, we used the same system prompt template provided by LLaVA authors to and we also adopted the MS-COCO annotation' bbased GPT-4 response' 12 and the released by I.Ia V.-.eA- authors as reference answers for LB-COCO variants and LB- respectively. For LB-Wild and WV-Bench, we generated reference answers with GPT--4o.
httppp/g..thcccoca..a..////////r 11 http:://s...cccoooococc///////// coco2014_val__0.jjsa
http://hhh..hoaacecc.///o/////aa heel/aa90_gpt__aneeee jsonl "attps://hugginggaccooo
31

---

Object hallucination detection and closed-form QA task. In contrast to open-ended tasks, all object hallucination detection and closed-form QA tasks provide a single ground truth answer as a form of discrete labels such as { Yes, No} and (A, B, C. D. .... For the multi-choice QA datasets, MMMU, MMStar, and ScienceQA, we attached a subfix prompt: "Answer in a character from the given choices directly." at the end of each question for answer formatting, while using the original question text for YES-or-NO datasets, MME and POPE, without a formatting prompt. We measured the exact matching accuracy Ex,y (x)) = v)] for these tasks.
J.
Effective Mutual Information Difference (EMID) and Jensen-Shannow Divergence (JSD). In addition to the evaluation with traditional metries, we also consider the EMID and JSD-based evalua- tion, which was recently proposed as an information-theorede approach to measure the robustness of MLLMs [19]. To compute the empirical estimates of MI, which is required for EMID compu- tation, we use the CLUB estimator [106] and reproduce the training and inference process of [19] by adopting image and text embeddings for the input image and text from CLIP-ViT-BAA [71] and KLM-RoBERTS. Basee [107] to replace X₂ and Xt, and also the text embeddings of --B-RaBERTa- Base for responses Y and Ye. To compute empirical estimates of JSD, we adopted the representation JSD estimator [108] on top of the CLIP- ViT-B323 and H.M-RoBERTTa Basse, too.
Representation analysis. Inspired by a recent work that reveals the importance of intermediate layer representation of the LLM backbone [109], we use the last input token embedding of the 24th layer (out of 32 layers in a 7B LLM backbone) for all experiments carried out in the representation space (Figure (b) right, Figure 8, Figure 13, and the JSD computation in Table 4).
B
Additional Results
B.1
Ablation Study
We first investigate two important hyperparameters for Vittle: (1) bottleneck layer index I and (2) KLD regularization strength 8, where we determined those parameter values based on the average performance on the clean POPE and LB-COCO, while not observing performance on perturbation datasets for fair model selection. We then further explore the impact of the interpolation coefficient a, which plays role in controlling the balance between the original representation and the bottleneck representation. Note that we could not conduct such an extensive search due to the computational burden of training 7B 13B scale models, so the hyperparameter values found here may not be optimal, and ttle can achieve better results with further hyperparameter tuning.
Baseline L24
L28
L31
Easeliie 0.01
0.05/
0.146
0/2/d
1.M
Relative Peoference
02
Rclative Proference P
43-
POPE
POPE V Statt
18-CCC
LB-CCC T d
POPE
POPE Stits
L8-COCO
LB-COCO Shits
Figure 12: Ablation study for the bottleneck layer index (left) and KLD regularization magnitude parameter 3 (right).
For bottleneck target layer ablation (Figure 12 left), we swept across {8. 16, 20, 24, 28, 31} out of 32 layers of the 7B- Bi-size LLM backbone. However, applying the bottleneck on the early layer failed to make the language modeling loss converge, so we only provided results for 24, 28, and 31 layers. We observed that intermediate layers (L24 and L28) achieve better results than the penultimate layer (L.31), and L,44 shows better results on POPE while L228 outperforms 1.24 on LB-COCO. In conclusion, applying the bottleneck to too early parts hinders shaping some shallow syntactic features that will be actively used at later parts of the layers [1]0, whereas applying to too late parts hurts output-specific alignment or formatting [111], which guide us to decide intermediate layer, i.c., 24th,
32

---

as default choice. This is in line with recent finding that the intermediate layer of LLM matters more than the early or later layers by showing that the quality measurements of the intermediate layer representations have stronger correlation with performance in downstream tasks [109]. Although we can search different layer indices for visual and textual tokens, we leave this to future work.
For KLD regularization strength parameter ablation (Figure 12 right), we swept across {0.01, 0.05, 0.1, 0.2, 1.0), and found that in the POPE dataset, strong regularization results in better performance, whereas it is not the case for LB-COCO. We choose 0.1 as our default, which induces balanced clean-data performance on these two tasks.
Table 11:
Ablation study for the representation interpolation coefficient a of the bottleneck layer. We observe that using the bottlenecked representation beyond the half portion of the total hinders the convergence of the language modeling loss.
Alpha
POPE
POPE Shifts Avg.
LB-COCK
LB-COCO Shifts Avg.
Baseline
86.98
84.12
77.8
72.3
0.1
87.22
84.20
77.9
73.1
0.25
87.34
84.47
75.6
73.1
0.5
87.71
84.90
76.7
73.0
0.75
Failed to converge
Failed to converge
We also explore the effect of the representation interpolation parameter a E [0, 1], which can be interpreted as gating mechanism to control the information flow. As a approaches one, the later parts of the LLM backbone (LLM head in our notation) mainly use the intormation-penallinn representation, while if a becomes smaller, the model strongly relies on the original representation. In Table 11, observe that using too large values of results in diverging language modeling loss, indicating that using strongly penalized representation only cannot predict proper response tokens sequence. Meanwhile, the larger value of induces better POPE performance, whereas the trend is inconsistent in the LB-COCC data set, which is consistent with the observations from the previous ablation study in B.
B.2 Full Results of Pair-wise Cosine Distance Comparison
We speculate that the performance degradation of MLLMs under perturbations originates from the representation discrepancy between clean and perturbed samples. That is, in the ideal case, a clean sample and its semantically equivalent perturbed sample should be closely mapped in the representation space, but current MLLMs did not shape the representation space in that way (see Figure and Figure 8). In Figure 13, provide the histograms of representation space pair-wise cosine distance between clean and perturbed examples in 27 types of perturbations. As we can see, Vittle (F) consistently mitigates the representation gap by reducing the pair-wit mair-wise distance diverse types of perturbations.
B.3 Full Results with LLaVA-v1.5-7B and LLaVA-v1.551332
Table 12 summarizes the overall results of our perturbation benchmarks on object hallucination detection (POPE) and open-ended QA tasks (LB-COCO). We note two findings here: (1) weight- space regularization methods, such as LoRA and WD failed to achieve reasonable performance; (2) although information maximization- based instruction tuning methods, such as ROSS and LIT, somewhat improve performance on POPE and its perturbation datasets, they greatly underperform Vittle, indicating non-trivial challenge to design a versatile instruction tuning objective that can improve MILMs on broad tasks. Meanwhile, we explore whether Vittle can be effective for a much larger model, e.g., a 13B-scale model. Table 13 shows that Vittle achieves consistent performance gains in object hallucination detection and open-ended QA tasks under distribution shifts, implying the scalability of our method.
33

---

(V) Gaussian Noise
(T) Word Delete
(V) Frost (T) Arabic
80
Ravoriie (1 003)
Raceline (8:541)
Basadine (0.1200
Count
Oum (0.023)
Count
Oura (0.137)
0.000
0:50 Cosline Distance Pained Samples
0.29
Cosine Distance of Paired Samples
Castiee Distance of Pained Semples
Shot Noise
(T) Word Swap
(V) Frost: (T) Greek
(0.6
Count
10-
Count
0.800
8.005
0.10
8,002
0.20
822
5.00
0.33
8:30
2.0
9.3
6.5
Cosino Distance of Poired Samples
Costne Distance Paired Samples
Cosine Distance Paired Samples
(V) Speckle
Word
Frost; Hindi
5
Boseline
5. 4
Bescline con)
Count
Dum 052)
count
Dune (0.103)
0.000
0,092 0.894
1.00 codo 0.112
8.00
od0o e a:s buoe 6l25 8359 8375
0.33
015
0.20
625
350
0.3 0.43
Samples
Paires Semples
(V) Defocus Blur
(T) Char Typo
(V) Zoom Blur ( Arabic
Count
Dum (3.04)
Beseline (0 0051
Court
Ours (8.00A)
Bapeline 00 053)
Boseline o7 1971 Dur (8.152)
6.00
0.00 02
Cosine
(V) Zoom Blur
(T) Char Delete
(V) Zoom Blur (T) Greek
Bundiir (0 0231 Oum (3.019)
5
Mescline (0:580)
Boudere 00 225) Ours (9.185
Count
Count
Ours 18
Cosinc Distainc of Paired Samples
Cosin: Distance Pairod Samples
Cos Distandc Raired Samples
(V) Fog
(T) Char Insert
(V) Zoom Blur : (T) Hindi
Busding
Bescline (0.040)
0.000 B0005 0a800 E.00s noice salss 2.0s2 Cosine Distarnce of Paired Samples
0.30
0,55
0,10
B.5
0.30
810
0.55 0.20
0:28
Casire Distance of Paired Samplee
Cosine Distance of Paired Samoles
Hindi
(T) Arabic
0544)
Count
E 12
0.000
3.065
C0lt
eats
1.006
5.10
0.1
B5e
0.5
6:35 Paired Sumples
0.13
0,55
013
E.1
0.20
635
1.30
0.10
Parie Samplis
Casire
Samp
Brightness
(T) Greek
(V) Geussian Noise Greek
Sealiw
Beseline 10, Ours (0 1621
Beewinee
Oves (9.052)
Count
Oues 28.655
Puired
(V) Contrast:
(T) Arabic
(V) Gaussien Noise: (T) Hindi
mu (0 0021 Oues 062)
Besuline (C.172) 10.500.1521
Baading (0 222) Ours 20.122
Count
Cosine Distancce Pared
Cosine Datance Parred
Cosine Distance Pared Samples
Figure 13: Pair-wise cosine distance of intermediate representations between clean LB-COCO and 27 versions of perturbed LB-COCO datasets. Vittle (F) consistently reduces the representa- tion gap between the clean samples and their semantically equivalent perturbed ones.
34

---

Table 12: Comparison with alternative training approaches. We compare Vittle with weight- space regularization methods, LoRA [83] and weight decay (WD), and two recent visual instruction tuning learning objectives, ROSS [84] and LIT [85] on LLaVA-vI. 1.-7B model. Evaluations are conducted on POPE, its nine visually perturbed variants (POPE V Pert.), LB-COCO, and its nine
(Visaally/extuaallyjnnnny perturbed variants, where we mark the best one as bold and the second
best one underlined.
Method
POPE
POPE Pert.
LB-COCO
LB-COCO V Pert.
LB-COCO Pert.
LB-COCO J Pert.
Baseline
86.98
84.12
77.8
73.4
72.2
62.3
LoRA
83.33
80.23
73.4
70.4
62.7
39.7
WD
87.22
83.97
74.1
72.1
73.0
59.5
ROSS
87.79
84.67
74.4
72.0
71.3
60.0
LIT
87.38
84.21
77.5
72.1
72.9
58.9
Vittle (L)
87.71
84.91
76.7
73.9
73.0
62.7
Vittle (F)
87.81
84,99
76.1
74.2
74.1
64.4
Table 13: Vittle LLaVA-v1.5-13BB model. We compare Vittle with the standard learning objective on LLaVA-v1.5-13B model that uses Vicuna-v1.5-13B as an LLM backbone. We set the bottleneck layer index = 36, interpolation coefficient 0.5, and bottleneck KLD regularization strength 3
0.1 Vittle outperforms baseline on perturbed datasets while showing rivaling
d
performance the clean dataset.
Method
POPE
POPE Pert. LB-COCO LB-COCO V Pert.
LB-COCO Pert.
LB-COCCO Pert.
Baseline
87.14
84.02
76,9
73,5
73.8
64.6
Vittle (L)
87.22
84.85
76.6
74.5
74.0
65,4
Vittle (F)
87.32
84.65
76.8
74.2
73.9
65.3
B.4
Further Investigation on Vittle
Peak memory allocation. Vittle hosts MIP blocks inside the LLM backbone to model the posterior distributions of the inner representations, and compute additional learning sigmals, e.g., KLD between priors and posteriors. In Table 5, we reported the wall-clock runtime, and here, we compare the maximum peak memory allocation across GPU chips (8 chips during the training and 1 chip during the inference) in Table 14. We that the memory overhead induced by Vittle is marginal across both training and inference.
Table 14: Memory comparison. We compare baseline and Vittle (F) in terms of the maximum peak memory allocation (gigabytes; GB) across GPU chips during training and inference.
Method
Peak Mem GB (train) Peak Mem GB (test)
Baseline
37.55
15.62
Vittle
38.98
15.84
Table 15: Response diversity comparison. We report scores of four representative measurements of text diversity on the models' responses to LB-COCO clean data queries.
Method
Distinct- (t)
Distinet-2 (t)
CR (L)
Hom RI. (L)
Baseline
0.2356
0.6117
3.234
0.155
Vittle (L)
0.2413
0.6279
3.212
0.147
Vittle (F)
0.2382
0.6260
3.238
0.144
Output diversity comparison. One may be concerned that learning a compressive representation with Vittle can hurt the diversity of the textual output, which is undesirable for a chat assistant. To investigate the output text diversity with and without bottleneck training, we evaluate with four common textual diversity metrics, Distinct n-gram [1 [1],], as well as the compression ratio (CR) and
35

---

Object Hallucination Detection Task Under Visual Perturbations
Accuracy
Gaussian Roise
Shott Noise
Speckle Noise
Delocus Blur
Zoom Blur
rog
frost
Brighmess
Contract
Arg. Part.
Open-ended QA Task Under Visual Perturbations
Relative ireference
Deseline
Vitlee (7)
Gaussian Noise
Shot Nosse
Speckle Noise
Detocus Blur
Zoom Blur
Fog
Frost
Brightness
Contrast
Avg. Part.
Open-ended QA Task Under Textual Perturbations
Relative Preference
Word Delete
Word Swap
Word Insert
Char typo
Char Delete
Char Insert
Hindi
Greek
Arable
Arg. Part.
Open-ended QA Task Under Joint Perturbations
Relative Preference
35
Baseline
Vittle (F)
Prest Arabie
Grest
Frost Hindi
Zoom Blur Arabic
Zoom Blur Greek
Zoom Blunkindii
Gaussian Noise Arabic
Gaussian Roise Greek
Gaussian Noceesiini
Pert.
Figure 14: Object hallucination defection performance on POPE perturbation datasets (top), and Open-ended QA performance on LB-COCO perturbation datasets (three below) of Prism- 7B. We enumerate the accuracy for the object hallucination detection task and relative preference score for the open-ended QA task of each method on perturbed datasets, where we observe consistent performance gains by Vittle.
the homogeneity score driven by Rouge-L (Hom RL) [113] over the outputs from each model trained by the baseline method and Vittle on the LB-COCO clean dataset.
B.5 Applicability to Other MILMS
We now investigate Vittle's effectiveness another recent MILM, Prism-7B, beyond LLaVA. As noted in Section A. Prism has quite a different design principle than L.LVVA with respect to the visual encoder and the training strategy, so it is suitable for investigating the versatility of Vittle
36

---

between models. Table 16 shows summarized results on our perturbation benchmarks" hallucination detection tasks, Vittle outperforms the standard cross-entropy only training baseline on clean and perturbed datasets. In open-ended QA tasks, Vittle consistently boosts performance in perturbation scenarios with large margins while maintaining performance on the clean dataset. The results of the perturbation-ssecific performance comparison are provided in Figure 14.
14 In object
Table 16: Vittle Prism-7B model. We compare Vittle (F) with the standard learning objective
under the Prism-7B model training regime that adopts two visual encoders (DINOv2 and SigLIP) and the single-stage training rather than two-stage training with single CLIP visual encoder. Vittle significantly improves perturbation-robustte
compared with a naive learning objective.
Method POPE
POPE V Pert. LB-COCO LB V Pert. LB-COCO T Pert. LB LB-COCO Pert.
Baseline 87.54
85,29
79.4
75.3
71.9
53.8
Vittle (F) 88.11
85.52
79.0
76.2
75.4
63.2
C Extended Literature Review
Compression for generalization. There is rich history in the machine learning field that connects compression of the model or its inner representation to generalization [114], from the classical learning theory with Occam's 's razor [115, 116] and Minimal Description Length [117, 118, 119] to IB principle [46, 47], by suggesting models that provide minimal and simplest representation of data generalize better [33, 118, 52, 120] analogy to human perception [13, 121, 15]. Recently, Wilson [122] proposed a new generalization bound for contemporary large-scall scale models where the compress- ibility of a learning algorithm plays key role in better generalization. According to that discussion, even the maximally flexible billion-scale model can have a small effective dimensionality (indicating the higher compressibility) by embracing soft inductive biases [123], such as, a regularization term, to the learning problem. On top of these, IB-objective of Vittle can be understood as a soft inductive bias seek a minimal sufficient representation that helps generalization for the challenging queries.
Robustness of fine-tuned foundation models. Although large-scale pre-trained models have appealing generalization capability across diverse data instances from different domains, their fine- tuned counterparts usually hurts that strong generalization capability while being adapting on task- specific in-distribution samples [124, 125]. This undesirable performance compromise between adaptation to in-distribution samples and generalization to samples from broad domains has spurred the community to work robust fine-luning of foundation models [1 [124, 125, 126, 127, 128, 129, 130). This line of work addresses the adaptation-robusteed
trade-off by (1) introducing a regularization term [128, 129]. (2) tweaking the training procedure [124, 126], (3) merging multiple models in the weight space [125, 131]. However, almost all of the existing robust fine-tuning literature has focused on a discriminative model, such as CLIP [71], under classification setups. Although there are a few works on robust instruction tuning of MILMs [132, 133], they do not specifically focus on improving robustness under diverse types of distribution shifts and propose a data-centric approach, i.e. expanding instruction tuning datasets in terms of quantity or diversity, that requires external MLLL-baased data generation process and/or careful post-processing from humans. In this work, we take a representation-cen/ff approach that modifies the learning objective of visual instruction tuning to efficiently enhance the robustness of MILMM under diverse distribution shifts (27 types in total).
"Due resource constraints, only explore Vittle (F) of our prior distribution instantiations.
37

---

D Derivation of Variational Bound for IB in MLLM
Here we provide a full derivation for the variational lower bound for IB. The derivation skeleton was mainly inspired by existing works [134, 32j. We begin with the mutual information term I X). Given the sequential nature of MLLM, we decompose both the input X = (Xu, X₂) and the latent representation Z (Z., Ze) into visual and textual components. We can then derive the following upper bound for X):
/ p(a,, 2) log p(x,=) drd= =
f p(x; 2) log
p(2|xr drdz
T(Z, x)
p(x)p(2)
p(z)
/ p(x,2) log p(=(z) drdz (KL()())lo(z))
r(z)
< / p(z,=) log p(z(2)
drdd
r(z)
/ plæv, Tes Zu: 2x) log pl t,, Ds tx)p((zn(())
dr. drrddddzddd
r(z))r(z)
/ p(an) / p(zoleus)1)) 21) / plzu (x)) p(zu)
p(eu(Te) de dendrddzydz
log
r(zo)
+ f p( 21) / p(zolze) p(z Ieo log
p(z, p(z l2
F2))
de do
r(2))
+ Ex..,, (DLL Øe, 24 (2)22))
(9)
where the first inequality holds given the non-negativity of DKr. p()) and p (e) = ((2u(zr,) due causal attention in MI.LM. Here, introduce r(z) r(zo, 2+) r(z, (r()))
factorizable variational approximation the true prior for the latent representation p(2). Next, for the output-relevant term T(Z,Y) we have the lower bound:
ply. 2) log p(y, z) dydz = / plu. plu.2) log p(ylz) p(x)
I(2,Y)
dydz
p(y)p(2)
/ p(y.2) )) log q ddddz + DKL (KL(l(l))ll(sol)))) / ply)
2 p(v.2) 2) log ((llz)dddz
p(., us =) log alyl=)d p(o))(ols))(s)))) log alyl =dv=d)ddyd,,
= Ex,y Exl [log [(yl=)]]
(10)
where p(x, y. 2) ((=))(=2=)))l) ((yl)) given the Markov assumption ++ ++ Z, and p(=|r, y) = p(2[x) holds given that the representation Z can not directly depend on Y, and the entropy term of y. i.e., J -py( log p(y)dy = HYY), is ruled out due its independence for optimization problem. Here, we replace the intractable p(ylz) 2) with variational approximation ((v)z) that will be parameterized by model. Finally, combining the lower bound of I (Z, Y) and the upper bound of I(Z, X) yields variational lower bound for the IB objective as follows,
IB(X, Y) 2 Ex.y [E2] [log [+][2
(E.. [PL (x) + Beu.ar (pLe x, +))))(==)))))
(11)
Asymmetric posterior p<z|z) modeling in practice. As we fed the ppre/post-bottlenedd
rep-
resentation interpolation q(vl)), this yields an asymmetric specification of p(z(z) being modeled in / 1(2, X) and [ZZ, Y). To be precise, given our posterior instantiation p(2(z) := N(2:; o²
(1 a²2 až into the LLM-head for for output decoding
I) in the upper bound of I(,,X). which is modeled by the LLM-stem for plus the bottleneck layer gø, we use (2 ap(()) + (1 a)c, (x)) T), where e = Jo (ax) denotes a constant vector, to model the p(z[z) in the lower bound of I I(Z,Y)
38

---

E
Missing Proof
E.1
Preliminary
We start by providing a definition of Mutual Information (MI) below.
Definition E.1 (Mutual Information (MD). For a joint distribution Pxy over X V, the mutual information with respect to Pxy is defined as,
(xy(x.
1(Px)) == Pxy [log Px Px(x)
(12)
If X is an instruction and Y is a corresponding response, we regard T(Pxy) as a relevance between the instruction and the response that can be seen a possible quantification of instruction following capability of MLLMs. Effective MI is defined based on the MI as follows:
Definition E.2 (Effective Mutual Information (EMI) [19]). Given the joint distribution Pxy and MLLM Pe parameterized with 6 the effective mutual information between the input and model response is defined as,
EMI(PxY; Pe) == I(Px).) I(Pxy),
(13)
where Pxys denotes the joint distribution between the input X and the output of the model Ye- Although the vanilla MI can also be used as a metric to evaluate models' output response by I ((xxy.). the target data distribution which is undesired when our interest is
the scale of it varies depending to compare performance of model across multiple domains which can be addressed by EMI Recall that ultimately interested in the performance difference of MLLMs across different datasets, and this can be captured by the EMI difference (EMID) as follows:
Definition E.3 (EMID). Let Pe X > y be an MLLM with parameters @ that produces an output response Ye given an input instruction X. For joint distributions Pxy and QXy, effective mutual information difference of Pe over P and Q is defined below.
EMID( Qxy: Pe) = [I(Pxvo) - I(Pxv)] - I[Qxy)]
(14)
By setting as an instruction tuning distribution (training data) and Q as an arbitrary test time distribution (evaluation data), prefer model that has smaller EMID value, which indicates better robustness under distribution shifts between P and Q. Now, based on the original theorem provided by Oh al. [19], we are ready to derive a new upper bound for EMID tailored to our representation-centric visual instruction tuning setup.
E.2 A New Upper Bound for Effective Mutual Information Difference
We first review Lemma 1 of Shui al. [135] and its adapted version, a conditional entropy bound [19] as follows.
Lemma E.4 (Lemma from Shui al. [135]). Let Z Z be the real-valued integrable random
variable, and denoting two distributions on a common space Z by P and Q such that Q is absolutely continuous w.r.t P. If for any function and X R such that Er(exp(X(f(z)) Er(((2))) < co.
then we have:
X(E== - EZ- (f))) DRI.(Q1IP)
+ log PooxE-lelex(([[U - EE-P((()))
Lemma E.5 (Conditional entropy bound [19]). Let f(x) == H(Q))z) and ((Q2)z) = maxaex H ((Q1zz), given the marginal distributions Px and Qx. and conditional distributions Pvix and Qy x, according Lemma E.4, we have a conditional upper bound:
i) Ezzp (QQy))) Ez-q ((())))(Q (2
Similarly, given the marginal distribution Px and Qx, and MLLM Pe, let f(z) ((Pe(-))) and (Po) := maxxex (P)() then, according to Lemma E.4, we have another conditional
upper bound:
ii E (HP((-) - E (H(Pe 2
39

---

Next, we should also need to formulate the relationship between JSD in the input space and JSD in the representation space, which is done through Lemma E.6.
Lemma E.6. Let f : X' + Z be an encoder that maps an input X to a representation Z. for the input distributions Px and Qx and f-induced representation distribution Pz and Qz. we have an inequality below,
+ Ex DKL (Qx Mx|=)) (15)
where Mx|= == Px|= Qx1=
Proof. We start from the definition of JSD,
Dis(PxIIQx) 2 - (Pxl(xx))M DKL(Qx||MM) Mx Px +x
By applying the chain rule of KLD under a deterministic map" 15 X + Z, we know that.
[xz)) = DRL DRL(P2|1M2) + /
| )dz
(Px((Px|Mxx)) + (Px(a)DnaPP22a1
+ (KL(Px(IMx))
Then, we have,
(is(PxIIQx) Das Dss(Pz|1Qz) + 2 "B(2z=rl (DKi. (PxjelPlllll +
which results in ineq. (15) by applying the triangular inequality after multiplying 2 on both sides. C
Now we derive a new upper bound for EMID, which is defined over the representation space rather than the previous Proposition E.7 (EMID upper bound). Let Pe be an MLLM that maps X (X,, to
defined over the input space [19] in Proposition E.7.
{Z., ) and then subsequently maps to Ye. Given joint distributions PxY = Px x Prix and
QXY = Qx Qyix. by assuming consistent conditional distributions over Z 7r Z, Z,, and Y|X between P and Q, we have an upper bound for EMID EMID(Pxy Qxy;Pe) as follow,
A (D) (Px, 11Q + (Px
Axxiz) + (H(P))) H(Py) + (H(Q).) HQr)). (16)
where and D indicate the entropy and square root Sensen-Shannor divergence (JSD), respec- tively, Ax Axiz == Ex~p || Ez DKL (Qx M| with a mixture distribution M P+Q and == m (Q + ((P..))
Proof. Given the entropy-based definition of the mutual information, T(PxY) = ((P)) (H let Pro, = Epxp(Po((tz) and Qy = pressed as follows,
then, EMID can be ex-
EMID( Qxy;
EMI(PxY: Pe) EMI[(XX; Pe)
(H(Pv,) - ( ((P))+(+
-(H(Q)))
H(Qvix))
(H(Pyix) ((Qyix)) + (B.--((((PPo(11))) -HaP)))(
+ +(H(Prr) ((P)) + - H(Qra))
< (H(Pyix) - (QQix))) (A)
+ (E (H.P))))))) (
+ (H(...) - HPv)) + (Qv) H(Qu))
(1 (17)
15 Note that although Vittle governs probabilistic encoder during training, it produces deterministic output during inference by using the learned posterior rather than random sample (Section 3.3).
40

---

Moreover, we have the following inequality for H(P x) proposed by [19],
- (Q))))) & JR..-(DD R..t0
(18)
By plugging inequalities Lemma E.5 and ineq. (18) into the incq. (17) replace the terms (A)
and (B), and given the consistent conditional distribution assumption for Y|X, i.e., Prix = Qrix we have much simpler upper bound follows,
EMID(Pxy Qxx; Pa) S H (2xxssxx1lxx (M(P))) H1 + - - H(Qv))l
where H := m [H (Qy + (Py)), ) Then, we can further replace the term (xx(PxlIQx) by using Lemma E.6 get bound defined by representation divergence below,
EMID(Xxy, Qxy; Ps)
s (P / DRL + (=..g//DKu/Qxxxx/MxM)))))
+ + - (Qr) H(Qr.)l-
(19)
Meanwhile, the chain rule of KLD and the definition of JSD with our consistency assumption for conditional distributions for Z,, Z and Z,, one can easily show below.
2ss Pz, 2, Qz, 2,) DxI. Pz, 2 [2. 2.) (2, || (1
Das Pz, 1lQz,) + Das Pss(Pz,112z)
(20)
Plugging Eq. 20 into ineq. (19) and applying the triangular inequality complete the proof.
L
F
Impact Statement
Multimodal large language models (MLLMs) today have many socictal applications. This work tackles the robustness of MILMs to distribution shifts between training and test time data. We observed consistent improvement of proposal Vittle in various types of visual and textual shifts, allowing users to trust the model more than before to safely use AI in a variety of environments. Moreover, although focused on the robustness perspective in this work, improved invariance- sensitivity trade-off also benefits the fairness-diseriinaateeen desideratum towards reliable AI. Meanwhile, even though its robustness distribution shifts was improved, there are still potential misuse cases with MLLMs that can affect humanity by producing systematically biased outputs, given the existence some adversarial data providers attackers.
trade-off, which is another crucial
41


Reviews:
[{"review_id": "jwdfFFxB6D", "reviewer": "Reviewer_RnJj", "summary": "The paper introduces Visual Instruction Bottleneck Tuning (Vittle), which inserts an information-bottleneck layer inside the LLM backbone to regularise representations. Robustness is evaluated on 30 distribution-shift scenarios created from POPE (object-hallucination) and LB-COCO (open-ended QA) perturbation suites.", "questions": "Questions for Authors\n\n1. Does Vittle still help when applied to recent models?\n\n2. How sensitive is performance to the IB weight β and to the insertion layer l?\n\n3. Why limit the bottleneck to instruction-tuning? Would pre-training with IB amplify robustness?\n\n4. Could the authors elaborate on when Vittle (L) vs (F) is preferable?\n\n5. Have you considered more realistic perturbations (partial occlusion, codec compression, ASR/OCR noise)?\n\n6. What is the inference-time latency and memory overhead beyond the ~1.5 % parameter increase and 20 % training-time rise?\n\n7. Does the bottleneck reduce answer diversity or expressiveness while lowering hallucination?\n\n8. Please clarify concrete advancements over the previous ICML-Workshop submission (data size, theory, experiments, scope).\n\nThese points should help strengthen your review and guide the discussion phase.", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n1. Solid theoretical grounding. The authors derive a variational IB lower bound and link it to EMID, a robustness metric.\n\n2. First end-to-end IB for MLLMs. Unlike prior projector-only work, Vittle places a learnable bottleneck within the LLM and trains it jointly.\n\n3. Clarity of exposition. The paper is well-structured with intuitive diagrams and ablation details.\n\nWeaknesses\n\n1. Marginal gains. Relative improvements on perturbed sets are mostly single-digit pp; absolute scores remain close to the baseline.\n\n2. Perturbation realism. Visual corruptions focus on generic CV transforms (brightness, blur, noise); they might not match real LMM deployment errors (occlusion, compression, camera shake).\n\n3. Older model backbone. Experiments centre on LLaVA-v1.5-7B/13B and Prism-7B; results on newer models are missing.\n\n4. Unclear novelty over ICML workshop version. The manuscript does not explicitly differentiate itself from the prior workshop paper.", "was_revised": true, "final_justification": "Dear authors,\n\nDuring the rebuttal period, they addressed many of the points I raised to some extent, but I believe it is difficult to draw a conclusion based solely on the experiments conducted within the short rebuttal window. For the new vision–language models, most of the experiments presented in the paper should be reproduced, and the current perturbation should also be applied to all vision-related experiments in the paper. Therefore, I will maintain my current score.", "rebuttal": "Thanks for taking the time to point out some important points that are worth discussing! Below, we provide responses to individual weaknesses (W) and questions (Q) you pointed out.\n\n> W1: Clarification on performance gains.\n\nWe acknowledge the reviewer’s point, and would like to emphasize that **achieving improvements across 30 different types of distribution shifts and multiple models is non-trivial**, especially given the strong baseline performance. Vittle delivers significant improvements in many cases, such as LB-COCO text shifts – Word Insert, Char Typo, and Arabic; most of the LB-COCO joint shift cases. We would like to note that these improvements, i.e., **3.5% ~ 9% improvement over the baseline, are not quite trivial**, and the improvements are also consistent across multiple experimental settings, which demonstrates the effectiveness of our method. \n\nThese robustness gains are particularly valuable as they come with only 1.5% parameter overhead and almost no inference-time cost increase. Importantly, Vittle also preserves performance on the original in-distribution datasets, ensuring that robustness does not come at the expense of standard task performance. We therefore believe the improvements represent meaningful enhancements for MLLMs.\n\n> W2, Q5: Considering more realistic shifts.\n\nWe thank the reviewer for raising this important point. Our perturbation set was designed to **cover a wide range of standardized, controllable shifts commonly adopted in robustness evaluation** for CV and (M)LLMs (e.g., [Dan et al. 2019; Qiu et al. 2024; Oh et al., 2025]), allowing systematic comparison across multiple models and settings. While these generic corruptions (brightness, blur, noise) do not exhaustively capture all real-world deployment errors, they provide a principled testbed for stress-testing robustness in a reproducible manner. Importantly, our study does not rely solely on synthetic perturbations: we also evaluate on long-tail, naturally collected benchmarks such as LB-Wild, LB-Wilder, and WV-Bench, which **reflect realistic distribution shifts** in user queries (see **Section 4.2**). Vittle shows consistent gains on these datasets as well, suggesting that the improvements are not limited to artificial settings. \n\nWe agree that occlusion, compression artifacts, and camera shake are important additional sources of shift. However, a lack of benchmark datasets on such realistic perturbations is a major obstacle on this path. Establishing a more realistic distribution shift benchmark for MLLMs is therefore an important direction for future work. Nonetheless, the current results already demonstrate that Vittle improves robustness across both controlled perturbations and naturally occurring distributions.\n\n\n> W3, Q1: Applicability to recent models.\n\nOur primary goal was not to chase state-of-the-art performance, but to demonstrate the effectiveness of the proposed training paradigm in a **standard, widely adopted setup that ensures fair comparison and easy reproducibility**. For this reason, we focused on the well-established LLaVA‑v1.5 training pipeline, which is the common benchmark in prior work [Wang et al., 2025; Zhou et al., 2025], and also reported results with one recent model Prism [Karamcheti et al., 2024] in the Appendix.\n\nThat said, we agree it is important to validate generality on newer models. To this end, we additionally evaluated Vittle on `LLaVA‑Mini` [Zhang et al., 2025], a recently proposed MLLM with an efficiency‑oriented compression module. Using the same configuration as LLaVA‑Mini (Vicuna‑7B backbone, Table 1 in Zhang et al. 2025), we observed that Vittle improved robustness on POPE visual shifts while preserving in‑distribution performance. \n\n| Method     | POPE  | POPE V Shift |\n|--|--|--|\n| Baseline   | 79.37 | 77.39        |\n| Vittle (F) | **81.07** | **78.32**        |\n\nMore broadly, given the rapid pace of MLLM development, it is inevitable that even newer models will emerge after the completion of this work. We emphasize that our contribution is a training framework that is designed to be **model‑agnostic** and easily applicable to alternative backbones.\n\n> W4, Q8: Novelty over ICML workshop version.\n\nWe would like to clarify that the ICML workshop is a **non-archival workshop without proceeding, and thus the NeurIPS submission should be evaluated independently on its own merits**. Workshop is mostly intended to share early ideas with the community. Based on the submission policy noted in Call for Paper NeurIPS 2025, papers presented at workshops (without proceeding) are permitted, whether they are substantially similar to the submitted version or not. Therefore, the policy does not warrant the need to claim the novelty over workshop version. We thank you for checking on this.\n\n> Q2: How sensitive is performance to the IB weight β and to the insertion layer l?\n\nWe provide a sensitivity analysis of both the IB weight $\\beta$ and the insertion layer $l$ in **Fig. 12 of Appendix**. We find that the choice of insertion layer plays a vital role in ensuring stable performance across diverse tasks. This aligns with findings from Skean et al. (2025), who showed that the selection of probing layers in LLMs strongly influences downstream outcomes; we observe a similar trend in the visual instruction tuning phase of Vittle. The IB weight $\\beta$ also affects performance, as values that are too small or too large can destabilize training, but overall, $\\beta$ is less sensitive than the choice of insertion layer.\n\n> Q3: Would pre-training with IB amplify robustness?\n\nThanks for the suggestion! Our chief goal of this work is to improve the robustness of MLLM to distribution shifts. However, it is hard to define “distribution shifts” for the pre-trained model because most of the recent MLLMs have undergone a hyper-scale pre-training with tons of data points. \n\nIn contrast, instruction tuning is conducted on a relatively small scale with a limited number of data points, which enables us to define the distribution shift between train and test well and helps us to derive a clear problem statement. Therefore, we confine our interest to the instruction tuning setup in our manuscript.\n\nInvestigating the effectiveness of IB in a pre-training setup is also an important direction. Although we are currently conducting an experiment on GPT-2 pre-training, we have yet to obtain results due to resource and time constraints, but we will keep you updated once the results are available.\n\n> Q4: Could the authors elaborate on when Vittle (L) vs (F) is preferable?\n\nAs discussed in **Section 4.2**, Vittle (F) tends to perform better under perturbations due to its stronger isotropic prior constraint, making it more robust to distribution shifts. In contrast, Vittle (L) leverages a learnable prior, which provides greater adaptability and yields stronger performance on long-tail queries and knowledge-intensive tasks without perturbations. In practice, we recommend using Vittle (F) when robustness to severe test-time perturbations is the priority, and Vittle (L) when adaptability to diverse, long-tail samples and preservation of general knowledge are more critical.\n\n\n> Q6: What is the inference-time latency and memory overhead?\n\n- In **Table 5** of our manuscript, we already report the inference-time latency comparison of our method with the baseline, and show that **Vittle achieves almost identical inference time with the baseline**. \n- In terms of memory overhead, we report train and inference-time peak memory below. As we can see, Vittle has almost comparable peak memory usage.\n\n| Method   | Peak Mem GB (train) | Peak Mem GB (test) |\n|----|-----|------|\n| Baseline | 37.55               | 15.62              |\n| Ours     | 38.98               | 15.84              |\n\n> Q7: Does the bottleneck reduce answer diversity or expressiveness?\n\nTo investigate the output text diversity with and without bottleneck, we evaluate a common textual diversity metric Distinct n-gram [Li et al. 2015], as well as compression ratio (CR) and Rouge-L-based homogeneity score (Hom RL) [SShaib et al. 2024] over outputs from each model on LB-COCO clean dataset.\n\n| Method     | Distinct-1 (&uarr;) | Distinct-2 (&uarr;) | CR (&darr;) | Hom RL (&darr;) |\n|---|-----|-----|----|----|\n| Baseline   | 0.2356     | 0.6117     | 3.234 | 0.155     |\n| Vittle (L) | **0.2413**     | **0.6279**     | **3.212** | 0.147     |\n| Vittle (F) | 0.2382     | 0.6260     | 3.238 | **0.144**     |\n\nThe result implies that Vittle does not hurt output diversity compared to the baseline and even improves it, achieving a favorable balance between invariance and sensitivity in the inner representation space to produce better responses per input.\n\n\n### Reference\n- Dan et al. 2019, Benchmarking Neural Network Robustness to Common Corruptions and Perturbations\n- Qiu et al. 2024, Benchmarking Robustness of Multimodal Image-Text Models under Distribution Shift\n- Oh et al. 2025, Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approach\n- Wang et al. 2025, Reconstructive Visual Instruction Tuning\n- Zhou et al. 2025, Learning to Instruct for Visual Instruction Tuning\n- Karamcheti et al. 2024, Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models\n- Zhang et al. 2025, LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token\n- Skean et al. 2025, Layer by Layer: Uncovering Hidden Representations in Language Models\n- Brown et al. 2020, Language Models are Few-Shot Learners\n- Penedo et al. 2024, The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale\n- Zellers et al. 2019, HellaSwag: Can a Machine Really Finish Your Sentence?\n- Li et al. 2015, A Diversity-Promoting Objective Function for Neural Conversation Models\n- Shaib et al. 2024, Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scores"}, {"review_id": "nFP34Ip2Ag", "reviewer": "Reviewer_wdcV", "summary": "This paper introduces the Information Bottleneck (IB) theory into end-to-end instruction tuning for Multimodal Large Language Models (MLLMs), aiming to improve model robustness to out-of-distribution inputs without increasing data volume or model size. The authors derive a variational IB lower bound tailored for autoregressive MLLMs, design lightweight and pluggable bottleneck layers, and validate the method across 30 distribution shift scenarios, demonstrating consistent improvements in both open-/closed-ended QA and object hallucination detection tasks.", "questions": "Please refer to the weakness.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. Simple and general implementation: A single pluggable MLP layer with two modes—fixed or learnable priors—allows reusability in existing training pipelines.\n\n2. Comprehensive experiments: The study covers three task types, 45 datasets, and 27 perturbation levels, including ablations, prior comparisons, and fine-grained severe shift analyses. Results consistently outperform standard instruction tuning and several baselines (e.g., LoRA, weight decay, information maximization methods).\n\n3. Clear writing and logical structure.\n\nWeaknesses:\n\n1. Mild performance regressions underexplored: Slight drops occur in some multi-disciplinary multiple-choice tasks (e.g., MMMU), but causes and trade-offs are not discussed.\n\n2. Lack of safety and bias analysis: Information compression may risk filtering critical information (e.g., leading to misdiagnoses), yet the paper does not explore or assess such potential downsides.\n\n3. Limited resource consumption reporting: Only training/inference time is reported, with no details on memory or GPU usage.", "was_revised": true, "final_justification": "I appreciate the authors' efforts in addressing my concerns. I have carefully read the rebuttal. I will keep my score.", "rebuttal": "Thanks for taking your time to provide this constructive feedback! We provide a response to the pointed out weaknesses (W) item by item.\n\n> W1: Mild performance regressions underexplored: Slight drops occur in some multi-disciplinary multiple-choice tasks (e.g., MMMU), but causes and trade-offs are not discussed.\n\nWe thank the reviewer for this insightful observation. We believe the observed regressions can be explained by the nature of the bottleneck principle: while it enhances robustness by reducing sensitivity to superficial perturbations, it may also filter out fine-grained visual details that are not explicitly anchored by instruction tokens. Since MMMU includes many OCR-related, knowledge-intensive splits (e.g., Accounting, Finance), this trade-off can lead to slight drops in performance. Notably, the learnable-prior variant, Vittle (L), mitigates this issue by adapting its prior to retain more task-relevant details, **recovering performance closer to the non-bottleneck baseline** (35.3 vs 35.6, Table 2). We view this as evidence that the prior design offers a lever to balance robustness against general VQA benchmark performance, and we will make sure to add this discussion in our manuscript. \n\n\n> W2: Lack of safety and bias analysis: Information compression may risk filtering critical information (e.g., leading to misdiagnoses), yet the paper does not explore or assess such potential downsides.\n\nWe agree that it is important to analyze the potential downsides of our method to ensure its reliable use in real-world applications. By reflecting Reviewer wdcV’s concern, we conducted three more experiments to evaluate the **output safety** with MM-SafetyBench [Liu et al. 2023], **fine-granular visual recognition capability** with OCRBench v2 [Fu et al. 2025], and output **textual diversity** [Li et al. 2015; Shaib et al. 2024].\n\n| Method     | MM-SafetyBench (&darr;) | OCRBench v2 Acc (&uarr;) | Distinct-1 (&uarr;) | Distinct-2 (&uarr;) | CR (&darr;) | Hom RL (&darr;) |\n|------------|---------------------|---------------|------------|------------|-------|-----------|\n| Baseline   | 0.6667              | **0.286**         | 0.2356     | 0.6117     | 3.234 | 0.155     |\n| Vittle (L) | **0.6026**              | 0.285          | **0.2413**     | **0.6279**     | **3.212** | 0.147     |\n| Vittle (F) | 0.6821              | 0.280         | 0.2382     | 0.6260      | 3.238 | **0.144**     |\n\n\nWe observe that (1) under jailbreak prompting, Vittle (L) induces stronger refusal and achieves better safety than the baseline, whereas Vittle (F) slightly underperforms the baseline; (2) as hypothesized in the response to W1, both Vittle (L) and (F) marginally hurt fine-grained visual recognition capability compared to the baseline; but (3) both Vittle (L) and (F) show higher output textual diversity across four measures overall. \n\n\n> W3: Limited resource consumption reporting: Only training/inference time is reported, with no details on memory or GPU usage.\n\nWe agree that a more complete resource analysis is valuable. In addition to the training/inference time reported in Table 5, we have now measured GPU memory usage. In the table below, we report the per-device maximum peak memory allocation during training and inference and the total GPU hours (single-gpu-basis) as below.\n\n| Method   | Peak Mem GB (train) | Peak Mem GB (test) | Total GPU hours (train) |\n|----------|---------------------|--------------------|-------------------------|\n| Baseline | 37.55               | 15.62              | 88                      |\n| Ours     | 38.98               | 15.84              | 106                     |\n\nOverall, the results confirm that Vittle improves robustness with minimal inference time computational and memory overhead, making it practical for deployment. \n\n### Reference\n- Liu et al. 2023, MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models\n- Fu et al. 2025, OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning\n- Li et al. 2015, A Diversity-Promoting Objective Function for Neural Conversation Models\n- Shaib et al. 2024, Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scores"}, {"review_id": "5G4vjKldu7", "reviewer": "Reviewer_eMVr", "summary": "This paper proposes a new approach, called Visual Instruction Bottleneck Tuning, to improve the robustness of multimodal large language models under distribution shifts. Instead of scaling data or model, they apply the Information Bottleneck (IB) principle to encourage the model to learn minimal sufficient representations. They derive a novel variational lower bound for the IB objective tailored for MLLMs, and instantiate it as a simple bottleneck layer inserted into the LLM backbone. They test Vittle on 45 datasets, showing consistent gains on various perturbation and long-tail settings, while maintaining strong performance on standard benchmarks.", "questions": "1. Did you try applying Vittle on purely language tasks (without images)? Will the IB principle still help for textual instruction tuning?\n\n2. How stable are your results across different random seeds? Given that sampling is used in bottleneck, is variance significant?\n\n3. Could you share intuition why the fixed standard Gaussian prior Vittle(F) sometimes works better under perturbations than the learnable prior?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper addresses a very practical and important problem, i.e., the robustness of MLLMs under distribution shifts. Although there exists several research works on this domain, it is still largely under-explored to me. \n\n2. The idea to integrate IB principle into end-to-end instruction tuning for large-scale MLLMs is novel and well motivated.\n\n3. I appreciate the theoretical analysis, especially how they connect their IB objective to EMID robustness metric.\n\n4. Experiments are very extensive, covering various types of distribution shifts and several long-tail benchmarks. The improvements are quite consistent\n\n5. The implementation is lightweight, only adds ~1.5% parameters, with nearly unchanged inference cost.\n\n**Weakness**\n\n1. The method requires tuning hyperparameters like α. It is unclear how sensitive final performance is to this.\n\n2. The paper focuses mainly on visual QA tasks. It is interesting to see whether the same approach can benefit more realistic tasks like agentic tasks.", "was_revised": true, "final_justification": "Most of my concerns are addressed. I will keep my previous score.", "rebuttal": "The authors sincerely appreciate your comments and are happy to hear that you find a couple of strengths in our work! We provide a response to the pointed out weaknesses (W) and questions (Q) item by item.\n\n> W1: The method requires tuning hyperparameters like $\\alpha$. It is unclear how sensitive final performance is to this.\n\nIn **Table 10 of Appendix**, we presented the ablation study for the $\\alpha$ parameter (along with $\\beta$ and bottleneck insertion layer $l$ in Figure 12). As we can see in the table below, Vittle consistently outperforms the baseline, indicating that the method is generally robust to this choice. However, when $\\alpha$ is increased beyond 0.5, we observed unstable training and exploding losses. This suggests that over‑relying on bottleneck‑generated representations can harm the stability of the instruction tuning phase, and thus $\\alpha$ should be set conservatively.\n\n| Alpha           | POPE V Shift |  LB-COCO T Shift |\n|-----------------|--------------|-----------------|\n| baseline (0)    |  84.12        |  72.3            |\n| Vittle L (0.1)  |  84.20        |  73.1            |\n| Vittle L (0.25) |  84.47        |  73.1            |\n| Vittle L (0.5)  | 84.90        |  73.0             |\n\n\n> W2: The paper focuses mainly on visual QA tasks. It is interesting to see whether the same approach can benefit more realistic tasks like agentic tasks. / Q1: Did you try applying Vittle on purely language tasks (without images)? Will the IB principle still help for textual instruction tuning?\n\nWe appreciate this thoughtful suggestion! Our manuscript focused on multimodal instruction tuning---an increasingly popular paradigm---where robustness to visual and textual perturbations is most critical. That said, we agree that extending Vittle to broader domains, such as agentic tasks and purely unimodal setups, is an exciting direction. In principle, the IB framework is modality‑agnostic, since it regularizes representations to discard nuisance factors while retaining task‑relevant information.\n\n\nTo substantiate this, we are performing two additional unimodal experiments during the rebuttal: \n\n\n1. **Language‑only setup**: We trained GPT‑2‑small [Brown et al. 2020] on FineWebEdu‑10B [Penedo et al. 2024] and evaluate the model on the clean and perturbed version of the HellaSwag validation split. \n2. **Vision‑only setup**: We fine-tune a pre-trained ViT-Base on ImageNet by following the configs of Wortsman et al. 2022, and evaluate the model on the clean and perturbed version of the ImageNet validation split. \n\n| Fine-tune Loss          | ImageNet (Clean) | ImageNet-C (Nine perturb. Avg) |\n|---------------|------------------|--------------------------------|\n| Cross-Entropy | 81.14            | 54.31                          |\n| w/ Bottleneck | **81.52**            | **56.19**                          |\n\nUnfortunately, we have yet to obtain results on Language-only setup due to resource and time constraints, but we will keep you updated once the results are available. However, the promising result on the vision model fine-tuning setup implies that our proposed recipe of the information bottleneck-based training is a general method that can be applied to diverse setups of large-scale AI modeling not only for visual instruction tuning.\n\n\n> Q2: How stable are your results across different random seeds? Given that sampling is used in bottleneck, is variance significant?\n\nWe thank the reviewer for this question. As suggested, we repeated key experiments with **three random seeds**, and report the aggregated performance for these runs below. Although Vittle shows some variation in final performance depending on the seed, the variance is not significant compared to the baseline’s own seed variance in terms of the coefficient of variation (CV), which is defined as STD/Mean to measure scale-normalized variance.\n\n| Method     | Mean ($\\mu$)   | STD ($\\sigma$)    | CV (${\\sigma\\over\\mu}$)    |\n|------------|--------|--------|--------|\n| Baseline   | 83.44 | 0.18 | 0.22 |\n| Vittle (F) | 84.84 | 0.21 | 0.24 |\n\n> Q3: Could you share intuition why the fixed standard Gaussian prior Vittle(F) sometimes works better under perturbations than the learnable prior?\n\nBoth fixed (F) and learnable (L) Gaussian priors of Vittle are **instance-independent priors** to enforce the bottleneck principle, but the **(L) prior is data-dependent** because it learns the distributional parameters on the entire training dataset (although it does not have an instance-level dependency), whereas the **(F) prior with standard Gaussian does not make any dependency on data**. Therefore, _the posterior distribution of input representations from Vittle (F) is more strongly compressed to decrease the KL divergence between posterior and prior than that of Vittle (L)._ \n\nIntuitively, the more compressive posterior representations imply stronger invariance to perturbations, and based on our theoretical analysis, **the more compressive representation induces the tighter upper bound of the effective mutual information difference (EMID)**, a measure of robustness, if the train and test input distributions share the support.\n\n\n\n### Reference\n- Brown et al. 2020, Language Models are Few-Shot Learners\n- Penedo et al. 2024, The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale\n- Wortsman et al. 2022, Robust fine-tuning of zero-shot models"}, {"review_id": "RV5lRtZujl", "reviewer": "Reviewer_haEx", "summary": "The paper proposes Visual Instruction Bottleneck Tuning (Vittle) to enhance the robustness of MLLMs under distribution shifts. The technique is built upon the information-bottleneck theory and specially designed for resolving the distribution shift problem of MLLMs.", "questions": "See Weaknesses.", "limitations": "See the third question in Weaknesses.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThis paper proposes a theoretically sound technique to resolve the distribution shift problem for MLLMs. The experimental results show the  effectiveness of the proposed method. The implementation is clear and can easily be reproduced.\n\nWeaknesses:\n1. It looks the performance of Vittle(F) and Vittle(L) varies across tasks. For example, Vittle(L) excels in long-tail scenarios, while Vittle(F) performs better under perturbations. Since in many benchmarks either Vittle(F) or Vittle(L) brings very limited or even negative performance gain (e.g., Vittle(F) in Table 2 and Vittle(L) in Figure 5 left), it seems Vittle requires task-specific hyper-param choice (F or L) under distribution shift.\n2. The relationship between IB and model robustness in distribution shift is not clearly stated. Why \"This property is particularly\n129 desirable for robust instruction tuning\"(Line 128-129)? Can IB enhance the robustness of other models in other tasks (e.g., CNNs in image classification) under distribution shift? \n3. Vittle introduces 20% extra training time and 1.5% extra parameters compared to baseline. Concerning that in many benchmarks Vittle brings very limited performance gain, is it possible that the difference simply comes from the increase of the training time and parameters? I think it is necessary for the authors to clarify this problem.", "was_revised": true, "final_justification": "The authors have answered my question. My only concern of this paper is the limited performance improvement. I thus keep the score of \"Borderline accept\".", "rebuttal": "We appreciate your productive comments and feedback, and happy to provide our response to the weaknesses (W) identified by you.\n\n> W1: It looks the performance of Vittle(F) and Vittle(L) varies across tasks. For example, Vittle(L) excels in long-tail scenarios, while Vittle(F) performs better under perturbations. Since in many benchmarks either Vittle(F) or Vittle(L) brings very limited or even negative performance gain (e.g., Vittle(F) in Table 2 and Vittle(L) in Figure 5 left), it seems Vittle requires task-specific hyper-param choice (F or L) under distribution shift.\n\nWe appreciate the reviewer’s insightful observation and agree that Vittle (F) and (L) emphasize different aspects of robustness. This is expected by design: the fixed prior (F) enforces stronger compression, yielding greater invariance and robustness under perturbations, while the learnable prior (L) introduces more flexibility, which benefits long-tail and knowledge-intensive tasks. Importantly, across 45 datasets and 30 shift scenarios, both variants can improve robustness relative to the baseline in most cases, and the magnitude of negative regressions is small. Thus, **rather than requiring task-specific tuning, the two variants represent different robustness–adaptability trade-offs**: (F) is preferable when robustness to perturbations is critical, while (L) is preferable when generalization to diverse long-tail queries is important. We will clarify this trade-off more explicitly in the paper and note that developing a unified adaptive prior that balances both regimes is an exciting direction for future work.\n\n\n> W2: The relationship between IB and model robustness in distribution shift is not clearly stated. Why \"This property is particularly 129 desirable for robust instruction tuning\"(Line 128-129)? Can IB enhance the robustness of other models in other tasks (e.g., CNNs in image classification) under distribution shift?\n\nThe connection between IB and robustness is that IB explicitly encourages representations to discard nuisance variability while retaining task‑relevant information, which naturally mitigates sensitivity to distribution shifts. In the instruction‑tuning context, this is particularly desirable because multimodal inputs often contain spurious or modality‑specific noise (e.g., background clutter, token irregularities). **Our theoretical analysis** using the effective mutual information difference (EMID) **further formalizes this intuition by showing that stronger compression yields a tighter bound on the train–test performance gap**.\n\nWe would also like to note that we already discussed IB’s effectiveness beyond MLLMs in Related Work (**L104–L117**), citing applications in CNNs and other architectures where IB improves robustness under corruptions and domain shifts. Other explicit examples of IB to enhance robustness under distribution shifts include DIB [Dubois et al. 2020] and IIB [Li et al. 2022]. Our contribution is to bring this principle to autoregressive multimodal instruction tuning, where robustness under distribution shift has been largely unexplored, and to provide both theoretical justification and large‑scale empirical validation.\n\n\nFor this rebuttal, we further conducted an experiment on a ViT-Base ImageNet fine-tuning setup for image classification task with the same configuration adopted in Wortsman et al. 2022 (i.e. fine-tuning of a CLIP pre-trained ViT over 10 epochs on ImageNet).\n\n| Fine-tune Loss          | ImageNet (Clean) | ImageNet-C (Nine perturb. Avg) |\n|---------------|------------------|--------------------------------|\n| Cross-Entropy | 81.14            | 54.31                          |\n| w/ Bottleneck | **81.52**            | **56.19**                          |\n\nWe observe that information bottleneck is still beneficial to improve robustness on this unimodal image classification setup with a pure vision model.\n\n> W3: Vittle introduces 20% extra training time and 1.5% extra parameters compared to baseline. Concerning that in many benchmarks Vittle brings very limited performance gain, is it possible that the difference simply comes from the increase of the training time and parameters? I think it is necessary for the authors to clarify this problem.\n\nWe thank the reviewer for raising this important point. We agree that it's important to understand this more clearly. \n\nFirst, the added parameter count is only 1.5%, localized to the bottleneck module, which is negligible compared to the 7B–13B LLM backbone; prior work shows such a small increase has no measurable effect on capacity. Second, we controlled for training time by conducting longer‑epoch baselines: extending baseline training by the same 20% brings no comparable robustness improvements to Vittle's improvements. Finally, our method preserves in‑distribution accuracy, unlike other regularization approaches with similar or higher cost (e.g., weight decay, information‑maximization baselines; Table 3 & Fig. 6).\n\n| Epoch | Method     | POPE V Shift | \n|-------|------------|--------------|\n| 1     | Baseline   |      83.38        |\n| 1.2 (20% longer)   | Baseline   | 84.17             |\n| 1     | Vittle (F) |      **84.74**        |\n\nThus, **the observed robustness improvements arise from the information‑theoretic bottleneck regularization introduced by Vittle, not from marginal increases in parameters or training time**. We will clarify this in the revision.\n\n### Reference\n- Dubois et al. 2020, Learning Optimal Representations with the Decodable Information Bottleneck\n- Li et al. 2022, Invariant Information Bottleneck for Domain Generalization\n- Wortsman et al. 2022, Robust fine-tuning of zero-shot models"}, {"review_id": "xFsLzksA92", "reviewer": "Reviewer_BHRh", "summary": "This paper investigates the enhancement of MLLM adaptability to perturbations by leveraging the information bottleneck concept. A bottleneck layer is inserted into LLaVA's LLM output layers. The authors provide theoretical grounding to demonstrate that adding this layer corresponds to the theoretical lower bound of the information bottleneck. Furthermore, experiments are conducted on 3 models (2 different-sized LLaVA and Prism-7B) to evaluate and enhance the robustness of MLLMs against perturbations.", "questions": "1. The article emphasizes the concept of IB, but its implementation closely resembles VIB normalization, with similar work also seen in CIB [A]. Could you highlight the innovation of this article in comparison to such VIB normalization methods?\n\n2. The bottleneck layer proposed in the article is designed to enhance robustness against input perturbations, as evidenced by the experimental results. How complementary are these benefits to common data augmentation schemes?\n\n3.The experiments in the article are primarily conducted on two - stream VLM. To increase the credibility of the results, could you provide additional experimental results on one - stream VLM?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The authors provide a theoretical justification for Vittle by connecting it to an information-theoretic robustness metric of MLLMs;\n2. The method proposed in this paper is simple and applicable for further research and application;\n3. The paper is well written.\n\nWeaknesses:\n1. The article emphasizes the concept of IB, but its implementation closely resembles VIB normalization, with similar work also seen in CIB [A]. Could you highlight the innovation of this article in comparison to such VIB normalization methods?\n\n2. The bottleneck layer proposed in the article is designed to enhance robustness against input perturbations, as evidenced by the experimental results. How complementary are these benefits to common data augmentation schemes?\n\n3.The experiments in the article are primarily conducted on two - stream VLM. To increase the credibility of the results, could you provide additional experimental results on one - stream VLM?\n\n[A]  Correlation information bottleneck: Towards adapting pretrained multimodal models for robust visual question answering. International Journal of Computer Vision, 132(1), 185–207.", "was_revised": true, "final_justification": "Q1: Theoretical distinction between CIB and Vittle: The authors' detailed clarification sufficiently explains the theoretical and operational differences between CIB and the proposed Vittle framework.\n\nQ2: Complementarity with Augmentation: The supplemental ablation studies have empirically demonstrated partial complementarity with standard augmentation techniques. This evidence is acceptable.\n\nQ3:  About the application on different VLM architectures: The results on the 2-stream structured VLM are also quite convincing.\n\n\nAccording to the above considerations,  I raise the score to 5.", "rebuttal": "Thank you for taking your time to give us the professional comments; We would be happy to answer your questions (Q) below.\n\n> Q1: The article emphasizes the concept of IB, but its implementation closely resembles VIB normalization, with similar work also seen in CIB. Could you highlight the innovation of this article in comparison to such VIB normalization methods?\n\nThank you for pointing out this relevant work, which we had missed. We agree that CIB [Jiang et al., 2023] is relevant and will cite it in the next revision. While both approaches are inspired by the information bottleneck principle, our contributions differ significantly in terms of objective formulation, implementation, and experimental scope:\n\n1.  **Objective formulation**: Although both CIB and ours use the variational lower bound for the mutual information $I(Z,Y)$, CIB adopted a multimodal-correlation driven upper bound for $I(Z,X)$, whereas we derive an autoregressive conditional language modeling-specific upper bound for $I(Z,X)$ with variational inference. **This results in distinct formulations tailored to sequence‑generation‑based instruction tuning** (See Eq. (6) of CIB paper and Eq. (4) of our paper).\n2. **Implementation**: CIB estimates $I(Z,X)$ with external neural estimators (e.g., CLUB, NWJ), which can introduce instability. By contrast, Vittle uses a closed‑form KL divergence formulation, avoiding the variance of estimator‑based methods. In addition, we proposed a novel interpolation between pre‑ and post‑bottleneck representations, where the interpolation coefficient $\\alpha$ is progressively increased during training. **This design is unique to our work, and plays a crucial role in achieving stable multimodal instruction tuning** (see Table 10 in the Appendix), as naive adoption of a hard bottleneck ($\\alpha$ from the start) leads to unstable training and poor downstream performance.\n3. **Experimental scope**: CIB evaluated robustness in a _multi‑choice VQA classification setup_ with BERT‑scale bi-directional attention transformer, which differ substantially from today’s autoregressive LLM‑based MLLMs. In contrast, our work applies Vittle to **billion‑scale causally-masked attention transfomer backbones in autoregressive instruction tuning, aligning with modern MLLM design**. This allows us to evaluate on a much broader set of tasks—including **open‑ended VQA**, closed‑form VQA, and hallucination detection, whereas CIB narrowly focused on closed-form VQA tasks. Moreover, our robustness evaluation is extensive, spanning **45 diverse datasets and 30 distribution shifts** across visual, textual, and joint shifts, providing one of the most comprehensive robustness studies to date for multimodal LLMs.\n\nThese points highlight the originality and unique contribution of this work, which we believe is a crucial milestone for robust instruction tuning of modern MLLMs that are quite unexplored yet.\n\n\n> Q2: The bottleneck layer proposed in the article is designed to enhance robustness against input perturbations, as evidenced by the experimental results. How complementary are these benefits to common data augmentation schemes?\n\nWe thank the reviewer for this thoughtful question. Vittle is complementary to data augmentation: augmentation methods improve robustness by exposing the model to perturbed inputs during training, while Vittle directly regularizes the internal representations via the information bottleneck, making them more invariant to nuisance factors. This difference in mechanism means the two approaches can potentially reinforce each other. In fact, preliminary experiments where we combined Vittle with standard visual augmentations (SimCLR Augmentation [Chen et al. 2020], AutoAugment [Cubuk et al. 2018], and RandAugment [Cubuk et al. 2019]) showed additive gains in robustness over using either approach alone.\n\n| Augmentation       | Method     | POPE V Shift |\n|--------------------|------------|--------------|\n| -         | Baseline   |       83.38       |\n| -         | Vittle (F) |       84.74       |\n| SimCLR Aug         | Baseline   |   85.00           |\n| SimCLR Aug         | Vittle (F) |   85.67           |\n| AutoAugment   | Baseline   |     84.99         |\n| AutoAugment   | Vittle (F) |     85.91         |\n| RandAugment | Baseline   |      84.39        |\n| RandAugment | Vittle (F) |      85.37        |\n\n\n\n> Q3: The experiments in the article are primarily conducted on two - stream VLM. To increase the credibility of the results, could you provide additional experimental results on one - stream VLM?\n\nWe would like to clarify that our main experiments are already conducted on one‑stream architectures, such as LLaVA‑v1.5, where visual features are projected into the LLM embedding space and processed jointly with text tokens by the same backbone. According to the definition in a vision-language pre-training survey paper [Chen et al. 2022], LLaVA belongs to the one-stream VLM. This setup is widely adopted in recent multimodal LLM research, ensuring fair comparison with prior work. To further demonstrate generality, we also evaluated Vittle on Prism-7B in Appendix, another strong one‑stream VLM, and observed consistent robustness improvements under distribution shifts. \n\n### Reference\n- Jiang et al. 2023, Correlation Information Bottleneck: Towards Adapting Pretrained Multimodal Models for Robust Visual Question Answering\n- Cheng et al. 2020, CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information\n- Belghazi et al. 2018, Mutual Information Neural Estimation\n- Cubuk et al. 2018, AutoAugment: Learning Augmentation Policies from Data\n- Cubuk et al. 2019, RandAugment: Practical automated data augmentation with a reduced search space\n- Chen et al. 2020, A Simple Framework for Contrastive Learning of Visual Representations\n- Chen et al. 2022, VLP: A Survey on Vision-Language Pre-training"}]

Author rebuttal:

Meta-review: This paper addresses the robustness of multimodal large language models under distribution shifts, by introducing visual instruction bottleneck tuning. The proposed method is grounded in the information bottleneck framework. The paper derives a variational lower bound, connects it to robustness metrics, and implements a practical algorithm. Experiments on several datasets across diverse shift scenarios show consistent improvements 

This paper received the reviewing comments from five reviewers. Reviewers raised concerns in (i) unclear technical details and advancement over previous work; (ii) more fine-grained experimental result analysis (such as answer diversity, expressiveness, safety, and bias analysis); (iii) the balance between computational efficiency and performance gains; (iv) the difference between the current form and previous workshop version. After the rebuttal, four reviewers are positive and reach a consensus. Reviewer RnJj is concerned about the concern (iv). As this is common nowadays for submitting the same work to a non-archival workshop and a top-tier conference within the machine learning community, and does not violate the confidence policy, this is not considered to be the reason for the paper to be rejected. 

The AC has checked the paper and the reviewers’ comments, and considers this to be solid work. Therefore, an acceptance recommendation is made. The authors are encouraged to incorporate the reviewers’ constructive feedback in the final version to further strengthen the clarity and impact of this work.
Decision: Accept (poster)