Title: Visual Instruction Bottleneck Tuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Large Language Models

Vision-Language Models

Instruction Tuning

Distribution Shifts

Robustness

Information Bottleneck
Abstract: Despite widespread adoption, multimodal large language models (MLLMs) suffer performance degradation when encountering unfamiliar queries under distribution shifts. Existing methods to improve MLLM generalization typically require either more instruction data or larger advanced model architectures, both of which incur non-trivial human labor or computational costs. In this work, we take an alternative approach to enhance the generalization and robustness of MLLMs under distribution shifts, from a representation learning perspective. Inspired by information bottleneck (IB) principle, we derive a variational lower bound of the IB for MLLMs and devise a practical implementation, Visual Instruction Bottleneck Tuning (Vittle). We then provide a theoretical justification of Vittle by revealing its connection to an information-theoretic robustness metric of MLLM. Empirical validation of multiple MLLMs on open-ended and closed-form question answering and object hallucination detection tasks over 45 datasets, including 30 shift scenarios, demonstrates that Vittle consistently improves the MLLM's robustness under shifts by pursuing the learning of a minimal sufficient representation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the enhancement of MLLM adaptability to perturbations by leveraging the information bottleneck concept. A bottleneck layer is inserted into LLaVA's LLM output layers. The authors provide theoretical grounding to demonstrate that adding this layer corresponds to the theoretical lower bound of the information bottleneck. Furthermore, experiments are conducted on 3 models (2 different-sized LLaVA and Prism-7B) to evaluate and enhance the robustness of MLLMs against perturbations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The authors provide a theoretical justification for Vittle by connecting it to an information-theoretic robustness metric of MLLMs;\n2. The method proposed in this paper is simple and applicable for further research and application;\n3. The paper is well written.\n\nWeaknesses:\n1. The article emphasizes the concept of IB, but its implementation closely resembles VIB normalization, with similar work also seen in CIB [A]. Could you highlight the innovation of this article in comparison to such VIB normalization methods?\n\n2. The bottleneck layer proposed in the article is designed to enhance robustness against input perturbations, as evidenced by the experimental results. How complementary are these benefits to common data augmentation schemes?\n\n3.The experiments in the article are primarily conducted on two - stream VLM. To increase the credibility of the results, could you provide additional experimental results on one - stream VLM?\n\n[A]  Correlation information bottleneck: Towards adapting pretrained multimodal models for robust visual question answering. International Journal of Computer Vision, 132(1), 185–207.\n\nQUESTIONS:\n1. The article emphasizes the concept of IB, but its implementation closely resembles VIB normalization, with similar work also seen in CIB [A]. Could you highlight the innovation of this article in comparison to such VIB normalization methods?\n\n2. The bottleneck layer proposed in the article is designed to enhance robustness against input perturbations, as evidenced by the experimental results. How complementary are these benefits to common data augmentation schemes?\n\n3.The experiments in the article are primarily conducted on two - stream VLM. To increase the credibility of the results, could you provide additional experimental results on one - stream VLM?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-26"}

comments: SUMMARY:
The paper proposes Visual Instruction Bottleneck Tuning (Vittle) to enhance the robustness of MLLMs under distribution shifts. The technique is built upon the information-bottleneck theory and specially designed for resolving the distribution shift problem of MLLMs.

STRENGTHS AND WEAKNESSES:
Strengths:
This paper proposes a theoretically sound technique to resolve the distribution shift problem for MLLMs. The experimental results show the  effectiveness of the proposed method. The implementation is clear and can easily be reproduced.

Weaknesses:
1. It looks the performance of Vittle(F) and Vittle(L) varies across tasks. For example, Vittle(L) excels in long-tail scenarios, while Vittle(F) performs better under perturbations. Since in many benchmarks either Vittle(F) or Vittle(L) brings very limited or even negative performance gain (e.g., Vittle(F) in Table 2 and Vittle(L) in Figure 5 left), it seems Vittle requires task-specific hyper-param choice (F or L) under distribution shift.
2. The relationship between IB and model robustness in distribution shift is not clearly stated. Why "This property is particularly
129 desirable for robust instruction tuning"(Line 128-129)? Can IB enhance the robustness of other models in other tasks (e.g., CNNs in image classification) under distribution shift? 
3. Vittle introduces 20% extra training time and 1.5% extra parameters compared to baseline. Concerning that in many benchmarks Vittle brings very limited performance gain, is it possible that the difference simply comes from the increase of the training time and parameters? I think it is necessary for the authors to clarify this problem.

QUESTIONS:
See Weaknesses.

LIMITATIONS:
See the third question in Weaknesses.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new approach, called Visual Instruction Bottleneck Tuning, to improve the robustness of multimodal large language models under distribution shifts. Instead of scaling data or model, they apply the Information Bottleneck (IB) principle to encourage the model to learn minimal sufficient representations. They derive a novel variational lower bound for the IB objective tailored for MLLMs, and instantiate it as a simple bottleneck layer inserted into the LLM backbone. They test Vittle on 45 datasets, showing consistent gains on various perturbation and long-tail settings, while maintaining strong performance on standard benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper addresses a very practical and important problem, i.e., the robustness of MLLMs under distribution shifts. Although there exists several research works on this domain, it is still largely under-explored to me. \n\n2. The idea to integrate IB principle into end-to-end instruction tuning for large-scale MLLMs is novel and well motivated.\n\n3. I appreciate the theoretical analysis, especially how they connect their IB objective to EMID robustness metric.\n\n4. Experiments are very extensive, covering various types of distribution shifts and several long-tail benchmarks. The improvements are quite consistent\n\n5. The implementation is lightweight, only adds ~1.5% parameters, with nearly unchanged inference cost.\n\n**Weakness**\n\n1. The method requires tuning hyperparameters like α. It is unclear how sensitive final performance is to this.\n\n2. The paper focuses mainly on visual QA tasks. It is interesting to see whether the same approach can benefit more realistic tasks like agentic tasks.\n\nQUESTIONS:\n1. Did you try applying Vittle on purely language tasks (without images)? Will the IB principle still help for textual instruction tuning?\n\n2. How stable are your results across different random seeds? Given that sampling is used in bottleneck, is variance significant?\n\n3. Could you share intuition why the fixed standard Gaussian prior Vittle(F) sometimes works better under perturbations than the learnable prior?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper introduces the Information Bottleneck (IB) theory into end-to-end instruction tuning for Multimodal Large Language Models (MLLMs), aiming to improve model robustness to out-of-distribution inputs without increasing data volume or model size. The authors derive a variational IB lower bound tailored for autoregressive MLLMs, design lightweight and pluggable bottleneck layers, and validate the method across 30 distribution shift scenarios, demonstrating consistent improvements in both open-/closed-ended QA and object hallucination detection tasks.

STRENGTHS AND WEAKNESSES:
Strengths:

1. Simple and general implementation: A single pluggable MLP layer with two modes—fixed or learnable priors—allows reusability in existing training pipelines.

2. Comprehensive experiments: The study covers three task types, 45 datasets, and 27 perturbation levels, including ablations, prior comparisons, and fine-grained severe shift analyses. Results consistently outperform standard instruction tuning and several baselines (e.g., LoRA, weight decay, information maximization methods).

3. Clear writing and logical structure.

Weaknesses:

1. Mild performance regressions underexplored: Slight drops occur in some multi-disciplinary multiple-choice tasks (e.g., MMMU), but causes and trade-offs are not discussed.

2. Lack of safety and bias analysis: Information compression may risk filtering critical information (e.g., leading to misdiagnoses), yet the paper does not explore or assess such potential downsides.

3. Limited resource consumption reporting: Only training/inference time is reported, with no details on memory or GPU usage.

QUESTIONS:
Please refer to the weakness.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Visual Instruction Bottleneck Tuning (Vittle), which inserts an information-bottleneck layer inside the LLM backbone to regularise representations. Robustness is evaluated on 30 distribution-shift scenarios created from POPE (object-hallucination) and LB-COCO (open-ended QA) perturbation suites.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1. Solid theoretical grounding. The authors derive a variational IB lower bound and link it to EMID, a robustness metric.\n\n2. First end-to-end IB for MLLMs. Unlike prior projector-only work, Vittle places a learnable bottleneck within the LLM and trains it jointly.\n\n3. Clarity of exposition. The paper is well-structured with intuitive diagrams and ablation details.\n\nWeaknesses\n\n1. Marginal gains. Relative improvements on perturbed sets are mostly single-digit pp; absolute scores remain close to the baseline.\n\n2. Perturbation realism. Visual corruptions focus on generic CV transforms (brightness, blur, noise); they might not match real LMM deployment errors (occlusion, compression, camera shake).\n\n3. Older model backbone. Experiments centre on LLaVA-v1.5-7B/13B and Prism-7B; results on newer models are missing.\n\n4. Unclear novelty over ICML workshop version. The manuscript does not explicitly differentiate itself from the prior workshop paper.\n\nQUESTIONS:\nQuestions for Authors\n\n1. Does Vittle still help when applied to recent models?\n\n2. How sensitive is performance to the IB weight β and to the insertion layer l?\n\n3. Why limit the bottleneck to instruction-tuning? Would pre-training with IB amplify robustness?\n\n4. Could the authors elaborate on when Vittle (L) vs (F) is preferable?\n\n5. Have you considered more realistic perturbations (partial occlusion, codec compression, ASR/OCR noise)?\n\n6. What is the inference-time latency and memory overhead beyond the ~1.5 % parameter increase and 20 % training-time rise?\n\n7. Does the bottleneck reduce answer diversity or expressiveness while lowering hallucination?\n\n8. Please clarify concrete advancements over the previous ICML-Workshop submission (data size, theory, experiments, scope).\n\nThese points should help strengthen your review and guide the discussion phase.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-09"}