Learning to Instruct for Visual Instruction Tuning
Authors: Zhihan Zhou
Feng Hong
Jiaan Luo
Yushi Ye
Jiangchao Yao
Dongsheng Li
Bo Han
Ya Zhang
Yanfeng Wang
Venue: NeurIPS 2025 poster ()
Keywords: Visual Instruction Tuning
Multimodal Large Language Models
Primary area: deep_learning

Abstract: We propose L2T, an advancement of visual instruction tuning (VIT). While VIT equips Multimodal LLMs (MLLMs) with promising multimodal capabilities, the current design choices for VIT often result in overfitting and shortcut learning, potentially degrading performance. This gap arises from an overemphasis on instruction-following abilities, while neglecting the proactive understanding of visual information. Inspired by this, L2T adopts a simple yet effective approach by incorporating the loss function into both the instruction and response sequences. It seamlessly expands the training data, and regularizes the MLLMs from overly relying on language priors. Based on this merit, L2T achieves a significant relative improvement of up to 9% on comprehensive multimodal benchmarks, requiring no additional training data and incurring negligible computational overhead. Surprisingly, L2T attains exceptional fundamental visual capabilities, yielding up to an 18% improvement in captioning performance, while simultaneously alleviating hallucination in MLLMs. Github code: https://github.com/Feng-Hong/L2T.

Learning to Instruct for Visual Instruction Tuning
Zhihan Zhou"! Feng Hong" Jiaan Luo'
Yushi Ye'
Jiangchao Y Dongsheng Li² Bo Han Ya Zhang Yanfeng Wang"
"Cooperative Medianet Innovation Center. Shanghai Jiao Tong University
Microsoft Research Asia 'Hong Kong Baptist University
ASchool of Artificial Intelligence, Shanghai Jiao Tong University
3 Institute of Artificial Intelligence for Medicine, Shanghai Jiao Tong University School of Medicine {zhihanzhou, feng. hong, Sunarker) edu.
Abstract
We propose L2T, an advancement of visual instruction tuning (VTT). While VIT equips Multimodal LLMs (MLLMs) with promising multimodal capabilities, the current design choices for VIT often result in overfitting and shortcut learning, potentially degrading performance. This gap arises from an overemphasis on instruction-following abilities, while neglecting the proactive understanding of visual information. Inspired by this, L2T adopts simple yet effective approach by incorporating the loss function into both the instruction and response sequences. It seamlessly expands the training data, and regularizes the MILMs from overly relying language priors. Based on this merit, L2T achieves a significant relative improvement of up to on comprehensive multimodal benchmarks, requiring no additional training data and incurring negligible computational overhead. Sur- prisingly, L.2T attains exceptional fundamental visual capabilities, yielding up 18% improvement in captioning performance, while simultaneously alleviating hallucination in MLLMs. Github code: wtw.gihhhu....onnggggeeee
Introduction
COCOI?
Clungi
Large Language Models (LLMs) have achieved significant progress and success. Built this, Multimodal LLMs (MLLMs) have garnered substantial attention in the re- search community. There has been surge in advancements based on the Visual In- struction Tuning (VIT) [Liu et al., 2023a] paradigm. By aligning language inputs and visual representations through simple con- nectors, and subsequently conducting end- to-end mnnitu on carefully designed multimodal instruction data, MLLMs [Liu et al., 2023a, b024a,b, Zhu et al., 2024, Tong et al., 2024] have achieved notable improvements across various multimodal tasks like visual question answering and image captioning.
Fivicti4
13:2
betip
112
2017
KM000
11.38
11333
C0X
Vamme12
VESLLaM--1
Figure 1: Performance comparison on 16 tasks be- tween L2T and VIT using different models, including Tiny VA Qwen2-0.55 [Zhou et al., 2024], LLaVA- 1.5 Vicuna-77 7B [Liu et al., 2024a], and LLaVA-1.5 Vicuna- 13B [Liu et al., 2024a]. The pretraining phase uses the LLLaVA-pretrain-S- -55kk dataset, while the fine- tuning phase employs the
dataset. Equal contribution. Work done during Feng Hong's intersship Microsoft Research Asia.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Typically, visual instruction tuning consists of two stages: pre-training and fine-tuning [Liu et al., 2023a, Zhu et al., 2024]. In the pre-training stage, connector is trained to align visual and language data. The fine-tuning phase is similar to instruction tuning in language models, where the model trained to generate responses based on multimodal instructions. However, recent research has shown that instruction tuning can lead to a series of issues, such as the knowledge degradation [Ghosh et al., 2024] and hallucinations [Rawte et al., 2023, Tonmoy et al., 2024]. Similar problems have also been observed in models that employ visual instruction tuning [Huang et al., 2023, Leng et al., 2024]. The potential causes may lie in overfitting and shortcut learning during instruction tuning [Sun et al., 2024a]. Figure 2 illustrates an example of shorteut, where the model might ignore visual content and generate responses based solely on language priors. Current advancements [Liu et al., 2024a,b, Bai et al., 2023b, Tong et al., 2024] in MLLMs implicitly mitigate such issues by leveraging larger, higher-quality, and more diverse training datasets, larger models, and improved pretrained initializations for visual and language backbones.
In this paper, we propose an orthogonal solution to improving visual instruction tuning: learning to instruct images regularizer, which we refer to as Learning to InstrucTT (L2T). Specifically, in addition to learning to gen- erate responses to given images and instructions usual, L2T also learn to generate instructions for images that exclude templates, which refer to special tokens and figh.frreuency.. low-information template tokens in the instructions. L2T en- hances VIT by: (I) expanding the training content mitigate overfitting without explicitly enlarging the training set (Sec- tion 3.4); and (2) learning to instruct images, which forces the model to focus more on the visual content and prevents learning some shorteuts (Section 2).
Training example
Image:
Instruction: What color is the plate? Answer the
question using single word phrase.
Response:
Instruction:
What is in the glass?
Response:
We conduct extensive experiments across a range of 16 tasks, comparing L2T and VIT on different models. As shown in Figure 1, L2T achieves a significant relative improvement of up to 6% over VIT in overall multimodal task performance. Moreover, L2T shows significant potential in alleviating hal- lucination issues in MLLMs, as demonstrated across four diverse hallucination benchmarks (Section 3.3). Through comprehensive and detailed experiments, L,T demonstrates the following advantages: (1) It improves the ability across different multimodal tasks compared the conventional vi- sual instruction tuning, especially those focusing more on visual content, such as OCR, captioning and hallucination mitigation. (2) Our method is orthogonal to existing research advancements and can further enhance MLLMs' performance through simple integration. (3) It shows advantages in im- proving performance while making minimal compromises training and inference costs.
LLM's 's responses w/O image
Prompt:
What color is the plate? Answer the
question using single word or phvase.
LLM:
White
Promi:
What the gloss? Answer che question using single word phrase.
L.I::
Water
Figure 2: An example where a pure language model provides correct an- swers based only on language pri- ors, without relying on visual content. This shows that learning to generate responses alone cannot prevent the model from taking shortcuts by ignor- ing visual content and relying solely on textual instructions.
Method
Problem Formulation. Taking LLaVA an example, typical model architecture comprises pre-trained large language model f()) with parameters 0y with the corresponding tokenizer and embedding layer t(-) (e.g., Vicuna [Peng et al., 2023]), a pre-trained visual feature extractor g(-) with parameters 0, (e.g., CLIP-ViT-L/14 [Radford et al., 2021J). and a cross-modal connector h(-) with parameters 0% such as linear layer or MLP. Let 0 {Ø,, 0g. 02} denote the set of all parameters. For an image Xv and a related text instruction X,, we obtain the corresponding textual response through the following forward process. As shown in Figure 3, the image Xy is forwarded through the visual feature extractor and cross-modal connector h, mapping it to visual tokens Hy = higg( (X))) in the language embedding space. Hy is then combined with the language tokens H,, ((X,) to form a sequence, which is forwarded into LLM f for autoregressive generation of the response sequence X^. The goal of visual instruction tuning is to train the model to exhibit strong multi-modal instruction-follovii capabilities.
2

---

Learn instruct
Learn to respond
Response X
Language Model
I Langunge Model
Cross-Modal Connector
Tokemizer &
What color Arower the question usig the plate? ja single word or phrose.
Emboddiing Layer
White
Visual Feslure Extractio
Image Ny
Instruction
N,
Image Xy
Instruction X,
Response X,
Figure 3: The model architecture using LLaVA as an example, and the data flow for generating responses from images and instructions.
Figure 4: Illustration of L2T. In addition to learn- ing to generate responses like VIT, L2T also learns to generate instructions that exclude tem- plates.
Visual Instruction Tuning (VIT). For given training sample triplet (Xv, X,, XA)', the sequence lengths of X, and XA are L, and LA. respectively. The training objective of VIT is to learn to generate the response X Specifically, it involves learning predict each token XA.: in XA, where i E (1,2... ......... LA). based on the image Xv, the instruction X,, and the preceding response sequence XA,<i- The loss function for the training sample is formulated the negative log-likelihood of the response given the image and the instruction:
L = - log pa (A Xv X,) - Σ lo X1, XA.ii).
(I)
Training is typically divided into two stages: pretraining and fine-tuning, In the pretraining stage, only the cross-modal connector is trained to align visual features with the language embedding space. In the fine-tuning stage, the entire model is trained end-to-end, with options to freeze the visual feature extractor and apply LoRA [Hu et al., 2022] fine-tuning.
Learning to Instruct (L2T). Building upon VIT, we propose extending its paradigm through learning to instruct images. Specifically, in addition to learning how to generate response given an image and an instruction, we also learn how to generate meaningful instruction for given image. For training sample triplet (Xv, X,, XA), we define the loss function L as the negative lokelikelihood of both the instruction and the response conditioned on the image:
log pop (X,, = Σ log po (1,, Xv. X1,t))
log pe (X,,) [v. X1, XA,<t)
Learn to Instract
Learn Respond
(2)
By learning to generate appropriate instructions for images. L2T achieves two key benefits. I)It
naturally expands the data the model learns to fit, helping to alleviate potential overfitting to some extent. 2) Learning to instruct images ensures that the model focuses on the image content, effectively preventing it from ignoring the visual input and relying solely on languages to generate responses.
To ensure the model learns meaningful content related to the image, we exclude the learning of certain irrelevant parts of instructions. Such irrelevant context primarily arises from two sources: (1) system templates and (2) task templates. Specifically, system templates refer to the tokens used to guide the MLLMs in adopting the role of helpful and polite Al assistant, or as the conversational clues distinguishing whether the content is generated by the "USER" or the "ASSISTANT". System templates can be easily removed when it is added. Task templates refer to tokens that indicate the task type and output format. We identify task templates by calculating the frequency of all sentences in the entire training dataset and selecting the most frequent ones. The detailed removed task templates can referred Appendix A. Note that the training data in the
Template Removal.
For simplicity, we use single-tun tur conversations as an example, which can be naturally extended to multi-tum conversations.
3

---

noce 1.304 VGT mose 1.224
man 00534
11131100 & 6.6. 3.3
0.00
(a) VC on VQA+2.
(b) VC on Doc.QA.
500 210
340 409 505
500
710
100 200 300
di 200 000 100
(a) Atlention of VIT.
(b) Attention of L2T.
Figure 5: The visual contribution (VC) distri- butions of VIT and L2T on the VQAv2 training data and VQA test data. Experiments are based on TinyLLaVA Qwen2-0.5B.
Figure 6: Visualization of attention weights. Darker colors indicate higher attention weights. Experi- ments are based on TinyLLaVA Qwen2-0.5B.
Instruction:
Instruction:
Instruction:
which Aed and Good grape mesmumi Arnwer the gunsion with singl weet
plaing on the moskue??
How you deschb
VII model:
The monbor shows itioge of compater mouse.
weather imge
penking thrugh beantfi
2009
model:
1.27 model
1.21 model:
There no masiar in the image The image fontures white Apple string
The weather is the image denk and eightzine, es the cankeal ride i It and people ar evjniop rice right
2019
Figure 7: Cases, each including image, instruction, and the responses generated by the VIT and L2T models, respectively. The models are based on LLaVA-1.5 Vicuna-7B. (Left) The VIT model shows better OCR capabilities. (Middle) The VIT model demonstrates robustness and effectively avoids being influenced by misleading information in the language instruction. (Right) The VIT model provides more comprehensive and accurate image description.
pretraining stage is constructed using image-caption pairs, with a set of template prompts serving as instructions, which are unrelated the image content. Therefore, we apply our method L.2T only during the end-to-end fine-tuning stage. In Figure 4, present illustration of L2T.
Analysis. To assess the rationality of L2T alleviating shorteut learning MILMs, we conduct experiments to gain deeper insight into which aspects of visual or instructional content are most influential in response prediction. Specifically, we first introduce visual contribution (VC), which quantifies the difference in the log-likelihood of the response when conditioned on both the image and instruction, versus when conditioned solely on the instruction:
VC =
X,) - log po (X Xv
= = X,)
where X, = 0 in the implementation denotes that the original visual input is replaced with random noise. In this spirit, VC captures the relative importance of visual signals in response prediction, isolating the contribution of visual information from any confounding effects due to the instructional signals. Figure 5 presents the VC of L.2T and VIT for both the training and testing datasets. Our findings reveal that 1.2T achieves a substantial relative improvement, with a 9% increase in VC, highlighting its effectiveness regularizing MLLMs to better utilize visual inputs.
To further substantiate these findings, we visualize the attention weights in L2T and VIT. As shown in Figure 6, L2T leads to stronger activation in the visual components of MLLM attention heads, indicating more effective utilization of visual inputs. This is consistent with the VC results. More experimental details and additional visualization results can be referred to Appendix F.
Moreover, Figure presents several cases, each consisting of an image, an instruction, and the responses generated by the VIT and L,2T models, respectively. It can be observed that by learning to instruct images, the 1.2T model demonstrates more precise and comprehensive image understanding. as well stronger robustness against misleading information in the language instructions.

---

3 Experiment
3.1 Setup
Model Architectures. (1) LinyLLaVA [Zhou et al., 2024]: We follow TinyLLaVA to utilize small- scale LLMs, including Qwen-2-0.5B (Yang et al., 2024] and Phi-2-3B [Gunasekar et al., 2023]. The SigLIP-400M [Zhai et al., 2023] is adopted as the vision encoder due to its effective performance in combination with small-scale LLMs. The multimodal connector follows LLaVA 1.5 Liu et al. [2024a] with a two-layer MLPP and GELU activation. (2) LLaVA 1.5, LLaVA-NeXT [Liu et al., 2024b]: For VL 1.5, the base LLMs, while for LLaVA-NeXTI Vicuna-v1.5-7B serves as the base LLM. The base vision encoder is CLIP-ViT-LA [Radford et al., 2021], and the connector is a two-layer MLP.
Vicuma-v1.5-27 [Peng et al.. 2023] and Vicuma-v1.5 1.B-13B [Peng et al., 2023] as
Training Datasets. (1) Pretraining Stage: We adopt the LLaVA 1.5 framework, utilizing the LLaVA- pretrain-558k data [Liu et al., 2024a] for all pretraining phases. (2) Finetuning Stage: For instruction tuning, we use the LLaVA-mis 11AAmixx-656 665k data [Liu et al., 2024a] on TinyLLaVA and LLaVA 1.5. For LeaVA-NeXT, we use the with diverse instruction data. More details can be referred to Appendix B.2.
[Liu et al., 2024b], expansion of LLaVA-mix-665k
Implementation Details. We train all models on NVIDIA 1100 GPUs, strictly following the training recipes of TinyLLaVA, LLaVA 1.5 and LLaVA-NeXT. See Appendix B. for more details.
Evaluation Benchmarks. We conduct a thorough evaluation of four distinct capabilities using 16 multimodal instruction datasets. These capabilities are categorized as follows: (1) General Visual Question Answering, assessed through VQAv2 [Goyal et al., 2017]. GQA [Hudson and Manning, 2019], ScienceQA [Lu et al., 2022], and VizWiz [Gurari et al., 2018 8]: (2) Comprehensive Multimodal Benchmarks, evaluated using MME [Fu et al., 2023]. MMMU [Yue et al., 2024], and MMStar [Chen et al., 2024b]; (3) Chart, Document, and OCR Understanding. evaluated using ChartQA [Masry et al., 2022]. TextVQA [Singh et al., 2019], DocVQA [Mathew et al., 2021]. IJ, and OCR Bench [Liu al., 2023b]; and (4) Image Captioning, assessed through COC02017 [Lin et al., 2014]. Flickr30k [Young et al., 2014], NoCaps [Agrawal et al., 2019], RefCOCO [Kazemzadeh et al., 2014], and TextCaps [Sidorov al., 2020]. The performance metries used include CIDEr [Vedantam et al., 2015] for all captioning tasks, the perception score for MME, and accuracy for the remaining tasks. The experiments are conducted using LMMs-Evall [Zhang et al., 2024]. In Section 3.4, some tasks are evaluated on the lite version, denoted as "-L".
3.2 Main Results
In Table 1, we present comprehensive evaluation of L2T across multiple benchmarks.
General Visual Question Answering. We first evaluate several widely used VQA benchmarks. including the most prominent VQAv2 benchmark and the visual reasoning-oriented oriented GQA. We also assess VizWiz, which requires predicting unanswerable questions, and ScienceQA, which covers a diverse range of scientific topics. Our results demonstrate that our proposed 1.2T achieves competitive VQA performance when compared to VIT.
We evaluate the multi-discipline multimodal understand- ing and reasoning capabilities of our L2T through MME, MMMU and MMStar benchmarks. These benchmarks span a wide range of disciplines, including Art & Design, Business, Science, Health & Medicine, Humanities & Social Sciences, Technology & Engineering, and encompass diverse tasks such as object presence, counting, spatial reasoning, commonsense inference, numerical computation, and code reasoning. Our results demonstrate that L2T consistently outperforms VIT, achieving an average relative improvement of up to 3.7%. This underscores the effectiveness of 1,2T in enhancing performance across diverse multimodal applications.
Comprehensive Multimodal Benchmarks.
Chart, Document, and OCR Understanding. We include a diverse set of OCR-related datasets to evaluate the performance of models locating, extracting and interpreting text from various data visualizations, including charts, diagrams, documents, and natural images, in order to accurately address the associated questions. The results indicate that our L2T significantly outperforms VIT across all OCR-related benchmarks, yielding an average relative improvement of 6.3%. This high-
5

---

Table 1: Performance comparison of VIT and L2T across 16 representative multimodal bench- marks. The benchmarks span four categories: General VQA, multi-discipline benchmarks, Char/Documen/OOCR understanding, and image captioning. We report the average relative im- provement for each category, along with the overall relative improvement across all benchmarks. "LLaVA-NeXT's training script is not fully apen-sourceds our implementation may differ slightly.
Visual Instruction Tuning (VIT)
Learning to Instruct (L2T)
Benchmark
LLNANNXX
TaplZaNA Qrax2435
TiulLaVa
Qna:2430
Vamm-133
Viause78
VQAv2
72.31
77.32
76.64
78.26
80.07
72.35
77:48
77.35
78.70
80.62
Denast QA
GQA
57.48
61.46
61.97
63.25
64.24
56.89
61.12
62.32
63.91
62.54
ScienceQA
58.55
71.78
69.46
71.39
70.10
59.89
71.00
68.96
71.34
70.60
QA
56.53
41.85
56,02
58.66
Average
+1.5%
+0.5 %
+0.7%
+0.2%
+1.1 %
MME
1189.99
1475.51
1506.26
1522.60
J477.28
1175.24
1431.12
1531.7
1533.63
1556.44
MMMU
29.67
36.67
36.33
34.33
32.33
37.89
35.56
MMStur
34,93
36.07
33.65
36.12
37.26
36.14
36.44
34.63
36.66
38,98
Average
+3.7% %
+0.4%
+2.2%
+1.3%
+2,8%
CAartQA
13.28
16:48
18:24
17.92
54.88
13.80
17.20
18.96
20.44
66.80
TexVVQA
44.48
52,10
46.09
48.73
64.82
45.
52.35
47.58
50.24
65.06
DCB
DocVQA
22.32
28.46
21.43
23.49
68.49
24.60
30.74
23.99
25.59
72.52
Dic. Uhabrnandinggg
OCR Bench
26.50
34.50
31.30
33.30
52.00
27.20
35.40
32.30
36.20
55.80
Average
%
+3.9%
+8.7%
+8.8%
COC02017
97.85
100.82
110.38
115.20
99.96
100.37
102.61
112.96
114,46
139.21
Irage Capiniing
Flickr30k
64.50
76.00
74.85
79.45
68.46
66.09
78.34
77.64
81.45
75.26
Cyploiing
NoCaps
101.13
109.15
88.35
93.08
103.60
114.10
ReCCOCO
17.50
30.59
29.76
34.22
33.82
27.55
52.98
42,24
56.16
35.36
Capianis
TextCaps
95.80
105.79
98.15
103.71
70.40
94.99
105 36
105.14
107.58
74.03
Inne Oxpinning
Average
+12.6%
+ +16.0%
+11.5%
14.11
+17.6%
Overall
+6.1%
+6.2%
+5.6%
+6.9%
+8.5%
lights the effectiveness of the proposed L2T in extracting fine-grainck visual information. This be attributed to the fact that our 1.2T encourages the proactive generation of instructions. In OCR-related datasets like TextVQA, the instructions require predicting "Reference OCR token" that captures the informative textual content within the image. This necessitates a more comprehensive understanding of the low-level information embedded in the images, thus enhancing OCR capabilities.
Image Captioning. We assess image-captioning. a representative image-text task widely used as a pretraining task for MLLMs. Our analysis includes classical captioning datasets such as COC02017 and Flickr30k, along with more specialized datasets such as TextCaps, which involves fine- grainne textual content description, and NoCaps, which represents out-of-domain scenarios. The results demonstrate that L2T achieves the most substantial performance improvement over VIT on captioning datasets, among the various capabilities evaluated, with an average relative improvement of up to 17.6%. The reasons are two-fold: (1) L2T motivates a more comprehensive understanding of the images, which guarantees the generation of accurate descriptions, and (2) L2T alleviates the negative effects of overfitting on instruction-followinn abilities, preserving more of the captioning skills acquired during the pretraining stage.
Finding. L2T consistently outperforms VIT across all benchmarks, achieving an average relative improvement of up to 8.5%. The most significant gains are observed in OCR-related and captioning datasets, with improvements of up to 8,8% and 17.6%. This suggests that L2T excels at regularizing MLLMs to pay more attention to visual input and strengthening their fundamental visual capabilities.
3.3 Hallucination Evaluation
In this section, evaluate the effectiveness of L2T in mitigating hallucinations, shown in Table 2.

---

Table 2: Hallucination evaluation results. We present the average aceuracy computed across ad- versarial, popular and random splits of POPE, along with per-instance CHAIR, and per-sentence CHAIR, (4) on COCO2014, with superscript G the greedy strategy and superseript B as beam search. Additionally, we report the Question Pair Accuracy (qAcc), Figure Accuracy (ʃ Acc) and All Accuracy (a Acc) for Hallusion Bench, along with the GPT score and hallucination rate MMHAL-
Bench. Best results are highlighted in bold. Experiments are hased on LLaVA-1.5 Vicuna-7B.
Datases
POPE
COC02014
HallasionBench
Metric
Adv.
Pop.
Random
CHAIR,
CHAIR
CHAIR, (.)
CHAIR
gácr
f
a.ccc
VIT
85.17
87.30
88.47
48.60
13:40
$3.40
14,90
10.33
19.36
43,85
1.T
85.60
87.90
88.47
46.20
11,80
51.40
13.10
10.77
19.36
44.90
Dataset
AIMHAL-Bench
Metric
Score
Hal. (1)
Attribute
Adversarial
Comparison
Counting
Relation
Environment
Holistic
Other
VIT
1.73
0.68
3.00
1.25
1.58
2,50
1.17
2,08
1.00
1.25
L2T
2.36
0.53
3.08
1.67
2.92
1.92
2.25
3.33
1.92
1.83
Results on POPE. POPE [Li et al., 2023b] is the most commonly used benchmark for object hallucination. It consists of "Yes or No" questions to evaluate the MLLMs' capability determine whether the given object is in the image. We report accuracies based on questions derived from adeversarial, popular and random sampling. 1.2T outperforms VIT in both adversarial and popular sampling settings, showcasing the effectiveness of 1.2T reducing object hallucinations.
Results CHAIR Evaluation. In addition to the discriminative evaluations on POPE, utilize the CHAIR metric [Rohrbach et al., 2018] as a complementary approach to evaluate object hallucination in image captioning tasks. Specifically, CHAIR quantifies the proportion of objects referenced in an image caption that are absent from the corresponding ground-truth label set. As shown in Table 2, our L2T consistently outperforms VIT, achieving improvements of 2.2% and 1.7% in terms of the CHAIR, and CHAIR, metric, respectively. This further demonstrates that our L2T effectively regularize MLLMs, reducing the generation of plausible but visually irrelevant contents.
Results on MMHAL- CHAIR mainly focuses on object hallucination by evaluating the pres- ence of objects. To provide a more fine-grained assessment, we further adopt MMHAL-Bench [Sun et al., 2024b] to evaluate a broader range of hallucination types. For evaluating the quality of the gen- erated content, we leverage GPT-4 to compute the GPT score and hallucination rate for comparison. The results demonstrate that L2T significantly outperforms VIT by 36% in average score and 22% in hallucination rate. Moreover, L2T consistently outperforms VIT across various hallucination types, with notable improvements in adversarial, comparison, relation, environment, and holistic settings.
Results on Halh. To further improve the comprehensiveness of hallucination evaluation, we use HallusionBench [Guan et al.. 2024] to include more scenarios with various disciplines, image types and input modalities. We report Question Pair Accuracy AgAcc), Figure Accuracy (f (ccc and All Accuracy (aAAc)) based on GPT-4 for comparison. The results demonstrate that L2T outperforms VIT, with improvements of 0.44% and 1.05% in terms of AAc and aAcc, respectively, highlighting its effectiveness in mitigating a wider range of failure modes multimodal hallucinations.
3.4 Further Analysis
Mitigate Overfitting from the Loss Perspective. In Figure 8, we present the loss distributions of our L.2T and VIT on the training dataset (VQAv2) and test datasets (DocVQA and NoCaps). The cross-entropy loss is calculated specifically for the response component, excluding the instruction part, even for L2T. The results reveal that L2T exhibits slightly higher loss than VIT on the training data but achieves lower loss unseen test data. This clearly demonstrates that our method, by learning to instruct images as a regularizer, effectively avoids overfitting on the training data and exhibits superior generalization performance.
Ablation on the Scale of Instruction Data. the instruction tuning phase on L2T, we present the performance of both the baseline VIT and L2T at 40%, 60%, 80% data levels in Figure 9()) Note that we use the same scale for the axes of both radar charts. It can be observed that, at the same data scale, L.2T significantly outperforms VIT. Specifically, from 40% to 80% instruction data, our method shows an average performance improvement over VIT across the eight metrics in Figure 9(a) by 5.9%, 13.0%. and 9.3%, respectively. The performance
To investigate the impact of different data volumes during
7

---

L2T. nxee 0.414
3.24
127. iose -2339
022
VIT mon- 0.380
VIT m.00-2.77
3.20
9.1
2.10
9.10
608
1700%
(a) Train loss on VQAv2. (b) Test loss Doc VQA.
(a) Instruction data scale. (b) Pretraining data scale.
Figure 8: (a) Training loss distribution of L2T and VIT on VQAv2. (b) Testing loss distribution of 1,T and VIT on Doc The standard cross- entropy loss is calculated for the response part. Experiments are based TinyLLaVA Qwen2- 0.5B.
Figure 9: Ablation on (a) instruction data scale and (b) pretraining data scale. The figure shows the performance of L2T and VIT using 40%, 60%. 80%, and 100% of the data, with another stage using the full dataset. Experiments are based on TinyLLa Qwen2-0.5B.
Table 4: Ablations on template removal. The table shows the performance of VIT, which learns to respond on XA, and L2T, which learns to instruct using variants of progressively removing system template X and task template XT. Here, x) refers to the instructions excluding both the system and task templates. Experiments are based on LinyLLaVA Qwen2-0.5B.
Method
Benchmark
x/
Xj
XA
VQAv2-I
VizWiz-L.
MMMU
MMStar
ChanQA
OCR Beneh
Flickrš0k-L
RefCOCO-L
64,02
29.90
29.67
34.93
13.28
26.50
70,28
14.81
14.48
73.79
65.26
29.52
31.22
35.85
13.76
27.40
74.21
21.97
-9%
66.34
29.74
32.33
36.14
13,80
27.20
74.30
23.11
+115
improvements suggest that our method benefits from increasing data volume, highlighting its potential for continuous improvement within the research trend of expanding data scaling laws.
Ablation on the Scale of Pretraining Data. We also investigate the impact of data volumes during the pretraining phase on L2T, with the results shown in Figure 9()). L2T demonstrates a clear advantage over the baseline at the same data scale. From 40% to 80% pretraining data, the average improvements are 10.9%, 8.7% and 8.5%, respectively. Interestingly, the performance with 80% pretraining data even surpasses that with 100% pretraining data. This "less-is-mon more" [Zhou al., 2023] phenomenon highlights the complexity of how pretraining data impacts final performance.
Ablation on Types of instruction data. We selected different types of data from the full dataset (mixed with 10% of other types of data), includ- ing QA data, GPT-generated data, se- lection data, grounding data, and cap- tioning data. We conduct instruction tuning using each of these data types, and the results are presented in Ta- ble 3. It can be observed that the im- provements of our method vary con- siderably across different data types, which we attribute primarily to the dif- ferences in instruction quality. The in- structions for grounding data consist only of fixed templates and almost ran- dom bounding box coordinates, making difficult for the model to gain improvements from learning such instructions. In contrast, the instructions for QA data contain wealth of information related to the image content, which enables our method to provide a significant improvement.
Table 3:
Performance comparison of VIT and L2T when using a single data type as the primary training data, includ- ing Choice, Grounding, GPT-generated, Captioning, and QA data, Experiments are based on LinyLLaVA Qwen2-0.5B.
Data
Method
VQAv2-L
MMMU
ChanQA ReiCOCO-L
Choice
VIT
52.76
31.56
12.56
20.45
L2T
50.90
31.22
11.88
25.33
+3%
Grounding
53.10
33.76
11.32
L2T
52.62
32.11
11.32
26.51
+0%
VIT
53,90
31.11
11.60
8.42
GPT-gen
L2T
55.16
31.78
11.68
11.34
+10%
Captioning
VIT
45.12
29.44
11.52
16.02
L2T
47.46
31.11
12.00
16.85
+5%
VIT
63.26
31.00
11.84
16.94
QA
62.28
33.00
12.36
Ablation on Template Removal. To gain more insights, we conduct experiments to assess the impact of removing each prompt template from the instructions, as shown in Table 4. Four experimental setups considered: (1) the baseline method, which learns only the answer component; (2) the
8

---

Table 6: Performance comparison VLM baseline Prism-7B. L2T yields consistent improvements across diverse benchmarks, highlighting its general effectiveness.
Task
GQA
VizWiz
TexiVQA
RefCoCo
RefCOCO+
RefCOCOg
POPF
VSR
A12D
VIT Prism-7B 61.92
55.36
52.80
56.70
50.70
52.70
88.00
53.20
55.50
1.2 Prism-7B
62.62
57.75
55.60
66.00
58,90
62.00
88.50
61.70
57.10
straightforward method, which learns both the full set of instructions and answers; (3) the method that learns the instructions excluding system and format messages (USER/ASSISTANT tokens), while still learning the answers; and (4) the method that learns the instructions without system/format messages and task- specific prompts. while learning the answers as well. The results indicate that progressively removing fask-irrelevant template tokens enhances the overall performance of L2T, preventing MLLMs from overfitting to redundant template tokens.
Computational Analysis. To validate the computa- tional efficiency of L2T, we conduct experiments on finetuning VL. Vicuna-7B using instructional data with varying instruction-to-reet to-respose length ra- tios L)). ranging from 0.05 20. The data are sampled from the training set. We report the average number of samples per second and the average num- ber of steps per second (averaged over 100 training steps). Figure 10 demonstrates that L2T achieves an average of 0.331 + 0.005 steps per second, while VIT achieves 0.334 + 0.005. Our L2T incur only a negligible computational overhead of less than 1 1%. Detailed results can be referred to Table 11 in the Appendix E.
0.4
VIT
0.3
0.2
0.3
0.05
0.1
10
20
Inéticilon--t--epponss Length Ratios
Figure 10:
Computational cost across dif- ferent Lı/LA Experiments are based on LLaVA-1.5 Vicuna-7B.
Evaluation another VLM baseline Prism-7B. To further verify the generalizability of L2T, we conduct additional experiments on another robust vision-language model (VLM) baseline, Prism [Karancheti et al., 2024], which surpasses LLaVA-1.5 by incorporating optimized train- ing strategies, advanced image preprocessing, and fused visual backbones such as SigLIP and DiNOv2. We apply L2T Prisn-DINOSigLIP-CCOO -ontioolspLPP-CCttoollee----- and compare it with the standard VIT-style Prism baseline across diverse benchmarks, including visual question answering (GQA, VizWiz, Tex))QA), localization (RefCOCO, RefCOCO+, RefCOCOg), and challenging reasoning tasks (POPE, VSR, AI2D). As shown in Table 6, L2T consistently improves performance across all benchmarks, demonstrating its effectiveness and broad applicability.
Effect on Textual Understanding. To examine whether L2T compro- mises language proficiency by altering the SFT data composition, we eval- uate the model on several text-only benchmarks, including MT-Bench, WildBench, MMLU, and AGIEval. We use the LLaVA-1.5 Vicuna-7B model as the baseline and apply L2T under identical training configurations. As shown in Ta- ble L,T achieves comparable results to the baseline on MT-Bench, MMLU, and AGIEval, while notably improving WildBench performance. These results indicate that learning to instruct functions as a synergistic regularization rather than a disruptive modification, enhancing visual grounding without compromising the model's core text comprehension capabilities.
Table 5: Performance comparison on text-only benchmarks. L2T maintains comparable or improved performance, indi- cating no degradation in core language ability.
Benchmark
MTBench
WildBench
MMLU
AGllvval
ViT LLaVAI,5-7B
5.35
-9.40
49.57
32.62
LiT LLaVAL5-7B
5.49
-6.27
49.18
32.49
Exploring the potential of L2T for self-improving instruction tuning. To illustrate the broader potential of L2T, we conduct pilot exploration where the model leverages its own generation capability to enhance itself. Starting from L.T model pretrained on a 100k subset of the LLaVA-mix- 665k dataset, we prompt it with image-only inputs to automatically produce 100k instruction instruction-response pairs. These self-generated samples are then merged with the original training data for continued fine-tuning. As shown in Table 7, this simple self- seff-bootstrapping process leads to consistent gains across multiple benchmarks. The results highlight 1.2T's potential to evolve through its own generated supervision, paving the way toward self-improving vision- languaee models.

---

Table 7: Exploring the potential of L2T for self- self-improving instruction tuning. Incorporating 100k model-generated instruction response pairs improves performance across diverse benchmarks.
Benchmark
VQA-2-L
VizWiz.L
MMMU
MMSTAR
ChariQA
OCR Bench
Plickr30lk.L.
ReCOCOLL
56.68
11.96
17.42
L2T TinyLLaV 0.5B w/ generated data
60.80
24.22
32.33
34.75
12.44
24.50
68.32
24.53
4 Related Work
Visual Instruction Tuning. Recent advances multimodal learning [Radford et al., 2021, Zhou et al., 2025, Zhao et al., 2025, Liu et al., 2025] have greatly enhanced the integration of visual and linguistic understanding. Building upon these foundations, the concept of Visual Instruction Tuning was first introduced in LLaVA [Liu al., 2023a] and MiniGPT-4 [Zhu et al., 2024], aiming to unify the understanding of vision and language by leveraging pre- vre-trained visual and language models. Common system architectures typically consist of (1) pre-trained visual model for encoding visual features, (2) a pre-trained large language model for interpreting images and user instructions and generating responses, and (3) cross-modal connector for aligning visual features with the language model's input. Visual resamplers, such as Qformer [Li et al., 2023a], can serve as an optional module to reduce the number of visual patches [Bai et al., 2023a, Dai al., 2023]. LLaVA-NeXT [Liu et al., 2024b] significantly enhances visual perception by using dynamic visual resolutions. DEEM [Luo al., 2024] replaces the traditional visual encoder with diffusion model, further enhancing visual perception. Cambrain-1 I [Tong et al., 2024] improves visual robustness through visual encoder routing, but it also introduces higher training overhead.
(Language) Instruction Tuning, Instruction tuning has emerged as a critical approach in aligning large language models (LLMs) with specific tasks domains. By fine-tuning language models on datasets composed of task instructions and corresponding responses, this approach has demonstrated its effectiveness in enhancing generalization to unseen tasks, as evidenced by models like Instruct- GPT [Ouyang et al., 2022] and Flan-PaLM [Chung et al., 2024]. Early explorations of instruction tuning achieved notable success using human- written completions [Bai al., 2022, Ouyang et al., 2022, Wei et al., 2022]. Recent studies [Wang et al., 2023, Honovich et al., 2023] have expanded on this by exploring how content generated by large language models can be used to construct instruction tuning datasets, further enhancing model capabilities.
The most relevant recent work to ours is IM [Shi al., 2024], which incorporates loss over instructions during the instruction tuning process of language models. However, our work differs significantly in the following key aspects: 1. Scope: Our work focuses on MLLMs, whereas IM is on language models. 2. Motivation: Our primary motivation is to encourage the model to pay greater attention to visual information and avoid shortcut learning based solely on textual cues. In contrast, IM is primarily aimed at mitigating overfitting. 3. Methodological Differences: We employ an automated approach filter out the impact of frequently occurring template instructions (see ablation study in Table 4), whereas IM only filters limited set of special tokens, such as "< user |>" 4. Scalability: We validate the effectiveness of L2T on a large-scale dataset of nearly 1M samples. whereas IM's evaluation is limited to much smaller datasets, showing performance saturation at just 13k samples.
5 Conclusion
In this paper, we propose L2T, which enhances multimodal capabilities by learning to instruct images as regularizer, rather than focusing only on learning to respond. This enables L2T to seamlessly expand the training data, reducing overfitting and proactively guiding the model to learn visual inputs more effectively, thereby avoiding shortcuts. Experimental results demonstrate the effectiveness of L2T across 16 multimodal tasks, highlighting its superior performance on OCR and image captioning tasks by placing greater emphasis on visual content. Furthermore, L2T significantly improves hallucination mitigation. is also worth noting that our method is orthogonal to existing advancements in MILMS and can be easily integrated into these methods with minimal compromises on computational costs. We believe that 1.2T has the potential to evolve the general VIT framework by mitigating overfitting, enhancing data efficiency, and promoting improved generalization in MLLMs.
10

---

Acknowledgement
This work is partially supported by the National Key R&D Program of China (No. 2022/D0160703), National Natural Science Foundation of China (No. 62306178) and STCSM (No. 221D/2229005), 111 plan (No. BP07 19010). BH is supported by RGC Young Collaborative Research Grant No. C2005-24Y, RGC General Research Fund No. 12200725, and NSFC General Program No. 62376235.
References
Harsh Agrawal, Peter Anderson, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, and Stefan Lee. nocaps: novel object captioning at scale. In ICCV, pages 8947-8956. IEEE, 2019.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large -aigunla model with versatile abilities. Arxiv, abs/2308. 12262263..196,,,, 2023a.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. ArXiv, abs/2402. 14289, 2023b.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac HatfeeldDDodss, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovit, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. Arxiv, abs/2204.05862, 2022.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui Hc, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, editors, ECCV, volume 15075 of Lecture Notes in Computer Science, pages 370-387. Springer, 2024a.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? CORR, abs/2403. 2024b.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction finctuned language models. J. Mach. Learn. Res., 25:70:1-70:53, 2024.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, NeurIPS, 2023.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng. Ke Li, Xing Sun, and Rongrong Ji. MME: comprehensive evaluation benchmark for multimodal large language models. CoRR, abs/2306. 13394, 2023.
Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Ramaneswaran S., Deepali Aneja, Zeyu Jin, Ramani Duraiswami, and Dinesh Manocha. A closer look at the limitations of instruction tuning. In ICML. OpenReview.net, 2024.
Yash Goyal, Tejas Khot, Douglas Summers-Stur Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 6325-6334, IEEE Computer Society, 2017.
11

---

Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang. Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision- language models. In CVPR, pages 14375-1438 14385- IEEE, 2024.
Suriya Gunasekar, Yi Zhang. Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giomo, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah. Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need. CoRR, abs/2306.11644, 2023.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, pages 3608-3617. Computer Vision Foundation IEEE Computer Society, 2018.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. In Anna Rogers, Jordan L. Boyd-Graber and Naoaki Okazaki, editors, ACL, pages 14409-1 14428. Association for Computational Linguisties, 2023.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR. OpenReview.net, 2022.
Qidong Huang, Xiaoyi Dong. Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. OPERA: alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocatee
CoRR, abs/231 1.19911, 2023.
Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 5700-6709 Computer Vision Foundation / IEEE, 2019.
Kushal Kafle, Brian L. Price, Scott Cohen, and Christopher Kanan. DVQA: understanding data visualizations via question answering. In CVPR, pages 5648-5656. Computer Vision Foundation / IEEE Computer Society, 2018.
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vims: Investigating the design space of visually-conditioned language models. In ICML. OpenReview.net, 2024.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L. Berg. Referitgame: Referring to objects in photographs of natural scenes. In Alessandro Moschitti, Bo Pang, and Walter Daclemans, editors, EMNLP, pages 787-798. ACL, 2014.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Min Joon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, ECCV, volume 9908 of Lecture Notes in Computer Science, pages 235-251. Springer, 2016.
Geewook Kim, Teakgyu Hong, Moonbin Yim, Yeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, ECCV, volume 13688 of Lecture Notes in Computer Science, pages
Springer, 2022.
Ranjay Krishna, Yuke Zhu. Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis., 123(1):32-3,, 2017.
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In CVPR, pages 13872-13882. IEEE, 2024.
12

---

Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, ICMI,, volume 202 of Proceedings of Machine Learning Research, pages 19730-19742. PMLR, 2023a.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, EMNLP, pages 292-305. Association for Computational Linguisties, 2023b.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, ECCV, volume 8693 of Lecture Notes in Computer Science, pages 440-555. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, NeurlPS, 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 26286-26296. IEFF, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024b.
Yikun Liu, Yajie Zhang, Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiangchao Yao, Yanfeng Wang, and Weidi Xie. Lamra: Large multimodal model as your advanced retrieval assistant. In CVPR, pages 4015-4025. Computer Vision Foundation IEEE, 2025.
Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, and Xiang Bai. On the hidden mystery of OCR in large multimodal models. CoRR, abs/2305, abs/2305.07895, 2023b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR. OpenReview.net, 2019.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, NeurlPS, 2022.
Run Luo, Yunshui Li, Longze Chen, Wanwei He, Ting-En Lin, Ziqiang Liu, Lei Zhang, Zikai Song, Xiaobo Xia, Tongliang Liu, Min Yang, and Binyuan Hui. DEEM: diffusion models serve as the eyes of large language models for image perception. Arxiv, abs/2405.15232, 2024.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: A vi- sual question answering benchmark requiring external knowledge. In CVPR, pages 3195-3204. Computer Vision Foundation IEEE, 2019.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq R. Joty, and Enamul Hoque. Chartqu: A benchmark for question answering about charts with visual and logical reasoning. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, ACL, pages 2263-22777 Association for Computational Linguistics, 2022.
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. Docvqa: A dataset for VQA on document images. In WACV, pages 2199 2208 IEEE, 2021.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. OCR-VQA: visual question answering by reading text in images. In ICDAR, pages 947-952. IÉEE, 2019.
Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. Lims know more than they show: On the intrinsic representation of LLM hallucinations. The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, 2025.
13

---

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, NeurlPS, 2022.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with GPT-4. CoRR, abs/2304.03277, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, ICML, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR, 2021.
Vipula Rawte, Amit Sheth, and Amitava Das. A survey of hallucination in large foundation models. CoRR, abs/2309.05922, 2023.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujü,, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 2018. pages 4035-4045. ACL, 2018.
Dustin Schwenk, Apoory Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-OKVQA: A benchmark for visual question answering using world knowledge. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, ECCV, volume 13668 of Lecture Notes in Computer Science, pages 146-162. Springer, 2022.
Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He, CODI: compressing chain-of-thought into continuous space via self-distillation. arXiv, abs/2502.21074, 2025.
Zhengxiang Shi, Adam X. Yang. Bin Wu, Laurence Aitchison, Emine Yilmaz, and Aldo Lipani. Instruction tuning with loss over instructions. In NewrIPS, 2024.
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: A dataset for image captioning with reading comprehension. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, ECCV, volume 12347 of Lecture Notes in Computer Science, pages 742-758. Springer, 2020.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In CVPR, pages 8317-8326. Computer Vision Foundation IEEE, 2019.
Zechen Sun, Yisheng Xiao, Juntao Li, Yixin Ji, Wenliang Chen, and Min Zhang. Exploring and miti- gating shortcut learning for generative large language models. In Nicoletta Calzolari, Min- Yen Kan, Véronique Hoste, Alessandro Lenei, Sakriani Sakti, and Nianwen Xue, editors, LREC/COLING. pages 6883-6893. ELRA and ICCL, 2024a.
Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang. Yiming Yang. Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented RLHF In Lun- Wei Ku, Andre Martins, and Vivek Srikumar, editors, ACL, pages 13308-131111
Association for Computational Linguistics, 2024b.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang. Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Arxiv, abs/2406. 166600 2024.
S. M. Towhidul Islam Tonmoy, S. M. Mchedi Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. A comprehensive survey of hallucination mitigation techniques in large language models. CoRR, abs/2401.01313, 2024.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In CVPR, pages 4566-4575, IEEE Computer Society, 2015.
14

---

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self gelf-generated instructions. In Anna Rogers, Jordan I.. Boyd-Graber and Naoaki Okazaki, editors, ACI., pages Association for Computational Linguistics, 2023.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetumed language models zero-shot learners. In ICLR. OpenReview.net, 2022.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li. Mingfeng Xue, Na Ni, Pei Zhang. Peng Wang, Ru Peng, Rui Men, Ruize Gao. Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang. Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report. CoRR, abs/2407. 10671, 2024.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Trans. Assoc. Comput. Linguistics, 2:67-78, 2014.
Xiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang. Huan Sun, Yu Su, and Wenhu Chen. MMMU: A massive multi multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In CVPR, pages 9556-9567. IEEE, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, pages 11941-11952. IEEE, 2023.
Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models. CoRR, abs/2407. 12772, 2024.
Zihua Zhao. Feng Hong, Mengxi Chen, Pengyi Chen, Benyuan Liu, Jiangehao Yao, Ya Zhang, and Yanfeng Wang. Differential iiferential-informedd sample selection accelerates multimodal contrastive learning. In ICCV, 2025.
Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang, Tinyllava: A framework of small-scale large multimodal models. arXiv, abs/2402. 10000....420 14289, 2024.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: less is more for alignment. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, NewrIPS, 2023.
Zhihan Zhou, Yushi Ye, Feng Hong, Peisen Zhao, Jiangchao Yao, Ya Zhang, Qi Tian, and Yanfeng Wang. Uncover the balanced geometry in long-tailed contrastive language-image pretraining. Mach. Learn., 114(4):106, 2025.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In ICLR. OpenReview.net, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All statements in the abstract and introduction are aligned with the main contribution of the paper: learning to instruct images as a regularizer for visual instruction tuning, All claims are supported by extensive experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We included the limitations of the work Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provided the details for reproducing the experimental results in Sec- tion 3.1, including training and model details. Additionally, in Appendix B.I, we provided a comprehensive overview of the hyperparameter configuration.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: For anonymity reasons, we have not made our code public. Upon acceptance, we will release our code on GitHuu.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips. wwtpppbicggguiiiecceceeeeooceeee gww.d/g////eeeeeee/oooccc Codeeebmissiiie wwweiisssssppioiooiooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the details, including the model architecture, implementation details, training data, evaluation benchmarks, hyperparameters, and more, provided in Section 3.1 and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due limited computational resources, we did not include error bars. This aligns with most work in visual instruction tuning, where error bars are typically omitted due to the high cost of multiple runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 3.1 and Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https /tte:r////eeepeeee cc Ceppullc/Etiiiiieeeeeeepiippeebe wtw.chhhiiiceieeeees
Answer: [Yes]
The research does not involve human participants or any risks related to privacy, bias, or societal harm, it fully follows the NeurIPS Code of Ethies.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discussed both potential positive societal impacts and negative societal impacts of the work in Appendix I.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
19

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
The datasets used in our research are all existing publicly available datasets, and this work primarily focuses on designing improved approach of visual instruction tuning.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All code, data, and models used are properly cited. We will include the license, copyright, and terms of use for the code asset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswithcom thcode. com/ comddataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Upon acceptance, we will provide a well-documented README file that includes clear instructions, details about the code and model, and running examples to ensure easy understanding and usage.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: /topp://neewee wtwpipppccc/ooocfeeeececmm /cc//onfeeeence///22///// for what should or should not be described.
22

---

A Detailed Removed Task Templates
In this section, we provide a comprehensive overview of the task templates removed during the training of TinyLLaVA, LLaVA 1.5, and LLaVA-NeXT, as shown in Table 8 and Table 9. Notably, -LaVA-NeXT-DDtt includes a more diverse set of instruction datasets, leading to wider variety of task templates. Consequently, larger number of task templates are removed compared to those discarded from the LLaVA-mix-6665k dataset.
Table 8: Detailed removed task templates for training TinyLLaVA, LLaVA 1.5.
Task Templates
Answer the question using a single word or phrase.
Answer with the option's letter from the given choices directly.
Provide one-eentence caption for the provided image. Reference OCR token:
Please provide short description for this region:
Please provide the bounding box coordinate of the region this sentence describes:
What is the title of this book?
What is the genre of this book?
What type of book is this?
Who is the author of this book?
Who wrote this book?
Table 9: Detailed removed task templates for training LLeVA-NeXT.
Yaok Templates
Asower the guoeties ising single went phaose
Asswar the guocion with CICCO fomur.
Provide shont docrigtau for the gince eqgann.
What is the slle ths book?
Anouer with the opston's leter from glee cheices dimetly.
Phoside the hourling hos coondroirc regionn the pives arrimnor dewebbo.
OCR image sectios by sectios, foon top to botoo, right. image, space
Du
catpst test.
oype
Who axhor thi book?
Doserite this
with GPT COCO Gorarr.
Provide cono-c captou for the pervabad image.
Psevide requesad infoemaaion dimeciy.
What is the grere of this book?
Asswer the greston with a sigge ward
Aiz the valaos in the chus presonsed is percentane seale?
cach bar stell sdid coker witboe patemm?
Which prong hos the anailin snreend valar?
Which group lox the bareu varerrdd witeo?
wu?
contan vacked
group?
proup?
Wlich bas
the smallst valued individual bas whole char?
indiiiad
chart?
Which fun omneer the vitued individaal ber while char??
What
indiviiial fur whale duun??
Hlw
thene?
Which lages vata??
What is the value of the lurgest bur?
Which har has the smollet value?
Does the chatt costaz any sezative valuo??
Acc the vass in the duat preseree is a logusitanic scile?
sigottim for the erollent aecemcy ammmed armes consure??
Which higgrthm los the lerent scrancy cummedd antres all catsorid?
lubcl socoad aroup froes lel??
chjest prefoned
saitber of poople scasseed acos all catepooies?
ohjec
by
poople
the cargoneet
What lubel
fios
Wlich
sumber scits susted
stous?
Which ohyet
peleened
What tanel of me
from the left
Which horn wid the nost on of unis sumced across al the stors??
How reas has there per grou?? Which checo the leas preforme
What a the ludcl
socord bares the keli?
How many grous hass thon??
Wiichet the tead grefored in any canegery?
alporiaa lus lovest accamay?
accunac sigetten highte: accurayy
dlprcatoon with lowesi sccuracy?
Which signtiim
las highoet asomsyy
dasaet?
higkes
Which object is the most prefous for dstwer?
Which algoritm fer linand
What lowcer acceracy reperved while clun??
catepac??
What the liaci of tae tland grogg bas fuoo the left?
What the Lubol the soored har from the botton in rack grapp?
What the bincl of the fire far froes the bottonn in each groap?
What the cifferznce batwon mast and lead moliened otyas??
What the dificerace betworn the bigast and the sttalien: value the chunt
How much more acconte the mos accunno algaldin compared to the lead accorate dlpetii??
What the label of the thied her from the left'
wnd
unite?
rass uzis
ner were sold?
Hon many people
skjeor?
Wlich
Hie
the willng cellie whole doun??
postir lase protomul obyeer?
Which izin wid leua usits skop??
How mass unis did the want sellung item sell is the viall chat!
Hlw many people ake the lead pesiemed ohjoc the whole chart?
Now many people like the moe perfomed abreci in the whole doue??
What the lubel thidd bet from the left cads gavep?
What the lotel of the fourth group burs from the lett.
What the ludi fire mnop hus frre botten!?
How mans more the mos and item wese sold cxmpueed to the lead sold iemn?
the secoad grop froor bottons?
bar from
B
More Details on Experimental Setup
B.1 More Details on Hyperparameter
Detailed hyperparameters for training TinyLLaVA, LLaVA 1.5 and LLaVA-NeXT are shown in Table 10. For TinyLLaVA and LLaVA 1.5, pretraining is conducted with learning rate of le-3 and a batch size of 256, while finetuning uses a learning rate of 2e-5 and a batch size of 128. For LLa pretraining uses a learning rate of 1e-3 and batch size of 128, with finetuning using
23

---

learning rate of le-5 and a batch size of 32. All experiments use the AdamW optimizer [Loshchilov and Hutter, 2019] and cosine decay schedule with a warm up ratio 0.03. During pretraining phase, only the multimodal connector is trained. In finetuning, LaVA and LLaVA 1.5 jointly train the connector and language model, while LLaVA-NeXT trains all parameters, including the vision encoder.
Table 10: Training Hyperparameters for TinyI.LaVA, LLaVA 1.5 and LLaVA-NeXT.
TinyLLLaA & LLaVA 1.5
LLaVA-NeXT
Hyperparameter
Pretrain
Finetune
Pretrain
Finetune
Learning rate (LR)
le-3
2e-5
le-3
le-5
LR warmup ratio
0.03
0.03
0.03
0.03
Batch size
256
128
128
LR schedule
cosine decay
cosine decay
cosine decay
cosine decay
Epoch
1
Optimizer
AdamW MLP
AdamW MLP, LLM
AdamW MLP
AdamW Vision enc., MLP, LLM
Trainable parameters
B.2 More Details on Training Data
In the finetuning stage of instruction tuning, we use the LLaV-xix--665k data [Liu al., 2024a] on LinyLLaVA and LLaVA 1.5, which incorporates diverse set of instruction-following datasets, including free conversational data (LLtVA-Instruct [Liu et al., 2023a]), visual question answer- ing (VQAv2 [Goyal et al., 2017], GQA [Hudson and Manning, 2019], OKVQA [Marino et al., 2019], A-OKVQA [Schwenk et al., 2022I), OCR and captioning (OCRVQA [Mishra al., 2019], TextCaps [Sidorov et al., 2020]), and visual grounding (RefCOCO [Kazemzadeh et al., 2014], VG [Kr- ishna al., 2017J). For LLaVA-NeXT, we use the A.L.VA-NeXT-s extends the LLaVA-mix-665k [Liu et al., 2024a] dataset with high-quality user instruction data from LAION-GPT-V, ShareGPT-4V [Chen et al., 2024a], and LLa VA-demee [Liu et al., 2023a], as well OCR, document, and chart data from VQA [Mathew et al., 2021], SynDog-EN EN [Kim et al., 2022], ChartQA [Masry et al., 2022], DVQA [Kafle et al., 2018], and A12D [Kembhavi et al., 2016].
[Liu al., 2024b], which
C More Details on CHAIR Evaluation
As mention in Section 3.3, we utilize the CHAIR metric to evaluate object hallucination in image captioning tasks. The metric assesses object hallucinations through two dimensions: per-instance CHAIR; and per-sentence CHAIR.. The former measures the fraction object instances that hallucinated, while the latter determines the proportion of sentences that contain least one hallucinated object. Definitions of these two metrics are formally provided:
[hallucinated objects)|
CHAIR;
|all objects mentioned]]"
CHAIR, (sentences with hallucinated object]|
H{ all sentences))
We follow [Huang et al., 2023] to perform the CHAIR evaluation the COC02014 dataset. Specifi- cally, we randomly select 500 images in the validation set. The decoding process uses the greedy strategy and the Beam search with Nbeam 5, with maximum of 512 tokens for new content generated.
D
More Cases
More cases of our I.2T compared to VIT are shown Figure 11. Our I.2T consistently outperforms VIT across range of scenarios, including OCR, image captioning, and hallucination mitigation.
24

---

Instruction:
Indruction:
What is witton in the imaee
Context: Scoriet mocows live the forests of Centrul and South Arrerica They eut seeds ond nuts. shope of hard nuts Which bird's beak ols udspeed crock lorge, hard nuts?
VIT Model:
caspted to open lorge.
The image fectures cursie
Centre
wring whec appears a signonuee message.
Afiicn grzy porot Whitesaped sickleh! Answer with the option's letter from the given chaices directly.
L.T Model:
The image fectures 0 handwmiten word "centree written in cursive.
12T Model:A
Irstruction:
Instrcttion
Provide onesemtence
coption for the provided inoge
Wiidi flowers opp, which flowers are an the bottom??
VII
VIT Model::
pulling large
The white flowers are on the top, and the ble flowers on the bortoon
pipe on traler.
1.2 Model:
L.T Modell:
Two men are stonding next to mack with a long nube
The bue flovers are on top, and the white flowers
Instruction
the which buttons should cantral the above?
VIT Madiet:
The buttons the sier the ieft the white counterops should be used control stove.
varicus postions on foant of stove, as pay ottention to the ones that correspond to your needs end use them accordingly.
L2T Model:
the image the stove is modem, digital stove with control ponel. To cororol the stove, you should buttons control pone,. These
be used anjusting the temperature, tirer settings and other functions the stove. The control purel is opiiaaly locoted on the fiont or side the scowe, and allws operate the stove safely and effoerdyy
Figure 11: More cases of L2T and VIT across OCR, image captioning, and hallucination mitigation tasks.
E
Computational Cost
Table 11 presents the computational cost of VIT and L2T across different length ratios. The experiments are conducted on NVIDIA A100 GPUs.
Table 11: Computational cost of VIT and L2T across different instruction instruction-to-respmne to-response length ratios. We report the number of samples per batch and steps per batch. Experiments are based on VL- Vicuna-7B.
Q/A Ratio
0.05
20
Metric
sample/s
step/s
sample/s
step/s
sample/s
step/s
sample/s
step/s
sample/s
step/s
VIT
2.617
0.327
2,692
0.337
2.645 2.676
0.331
2,708
0.339
2.678
0.335
L2T
2.640
0.330
2.623
0.328
0.334
2.613
0.327
2.708
0.338
F
Visualization of Attention Heads
Recent observations [Orgad et al., 2025, Shen et al., 2025] show that the hidden activation of the token preceding the answer (the colon ".* in the prompt "ASSISTANT.") encodes more information than the output logits. Inspired by this, we select this token's 's hidden activation for visualization. Specifically, we visualize 14 attention heads from the last layer of a 24-layer transformer during inference, focusing on their activation across input tokens, particularly image tokens, to compare VIT and L.2T. Data samples are randomly selected from VQAv2 [Goyal et al., 2017], GQA [Hudson and Manning, 2019] and OKVQA [Marino et al., 2019]. Detailed information about the selected examples is provided Table 12, and the visualizations of the corresponding attention heads are presented in Figure 6 and Figure 12.
25

---

Table 12: Examples for visualization of attention heads.
Prompt
Image
Visualization
A chat between a curious user and an ar- tificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image>>lnOn which side the photo is the faucet, the right or the lnnserr the question using a sin- gle word or phruse. ASSISTANT:
Figure 6
A chat between curious user and an artifi- cial intelligence assistant. The assistant gives helpful, detailed, and polite answers the user's questions. USER: <image>inIs the car to the left or to the right of the manwennnswr the question using single word or phrase. ASSISTANT:
Figures 12(a) and 12(b)
A chat between a curious user and an artifi- cial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image>in Where these red vegetables imported to the exported from the us?nnnsswr the question using a single word or phrase. ASSISTANT:
Figures 12(c) and 12(d)
A chat between a curious user and an ar- tificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image>inIn which knee is the hole the women's PantsPnnAnswer the question using a single word or phrase. ASSISTANT:
Figures 12()) and 12(f)
G Discussion on the Scope of This Work
Recent vision languae models (VLMs) often struggle with challenges such as hallucination and shortcut learning, which can limit their reliability and generalization. Addressing these issues requires careful consideration of the interactions between different training stages, including Supervised Fine-Tuning (SFT), rule-based Reinforcement Learning (RL), and Reinforcement Learning from Human Feedback (RLHF).
Our work focuses on improving the foundational SFT stage. We posit that the quality of early instruction tuning directly influences the effeetiveness and safety of subsequent alignment. Indeed, recent studies indicate that applying RL RLHF models already prone to hallucination or shorteut learning can reinforce these issues, leading to confidently incorrect outputs.
L2T addresses this problem at its root by reducing shortcut learning and enhancing visual grounding during SFT. A stronger SFT foundation provides a more reliable base for RLHF, enabling safer and more effective alignment. While recent work has emphasized RL and reward modeling as post-training interventions, foundational flaws in SFT can persist or even intensify during alignment. Thus, improving sarly-stage instruction tuning serves as a preventive measure rather than relying solely on corrective post-tree training methods.
Overall, this perspective highlights the broader impact of L2T: by enhancing the base model during SFT, facilitates downstream alignment and helps define the scope of our contribution in enabling safer and more generalizable multimodal training strategies.
26

---

Image Tokens
Image Tokens
4 3 0
3
Heal Indles
Head Irdex
6
7
7
8
8
13 13 10
10
12
100
200
300
400
500
600
700
0
100
200
300
400
500
600
700
Position
Sequence
Position Sequence
(a) Case Attention weights VIT.
(b) Case Attention weights L2T.
Image Tokens
Image Tokens
Head Indexx
5
Head Index
8
13 1312 11 10
100
200
300
400
500
600
700
100
200
300
400
500
600
700
Position Sequence
Position in Soquenee
(c) Case 2: Attention weights in VIT.
(d) Case 2: Attention weights in L.T.
Image Tokens
Image Tokens
4 2
Head
Head
13 11 10
12
100
200
400
500
600
700
100
200
300
400
500
600
700
Position in Sequence
Position in Sequence
(e) Case Attention weights in VIT.
(I) Case 3: Attention weights in L2T.
Figure 12: Visualization of attention weights in VIT and L.TT for additional cases. Darker colors indicate higher attention weights. Experiments are based on LLnYLLaVA Qwen2-0.5B.
H
Limitations
Despite the promising capabilities underscored by L2T, several limitations must be acknowledged. First, the performance improvements achieved by L2T mainly stem from the informative instruction content that aids in better understanding visual inputs. In contrast, redundant instruction content, such as system or task templates, do not contribute to performance gains, as they lack relevant information about the visual content. Second, it is critical to ensure that the instructions do not contain harmful or
27

---

biased content. The presence of such content could raise concerns about the faimess and reliability of MLLMs, making them more susceptible generating flawed, biased, or even harmful responses.
I
Broader Impact
This work focuses improving vision instruction tuning techniques, advancing the multimodal capabilities. can benefit applications such as assistive technologies, education, and content cre- ation. For example, individuals with visual impairments could gain access enhanced image-to-text descriptions, improving accessibility and inclusivity. However, alongside the development of multi- modal technologies, it is crucial address the risks of bias and harmful content. The model may inadvertently propagate societal biases embedded in the training data, leading to discriminatory or inappropriate outputs. Furthermore, enhanced capabilities could be exploited to generate convincing but harmful misinformation or deepfakes, posing risks to societal trust and safety. To responsibly advance this technology, we emphasize the importance of robust mitigation strategies, including diverse and inclusive training data, bias detection tools, and ethical safeguards against misuse, Ensur- ing that these technologies serve the public good requires ongoing collaboration among researchers, developers, and policymakers, fostering innovation while minimizing unintended negative impacts.
28


Reviews:
[{"review_id": "m2K0DCz7QJ", "reviewer": "Reviewer_gAGN", "summary": "This paper primarily investigates a method to enhance the performance of multimodal large models by simultaneously computing the loss for instructions during the Supervised Fine-Tuning (SFT) phase. The authors claim that this approach acts as a form of \"regularization,\" effectively mitigating hallucinations and improving overall performance, which is verified on many benchmarks.", "questions": "I kindly suggest that the authors consider adopting a broader perspective, such as examining the closed-loop interactions between SFT, rule-based RL, and RLHF, to effectively tackle challenges like hallucination and shortcut learning in VLMs. Furthermore, it would be beneficial to stay abreast of recent advancements in post-training methodologies within the language model field, with a particular emphasis on addressing key and fundamental issues.", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Pros\n\nThe authors address an intriguing issue—how to enhance the perception capabilities of multimodal large models while reducing hallucinations. The proposed method demonstrates a notable improvement in benchmark evaluations.\n\n### Cons\n1. I find the paper's approach to be rather superficial. SFT is a highly nuanced process that demands careful attention to various aspects, including the proportion of different tasks and the quality of the response patterns (https://arxiv.org/pdf/2503.01307v1). By introducing a loss for instructions, the method effectively increases the SFT data by 50%, which is dedicated to learning how to instruct. Such a significant alteration in the SFT ratio is bound to have adverse effects, such as a potential decline in text comprehension abilities. This could be observed in evaluations like mtbench (https://arxiv.org/abs/2306.05685), wildbench (https://arxiv.org/abs/2406.04770), and fundamental text capabilities such as BBH, MMLU, and AGIEval. A decrease in language proficiency would inevitably lead to a systemic deterioration in response quality.\n\n2. The baseline used in the article is relatively weak. I suggest testing the method on a more robust VLM baseline to determine if it still yields benefits.", "was_revised": true, "final_justification": "Thanks for the authors' feedback. The rebuttal partly addresses my concerns, so I raise the final rating to borderline accept. The authors may consider adding the discussions in the rebuttal into the camera-ready version if the paper are accepted.", "rebuttal": "We sincerely thank the reviewer for the time and effort spent evaluating our work. We appreciate your recognition of the motivation behind our method and its effectiveness in improving perception and reducing hallucinations in multimodal large models. Below, we address your concerns in detail.\n\n> **Q1:** I find the paper's approach to be rather superficial. SFT is a highly nuanced process that demands careful attention to various aspects, including the proportion of different tasks and the quality of the response patterns (https://arxiv.org/pdf/2503.01307v1). By introducing a loss for instructions, the method effectively increases the SFT data by 50%, which is dedicated to learning how to instruct. Such a significant alteration in the SFT ratio is bound to have adverse effects, such as a potential decline in text comprehension abilities. This could be observed in evaluations like mtbench (https://arxiv.org/abs/2306.05685), wildbench (https://arxiv.org/abs/2406.04770), and fundamental text capabilities such as BBH, MMLU, and AGIEval. A decrease in language proficiency would inevitably lead to a systemic deterioration in response quality.\n\n**A1:** We thank the reviewer for this insightful comment regarding the potential impact of our method on the model's core language abilities.\n\nWe respectfully argue that for an advanced Multimodal LLM, the ability to generate a relevant instruction from an image is a hallmark of deeper comprehension, not a task that should harm its language skills. We view \"learning to instruct\" as a synergistic form of regularization. It compels the model to more deeply process visual information to understand the context and task, thereby reducing its reliance on \"shortcut learning\" where it might ignore the image and guess based on language priors alone. This should lead to more robust and grounded reasoning, not weaker language abilities.\n\nTo empirically address your concern, we have conducted new experiments on the text-only benchmarks you suggested. The results for the LLaVA-1.5 Vicuna-7B model are as follows:\n\n\n\n| Benchmark                 | MTbench | WildBench | MMLU  | AGIEVAL |\n|---------------------------|---------|-----------|-------|---------|\n| ViT LLaVA1.5-7B           | 5.35    | -9.40     | 49.57 | 32.62   |\n| LiT LLaVA1.5-7B           | 5.49    | -6.27     | 49.18 | 32.49   |\n\n\nAs the results demonstrate, our LIT-trained model achieves comparable performance to the ViT baseline on MT-Bench, MMLU, and AGIEVAL, while showing a significant improvement on WildBench. This demonstrates that our method does not cause the hypothesized \"decline in text comprehension abilities.\"\n\nTherefore, we believe LIT acts as an effective regularizer that enhances visual grounding and mitigates hallucination, without making a trade-off in the model's fundamental language proficiency.\n\n\n\n\n> **Q2:** The baseline used in the article is relatively weak. I suggest testing the method on a more robust VLM baseline to determine if it still yields benefits.\n\n**A2:** We thank the reviewer for their constructive suggestions. To address this, we evaluate our LiT on more robust VLM, Prism [1], which outperforms LLaVA-1.5. Prism incorporates several advanced techniques, including optimized training, image preprocessing, and fused visual backbones like SigLIP and DiNOv2, making it a more powerful baseline. \nWe conducted experiments based on Prism-DINOSigLIP-Controlled-7B, applying LiT during training and comparing its performance to the standard VIT-style baseline across several benchmarks, including VQA (GQA, VizWiz, TextVQA), Localization (RefCOCO, RefCOCO+, RefCOCOg), and challenge tasks (POPE, VSR, AI2D).\n\n\n| Task         | GQA   | VizWiz | TextVQA | RefCOCO | RefCOCO+ | RefCOCOg | POPE  | VSR   | AI2D  |\n|--------------|-------|--------|---------|---------|----------|----------|-------|-------|-------|\n| ViT Prism-7B | 61.92 | 55.36  | 52.80   | 56.70   | 50.70    | 52.70    | 88.00 | 53.20 | 55.50 |\n| LiT Prism-7B | 62.62 | 57.75  | 55.60   | 66.00   | 58.90    | 62.00    | 88.50 | 61.70 | 57.10 |\n\nThe results show that LiT consistently improves performance across diverse tasks. These findings highlight that LiT leads to consistent, significant improvements even with a stronger baseline like Prism. This suggests that our method has broad applicability and can enhance the performance of advanced VLMs, underscoring its effectiveness in improving visual instruction tuning. We will include the experiment and discussion in our revised manuscript.\n\n\n\n> **Q3:** I kindly suggest that the authors consider adopting a broader perspective, such as examining the closed-loop interactions between SFT, rule-based RL, and RLHF, to effectively tackle challenges like hallucination and shortcut learning in VLMs. Furthermore, it would be beneficial to stay abreast of recent advancements in post-training methodologies within the language model field, with a particular emphasis on addressing key and fundamental issues.\n\n**A3:** We sincerely thank the reviewer for their insightful and forward-looking suggestions.\n\nWe agree that the interplay between SFT, RL, and RLHF is a crucial area that warrants more attention. Our work is grounded in the belief that the quality of the initial SFT stage directly influences the effectiveness of subsequent alignment [2]. Recent studies [3,4] suggest that applying RL to models already prone to hallucination or shortcut learning can reinforce these issues, resulting in confident but incorrect outputs.\n\nLIT addresses this problem at its root. By reducing shortcut learning and enhancing visual grounding during SFT, it produces a more reliable base model for RLHF. A stronger SFT foundation enables safer and more effective alignment.\n\nWhile recent advances have focused on RL [5] and reward modeling [6], we believe the foundational role of SFT remains important. Early-stage flaws can persist and even intensify during RL. LIT serves as a preventive measure, aiming to improve model behavior before alignment rather than correcting it afterward.\n\nWe will incorporate these points into the revised manuscript to better emphasize the fundamental importance of SFT and the broader impact of our approach.\n\n[1] Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models. ICML 2024.\n\n[2]  Training language models to follow instructions with human feedback. NeurIPS 2022.\n\n[3] Learning Auxiliary Tasks Improves Reference-Free Hallucination Detection in Open-Domain Long-Form Generation. ACL 2025.\n\n[4] Exploring and Mitigating Shortcut Learning for Generative Large Language Models. 2024.\n\n[5] Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Models. 2025.\n\n[6] R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning. 2025."}, {"review_id": "3I5eEC2b4D", "reviewer": "Reviewer_S2mP", "summary": "* The paper focuses on a well-known limitation of Visual Instruction Tuning — that it can lead to “shortcut learning,” where models come to rely on language priors instead of the visual input.  \n* To address this problem, it proposes Learning to Instruct (LIT), a simple but effective modification to the instruction tuning training objective.  \n  * → Instead of only calculating loss on the model's *response*, LIT also includes the loss from predicting the *instruction* itself, conditioned on the image.  \n  * This acts as a regularizer, forcing the model to ground its understanding more deeply in the visual content.  \n* The method requires no extra data or significant compute overhead and demonstrates consistent performance gains across several models and 16 benchmarks, with particularly strong improvements in OCR & Chart and Image Captioning.", "questions": "* **Controlling \\# tokens in the loss calculation**: The proposed LIT method computes the loss over more tokens (instruction \\+ response) than the VIT baseline (response only). This raises a question about a potential confounding variable: *could the observed benefits be partially attributed to simply having more tokens contributing to the gradient per sample, rather than the specific regularizing effect of predicting the instruction's content?* Have you considered or can you suggest an experiment to control for this and help disentangle these two potential effects?  \n  * Perhaps this can be accomplished by using the same number of tokens from the system prompt, which should be uninformative.  \n* **Method Name:** The acronym \"LiT\" is already widely recognized in the community for \"Locked-image Tuning\" ([https://arxiv.org/abs/2111.07991](https://arxiv.org/abs/2111.07991)). I am concerned that the current “LIT” name will get overshadowed by the well-established LiT method. Maybe it is worth considering an alternative name?  \n* **Visualizations & Clarity:** please see questions in the Weaknesses section.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n* The paper addresses the important problem of shortcut learning with a clever, simple, and highly practical method. Including the instruction in the loss is a relatively simple modification to Visual Instruction Tuning that could be easily adopted by the community with negligible overhead.  \n* The LIT method shows consistent performance improvements over strong VIT baselines across multiple model scales.  \n  * Gains are particularly impressive on vision-centric tasks like OCR, chart understanding, and especially image captioning, where it improves CIDEr by nearly 18% on average.  \n* **Novel Analysis of Shortcut Learning:** The \"Visual Contribution\" metric introduced is intuitive and simple to compute, and provides a rigorous way to quantify the model's reliance on visual information. This experiment strengthens the paper's core claim that LIT improves visual grounding.  \n  * → I am unaware of previous similar analyses. Most papers rely on the *overall* accuracy difference between “blind” and normal (vision-enabled) models on a dataset, which gives a single dataset-level number. The Visual Contribution enables a more precise per-example measure.  \n* Presentation of the method is clear, Figs. 3 & 4 clearly convey the method.\n\n**Weaknesses:**\n\n* **Poor Data Visualization:** The paper's clarity is significantly hampered by poor choices in data visualization, which make the results difficult to interpret.  \n  * The primary results in Figs. 1 & 9 are presented as radar charts, which are known to be problematic for making clear comparisons between series (see [this](https://www.darkhorseanalytics.com/blog/radar-more-evil-than-pie) and [this](https://www.data-to-viz.com/caveat/spider.html)).  \n    * Fig. 9 is especially bad — it is impossible to discern any clear takeaway from the chart.  \n    * Redoing these visualizations in another format such as a line or bar chart could really help readers glean the message more readily.  \n  * Fig. 10: the ylims on this chart make no sense. The average is reported to 3 decimals in the paper, but the chart ylims range is 400x larger. The differences between the series is almost undistinguishable from the chart. Maybe consider ylims such as (0.3, 0.35) to zoom in on the relevant portion?  \n* **Minor Clarity Issues:**  \n  * Some sentences in the paper are difficult to parse and could be improved with grammar checking and rephrasing for better readability (e.g., L52-54).  \n  * Fig. 11: what specific models generated these? Should specify in the caption?", "was_revised": true, "final_justification": "all concerns addressed. this is a strong paper worthy of acceptance.", "rebuttal": "We sincerely thank the reviewer for the encouraging and constructive feedback. We appreciate your recognition of our method’s simplicity, effectiveness, and novel analysis. Below, we address your suggestions in detail.\n\n> **Q1:** Poor Data Visualization: The paper's clarity is significantly hampered by poor choices in data visualization, which make the results difficult to interpret.\n> The primary results in Figs. 1 & 9 are presented as radar charts, which are known to be problematic for making clear comparisons between series (see this and this).\n> (1) Fig. 9 is especially bad — it is impossible to discern any clear takeaway from the chart.\n> (2) Redoing these visualizations in another format such as a line or bar chart could really help readers glean the message more readily.\n> Fig. 10: the ylims on this chart make no sense. The average is reported to 3 decimals in the paper, but the chart ylims range is 400x larger. The differences between the series is almost undistinguishable from the chart. Maybe consider ylims such as (0.3, 0.35) to zoom in on the relevant portion?\n\n**A1:** We appreciate the reviewer’s detailed suggestions. We will (i) replace the radar charts in Figs. 1 and 9 with bar or line charts to improve comparison and clarity, (ii) adjust Fig. 10’s y‑axis limits and ticks to make differences discernible in the revision.\n\n> **Q2:** Minor Clarity Issues:\n> (1) Some sentences in the paper are difficult to parse and could be improved with grammar checking and rephrasing for better readability (e.g., L52-54).\n> (2) Fig. 11: what specific models generated these? Should specify in the caption?\n\n**A2:** We thank the reviewer for their constructive suggestions. To address them, we provide clarifications below and will revise the manuscript accordingly.\n\n- For the sentence in Lines 52–54, our intended meaning is as follows: In addition to learning to generate responses from images and instructions, LIT is also trained to generate instructions for given images. During instruction generation, LiT is supervised to avoid templates, special tokens, and high-frequency, low-information phrases commonly found in instructions.\n- For Fig. 11, the examples were generated by models based on LLaVA‑1.5 Vicuna‑7B. \n\nWe will carefully proofread the entire paper for clarity (with particular attention to long sentences) and will include the model specification in the Fig. 11 caption.\n\n\n\n\n> **Q3:** Controlling # tokens in the loss calculation: The proposed LIT method computes the loss over more tokens (instruction + response) than the VIT baseline (response only). This raises a question about a potential confounding variable: could the observed benefits be partially attributed to simply having more tokens contributing to the gradient per sample, rather than the specific regularizing effect of predicting the instruction's content? Have you considered or can you suggest an experiment to control for this and help disentangle these two potential effects?\n> Perhaps this can be accomplished by using the same number of tokens from the system prompt, which should be uninformative.\n\n**A3:** We appreciate the reviewer’s insightful questions. Follow your valuable suggestion, we conducted a equal-token comparison that fixes the aggregate number of tokens contributing to the loss and varies only instruction informativeness. From LLaVA‑mix‑665k, we selected a 78k subset and compared model trained on: (i) an uninformative condition using fixed system and task templates $(\\mathbf{X}_I^S,\\mathbf{X}_I^T)$ paired with the answer $\\mathbf{X}_A$; and (ii) an informative condition using the instruction with both templates removed $\\mathbf{X}_I^{\\backslash S,T}$, also paired with $\\mathbf{X}_A$. We matched the dataset‑level instruction token counts across the two conditions, yielding comparable averages and ensuring that differences reflect instruction content rather than token count. (Response tokens $\\mathbf{X}_A$ are the same across conditions, so the total token count is also matched.)\n\n| Condition (same 78k subset)     | Learning Content Used for Loss                                                                          | Avg. Tokens Contributing to Loss (Reported w/o $\\mathbf{X}_A$)  |\n|-------------|--------------------------------------------------------------------------------------------|----------------------|\n| Uninformative  | system template $\\mathbf{X}_I^S$, task template $\\mathbf{X}_I^T$ and response $\\mathbf{X}_A$       | 73.274               |\n| Informative (LiT)            | instruction excluding both the system and task templates $\\mathbf{X}_I^{\\backslash S,T}$ and response $\\mathbf{X}_A$ | 73.273               |\n\nWe then trained two model variants that differ only in the instruction‑side supervision used in the loss (informative vs. uninformative condition). The model trained on uninformative tokens failed to produce meaningful outputs in evaluation, instead replicating the template and yielding near‑zero scores on VQA, OCR, and image captioning benchmarks. These results indicate that LIT’s gains do not come from using more tokens in the loss but from informative instruction supervision, which regularize shortcut learning and places greater emphasis on the visual signal. We will incorporate the experiments into our revised manuscript.\n\n> **Q4:** Method Name: The acronym \"LiT\" is already widely recognized in the community for \"Locked-image Tuning\" (https://arxiv.org/abs/2111.07991). I am concerned that the current “LIT” name will get overshadowed by the well-established LiT method. Maybe it is worth considering an alternative name?\n\n**A4:** We appreciate the naming concern. To avoid confusion with “LiT” (Locked‑image Tuning), we will reconsider the acronym and are inclined to adopt Learning to Instruct (L2T)."}, {"review_id": "9UFzStBQ3B", "reviewer": "Reviewer_Qwwm", "summary": "This paper introduces a simple-yet-effective instruction tuning method named LIT: by adding auxiliary training objectives during the SFT (Supervised Fine-Tuning) process, the model learns how to ask questions during training, rather than just learning to answer them. Ultimately, results show that this method can improve the model performance based on llava-series model.", "questions": "Please refer to the Strengths and Weaknesses.", "limitations": "Please refer to the Strengths and Weaknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The entire article is coherent and clear, with the method being concise and effective. The following are some issues that require further clarification:\n\n1. This seems to be a general method. Apart from the LLava series, have you considered validating its effectiveness on other models?\n\n2. Is there any source or basis for the design of this LIT approach?\n\n3. This method appears somewhat similar to multimodal pretraining. What are the main design between the two?", "was_revised": true, "final_justification": "Thank the author for the response. I think the response to Q1 could better help the author understand this work. Overall, I tend to maintain the score (rating 4), and I'm happy to see this submission appearing in the conference for the community.", "rebuttal": "We thank the reviewer for the positive and clear feedback. We appreciate your recognition of the method’s simplicity and effectiveness. Below, we address your questions regarding model generalization, the design motivation behind LIT, and its relation to multimodal pretraining.\n\n> **Q1:** This seems to be a general method. Apart from the LLava series, have you considered validating its effectiveness on other models?\n\n**A1:** We sincerely thank the reviewer for recognizing the general applicability of our method. Following your valuable suggestion, we have conducted additional experiments on Prism-7B [1], another powerful MLLM in addition to the LLaVA series. We conducted experiments based on Prism-DINOSigLIP-Controlled-7B, applying LiT during training and comparing its performance to the standard VIT-style baseline across several benchmarks, including VQA (GQA, VizWiz, TextVQA), Localization (RefCOCO, RefCOCO+, RefCOCOg), and challenge tasks (POPE, VSR, AI2D).\n\n\n| Task         | GQA   | VizWiz | TextVQA | RefCOCO | RefCOCO+ | RefCOCOg | POPE  | VSR   | AI2D  |\n|--------------|-------|--------|---------|---------|----------|----------|-------|-------|-------|\n| ViT Prism-7B | 61.92 | 55.36  | 52.80   | 56.70   | 50.70    | 52.70    | 88.00 | 53.20 | 55.50 |\n| LiT Prism-7B | 62.62 | 57.75  | 55.60   | 66.00   | 58.90    | 62.00    | 88.50 | 61.70 | 57.10 |\n\nAs the results demonstrate, LIT brings consistent and significant performance gains to Prism across a wide range of capabilities. This further substantiates our claim that LIT is a general and widely applicable method for improving visual instruction tuning, with benefits that are not confined to a specific model family. We will include the experiment and discussion in our revised manuscript.\n\n> **Q2:** Is there any source or basis for the design of this LIT approach?\n\n**A2:** Thank you for the question. The design of the LIT method is primarily motivated by direct observations of two prevalent issues in existing models: overfitting and shortcut learning.\n\nConceptually, LIT draws some inspiration from two well-established ideas in the machine learning literature:\n\n- Auxiliary Tasks in Self-Supervised Learning [2,3]: We treat \"instruction generation\" as an auxiliary task alongside the primary task of \"answer generation\". This auxiliary objective compels the model to develop a deeper understanding of the image content, acting as an effective regularizer that enhances the model’s generalization ability.\n- Reward Hacking in Reinforcement Learning [4,5]: During training, models often exploit language priors to \"guess\" answers while ignoring the visual input, which is an issue closely resembling the \"reward hacking\" phenomenon in reinforcement learning. LIT addresses this by restructuring the learning objective, requiring the model to generate image-grounded instructions. This effectively blocks the shortcut and ensures the model acquires genuine cross-modal understanding.\n\nTherefore, LIT offers a principled and practical solution that mitigates shortcut learning and overfitting, ultimately leading to more robust and faithful vision-language reasoning. We will incorporate the discussion into our revised manuscript.\n\n> **Q3:** This method appears somewhat similar to multimodal pretraining. What are the main design between the two?\n\n**A3:** Thank you for the excellent question, which allows us to clarify this important distinction. While both processes involve learning from image-text data, our Learning to Instruct (LIT) method is fundamentally different from multimodal pre-training in its stage, objective, and methodology.\n\nIn essence, pre-training teaches the model a general association between images and text, while LIT is a specific regularization technique applied during the fine-tuning stage to improve the model's ability to follow instructions and reason about visual content.\n\n\n\n| Aspect         | Multimodal Pre-training                                                                                                        | Learning to Instruct (LIT)                                                                                                                                           |\n| -------------- | ------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| Training Stage | The initial phase designed to align general visual features with the language model's embedding space.                         | A technique applied exclusively during the subsequent fine-tuning stage, where the model learns to follow specific instructions.                                     |\n| Objective      | General vision-language feature alignment. The goal is to train a connector that maps visual features into the language space. | Regularization to improve instruction following and reduce shortcut learning. It forces the model to focus on visual content rather than relying on language priors. |\n| Training Data  | Typically uses large-scale, descriptive image-caption pairs.                                                                   | Operates on curated instruction-response datasets (e.g., VQA, OCR, reasoning data).                                                                                   |\n|      Methodology          |       Training only a cross-modal connector while keeping the vision and language models frozen.                                     |                                                                             Extends the standard fine-tuning loss to include the task of generating the instruction itself, conditioned on the image.                                                                                          |\n\nIn summary, LIT is not a pre-training method but a novel fine-tuning technique designed to regularize the model, enhance its visual grounding, and mitigate issues like hallucination that arise during the instruction-tuning process. We will incorporate the clarifications and discussion into our revised manuscript.\n\n\n[1] Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models. ICML 2024.\n\n[2] Survey on self-supervised learning: Auxiliary pretext tasks and contrastive learning methods in imaging. Entropy 2022.\n\n[3] Self-Supervised Generalisation with Meta Auxiliary Learning. NeurIPS 2019.\n\n[4] Concrete Problems in AI Safety. 2016.\n\n[5] Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective. Synthese 2021."}, {"review_id": "LCsvwiGXRC", "reviewer": "Reviewer_xAmq", "summary": "The paper proposes a simple yet effective method to improve visual instruction tuning by incorporating the loss function into both the instruction and response sequences. This approach addresses the overfitting and shortcut learning issues commonly found in Visual Instruction Tuning (VIT). The authors conduct extensive experiments across 16 different multimodal benchmarks. The results demonstrate that LIT outperforms VIT in most cases, with an average relative improvement of up to 8.5%. This comprehensive evaluation provides strong evidence of the effectiveness of LIT.", "questions": "Please refer to [Weaknesses].", "limitations": "Please refer to [Weaknesses].", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "[Strengths]\n* The shortcut learning issues are important in visual instruction tuning. This paper points out that masking the instruction in visual instruction tuning is not correct. This solution is interesting and has been ignored by the community.\n* The proposed method is simple and effective.\n\n[Weaknesses]\n* The process of removing task templates in LIT is somewhat ad-hoc. The authors identify task templates by calculating the frequency of sentences in the training dataset and selecting the most frequent ones. However, this approach may not be robust across different datasets or tasks. A more systematic and adaptive method for template removal could enhance the generalizability of LIT.\n* This solution is effective in VQA. As discussed in SP, for the task with a fixed template, this method does not work. Can we design some methods to select which task should not mask the instruction during loss calculation?\n* After the training, the model can generate the instructions for a given image. It is interesting to make some generated instructions. Will these instructions be used as training data to further train the model?", "was_revised": true, "final_justification": "It is an interesting paper. My corcerns are well-sovled.", "rebuttal": "We thank the reviewer for the thoughtful comments and for recognizing the importance of addressing shortcut learning in visual instruction tuning. We appreciate your positive feedback on the simplicity and effectiveness of our method. Below, we respond to your suggestions in detail.\n\n> **Q1:** The process of removing task templates in LIT is somewhat ad-hoc. The authors identify task templates by calculating the frequency of sentences in the training dataset and selecting the most frequent ones. However, this approach may not be robust across different datasets or tasks. A more systematic and adaptive method for template removal could enhance the generalizability of LIT.\n\n**A1:** We appreciate the reviewer’s insightful questions. Following your valuable suggestion, we conducted an adaptive method using visual contribution (VC) at a more fine-grained token level, as detailed in Section 2.  Specifically, we computed the VC for each token in the instruction part, defined as:\n\n$\\mathrm{VC}\\^\\mathrm{I}\\_i = \\log p\\_{\\theta} (\\mathbf{X}\\_{I,i}|\\mathbf{X}\\_V,\\mathbf{X}\\_{I,<i}) - \\log p\\_{\\theta} (\\mathbf{X}\\_{I,i}|\\mathbf{X}\\_V=\\emptyset,\\mathbf{X}\\_{I,<i}).$\n\nThis VC score allows us to identify the least important tokens with the lowest visual contribution across the dataset. By setting a threshold of 20%, we filtered out the most irrelevant tokens, intuitively removing task templates that likely conveyed minimal information related to the images. For experimental details, we first pretrained a TinyLLaVA Qwen 0.5B model to estimate token-level VC. The resulting VC scores were then used to train a new model from scratch, during which meaningless tokens were removed based on their low VC values. \n\n| Benchmark                     | VQAv2-L | VizWiz-L | MMMU  | MMSTAR | ChartQA | OCR Bench | Flickr30k-L | RefCOCO-L |\n|------------------------------|------------|-------------|-------|--------|---------|------------|----------------|---------------|\n| ViT TinyLLaVA-0.5B           | 64.02      | 29.90       | 29.67 | 34.93  | 13.28   | 26.50      | 70.28          | 14.81         |\n| LiT TinyLLaVA-0.5B w/ VC removal | 64.46  | 30.42       | 31.00 | 36.42  | 13.88   | 27.10      | 74.15          | 20.42         |\n| LiT TinyLLaVA-0.5B           | 66.34      | 29.74       | 32.33 | 36.14  | 13.80   | 27.20      | 74.30          | 23.11         |\n\n\nThe results revealed that the VC-based removal did not consistently outperform our template removal method. Additionally, it introduces significant computational overhead, as it requires pretraining and inference on the entire dataset. In comparison, our proposed template removal remains more efficient and effective. However, we agree that the frequency-based removal approach may lack robustness across different datasets or tasks and will consider more adaptive and efficient template removal strategies in future work. We will incorporate this point to the conclusion of the revised manuscript.\n\n\n> **Q2:** This solution is effective in VQA. As discussed in SP, for the task with a fixed template, this method does not work. Can we design some methods to select which task should not mask the instruction during loss calculation?\n\n**A2:** We sincerely appreciate the reviewer for raising this thoughtful question. We would like to clarify as follows:\n- While the performance gains varies across tasks, we would like to clarify that the application of LIT does not lead to performance degradation or incur additional computational overhead. Even for tasks with modest gains, LIT maintains performance comparable to the baseline. Therefore, selectively applying LIT may not be necessary. A unified application ensures simplicity, generality, and no additional computational cost, without compromising the model’s overall performance.\n- The tasks where our method shows relatively limited gain tend to have a key characteristic: their instructions are mostly composed of fixed, uninformative templates. These templates are effectively removed through the template removal process in LiT. For example, in grounding tasks, an instruction such as \"Please provide the bounding box coordinate of the region this sentence describes: a red jeep\" is reduced to just \"a red jeep.\" Therefore, there is no need to design an additional mechanism to selectively avoid learning from such tasks because the template removal process already handles this. In fact, the limited improvement of LiT on specific tasks is due to the removal of numerous templates in the instructions. The underlying mechanism is that the remaining few tokens convey insufficient information, making it challenging to learn effective instructions.\n\nWe will incorporate the clarifications and discussion into our revised manuscript.\n\n> **Q3:** After the training, the model can generate the instructions for a given image. It is interesting to make some generated instructions. Will these instructions be used as training data to further train the model?\n\n**A3:** Thank you for the excellent question. It indeed opens up a promising direction for extending the capabilities of our LiT method. In response to your suggestion, we conducted an experiment to evaluate the feasibility of using model-generated instructions as additional training data for further improving the model. Specifically, we first pretrained the LiT model on a randomly selected 100k subset of the LLaVA‑mix‑665k dataset. Then, we used image-only inputs to guide the model to generate 100k instruction-response pairs. These synthesized samples were then integrated with the original 100k training samples for further training.\n\n| Benchmark                        | VQAv2-L | VizWiz-L | MMMU  | MMSTAR | ChartQA | OCR Bench | Flickr30k-L | RefCOCO-L |\n|----------------------------------|---------|----------|-------|--------|---------|------------|--------------|------------|\n| LiT TinyLLaVA-0.5B               | 56.68   | 23.20    | 31.78 | 33.46  | 11.96   | 23.40      | 68.55        | 17.42      |\n| LiT TinyLLaVA-0.5B w/ generated data | 60.80   | 24.22    | 32.33 | 34.75  | 12.44   | 24.50      | 68.32        | 24.53      |\n\n \nThe experimental results demonstrate that incorporating the additional 100k model-generated instruction-response pairs into further training leads to noticeable performance gains. This finding highlights the potential of LiT to improve through self-generated supervision, suggesting a promising direction toward developing self-evolving vision-language models via a closed-loop instruction–response mechanism.\n\nWe will include this experiment in the revised manuscript and explore this further in a systematic way in future work. We thank the reviewer again for the constructive idea."}]

Author rebuttal:

Meta-review: The paper introduces Learning to Instruct (LIT), a simple yet effective modification to the visual instruction tuning objective for Multimodal Large Language Models (MLLMs). Instead of only computing the loss on the model's response, the proposed method also includes the instruction sequence in the loss calculation. This approach is designed to act as a regularizer, forcing the model to better ground its understanding in the visual input, thereby mitigating common issues like shortcut learning and hallucination. Reviewers were generally positive about the paper's clear motivation and the method's simplicity and novelty, though they initially raised significant questions regarding the method's potential side effects, the strength of the baselines, and the underlying cause of the performance gains. The authors' thorough and constructive rebuttal, which included substantial new experiments, successfully addressed nearly all of these concerns, significantly strengthening the paper and garnering strong support from the reviewers.

# Summary Of Reasons To Publish:
1) The core idea is recognized by reviewers as clever, practical, and easy to implement, addressing the important problem of shortcut learning in MLLMs.
2) The authors provided extensive new experiments during the rebuttal phase that confirmed the method's effectiveness. This includes: 2.1) Demonstrating that the method does not degrade performance on language-only benchmarks (e.g., MMLU, MT-Bench), directly addressing a key concern from Reviewer gAGN. 2.2) Validating the approach on a stronger, more recent baseline (Prism-7B), as requested by Reviewers gAGN and Qwwm, showing consistent performance improvements.
3) The author-reviewer discussion yielded significant new insights that strengthen the paper's contribution. Specifically: 3.1) A new ablation study, suggested by Reviewer S2mP, confirmed that performance gains stem from the informative content of the instruction, not merely an increased token count in the loss calculation. 3.2) An experiment inspired by Reviewer xAmq demonstrated a promising self-improvement cycle, where model-generated instructions could be used to further boost performance.

# Summary Of Suggested Revisions:
To ensure the final version reflects the significant improvements made during the review process, the authors are expected to incorporate the following points discussed and agreed upon in their rebuttal:
1) The final paper must include the comprehensive results from the new experiments conducted for the rebuttal, including the language-only benchmark evaluations, the stronger Prism-7B baseline results, the equal-token control experiment, and the self-improvement cycle experiment.
2) As agreed with Reviewer S2mP, the problematic radar charts should be replaced with clearer formats like bar or line charts to improve interpretability.
3) To avoid confusion with a prior method, the acronym should be changed from "LIT" to "L2T" (Learning to Instruct), as suggested by Reviewer S2mP and accepted by the authors.
4) The manuscript should be updated to include the clarifications provided to Reviewer Qwwm, particularly the detailed distinction between this fine-tuning technique and standard multimodal pre-training.
5) The discussion should incorporate the acknowledged limitations regarding the ad-hoc nature of template removal, as raised by Reviewer xAmq, and frame it as a direction for future work.
Decision: Accept (poster)