Be Confident: Uncovering Overfitting in MLLM Multi-Task Tuning
Authors: Wenke Huang
Jian Liang
Guancheng Wan
Didi Zhu
He Li
Jiawei Shao
Mang Ye
Bo Du
Dacheng Tao
Venue: ICML 2025 poster (2025)
Keywords: Multimodal Large Language Models+Fine-Tuning+Open-Response Overfitting
Primary area: general_machine_learning->transfer_multitask_and_metalearning

Abstract: Fine-tuning Multimodal Large Language Models (MLLMs) in multi-task learning scenarios has emerged as an effective strategy for achieving cross-domain specialization. However, multi-task fine-tuning frequently induces performance degradation on open-response datasets. We posit that free-form answer generation primarily depends on language priors, and strengthening the integration of visual behavioral cues is critical for enhancing prediction robustness. In this work, we propose Noise Resilient Confidence Alignment to address the challenge of open-response overfitting during multi-task fine-tuning. Our approach prioritizes maintaining consistent prediction patterns in MLLMs across varying visual input qualities. To achieve this, we employ Gaussian perturbations to synthesize distorted visual inputs and enforce token prediction confidence alignment towards the normal visual branch. By explicitly linking confidence calibration to visual robustness, this method reduces over-reliance on language priors. We conduct extensive empirical evaluations across diverse multi-task downstream settings via popular MLLM architectures. The comprehensive experiment demonstrates the effectiveness of our method, showcasing its ability to alleviate open-response overfitting while maintaining satisfying multi-task fine-tuning performance.

Be Confident:
Uncovering Overfitting in MLLM Multi-Task Tuning
Wenke Huang" 11
Jian Liang"
Guancheng Wan' Didi Zhu 2
He Li
Jiawei Shao
Mang Ye
Bo Du
Dacheng Tao
Abstract
Connector
Large Language Model
Fine-tuning Multimodal Large Language Mod- els (MILMs) in multi-task learning scenarios has emerged an effective strategy for achieving cross-domain specialization. However, multi-tast fine-tuning appears performance degradation on open-response datasets. We posit that free-form answer generation primarily depends on language priors, and strengthening the integration of visual behavioral cues critical for enhancing predic- tion robustness. In this work, we propose Noise Resilient Confidence Alignment to address the open-response overfitting challenge during multi- task fine-tuning. Our approach prioritizes main- taining consistent prediction patterns MILMs across varying visual qualities. To achieve this, we synthesize distorted visual inputs and enforce token prediction confidence alignment towards normal visual branch. By explicitly linking confi- dence calibration to visual robustness, this method reduces over-reliance on language priors. We con- duct extensive empirical evaluations across di- verse multi-task downstream via popular MILM architectures. The comprehensive experiment demonstrates our effectiveness, showcasing its ability to alleviate open-response overfitting while maintaining satisfying multi-task performance.
Visual Encoder
Bpen-Response
Briefly desorthe the content of the image.
TV
Three men in ... catch rugby ball.
Fixed-Choics
2
How many people are there? A. 1 C33
73
Drop!
99
70.33
92
88.60
67
64.67
85
78
61
71
55.45
55
64
Zero-Shot
Fine-Tunfnig Epoch
Figure 1: Background and Problem. Fine-tuning Multimodal Large Language Model (MLLM) Multi-Task scenarios enables model to acquire diverse specialization capabilities. This pro- cess involves training (#$) both the connector and LLM modules for E epochs. We notice that during tuning process. -pen-Response task reveals notable performance degradation. while Fixed- Choice task demonstrates consistent stability. Experiments conducted on Flickr30k and ScienceQA with VILA architecture.
1. Introduction
Driven by the remarkable success of Large Language Model (LLM) in the natural language processing and understand-
ing (Devlin et al., 2018; Radford et al., 2019; Brown et al., 2020; Chowdhery et al., 2022; OpenAI, 2023), Many efforts have been made to extend LLM to Multimodal Large Lan- guage Model (MLLM)), which have demonstrated remark- able capabilities in generating coherent and contextually relevant descriptions from visual inputs (Liu et al., 2023b; Dai al., 2023; Lin et al., 2023; Liu et al., 2023a). Specif- ically, MLLM generally follows the paradigm to fuse the pre-trained vision encoder (Radford et al., 2021: Dosovit- skiy et al., 2021) into the representation space of the Large Language Model, e.g., LLaMA (Touvron et al., 2023) and Vicuna (Chiang et al., 2023), via the connector module (Dai et al., 2023; Liu et al., 2023b; Luo et al., 2024). Consider-
"Equal contribution; Work done during internship TeleAI 'National Engincering Research Center for Multimedia Soft- ware, School of Computer Science, Wuhan University, Wuhan, China Department of Computer Science and Technology, Zhe- jiang University. Hangzhou, China IInstitute of Artificial Intel- ligence (TeleAl). China Telecom, China 'Nanyang Technologi- cal University, Singapore. Correspondence to: Mang Ye <ye- mang @ www..edu.cn..... Bo Du <dubo whu.edu.cn>>
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Be Confident: Uncovering Overfitting in MLLM Multi-Task Tuning
ing that Multimodal Large Language Model is optimized on millions of multimodality instluction-follovingg datasets (Lin et al., 2014; Singh et al., 2019; Mishra et al., 2019), it brings powerful generalization ability on different related tasks. Despite this, MILMM still performs poorly on different downstream tasks (Ren et al., 2023; Zhou et al., 2024).
augmentation strategies (e.g., color jitter, random cropping) (Tian al., 2020; Chen al., 2024a; Woo et al., 2024; Kim et al., 2024b) can drift from the original answer. To ad- dress this issue, we propose the Noisy Visual Mixup which utilizes the Gaussian noise to construct the distorted visual signal, which does not modify the semantic level knowledge. Second, the inherently variable prediction length of MLLM makes straightforward contrastive regularization unstable and potentially disruptive. To overcome this problem, we introduce the Token Confidence Alignment, which calcu- lates the overall prediction token confidence and expects to achieve the contidence-level alignments. The rationale behind this is that beyond local empirical minimization, we encourage the noisy visual input to bring coherent predic- tion confidence with the original ones to alleviate textual bias and encourage visual representation ability. For thor- ough examination, we conduct experiments multi- mulli-tasking scenarios. We mix up and fine-tune on two major down- stream tasks, i.e., open-response and fixed-choice. The main contributions are summarized as follows:
Towards adapting to the target domain, the straightforward solution is fine-tune on the task-specific dataset (Zhou et al., 2024; Han et al., 2024). With respect to existing ex- plorations, they normally focus on single-task adaption, it would be simple to construct the personalized fine-tuning paradigm. However, facing the multi-task tuning require- ment, we are curious what challenge would incur. To visu- alize our exploration, we conducted extensive experiments on two major downstream tasks: open-response and fixed- choice tasks. We randomly select datasets from these two views to combine the fine-tuning resource. As shown in Fig. 1, with longer training epochs, the open-response task appears the overfitting tendency (Guo et al., 2017; Lak- shminarayanan et al., 2017) and brings the performance degradation on the corresponding testing dataset. We fur- ther investigate why Multimodal Large Language Model (MLLM) overfitting excites. As noted in previous studies (Favero al., 2024; Wang al., 2024a; Li et al., 2023b), MLLM tends to rely on the language priors to generate de- scription sentences, consequently allocating less attention to visual information. Therefore, particularly in open-response tasks, MLLM tends to overfit to the current distribution, resulting in constrained specialization performance and an increased likelihood of fabricating content (Huang et al., 2023; Gekhman et al., 2024). Thus, a fundamental question naturally emerges: How to alleviate open-response overfil- ting during multi-task MLLM fine-tuning
We focus on the multi-task Multimodal Large Language Model fine-tuning scenario and reveal that open rpenores tasks, such as image captioning, exhibit performance degradation due to language prior overfitting.
We propose Noise Resilient Confidence Alignment (NRCA), method designed to mitigate the open-response overfitting phenomenon by enhancing visual cues dur- ing Multimodal Large Language Model tuning process. Generally speaking, we construct noisy visual inputs and encourage token confidence alignment with the normal branch reinforce visual representation.
We perform a comprehensive analysis on multi-task sce- narios, including both open-response datasets (Flickr30k (Young et al., 2014) and COCO-Cap (Lin et al., 2014)) and fixed-choice datasets (ScienceQA (Lu et al., 2022) and IconQA(Lu al., 2021)). Experiments are conducted on the VILA (Lin et al., 2023) and LLaVA (Liu et al., 2023b) architectures. Through series of ablation studies, the promising results empirically validate the effective- ness of NRCA in alleviating open-response overfitting and enhancing multi-task fine-tuning overall performance.
The naive solution is to establish asynchronous training epochs, i.e., shorter for caption and longer for VQA. How- ever, this operation would incur catastrophic forgetting on the caption task. In this work, we seek to mitigate overfitting in MLLM fine-tuning by examining the problem from in- ner visual behavior perspective. Specifically, we find that when confronted with low-quality images, MLLM increas- ingly relies on textual modality prior information rather than visual cues (Leng et al., 2024; Favero et al., 2024; Wang et al., 2024a; Li al., 2023b). This observation suggests that maximizing the mutual information between different visual input views during fine-tuning could help counterbal- ance the textual bias. Motivated by this insight, we propose Noise Resilient Confidence Alignment (NRCA), a method inspired by contrastive learning paradigms that encourage augmentation invariance and instance discrimination (Ye et al., 2019; 2020; He et al., 2020; Chen et al., 2020; Huang et al., 2021; Wu et al., 2022; Wang et al., 2024c). However, directly applying standard contrastive paradigm in MLLM for visual coherence two challenges. First, common data
2. Related Works
2.1. Multimodal Large Language Models
Motivated by the great progress of Large Language Model (LLM) (Radford al., 2019; Brown al., 2020; OpenAI,
2023; Touvron et al., 2023; Shen et al., 2024b; Jin et al., 2024; 2025), researchers have been actively exploring ways to combine pre-trained LLM and vision encoders to con- struct the end-to-end Multimodal Large Language Model (MLLM) capable of processing image-text. Existing so- lutions follow to feed the visual features produced by a pre-trained vision encoder into the LLM module with the
2

---

Be Confident: Uncovering Overfitting in MLLM Multi-Task Tuning
visual projector, e.g., Flamingo (Alayrac et al., 2022), BLIP (Li al., 2022a; 2023a), InstructBLIP (Dai et al., 2023), QWen-VL (Bai et al., 2023), LLaVA (Liu et al., 2023b;a; Zhu et al., 2024c; Li et al., 2024a; Luo et al., 2024), VILA (Lin et al., 2023; Fang et al., 2024), Cambrian-1 (Tong et al., 2024). Although MLLM brings the powerful generalization ability on a wide range of tasks (Marino et al., 2019; Lu et al., 2021; Lin et al., 2014; Kazemzadeh et al.. 2014) but appears limited performance on specific domains. Thus, fine-tuning MLLM on the target task brings reasonable solution for boosting downstream performance (Zhou et al., 2024).
Shen et al., 2024a; Guo et al., 2024) and Update Calibration (Panigrahi et al., 2023; Chen et al., 2024b). The former solution introduces additional external vision signals, egg,
high resolution. object grounding. However, these solutions increase the computational burden and limit architecture compatibility due to the added parameters in pre-trained models (Huo et al., 2024). tion methods derive two main research directions. O Par- tial Update Mask (Li et al., 2022b; Ansell et al., 2022; Li et al., 2023c; Yu et al., 2024; Zhang et al., 2024d;a; Zhu et al., 2024a; Lu et al., 2024b; Hui et al., 2024; Chen et al., 2024b) selects the candidate parameters group to alleviate the overfitting phenomenon. For example, Random Mask- Tuning randomly selects half of the learnable parameters and freezes the rest. @ Stiff Penalty Regularization (Kirk- patrick et al., 2017; Zenke et al., 2017: Xuhong et al., 2018; Ritter et al., 2018; Buzzega et al., 2020; Li et al., 2020; Panigrahi et al., 2023) measures the parameter importance degree to construct the parameter weight updates penalty, However, existing approaches primarily focus on single-task adaptation and indiscriminately apply anti-overfitting objec- tives across all training samples, failing to address the fitting condition inconsistencies in multi-task setting (Dong et al.. 2023) such as open-response and fixed-choice objectives. In this work, we focus on MILM multi-task fine-tuning and reveal that open-response tasks are particularly susceptible to overfitting due to excessive dependence on textual modal- ities. To mitigate this limitation, we propose leveraging in- trinsic visual behavior patterns to strengthen cross-modal representation robustness. To address this issue, we incor- porate. Specifically, we synthesize perturbed visual inputs through Gaussian noise injection and enforce confidence consistency between predictions from corrupted and pristine visual branches. This dual-branch alignment mechanism effectively reduces textual bias dominance while preserv- ing semantic fidelity, effectively counteracting overfitting tendencies during MLLM optimization.
contrast, Update Calibra-
2.2. Fine-Tune MLLM
Pre-trained MLLM on large-scale datasets can be easily transferred to downstream vision tasks through finettuning (Huang et al., 2025). Existing solutions can be primarily cat- egorized into two types. 1) Reparameterization Fine-Tuning (Hu et al., 2022; Zhang et al., 2023; Wang et al., 2023; Hao et al., 2024; Liu et al., 2024e; Lin et al., 2024; Hu et al., 2024; Bi al., 2025a;;; Liang et al., 2025) normally apply low-rank matrices to approximate weight changes during fine-tuning and can merge with pre-trained weights prior to inference. Despite certain advantages, it faces several limita- tions that hinder application. First, it normally achieves sub- optimal performance compared with direct weight updates (Biderman et al., 2024; Shuttleworth et al., 2024). Second, it introduces additional parameter module, increasing the complexity of achieving universal architectural com- patibility. II) Partial Fine-Tuning (Li et al., 2022b; Ansell et al., 2022; Li et al., 2023c; Yu et al., 2024; Zhang et al., 2024d;a; Zhu et al., 2024a; Lu et al., 2024b) focuses on directly optimizing selected candidate elements in MLLM, such as visual connectors and LLM blocks, to align with the requirements of downstream tasks. This paradigm enjoys the cethitecture-agnooiii property and thus largely bene- fits MLLM transferability. In our work, we adhere to the partial fine-tuning and further provide detailed a learnable parameter discussion in the following See. 4.1.
3. Methodology
3.1. Preliminary
2.3. Overfitting Behavior in MILM Fine-Tuning
Given the Multimodal Large Language Model (MLLM #) architecture, the MILM model typically includes three mod- ules: visual encoder f (Dosovitskiy et al., 2021), LLM g. and the connector module yp (Liu et al., 2023b; Dai et al., 2023: Liu et al., 2023a: Lin et al., 2023). For a query in- stance, the input consists of both visual image x" and textual instruction w'. Denote the vocabulary dictionary as V and language response = [yI.... .t..... E RT. Thus, Yt E V and t E TY. T denotes the to- ken length. To be precise, we extract the visual features m = f(x=",, and then apply the trainable projection yp to convert m² into language embedding tokens, h" = 6 mm.
For downstream fine-tuning, the neural networks are de- fault optimized based on the cross entropy (De Boer et al., 2005) which has been shown to overfit to the current distri- bution (Platt et al., 1999; Amodei et al., 2016; Guo al., 2017). In the context of MILMM, over-fitiing on new factual knowledge encourages MLLM prone to fabricating content, amplifying the risk of hallucinations (Huang et al., 2023; Gekhman et al., 2024). Substantial research efforts have been directed towards compensating vision shortcomings and could be mainly classified into two types: Vision En- hancement (Luo et al., 2024; Shang et al., 2024; Ghosh et al., 2024; Zhao et al., 2024; Li al., 2024a; Shi et al., 2024;
3

---

Be Confident: Uncovering Overfitting in MLLM Multi-Task Tuning
downstream performance. Inspired by the success of the contrastive paradigm (Wu al., 2018; Ye et al., 2019; 2020; Chen et al., 2020; Khosla et al., 2020; Chen et al., 2022), we aim to encourage the MLLM to generate consistent predic- five behavior for both distorted and normal visual inputs.
MULMM
Eq. (5b)
Eq 2bb
P2
Noisy Visual Mixup. Compared to popular contrastive augmentation such color jitter and random cropping (Xiao et al., 2021: Zhao al., 2021; Wang al., 2022: 2024c), would largely bring the hallucination prediction and fail to align with the original feedback. Thus, we model the inherent characteristics of the current image by introducing random Gaussian distribution based on its own feature space. This approach leverages the image unique feature distribution to generate stochastic variations, capturing both its intrinsic structure and potential perturbations.
o(u
P
Eq. (2a)
MILN
Eq.
Sa)
@ Briefly describe the image.
Lady waits at the train station.
# fu(""),
fo(""),
(2a)
3" = (N
5),0"
(2b)
Token Prob.
Token Label
Add
Token Conf. Gaussian Dist.
Noisy Part
Normal
IA and o² respectively denote the mean and variance of the distribution. N specifies the Gaussian distribution. The 8 denotes the noisy ratio and is default set as 0.5. Then, we feed both normal x" and noisy 5" visual information with the corresponding text prompt x² into the MLLM and respectively obtain the contextual word prediction logits as 2 = =(h", h²) and غ = g(h", h²)..
Figure 2: Schematization of NRCA. We propose Noisy Visual Mixup. which leverages Gaussian distribution to generate noisy visual inputs. Additionally, we introduce Token Confidence Align- ment, designed align the confidence of noisy tokens with the normal visual branch behavior, thereby enhancing visual represen- tations during multi-task fine-tuning. The process is illustrated via logits output with the token length = 3 and class number V =4 Best viewed in color. Zoom in for details. Refer to Sec. 3.2
Token Confidence Alignment. Compared with traditional classification tasks, the prediction = E T for MLLM appears high output dimensions attributed to vocabulary scale, e.g., c 32000 in LLaMA, As result, directly con- ducting distribution alignment faces challenges related to the imbalance between head-tail knowledge variations (Gu et al., 2024; 2025; Kim et al., 2024a; Ko et al., 2024), and fails to adequately reflect the reliability of the ground-truth signal. Therefore, in this work, we propose aligning token confidence rather than token distribution mitigate overfit- ting during the fine-tuning stage of MLLM. Specifically, we first measure the prediction probability for the token t via the following formulation:
And textual token as h² = Tokenize(z'). Next, we combine both visual and textual tokens and pass them into the LLM module g to generate the logits output = = g(h", (f),, In our work, following previous MLLM fine-tuning works and benchmarks (Zhou et al., 2024; Zhu et al., 2024a; Han et al., 2024) to select the trainable parameters. For the visual mod- ule, we freeze the vision encoder and tune visual connector module sp. For LLM aspect, LLM includes several trans- former blocks and selects the candidate block layers set / for optimization as [T). Thus, we obtain learnable modules as w = {4,, g []]] Recall the fine-tuning distribution D, the default MILM optimization procedure follows:
P. (zt), P. = ((t) E R
arg min [ex" at se (g(h", ,1)).)).
(1)
(3)
p = | p ==p=)== E TTC
The normally utilizes the Pooss-E term (De Boer al., 2005) for regular optimization target.
a means the so ftmaax function. Then, we further obtain the overall token prediction confidence based on the ground- truth labels y {yr)T= as:
3.2. Noise Resilient Confidence Alignment
Motivation. In general, features derived from the standard visual output layer are effective for reasoning across Multi- modal tasks. However, due to the language prior bias (Leng al., 2024; Favero et al., 2024; Liu et al., 2024a; Wang et al., 2024b; Zhang et al., 2024b), Multimodal Large Language Model (MLLM), particularly during the fine-tuning stage for caption generation, tend to rely on the language- intuition. Therefore, in our work, we focus on enhancing the visual representation to mitigate overfitting and improve
I 7 p i-
p
(4)
Naturally, we encourage the noisy prediction confidence ř to align with the normal ones I. Furthermore, applying the same penalty strength across all samples fails to account for sample-specific fitting conditions. To address this, we introduce the empirical loss as a guiding factor, allowing for tailored penalty allocation for each query sample. We introduce the following regularization term.
4

---

Be Confident: Uncovering Overfitting in MLLM Multi-Tass Task Tuning
Noise
Normal
1
LoR
_
1y. (pe(
(5a)
T
1=1
74
I
LNRCA=
x LCE
(5b)
11
57.50
I
where 1 denotes the one-hot encoding of yr.We We detach the normal branch signals in Eq. (5b) to avoid the MLLM optimization degradation. Empirically, we identify open- response samples based their label length, applying this operation separately to these samples. Specifically, we ob- serve that fixed-choice sample labels typically consist of a single word followed by the [EOS] ] token, with the label length as 2. Finally, we carry out the following optimization objective in fine-tuning phase:
54
Grodling
Hig
Oan
Figure 3: Noise Evaluation Comparison. We evaluate noisy and normal samples Flickr30k using LLaVA with E 3. Normal Full Fine-Tuning shows a limited performance gap between noisy and normal visual signals, revealing a tendency of language prior reliance. In contrast, our approach demonstrates a notable per- formance improvement by emphasizing the importance of visual information. See Sec. 3.3 for details.
= LCE + ALNRCA-
(6)
represents the penalization nyper-paramette that controls the strength of confidence alignment. We set vide the corresponding ablation analysis in Sec. 4.2. Over- all, the MLLM model is regularized to preserve the down- stream distribution while promoting invariance in distorted confidence. As result, our method effectively mitigates downstream open-response overfitting by enhancing visual behavior alignment. We provide a algorithm description in Algorithm and the methodological framework in Fig. 2.
Table I: Attention Allocation and Performance Comparison. We conduct the evaluation the Flickr30k with noisy visual input. We tuning on the Sciie30ee+SiieeOe. datasets based on LLaVA architecture. Please see Sec. 4.2 for detailed discussion.
and pro-
Metries
Full FT Ran Mask Mag Mask Grafting 12-Reg
Ours
System (All)
0.6837
0.6966
0.6927
0.6673
0.6791
0.6677
Prompt (All)
0.29
0.1297
0.1280
0.1241
011181
0.25
Visual (AlI)
0.1933
0.1737
0.1793
0,2084
0.2029
0.2098
System (Mid)
0.5865
0.5874
0,5859
0,5726
0,5826
0.5574
Prompt (Mid)
0.1733
0.1843
0.1832
0.1664
0.1623 0.1576
3.3. Discussion and Limitation
Visual (Mid)
0.2402
0.2282
0.2309
0,2609 0,2552
0.2850
Gaussian Noise Meets MILM. Existing MLLM research often employs Gaussian noise (Goodman, 1963; Tüske et al., 2015; Variani et al., 2015; Hayashi & Uchida, 2019; Ardiz- zone et al., 2020) to generate low-quality visual inputs, thereby amplifying language priors (Xiao et al., 2024). A common approach to mitigating MLLM hallucinations is to adjust next-token logits in a contrastive manner using the normal prediction distribution (Leng et al., 2024; Woo et al., 2024; Zhu et al., 2024b; Chen et al., 2024a; Huo et al., 2024; Zhang et al., 2024b; Xiao et al., 2024; Liu al., 2024b). The rationale behind this approach is that disturbed inputs significantly exacerbate hallucinations, while contrastive decoding mitigates this effect by subtracting hallucinated concepts from the original distribution, thereby reducing confusion. Consequently, existing methodologies often treat noisy visual behavior as an optimization trap and disregard the corresponding feedback. In contrast, our work intro- duces a novel MLLM multi-task tuning approach address open-response overfitting by encouraging prediction confi- dence alignment between normal and noisy visual signals. This strategy compels the network to fully leverage its vi- sual processing capabilities, achieving consistent prediction confidence even when faced with degraded visual inputs.
domain-specializee experts. However, the limited size of downstream datasets poses a significant challenge, often leading to overfitting on target-specific behaviors and catas- trophic forgetting of pre-trained knowledge. Existing stud- ies predominantly focus on mitigating catastrophic forget- ting in MLLM, where fine-tuning risks compromise the gen- erality achieved during pre-training, thus causing upstream performance degradation. Consequently, prior research has primarily aimed to balance generalization and specializa- tion, ensuring robust performance on both seen and unseen tasks. In contrast, our work addresses a more practical see- nario by fine-tuning MILM for multi-task applications. We argue that leveraging MLLM to achieve multi-task special- ization is more efficient approach than the conventional one-to-one fine-tuning paradigm. However, when dealing with multi-task distributions, we identify a critical issue: MLLM frequently experiences performance degrudation in open-response tasks, e.g., image caption understanding, as shown in Fig. 1. We hypothesize that this degradation arises because open-response tasks tend to rely excessively on language priors rather than leveraging visual information, as confirmed in Fig. 3. Full Fine-Tuning brings limited per- formance gap towards the input visual quality. Furthermore, as for Multimodal Large Language Model (MLLM), input tokens include System Tokens, Prompt Tokens and Visual
Conceptual Difference. Fine-tuning the MLLM for down- stream tasks is a straightforward approach to developing
5

---

Be Confident: Uncovering Overfitting in MLLM Multi-Tass Tuning
Table Incremental Resource Cost with Overall Performance Comparison. Accuracies are derived from Flickr30k & Sci- enceQA based on VILA for E' 3. O denotes the complexity degree. indicates unchanged values. 1 means improved accuracy compared with Full FT, Please refer to Sec. 3.3.
1114
I lick33kk
ScienceQA
Metrics
Full FT 1.2-Reg Ran Mask
Our NRCA
Resource Cost
O(10))
((())
+1])
Performance
79.47
79.52
83.02
/3.24
01
02
0.3
0.5
D.5
62
055
0.5
0.7
0.2
0.3
0.5
0.7
0.8
1.0
Table 3: Ablative Experiments for LNRCA Formulation in
AVG
79.47 (83.9) ps 13.77p4.00 24
(82.67 1s:00
82.91
82.51
13.00
(2.12
Eq. (5b). Please see Sec. 4.2 for detailed discussion.
Figure 4: Ablation on Noisy Ratio ö in Eq. (2b) effect for image captioning (Left). visual question-answert
Cee
Flickr30k SQA AVG COCO-Cap SQA
AVG
(Middle), and over-
all performance (Right) with VILA. 5 = 0 degrades into the Full Fine- Tuninnn means improved accuracy compared with Full FT. Refer to See. 4.2 for the explanation.
Fine- Tine-Tune with VILA architecture
Zero-shot
55.45
|68.96|
62.20
72.57
68.96
70.76
X (Full FT)
70.33
188660
79.47
109.58
190.33
99.96
70.87
190.43
80.65
114.83
89.59
102.21
75.10
190.23
82.67
120.05
189 94
105.00
Fine-Tune with LLaVA architecture
Zero-shot
25.31
|69.56|
47.44
40.28
69.56
54.92
X (Full
63.11
88.3
75.71
101.76
188.65
95.21
66.82
|87.56
77.19
104.39
87.80
96.10
72.39
186.61
79.50
114.54
18632
100.43
VILA
ECCCO-CIP BSCKBOOA
Tokens and appear different contribution for the prediction output. Thus, we utilize the attention map between the first output token and the input token under the noisy visual input to visualize the contribution of different input token. Besides, recent works (Che al., 2025; Zhang et al., 2025; 2024c) have shown that the MILM tends to extract object information from the image at the middle layers. Thus, we plot the all layers and middle layers attention allocation for the first output token in the following Tab. 1. The results re- veal that our method allocates more attention weights on the visual cue. Therefore, both performance comparisons and attention analyses show that our method reduces reliance language priors and improves overall model performance by encouraging consistent predictions, even under distorted visual highhighggn the value of posterior guidance over prior dependence.
Zare Aloc
afficottt
LLaVA
DESCO Cap mncri
Figure Ablation on the hyper-parameter À in Eq. (6) for the proposed method with different MLLM architectures. We default set X= 2 in the following experiments. See Sec. 4.2. for details.
refers to Sec. 4.3. Furthermore, when the downstream dis- tribution consists solely of fixed-choice tasks, MLLM does not encounter significant performance degradation. How- ever, open-response tasks are crucial for unlocking MI.LM ability to generate free-form expressions, which go beyond the limitations of traditional classification. the context of multi-task settings, we argue that open-response tasks play an essential role in showcasing the full potential of dialogue experience of MLLM, even though they introduce challenges such as performance drops due to overfitting on language information priors.
Limitations. Our method, NRCA constructs additional noisy visual inputs to obtain the corresponding prediction confidence and expects confidence alignment with the nor- mal branch. As a result, our method incurs extra storage requirements for the noisy input ë" and the logits output sig- nals ž. However, existing methods typically require storing the full pre-trained parameters to apply parameter stiffness restrictions or sparse mask updating strategies, which heav- ily depend on the MLLM model scale (O11)) As shown in Table 2, while NRCA introduces incremental storage complexity as ((B x ([G") + {)))), this cost is relatively small compared to existing methods. Additionally, NRCA achieves the highest performance of 83.24, demonstrating that effectively balances resource efficiency and down- stream performance. Detailed performance comparison
4. Experiments
4.1. Experimental Setup
Architecture and Datasets. Adhering to the Multimodal Large Language Model (MLLM) paradigm, we evaluate the effectiveness of our proposed methods using two widely adopted models, LLaVA et al., 2023b) and VILA(Lin et al., 2023), as the foundational MLLM for our experi- ments. We categorize the four downstream datasets into two task types: image captioning (open-response) and visual
6

---

Be Confident: Uncovering Overfitting in MLLM Multi-Task Tuning
Table 4: Comparison with the state-of-the-ard Multimodal Large Language Model (MLLM) Fine-TTuning Solutions on the multi-task setting: image captioning (Flickr30k and COCO-Cap) and visual question-answering (ScienceQA and IconQA) based on VILA and
LLaVA architectures with tuning epoch E' = 3. We mark the Best in bold and Second in underline across different tuning methods. + means improved accuracy compared with Zero-shot. Please refer to Sec. 4.3 for relative explanations.
Methods Flickr30k ScienceQA
AVG
COCO-Cup ScienceQA
AVG
Flickr30k
IconQA
AVG
COCO-Cup
IconQA
AVG
Tune with VILA architecure
Zero-shot
55.45
68.96
62.20
72.57
68.96
70.76
55.45
55.57
55.51
72.57
55.57
64.07
FT
70.33
88.60
79.47
T09.58
90.33
99.96
68.20
86.62
77.41
108. 16
86.91
97.53
Ran Mask
74.77
91.27
83.02
113.86
90.68
102.27
73.16
86.57
79.87
113.59
86.64
100.12
Mag Mask
75.37
91.08
83.23
114.12
90.28
102.20
73.13
87.32
80.23
113.29
87.11
100.20
Grafting
70.23
89.14
79.69
109.02
90.93
99.98
69.01
86.67
77.84
107.73
87.44
97.59
1.2-Reg
68.71
90.33
79.52
108.94
90.48
99.71
69.3
87.22
78.26
108.69
87.00
97.85
76.00
90.48
83.24
122.27
89.64
105.96
75.98
86,99
81.49
120.82
86.19
103.51
NRCA
20.55
+ 21.52
+ 21.04
+ 49.70
+ 20.68
+ 35.20
20.53
+ 31.42
+ 25.98
48.25
+ 30.62
+ 39.44
Fine-Ture with LLaVA architecture
Zero-shot
25.31
69.56
47
40,28
69.56
54.92
25:31
48.42
36.87
40,28
48.42
44.35
FT
63.11
88.30
75.71
T01.76
88.65
95.21
64.14
82.68
73.41
101.6
83.06
92.37
Ran Mask
67.56
85.62
76.59
105.24
85.82
95.53
66.64
79.65
73.15
104.86
79.69
92.28
Mag Mask
66.72
85.72
76.22
105.67
85.57
95.62
66.70
79.54
73.12
105.94
79.56
92.75
Grafting
63.30
89.44
76.37
100.47
89.64
95.06
66.35
83.91
75.13
101.20
83.76
92.48
1.2-Reg
64.78
90.03
77.41
101.05
89.94
95.50
65,24
83.95
74.60
101.92
83.72
92.82
75.49
83.84
79.67
121.50
83.94
102.72
74.31
77.60
75.96
118.74
77.83
98.29
NRCA
50.18
14.28
32.23
81.22
+ 14.38
+ 47.80
49,00
29.18
+ 39.09
+ 78.46
+ 29.41
+ 53,94
question-answering (fixed-choice), as follows:
For each dataset, we randomly sample 10k instances from the training set. We randomly select datasets from the above mentioned captioning and VQA tasks construct the multi- task training datasets.
Flickr30k (Young et al., 2014): Design for understanding the visual content of images associated with linguistic expressions, it contains 31, 000 images from Flickr, each paired with 5
reference sentences.
Counterparts. We focus on exploring anti-overfitting MLLM fine-tuning methods and mainly compare with the Partial Update Mask and @ Stiff Penalty Regularization solutions as the following formulations:
COCO-Cap (Lin et al., 2014): Comprise over 330, 000 images with more than 1.5 million captions, this dataset provides 5 human-generated captions per image for both training and validation.
ScienceQA (Lu et al., 2022): Collect from elementary and high school science curricula. This dataset includes approximately 21, 000 multimodal multiple-choice ques- tions spanning diverse science topics.
Full Fine-Tuning (Full FT) [arXiv'05] (De Boer et al., 2005): Default optimize full candidate parameters towards the downstream distribution.
L2- Reguularrizaton (L2-Reg) [PNAS'17] (Kirkpatrick et al., 2017): Add L2 regularization term with the regularization hyper-parameter, i.e., 1e-3, to the original loss function. It focuses on keeping the fine-tuning model closer to the pre-trained model, thereby mitigating forgetting.
IconQA (Lu et al., 2021): Focus on abstract diagram understanding and cognitive reasoning in real diagram.
Specifically, we follow the training settings from prior works (Zhou et al., 2024; Zhu et al., 2024a) and utilize official optimization scripts!. For fixed-choice tasks (Sci- enceQA and IconQA), we use the textual prompt: "Answer with the option's 's letter from the given choices directly." For open-response tasks (Flickr30k and COCO-Cap), we collect five manually written instructions and randomly sample one as the prompt for each caption, as follows:
Grafting [ICML'23] (Panigrahi et al., 2023): Localize newly acquired skills inside fine-tuned language mod- els, which could be regarded as L, regularization with the penalty weigh, Le., 1e-6.
Random Mask- Muning (Ran Mask) [arXiv' 24] (Hui et al., 2024): Randomly update half ratio of parameters within each transformer layer at each iteration while freezing the other elements. We default set the sampling ratio as 50%.
" DDeccribe the image as simply as possible with a sentence or phrase"
Magnitude Tuning (Mag Mask) [NeurIPS' 15] (Han et al., 2015)): Select and maintain elements with relative large weight magnitude with the pre-defined sampling ratio. The default updating proportion is set as 50%.
"Give a brief summary of what you see"
" Provide a short description of the image"
"Write short description for the image"
Brieffl describe the content of the image"
Implementation Details. We follow the official reposito-
7

---

Be Confident: Uncovering Overfitting in MLLM Multi-Task Tuning
Table Comparison with the state-of-the-ard Multimodal Large Language Model (MLLM) Fine-Tuuning Solutions on the multi-task image captioning (Flickr30k and COCO-Cap) and visual question-answering (ScienceQA and IconQA) based on VILA and LLaVA architectures with tuning epoch E' = 5. We mark the Best in bold and Second in underline across different tuning methods. + means improved accuracy compared with Zero-shot. Please refer to Sec. 4.3 for relative explanations.
setting:
Methods
Flickr30k
ScienceQA
AVG
COCO-Cap ScienceQA
AVG
Flickr30k
IconQA
AVG
COcO-Cap
IconQA
AVG
Fine-Tune with VILA architecture
Zero-shot
55.45
68.96
62,20
72.57
68.96
70.76
55.45
55.57
55,51
72.57
55.57
64.07
Full FT
64.67
91.08
77.88
102.23
91.22
96.73
65.42
87.68
76.55
100.40
87.29
93.85
Ran Mask
66.50
90.98
78.74
106.52
90.98
98.75
67.43
86.95
77.19
105.4
87.21
96.31
Mag Mask
69.17
91.77
80,47
105.16
91.22
98.19
67.34
87.24
77.29
104.98
86.87
95.92
Grafting
64.76
91.03
77.90
100.16
90.13
95.15
62.4
87.82
75.11
99.43
87.43
93.43
1.2-Reg
63.13
91.22
77.18
101.98
91.67
96.83
61.72
87.86
74.79
100.34
87.46
93.90
70.87
90.58
80.73
117.54
90.78
104.16
74.95
87.22
81.08
117.99
86,92
102.46
NRCA
15.42
21.62
18.53
44.97
21.82
33.40
19.50
31.65
25.57
445.42
31.35
| 38.39
Fine- with LLaVA architecture
Zero-shot
25.31
69.56
47.44
40.28
69.56
54.92
25.31
48.42
36.87
40.28
48.42
44.35
Full FT
61.46
91.08
76.27
92.82
90.88
91.85
60.51
84.69
72.60
92.19
84.48
88.34
Ran Mask
62.31
88.84
75.58
95.91
88.30
92.11
62.14
82.66
72.40
91.98
81.62
86.80
Mag Mask
63.10
88.55
75.83
96.88
88.50
92.69
61.37
82.43
71.90
95.23
82.79
89.01
Grafting
61.40
91.82
76.61
92.91
91,92
92.42
101.2
83.76
92.48
90.84
85.16
88.00
L2-Reg
59.77
91.52
75.65
91.76
91.67
91.72
59.18
85.99
72.59
91.52
85.32
88.42
73.04
87.11
80.08
120.81
82,80
101.81
74.51
81.65
78.08
120.31
78.34
99.33
NRCA
47.73
17.55
.3.64
80.53
13.24
46.89
.99.00
33.23
41,21
80.03
29.92
54.99
ries2.3 to conduct the fine-tuning procedure. With respect to the training scenarios, we first randomly sample 10, 000 instances from each dataset and combine them as multi-task tuning datasets. As for the optimization details, the learning rate ir in LLaVA (Liu al., 2023b) is 2e 4 for L.M and 2e -- 5 for the visual projector. For VILA(Lin et al., 2023), we uniformly set the learning rate to le - 4. The training epochs are set to E = 3 and E = 5. The training batch size B is set to 16 by default. The fine-tuning block for LLM is the last N- = 2 layers. Regarding the experimental conditions, all experiments are conducted on NVIDIA 4090 GPUs, each with 24GB of memory. Due the limited computational resources, we select the LLaVA-115-77 for LLaVA and VILA1.5-3B for VILA.
(NRCA) in Tab. 3. The ablation results demonstrate that directly encouraging prediction behavior consistency effec- tively mitigates the open-response overfitting phenomenon. Moreover, incorporating empirical loss guidance, which accounts for sample difficulty, further enhances the over- all performance. Thus, combining Noisy Visual Mixup and Token Confidence Alignment acquires satisfying down- stream multi-task performance that coincides with our mo- tivation of exploiting the noisy resilience to alleviate the open-response overfitting during Multimodal Large Lan- guage Model tuning.
Noisy MixUp Ratio. The parameter 8, introduced in Eq. (2b), controls the Gaussian distortion ratio. As illus- trated in Fig. 4, increasing the distortion ratio poses chal- lenges for confidence alignment in open-response tasks, resulting in limited performance improvements. Generally, as d increases, trade-off between open-response and fixed- choice performance becomes more pronounced, with im- provements diminishing under higher noise ratios. For con- sistency and convenience, we set 8 = 0.5 across different scenarios in the subsequent experiments.
Evaluation Metrics To evaluate the performance of MLLM across different downstream task types, we utilize CIDER (Vedantam et al., 2015) and Top-1 Accuracy metrics for image captioning and visual question-answering tasks, respectively. Additionally, we compute the mean results to represent the overall downstream performance.
4.2. Diagnostic Analysis
Control Hyper-Parametes À in Eq. (6). The Sec. 4 quan- tifies the effect of hyper-parameter X, which measures the strength of token confidence difference penalization on dif- ferent scenarios. Specifically, the open-response and fixed- task trade-off performance progressively mounts À en- larges, and the improvement presents marginal under strict parameter stiffness. For convenience, we choose the X = 2 for different scenarios in the following experiments.
We perform ablation studies on two scenarios: Flickr30k & ScienceQA and COCO-Cap & ScienceQA, utilizing both the VILA and LL architectures with tuning epoch of E= to facilitate an in-depth analysis.
Proposed Training Objective. We quantitatively ana- lyze the proposed Noise Resilient Confidence Alignment
hhtts//giittbbcco//oooiiaaa////u hhapss/gtttab.cco///Vaa////LLL
8

---

Be Confident: Uncovering Overfitting in MLLM Multi-Task Tuning
4.3. Comparison to State-of-the-Arts
introduces prediction confidence calibration that shows the model-agnostie property. NRCA has been validated across various scenarios, demonstrating its effectiveness and high- lighting its potential for broader applications.
We compare our NRCA against related approaches on the multi-tas, scenarios:
& visual question- answering tasks. As shown in Tab. 4 and Tab. 5, several key observations can be made. First, Partial Update Mask does not fundamentally mitigate overfitting but merely slows the fitting process of MLLM. This is evident from its limited performance gains, even with extended tuning epochs E and larger model architectures, e.g., scaling from 3B to 7B. Specifically, Random Mask-Tuning applies stochastic masks to parameters, injecting noise to reduce reliance specific sub-networks and implicitly regularize by limit- ing parameter co-adaptation. While effective on smaller models (VILA-3B), its performance drops on larger ones (LLaVA-7B), suggesting its effect comes from constrained optimization rather than addressing language prior domi- nance. Second, Stiff Penalty Regularization is highly sensi- tive to regularization strength, making it difficult to main- tain stable performance improvements across varying set- tings with consistent hyper-parameters For instance, in VILA with COcO-Cap and IconQA under E = 5, Graft- ing demonstrates inferior performance compared to Full Fine-Tuning, highlighting its limitations in achieving robust and consistent results. Third, our method NRCA promotes prediction confidence alignment and consistently delivers robust results across various downstream tasks and model architectures. Moreover, as demonstrated in Tab. 2, our approach requires fewer computational resources compared to existing methods, highlighting its efficiency.
A Acknowledgement
This work is supported by National Natural Science Foun- dation of China under Grant (62225113, 62361 166629, 62176188, 623B2080), the National Key Research and Development Program of China (2023YFC2705700, 2024YFC3308400), and the Wuhan University Undergradu- ate Innovation Research Fund Project. The supercomputing system at the Supercomputing Center of Wuhan University supported the numerical calculations in this paper. Dr Tao's research supported by NTU RSR and Start Up Grants.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential socictal consequences of our work, none of which we feel must be specifically highlighted here.
References
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J. L., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M. a., Barreira, R., Vinyals, O., Zisser- man, A., and Simonyan, K. Flamingo: a visual language model for few-shot learning. In NeurIPS, volume 35, pp. 33716-23736 2022.
5. Conclusion
In our work, we focus on fine-tuning Multimodal Large Lan- guage Model (MLLM) in multi-task scenarios to achieve multi-domain specialization. However, we observe a criti- cal issue: open-response distributions appear performance degradation during the training process. We attribute this degradation to over language prior reliance. To address this, we propose Noise Resilient Confidence Alignment (NRCA), which aims enhance the effect of visual cues in the pre- diction process. Generally speaking, we leverage Gaussian distortions to construct noisy visual inputs and encourage overall token confidence alignment with the normal visual branch behavior. Compared to existing methods, our ap- proach offers two key advantages: First, Resource Cost De- cline: NRCA conducts output calibration without requiring storage of pre-trained Multimodal Large Language Model weights to counteract overfitting phenomenon, making ours computationally efficient compared to existing methods. Second, H/yeerParametter Stability: Related methods of- ten rely on pre-defined mask ratios or decorate parameter stiffness control, which are highly correlated with the ar- chitecture and scale of MLLM. In contrast, our method
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schul- man, J., and Mané, D. Concrete problems in ai safety. arXiv preprint DrXivv/606.000 06565, 2016.
Ansell, A., Ponti, E. M., Korhonen, A., and Vulić, I. Com- posable sparse fine-tuning for cross-lingual transfer. In ACL, 2022.
Ardizzone, L., Mackowiak, R., Rother, C., and Köthe, U. Training normalizing flows with the information bottle- neck for competitive generative classification. In NeurlPS, volume 33, pp. 7828-7840, 2020.
Bai, J., Bai, S.,, Yang. S.,, Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, Qwen-vl: A versatile vision- language model for understanding, localization, text read- ing, and beyond. arXiv preprint arXiv:2308. /2966, 2023.
Bi, J., Wang, Y., Chen, H., Xiao. X., Hecker, A., Tresp, V., and Ma Y. Llava steering: Visual instruction tun-
9

---

Be Confident: Uncovering Overfitting in MLLM Multi-Tass Tuning
ing with 500x fewer parameters through modality linear representationnssteriing In ACI., 2025a.
Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. Instructblip: Towards general-purpose vision-language models with instruction tuning. In NeurIPS, 2023.
Bi, J., Wang, Y., Yan, D., Xiao, X., Hecker, A., Tresp, V., and Ma, Y. Prism: Self-pruning intrinsic selection method for training- free multimodal data selection. arXiv preprint arXiv:2502.1211
De Boer, P.-T., Kroese, D. P., Mannor, S., and Rubinstein, R. Y. A tutorial on the cross-entropy method. Ann. Oper. Res., pp. 19-67, 2005.
2025b.
Biderman, D., Ortiz, J. G., Portes, J., Paul, M., Greengard, P., Jennings, C., King, D., Havens, S.. Chiley, V., Frankle, J. et al. Lora learns less and forgets less, TMLR, 2024.
Devlin, J., Chang. M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. arXiv preprint arXiv: 110.:0081 2018.
Brown, T., Mann, B., Ryder, N., Subbiah, M. Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., et al. Language models are few-shot learners. In NewrlPS, pp. 1877-1901, 2020.
Dong. G., Yuan, H., Lu, K., Ki, Li, C.,, Xue, M., Liu, D., Wang, W., Yuan, Z., Zhou, C., and Zhou, J. How abilities in large language models are affected by supervised fine-tuning data composition. arXiv preprint arXiv: 10.05599, 2023.
Buzzega, P., Boschini, M., Porrello, A., Abati, D., and Calderara, S. Dark experience for general continual learn- ing: strong, simple baseline. In NeurlPS, pp. 15920- 15930, 2020.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICIR, 2021.
Che, L., Liu, T. Q., Jia, Qin, W., Tang, R., and Pavlovic, V. Eazy: Eliminating hallucinations in Ivlms by ze- roing out hallucinatory image tokens. arXiv preprint arXiv:2503.07772. 2025.
Fang, Y., Zhu, L., Lu, Y., Wang, Y., Molchanov, P., Cho, J. H., Pavone, M., Han, S., and Yin, H. vila?: Vila augmented vila. arXiv preprint arXiv:2407.1445 2024.
Chen, B., Lyu, X., Gao, L., Song, J., and Shen, H. T. Al- leviating hallucinations in large vision-language mod- els through cettuciinatii-inndcee optimization. arXiv preprint ar ar0iv:2405. 15356, 2024a.
Favero, A., Zancato, I.., Trager, M., Choudhary, S., Perera, P., Achille, A., Swaminathan, A., and Soatto, S. Multi- modal hallucination control by visual information ground- ing. CVPR. pp. 14303-14312, 2024.
Chen, M., Fu, D. Y., Narayan, A., Zhang, M., Song, Z., Fatahalian, K., and Ré, C. Perfectly balanced: Improving transfer and robustness of supervised contrastive learning. In ICML. pp. 3090-3122, 2022.
Gekhman, Z.,, Yona, G., Aharoni, R., Eyal, M., Feder, A., Reichart, R., and Herzig. J. Does fine-tuning llms on new knowledge encourage hallucinations? arXiv preprint arXiv:2405.05904 2024.
Chen, T., Korbblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In ICML, pp. 1597-1607, 2020.
Ghosh, S., Evuru, C. R., Kumar, S., Tyagi, U., Nieto, O., Jin, Z., and Manocha, D. Visual description grounding re- duces hallucinations and boosts reasoning in Ivlms. arXiv preprint ArXiv:2405. 15683, 2024.
Chen, Y., Wang, S., Lin, Z., Qin, Z., Zhang, Y., Ding, T., and Sun, R. Mofo: Momentum
mamenntu-filterred filtered optimizer for mitigating forgetting in Ilm fine-tuning. arXiv preprint arXiv:2407.20999, 2024b.
Goodman, N. R. Statistical analysis based on a certain mul- tivariate complex gaussian distribution (an introduction). The Annals of mathematical statistics, 34(1):152-177, 1963.
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z.,, Zhang, H., Zheng, L., Zhuang. S.. Zhuang. Y., Gonzalez, J. E., et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https:/vvicna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Gu, Y., Dong, I.., Wei, F,, and Huang, M. Minillm: Knowl- edge distillation of large language models. ICLR, 2024.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. arXiv preprint 2rXiv:2204.0231/1 2022.
Gu, Y., Zhou, H., Meng, F., Zhou, J., and Huang. M. Mini- plm: Knowledge distillation for pre-training language models. In ICLR, 2025.
10

---

Be Confident: Uncovering Overfitting in MLLM Multi-Tass Task Tuning
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In ICML, pp- 1321-1330, 2017.
Huo, F., Xu, W., Zhang, Z., Wang, H., Chen, Z., and Zhao, P. Self-introspective decoding: Alleviating hallucina- tions for large vision-language models. arXiv preprint arXiv:2408 000:00032,022 2024.
Guo, Z., Xu, R.,, Yao, Y., Cui, J., Ni, Z., Ge, C., Chua, T.-S., Liu, Z., and Huang, G. LLaVA-UHD: an Imm perceiving any aspect ratio and high-resolution images. In ECCV, 2024.
Jin, C.,, Che, T., Peng, H., Li, Y., Metaxas, D., and Pavone, M. Learning from teaching regularization: Generalizable correlations should be easy to imitate. In NeurlPS, 2024.
Jin, C.,, Peng, H., Zhang, Q., Tang, Y., Metaxas, D. N.. and Che, T. Two heads are better than one: Test-time scaling of multi-agent collaborative reasoning. arXiv preprint arXiv:2504.09772, 2025.
Han, S., Pool, J., Tran, J., and Dally, W. Learning both weights and connections for efficient neural network. In NeurIPS, 2015.
Han, 7.,, Gao, C., Liu, J., Zhang, S. Q., et al. Parameter- efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403. 140000000.,4088, 2024.
Kazemzadeh, S., Ordonez, V., Matten, M., and Berg, T. Referitgame: Referring to objects in photographs of natu- ral scenes. In EMNLP, pp. 787-798, 2014.
Hao, Y., Cao, Y., and Mou, L. Flora: Low-rank adapters are secretly gradient compressors. In ICMI,, 2024.
Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., and Krishnan, D. Supervised contrastive learning. In NeurIPS, 2020.
Hayashi, H. and Uchida, S. A discriminative gaus- sian mixture model with sparsity. arXiv preprint arXiv:1911.06022 2019.
Kim, G., Jang, D., and Yang, E. Promptkd: Distilling student-friendly knowledge for generative language mod- els via prompt tuning. arXiv preprint urXiv:2402. 12842, 2024a.
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Mo- mentum contrast for unsupervised visual representation learning. CVPR, pp. 9729-9738, 2020.
Kim, S., Cho, B., Bae, S., Ahn, S., and Yun, S.-Y. Vacode: Visual augmented contrastive decoding. arXiv preprint arXiv:2408.05337, 2024b.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. ICLR, 2022.
Kirkpatrick. J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., A., al. Overcoming catastrophic forgetting in neural networks. PNAS, pp. 3521-3526, 2017.
Hu, J., Zhang, J., Qi, L.,, Shi, Y., and Gao, Y. Learn to preserve and diversify: Parameter-efficient group with orthogonal regularization for domain generalization. In ECCV, 2024.
Ko, J., Kim, S., Chen, T., and Yun, S.-Y. Distillm: Towards streamlined distillation for large language models. In ICML, 2024.
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng. W., Feng, X., Qin, B., al. survey hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM TIS, 2023.
Lakshminarayanan. B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurIPS, 2017.
Leng, S., Zhang, H., Chen, G., Li, X., Lu, S., Miao, C., and Bing, L. Mitigating object hallucinations in large vision- language models through visual contrastive decoding. In CVPR, pp. 13872-13882, 2024.
Huang, W., Yi, M., and Zhao, X. Towards the generalization of contrastive self-supervised learning, arXiv preprint arXiv:2111.00443, 2021.
Huang, W. Liang, J., Guo. X., Fang, Y., Wan, G., Rong, X., Wen, C., Shi, Z., Li, Q. Zhu, D., et al. Keeping yourself is important in downstream tuning multimodal large language model. arXiv preprint arXiv:2503.0454, 2025.
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Li, Y.. Liu, Z., and Li, C. Llava- onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024a.
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-inen -magg pre-training for unified vision-language understanding and generation. In ICML, pp. 12888- 12900. PMLR, 2022a.
Hui, T., Zhang, Z., Wang. S., Xu, W., Sun, Y., and Wu, H. Hft: Half fine-tuning for large language models. arXiv preprint XrX4v:2404. 18466, 2024.

---

Be Confident: Uncovering Overfitting in MLLM Multi-Task Tuning
J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping eeguuag--magee pre-training with frozen image encoders and large language models. In ICML, pp. 19730-19742. PMLR. 2023a.
Liu, S., Zheng, K., and Chen, W. Paying more attention image: A training-free method for alleviating hallucina- tion in Ivims. arXiv preprint arXiv:2407.21771 20244.
Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-.. F., Cheng, K.-T., and Chen, M.-H. Dora: Weight- decomposed low-rank adaptation. In ICML, 2024c.
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogeneous networks. In MLSys, 2020.
Lu, P., Qiu, L.,, Chen, J., Xia, T., Zhao, Y., Zhang, W., Yu, Z., Liang, X., and Zhu, S.-C. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In NeurlPS, 2021.
Li, Y., Luo, F., Tan, C., Wang, M., Huang, S., Li, S., and Bai, J. Parameter-efficient sparsity for large language models fine-tuning, In IJCAI, 2022b.
Lu, P., Mishra, S., Xia, T., Qiu, L.., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurlPS, 2022.
Li, Y. Du, Y., Zhou, K., Wang. J., Zhao, W. X., and Wen, J.-R. Evaluating object hallucination in large vision- language models. In EMNLP, 2023b.
Li, Y., Yu, Y., Zhang, Q., Liang, C., He. P.. Chen, W., and Zhao, T. Losparse: Structured compression of large language models based on low-rank and sparse approxi- mation. In ICML, pp. 20336-20000 20350. PMLR, 2023c.
Lu, Y., Qian, B., Yuan, C., Jiang, H., and Wang, X. Con- trolled low-rank adaptation with subspace regularization for continued training on large language models. arXiv preprint arXiv:2410. 16801, 2024a.
Lu, Z., Fan, C., Wei, W., Qu, X., Chen, D., and Cheng, Y. Twin-merging: Dynamic integration of modular expertise in model merging. In NeurlPS, 2024b.
Li, Z., Chen, C., Xu, T., Qin, Z., Xiao, J., Sun, R., and Luo, Z. Entropic distribution matching in supervised fine-tuning of llms: Less overfitting and better diversity. arXiv preprint ar arXiv:2408. 0nl:2208./66733 /6673, 2024b.
Luo, G., Zhou, Y., Zhang, Y., Zheng, X., Sun, X., and Ji, R. Feast your eyes: Mixture-of-re of-irr-uutrrrotiiiin adaptation for multimodal large language models. arXiv preprint Stt::2403303003 2024.
Liang, J., Huang, W., Wan, G., Yang, Q., and Ye, M. Lo- rasculpt: Sculpting lora for harmonizing general and spe- cialized knowledge in multimodal large language models. In CVPR, 2025.
Marino, K.,, Rastegari, M., Farhadi, A., and Mottaghi, R. Ok- vqa: A visual question answering benchmark requiring external knowledge. In CVPR, 2019.
Lin, J., Yin, H., Ping, W., Lu, Y., Molchanov, P., Tao, A., Mao, H., Kautz, J., Shoeybi. M.. and Han, S. Vila: On pre-training for visual language models. In CVPR, 2023.
Mishra, A., Shekhar, S., Singh, A. K., and Chakraborty, A. Oer-vqa: Visual question answering by reading text in images. In ICDAR, pp. 947-952, IEEE, 2019.
Lin, T., Liu, J., Zhang, W., Li, Z., Dai, Y., Li, H., Yu, Z., He, W., Li, J., Jiang, H., et al. Teamlora: Boosting low- rank adaptation with expert collaboration and competition. arXiv preprint 24Xx:2408.09856 2024.
OpenAL. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Panigrahi, A., Saunshi, N., Zhao, H., and Arora, S. Task- specific skill localization in fine-tuned language models. In ICML, pp. 27011-27033, PMLR, 2023.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In ECCV, pp. 740-755, 2014.
Platt, et al. Probabilistic outputs for support vector ma- chines and comparisons regularized likelihood meth- ods. Advances in large margin classifiers, pp. 61-74, 1999.
Liu, H. Li, C., Li, Y., and Lee, J. Improved baselines with visual instruction tuning. In CVPR, 2023a.
Radford, A., Wu, J., Child, R.. Luan, D., Amodei, D.,. Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9 2019.
Liu, H., Li, C., Wu, Q. and Lee, Y. Visual instruction tuning. In NeurlPS, 2023b.
Liu, H., Xue, W., Chen, Y., Chen, D., Zhao, X., Wang, K., Hou, L.. Li, R., and Peng, W. A survey on halluci- nation in large vision-lane sision-a models. arXiv preprint arXiv:2402.000 00253, 2024a.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748-8763. 8763, 2021.
12

---

Be Confident: Uncovering Overfitting in MLLM Multi-Task Tuning
Ren, Y., Guo, S., Bae, W., and Sutherland, D. J. How to prepare your task head for finetuning. In ICLR, 2023.
Vedantam, R., Lawrence Zitnick, C., and Parikh, D. Cider: Consensus-based image description evaluation. In CVPR, pp. 4566-4575, 2015.
Ritter, H., Botev, A., and Barber, D. Online structured laplace approximations for overcoming catastrophic for- getting. In NeurlPS, 2018.
Wang, F., Xiang, X., Cheng, J., and Yuille, A. L., Normface: L2 hypersphere embedding for face verification. In ACM MM, pp. 1041-1049, 2017.
Shang, Y., Zeng, X., Zhu, Y., Yang, X., Fang, Z., Zhang, J., Chen, J., Liu, Z., and Tian, Y. From pixels to tokens: Revisiting object hallucinations in large vision- laiioge models. arXiv preprint arXiv:2410.6679) 2024.
Wang. J., Ming, Y., Shi, Z. Vineet, V., Wang, X., and Joshi, N. Is picture worth thousand words? delving into spatial reasoning for vision language models. In NeurlPS, 2024a
Shen, I.., Chen, G., Shao, R., Guan, W., and Nie, L.. Mome: Mixture of multimodal experts for generalist multimodal large language models. arXiv preprint arXiv:2407.12700 2024a.
Wang. P., Panda, R., Hennigen, L. T., Greengard, P., Kar- linsky, L., Feris, R., Cox, D. D., Wang, Z., and Kim, Y. Learning to grow pretrained models for efficient trans- former training. In ICLR, 2023.
Shen, Y., Shao, J., Zhang, X., Lin, Z., Pan, H., Li, D., Zhang, J., and Letaief, K. B. Large language models empowered autonomous edge ai for connected intelligence. IEEE Communications Magazine, 62(10): 1401))640-14,, 2024b.
Wang. X., Pan, J., Ding, L., and Biemann, C. Mitigat- ing hallucinations large vision-language models with instruction contrastive decoding. In ACL, 2024b.
Shi, M., Liu, F., Wang, S., Liao, S., Radhakrishnan, S., Huang, D.-A., Yin, H., Sapra, K., Yacoob, Y., Shi, H., et al. Eagle: Exploring the design space for multi- modal Ilms with mixture of encoders. arXiv preprint arXiv:2408. 15998, 2024.
Wang. Y., Zhang, Q. Wang, Y., Yang, J., and Lin, 7. Chaos is a ladder: A new understanding of contrastive learning, In ICLR, 2022.
Wang. Y., Zhang, J., and Wang, Y. Do generated data always help contrastive learning? In ICLR, 2024c.
Shuttleworth, R., Andreas, J., Torralba, A., and Sharma, P. Lora vs full fine-tuning: An illusion of equivalence. arXiv preprint arXi::2410.21226 2024.
Woo, S., Jang, J., Kim, D., Choi, Y. and Kim, C. Ritual: Random image transformations as univer- sal anti anti-hallucination lever in Ivims. arXiv preprint 24i.:2400.1782111 2024.
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In CVPR, pp. 8317-8326, 2019.
Wu, C., Wu, F., and Huang, Y. Rethinking infonce: How many negative samples do you need? In IJCAI, 2022.
Tian, Y., Sun, C., Poole, B., Krishnan, D., Schmid, C., and Isola, P. What makes for good views for contrastive learning? In NeurlPS, volume 33, pp. 6827-6839, 2020.
Wu, Z.,, Xiong, Y., Yu, S. X., and Lin, D. Unsupervised fea- ture learning via non -on-maraetttic instance discrimination. In CVPR, pp. 3733-3742. 2018.
Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., Yang, J., Yang, S., Iyer, A., Pan, X., al. Cambrian-1: 1: A fully open, vision-centric exploration of multimodal lms. arXiv preprint arXiv:2406. /6860, 2024.
Xiao, T., Wang. X., Efros, A. A., and Darrell, T. What should not be contrastive in contrastive learning. In ICLR, 2021.
Xiao, Y., Lin, Y., and Chiu, M.-C. Behavioral bias of vision- language models: A behavioral finance view. arXiv preprint arXiv:2409. 15256, 2024.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprini arXiv:2302.1 13971, 2023.
Xuhong, L., Grandvalet, Y., and Davoine, F. Explicit induc- tive bias for transfer learning with convolutional networks. In ICML, pp.
Tüske, Z., Tahir, M. A., Schlüter, R., and Ney, H. Integrating gaussian mixtures into deep neural networks: Softmax layer with hidden variables. In ICASSP, pp. 4289 IEEE, 2015.
PMLR, 2018
Ye, M., Zhang, X., Yuen, P. C., and Chang, S.-F. Unsu- pervised embedding learning via invariant and spreading instance feature. CVPR, pp. 6210-6219, 2019.
Variani, E., McDermott, E., and Heigold, G. A gaussian mixture model layer jointly optimized with discriminative features within a deep neural network architecture. In ICASSP, pp. 4270 4274. IEFF, 2015.
Ye, M., Shen, J., Zhang, X., Yuen, P. C., and Chang, S.-F. Augmentation invariant and instance spreading feature for softmax embedding. IEEE PAMI, 2020.
13

---

Be Confident: Uncovering Overfitting in MLLM Multi-Task Tuning
Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2:67-78, 2014.
Zhu, T., Liu, Q. Wang, F., Tu, Z., and Chen, M. Unravel- ing moos-mmdality knowledge conflicts in large vision- language models. arXiv preprint arXiv:2410.036 2024b.
Yu, L., Yu, B., Yu, H., Huang, F,, and Li, Y. Language mod- cls are super mario: Absorbing abilities from homologous models as a free lunch. ICML, 2024.
Zhu, Y., Zhu, M., Liu, N., Ou, Z., Mou, X., and Tang, J. Llava-phi: Efficient multi-modal assistant with small lan- guage model. arXiv preprint arXiv:2401.02330, 2024c.
Zenke, F., Poole, B., and Ganguli, S. Continual learning through synaptic intelligence. In ICML, pp- 3987-3995. PMLR, 2017.
Zhang, Q. Chen, M., Bukharin, A., He, P., Cheng, Y., Chen, W., and Zhao, T. Adaptive budget alloca- tion for parameter-efficient fine-tuning, arXiv preprint arXiv:2303. 2023.
Zhang. W., Janson, P., Aljundi, R., and Elhoseiny, M. Over- coming generic knowledge loss with selective parameter update. In CVPR, pp. 24046-24056, 2024a.
Zhang, X., Quan, Y., Shen, C., Yuan, X., Yan, S., Xie, L., Wang, W., Gu, C., Tang, H., and Ye, J. From redundancy to relevance: Information flow in Ivims across reasoning tasks. NAACL, 2025.
Zhang, Y.-F., Yu, W., Wen, Q., Wang, X., Zhang, Z., Wang, L., Jin, R., and Tan, Debiasing large visual language models. arXiv preprint 055222, 2024b.
Zhang, Z., Yadav, S., Han, F., and Shutova, E. Cross-modal information flow in multimodal large language models. In CVPR, 2024c.
Zhang, Z., Zhang, Q. Gao, Z., Zhang, R., Shutova, E., Zhou, S., and Zhang, S. seudient-based parameter selection for efficient fine-tuning. In CVPR, pp. 18566-285777 2024d.
Zhao, L., Deng, Y., Zhang, W., and Gu, Q. Mit- igating object hallucination in large vision-language models via classifier-free guidance. arXiv preprinl arXiv:2402.08680., 2024.
Zhao, N., Wu, Z., Lau, R. W., and Lin, S. What makes instance discrimination good for transfer learning? In ICLRR 2021.
Zhou, X., He, J.. Ke, Y.. Zhu, G., Gutiérrez-Basulto, V., and Pan, Z. An empirical study on parameter-efficient fine-tuning for multimodal large language models. In ACI., 2024.
Zhu, D., Sun, Z., Li, Z., Shen, T., Yan, K., Ding, S., Kuang, K., and Wu, C. Model tailor: Mitigating catastrophic for- getting in multi mootil large language models. In ICML, 2024a.
14

---

Be Confident: Uncovering Overfitting in MLLM Multi-Tass Tuning
APPENDIX
A. Notation Table
We provide the notation table in Tab. 6.
Table 6: Notations table.
Description
Description
MILLM
f Vision encoder
Connector
9
Large Language Model
Language response
T Label Length
V Vocahulary dictionary
N LLM tuning blocks
x Visual image
N Gaussian distribution
4" Noisy visual image
x¹ Textual instruction
Figure Empirical Loss on both Open-Response and Fixed- Choice tasks. Experiments are based on the LLaVA with tuning epoch == We observe stepwise loss tendency in the former. The open-response distribution compels MLLM to wer-memoriie each sample during every training epoch, resulting a jumping loss behavior. Please refer to Appendix C.
m"
Visual feature
h" Visual token embedding
h' Text token embedding
Logits output
Pr Token Probability
Y: Token Confidence
Overall token prediction
0
softmax function
Fine- TuninT epoch
Downstream dataset
overfitting. As a preliminary step, we revisit the classi- cal MILM fine-tuning optimization loss. Specifically, we suppose (x", ,) as the query instance from the training dataset, where y E RT*|V| is the ground-truth truth label. We obtain the prediction token logits = 0(x", x²)). Then, formulate the cross-entropy (De Boer al., 2005) as:
C Loss function
Control hyper-panee ayper-parameter
Learning rate
B Training batch size
B. Algorithm
We provide the algorithm description in Algorithm 1,
1
Algorithm NRCA
LCE
T
ly, log(a<=));
Input:
Fine- Tine-Tunin epoch F., Downstream dataset Overall MILLM Network 0, Trainable parameter module w
1
1y, log(p:),
(7)
Output: The optimized selected MLLMM module w
T
for = 1,2,... do for (x", xx.,." D do
1 T
log(p?'),
x Construct Noisy Vision view x/
n. (x") via Eq. (2a)
8" + (x",,,0) ) by Eq. (2b)
where ly, denotes the one-hot encoding of yr, and repre- sents the softmar operation (Wang et al., 2017). Thus, to some extent, minimizing LCE can be interpreted as maxi- mizing the prediction confidence for each token, p
h =((f(x)) ) and h" = x(f("))
A' = Tokenize(x")
# = o(h", h²t) and g(h",
/ Token Confidence Alignment +/
Consequently, MLLM, with its large-scale network capacity, often encounters less challenging scenarios when handling downstream tasks with limited scope, making it prone to memorizing the training samples. In open-response tasks, long ground-truth label sequence is typically involved. As a result, during empirical loss minimization, each token prediction confidence naturally increases to align with the target distribution. contrast, for fixed-choice tasks, where only single ground-truth token label usually exists (i.e.. T 1), responses are constrained predefined options such as A, B, C, or D. This constrained candidate label space makes it more challenging to enforce memorization of each individual sample.
Pe o(2)), Pe (2)):
// Token Prob.
p
[p T
// Token Conf.
LNRCA (P) through Eq. (5b)
LCB + (p.y) in Eq. (5a)
C Lce + ACNRCA
== w V
// Update Param.
end
end
C. Rationale Analysis
We conduct a theoretical discussion to investigate the ra- tionale for regularizing the prediction confidence the Multimodal Large Language Model (MI.LM) to mitigate
Therefore, we respectively plot the training loss curve LCE for both open-response and fixed-choice tasks. As shown
15

---

Be Confident: Uncovering Overfitting in MLLM Multi-Task Task Tuning
Table Comparison with the state-of-the-art Multimodal Large Language Model (MLLM) Fine-Tuuning Solutions on the multi-task setting: image captioning (Flickr30k and COCO-Cap) Cap) and visual question-answering (ScienceQA and IconQA) based on the VILA architectures with tuning epoch E' 3. We mark the Best in bold and Second in underline across different tuning methods. means improved accuracy compared with 7ero-shot. Please refer to Appendix D) for relative explanations.
Methods Flickr30k ScienceQA
AVG
COCO-Cup
ScienceQA
AVG
Flickr30k
IconQA
AVG
COcO-Cup
IconQA
AVG
Tune with B 32
Zero-shot
55.45
68.96
62.20
72.57
68.96
70.76
55.45
55.57
55.51
72.57
55.57
64.07
FT
71.73
90.28
81.01
T11.22
90.13
100.68
71.67
86.68
79.18
110.86
87.46
99.16
Ran Mask
76.90
90.98
83.94
117.07
90.88
103.98
75.21
86.65
80.93
116.84
87.11
101.98
Mag Mask
77.89
90.78
84.34
117.35
91.27
104.31
76.13
86.97
81,55
116.93
86.91
101.92
Grafting
73.29
90.63
81.96
112.28
90.18
101.23
70.58
87.11
78.85
110.72
87.30
99.01
1.2-Reg
73.92
90.68
82.30
111.5
90.48
100.99
71.73
87.06
79.40
110.76
87.35
99.06
79.63
90.08
84.86
122.50
90.73
106.62
80.96
86.62
83.79
122.60
86.11
104.35
NRCA
24.18
+ 21.12
+ 22.66
49,93
+ 21.77
+ 35.86
25.51
+ 31.05
+ 28.28
+ 50.03
+ 30.54
+ 44.228
Fine-Tune with B = 24
Zero-shot
55.45
68.96
62.20
72.57
68.96
70.76
55,45
55.57
55.51
72.57
55.57
64.07
FT
71.57
90.68
81.13
110.31
90.48
100.40
71.63
87.08
79.35
109.72
86.49
98.11
Ran Mask
75.62
90.23
82.93
115.83
90.73
103.28
75.55
87.03
81.29
116.13
87.27
101.70
Mag Mask
75.16
90.63
82,90
115.35
90.68
103.02
74.27
87.27
80.77
115.10
87.16
101.13
Grafting
71.64
90.03
80.84
110.09
90.53
100.31
68.88
87.1
77.99
110.21
86.73
98.47
1.2-Reg
71.31
90.08
80.70
109.81
90.38
100.10
69.75
87.38
78.57
109.65
86.78
98.22
81.20
90.18
85.69
123.22
90:43
106.83
80.93
86.15
83.54
123.84
86.89
105.37
NRCA
25.75
21.22
23.49
50.65
21.47
+ 36.07
25.48
30.58
+ 28.03
51.27
31.32
+ 41.30
in Appendix C, the training loss for open-response tasks does not exhibit a progressive loss tendency but instead demonstrates a stepwise loss behavior. We hypothesize that this phenomenon is because MLLM tends to memorize each sample rather than explore the underlying distribution pat- terns, also observed in (Li et al., 2024b; Lu et al., 2024a). In contrast, it is challenging to map the same choice to different multi-modal inputs for ixed-choice settings. Consequently, MILLM is encouraged to analyze the multi-modal inputs and make informed decisions.
T
min
Jr (21)
Tye (2x)
(9)
Assuming that Ty. (2+) 2 nu (2:) holds (a natural assump- tion since clean inputs tend to yield higher confidence), the optimization objective becomes:
min (o - ove (2t)),
(10)
With recall of the empirical open- open-response task minimiza- tion, it would blindly increase the prediction confidence, P Therefore, in our work, we aim to reduce the reliance of MLLM on language priors and instead emphasize the role of visual features in prediction behavior. Generally, we encourage noisy visual inputs to achieve consistent predic- tion confidence comparable to the normal branch, thereby mitigating learning shorteuts.
which encourages the perturbed-input confidence og (2) to approach that of the clean input op (zt). Since y, = g ( (z"),, he), where f is the visual encoder and & and g denote the projection and decoding modules, this objective drives the network to produce robust visual representations that are invariant to input perturbations. As a result, our model is encouraged to ground its predictions more faith- fully in visual evidence rather than language priors, thereby mitigating overfitting by preventing overconfidence on indi- vidual training samples. We provide the full implementation and formulation Sec.
To formalize this, we denote I and i as the average predic- tion confidence over ground-trute tokens under clean and perturbed visual inputs (see Eq. (4)), respectively. Let 2, and & be the predicted logits at position f, and let o-) denote the softmax function. The NRCA regularization seeks to minimize the relative deviation:
D. Additional Experiments
We provide the experiment comparison with different train- ing batch sizes of B E (32, 24} in Tab. 7. We conduct experiments on the VILA architectures on the four down- stream datasets with the tuingg epochs as = 3. Compared with existing Multimodal Large Language Model methods, our method consistently demonstrates effectiveness.
min 1
/ = min
I
Σ
T
(e) - Tze (5a)
(8)
T
1
Since I is detached from the computational graph, this sim- plifies to:
16


Reviews:
[{"review_id": "WeO0Nam2px", "reviewer": "Reviewer_H4RG", "summary": "In the reviewed paper, the authors identify overfitting in open-response tasks as a significant challenge in multi-task multimodal large language model (MLLM) fine-tuning. They propose a novel method called NRCA, which aligns prediction confidences between noisy and normal visual inputs to improve visual representation learning. The paper presents comprehensive empirical evaluations that demonstrate the effectiveness of the NRCA approach in addressing the identified issue.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer H4RG:\n\nThank you very much for your affirmation of our work, as well as the insightful concerns and questions you have raised. We have carefully considered each comment and provided responses.\n\n**Q1: Multi-Task Specialization Definition** (Claims And Evidence)\n\nA1: Multi-Task Specialization refers to fine-tuning a pre-trained model for a specific downstream application, which normally has a limited amount of data, compared with the pre-training data scale. For example, the authors of LLaVA claim that it involves $150k$ unique language-image instruction-following samples. However, the specialization application samples are much more limited, i.e., $10k$ for target task. Thus, tuning on the downstream distribution tends to lead to overfitting on the specialization task. And we point out that open-response tasks easily appear the overfitting behavior within the multi-task specialization. We attribute this degradation to language prior reliance and propose Noise Resilient Confidence Alignment to enhance the effect of visual cues in the token prediction. Therefore, our method is applicable to Multimodal Large Language Model (MLLM) training itself but benefits more in the specialization tuning process. We will provide  a detailed multi-task specialization definition in our revised manuscript. Thanks for your valuable suggestions!\n\n**Q2: Language Prior Reliance Visualization Analysis**  (Experimental Designs Or Analyses & Questions For Authors)\n\nWe assume that for Multimodal Large Language Model (MLLM), input tokens include System Tokens, Prompt Tokens and Visual Tokens and appear different contribution for the prediction output. Thus, we utilize the attention map between the first output token and the input token under the noisy visual input to visualize the contribution of different input token.  Besides, recent works [1,2,3] have shown that the MLLM tend to extract object information from the image at the middle layers. Thus, we plot the all layers and middle layers attention allocation for the first output token in the following Table. The results reveal that our method allocates more attention weights on the visual cue. Besides, in Figure 3 (on Page 5), Full Fine-Tuning shows a limited performance gap between noisy and normal visual signals, revealing the reliance on language prior rather than understanding the visual information. Therefore, both the attention allocation analysis and performance comparison demonstrate that our method effectively reduces reliance on language priors and enhances the overall performance of the model.\n\n*Table: Attention allocation and performance comparison with relative methods. We conduct the evaluation on the Flickr30k with noisy visual input. We tuning on the Flickr30k+ScienceQA datasets based on LLaVA architecture.*\n|Metric |Full-FT | Rand Mask | Mag Mask | Grafting | L2-Reg | Ours |\n|----|---|---|---|---|---|---|\n| System   (All) | 0.6837 | 0.6966 | 0.6927 | 0.6673 | 0.6791 | 0.6677 | \n| Prompt   (All)  | 0.1229 | 0.1297 | 0.1280 | 0.1241 | 0.1181 | 0.1225 | \n| **Visual**  (All) | 0.1933 | 0.1737 | 0.1793 | 0.2084 | 0.2029 | **0.2098** | \n| System   (Mid)  | 0.5865 | 0.5874 | 0.5859 | 0.5726 | 0.5826 | 0.5574 |\n| Prompt   (Mid)  | 0.1733 | 0.1843 | 0.1832  |0.1664 | 0.1623 | 0.1576 |\n| **Visual**  (Mid)| 0.2402 | 0.2282 | 0.2309 | 0.2609 | 0.2552 | **0.2850**|\n| Accuracy| 62.25 |57.00 | 60.07|   60.26 | 59.78 | **67.89** |\n\n**Attributed to rebuttal format constraints, we will include attention map in our final version. Thanks for valuable suggestions!**\n\n[1] EAZY: Eliminating Hallucinations in LVLMs by Zeroing out Hallucinatory Image Tokens, arXiv, 2025\n\n[2] From Redundancy to Relevance: Information Flow in LVLMs Across Reasoning Tasks, NAACL, 2025\n\n[3] Cross-modal Information Flow in Multimodal Large Language Models, CVPR, 2025\n\n**Q3: Table Design** (Other Comments Or Suggestions)\n\nThank you for your suggestion. We will reorganize the formulation of Table 2 to make it clear and easy to understand!", "claims_and_evidence": "The following claims are not well supported\n\n`We argue that leveraging MLLM to achieve multi-task specialization is a more efficient approach than the conventional\none-to-one fine-tuning paradigm.`\n\nHow to define the multi-task specialization? The original MLLM training itself is somehow a `multi-task specialization`. Is this method applicable to MLLM training itself?", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "None", "experimental_designs_or_analyses": "Partially.\n\nAlthough the author asserts that NRCA addresses the issue of relying on the language prior, the supporting experiments and analysis unconvincing in demonstrating that the final improvement stems from resolving this problem. Including a visualization could aid readers in better understanding the proposed method and its effectiveness.", "relation_to_broader_scientific_literature": "A common method for reducing hallucinations in multimodal large language models (MLLMs) involves adjusting the next-token logits contrastively based on the standard prediction distribution. This paper primarily distinguishes itself through its unique approach to loss design.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The paper discusses its limitations, which is commendable. It also provides the code, enhancing transparency and reproducibility. Additionally, the design of the NRCA loss function is cleverly crafted, showcasing an innovative approach.\n\nThe limitation comes from the additional forward/backward pass.", "other_comments_or_suggestions": "Table 2 is a bit confusing without checking main text.", "questions_for_authors": "Could you design experiments to confirm that the observed improvement genuinely results from addressing the issue of reliance on the language prior?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OBhIwkC3t8", "reviewer": "Reviewer_ELAr", "summary": "This paper introduces \"Noise Resilient Confidence Alignment\" (NRCA), a method to address overfitting in multi-task fine-tuning of Multimodal Large Language Models (MLLMs). The authors observe that while fine-tuning MLLMs on multiple tasks, performance on open-response tasks (like image captioning) degrades over time, whereas fixed-choice tasks remain stable. They attribute this to MLLMs over-relying on language priors rather than visual information. The proposed solution injects Gaussian noise into visual inputs and enforces alignment between prediction confidence on noisy and clean inputs, thereby enhancing visual representation in the model. Experiments across multiple benchmarks (Flickr30k, COCO-Cap, ScienceQA, IconQA) with different MLLM architectures (VILA, LLaVA) show NRCA consistently outperforms baseline methods and other fine-tuning approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ELAr:\n\nWe sincerely thank you for your valuable feedback and hope our responses adequately address your concerns and merit a score revision.\n\n**Q1: Theoretical Claims of NRCA and Its Connection to Visual Representation**\n\nA1: Denote $I$ and $\\tilde{I}$ as the average confidence over ground-truth tokens under clean and perturbed visual inputs (see Eq 4). Let $\\tilde{z}_t$ and $z_t$ be the predicted logits at position $t$, and let $\\sigma(\\cdot)$ denote the softmax function.\n\nThe confidence alignment objective of $\\mathcal{L}_{\\text{NRCA}}$ is:\n\n$$\n\\min \\left| 1 - \\frac{\\tilde{I}}{I} \\right|\n= \\min \\left| \\frac{1}{T} \\sum_{t=1}^T \\frac{\\sigma _{y_t}(z_t)-\\sigma _{y_t}(\\tilde{z}_t)}{I} \\right|.\n$$\nSince $I$ is detached from the computational graph, the optimization reduces to:\n\n$$\n\\min\\left| \\sum_{t=1}^T \\sigma _{y_t}(z_t)-\\sigma _{y_t}(\\tilde{z}_t) \\right|.\n$$\nAssuming $\\sigma _{y_t}(z_t) \\geq \\sigma _{y_t}(\\tilde{z}_t)$ for all $t$ (a natural assumption since clean inputs typically yield higher confidence), the objective becomes:\n\n$$\n\\min \\sum_{t=1}^T \\left( \\sigma _{y_t}(z_t)-\\sigma _{y_t}(\\tilde{z}_t) \\right).\n$$\nThis encourages the ground-truth token probabilities under perturbed inputs to approach those under clean inputs, i.e.\n$$\n\\sigma _{y_t}(\\tilde{z}_t) \\to \\sigma _{y_t}(z_t).\n$$\nSince $\\tilde{z}_t=g(\\phi(f(\\tilde{x}_v)),h_t)$, the loss encourages the learnable modules ($\\phi$ and $g$) to produce robust visual features that preserve confidence on ground-truth tokens under perturbations, thereby improving visual representation quality.\n\n**Q2: Theoretical Connection Between Language Priors and Open-Response** \n\nA2: We consider token-level prediction in autoregressive multimodal generation, where the model learns the probability of a target sequence $y=(y_1, \\dots, y_T)$ given a visual input $x^v$ and textual prompt $x^t$. The objective is to minimize the negative log-likelihood $L_{\\text{CE}}.$\n\nIn open-response tasks such as image captioning (eg Flickr), the output is a multi-token sentence of length $T=k$, with autoregressive factorization:\n$$\np(y \\mid x^v, x^t)=\\prod_{t=1}^k p(y_t \\mid y_{\\lt t}, x^v, x^t).\n$$\nDue to the richness and uniqueness of each output, the model may overfit by memorizing input-output mappings and relying on linguistic priors in $y_{\\lt t}$ rather than visual cues.\n\nIn contrast, fixed-choice tasks (e.g., IconQA) involve single-token outputs ($T=1$) from a small label set:\n$$\np(y \\mid x^v, x^t)=p(y_1 \\mid x^v, x^t),\n$$\nwhere prefix conditioning is absent and language priors are less effective. The model must therefore rely more on visual input, reducing overfitting.\n\nThis analysis complements Appendix C and the observations in Fig 6, further clarifying the connection between language priors and overfitting in open-ended tasks.\n\n**Q3: Essential References Discussion**\n\nA3: Prior work on **modality bias** shows that MLLMs often over-rely on textual priors, causing visually misaligned hallucinations[1]. Contrastive decoding compares token distributions across views but adds inference cost[2]. Our method aligns token-level confidence during fine-tuning to improve understanding.\n\n\nFor **vision-language alignment**, existing methods either build diverse instruction data or use reinforcement learning with designed reward signals, both requiring costly annotation or tuning[3]. Our method avoids these overheads and integrates directly into standard fine-tuning.\n\n[1] Multi-modal hallucination control by visual information grounding,CVPR2024\n\n[2] Mitigating Object Hallucinations in LVLMs through Visual Contrastive Decoding,CVPR2024\n\n[3] Visual-RFT: Visual Reinforcement Fine-Tuning,arXiv2025\n\n**Q4: Random Mask Tuning Effectiveness Discussion** \n\nA4: Random Mask Tuning applies stochastic masks to parameters, injecting noise to reduce reliance on specific subnetworks and implicitly regularize by limiting parameter co-adaptation. While effective on smaller models (VILA3B), its performance drops on larger ones (LLaVA7B), as shown in Tab 4, suggesting its effect comes from constrained optimization rather than addressing language prior dominance. In contrast, our method perturbs visual inputs and aligns token-level confidence with the clean visual branch, enhancing visual grounding and ensuring consistent performance across model scales. We will include a detailed analysis in the manuscript.\n\n**Q5: Visual Attention Patterns** \n\nA5: Due to limited space, please refer to **Q2 in Response to Reviewer H4RG**.\n\n**Q6: Fixed-Choice Tasks Performance**\n\nA6: The phenomenon of modest gain in fix-choice task is also present in several counterparts designed to alleviate overfitting. Though some of them perform marginally better on fixed-choice tasks, they tend to incur trade-offs and underperform on open-response tasks. By enhancing visual robustness through confidence alignment, our method achieves a better trade-off and consistently surpasses all baselines in overall multi-task performance.", "claims_and_evidence": "The paper's primary claim that open-response tasks degrade during multi-task fine-tuning is well-supported by empirical evidence in Figure 1, showing performance drops on Flickr30k while ScienceQA remains stable. The claim that NRCA alleviates this issue is supported by extensive experiments across different datasets, model architectures, and fine-tuning parameters.\nHowever, the claim that language priors are the root cause of this degradation is less convincingly established. While the authors observe that models produce similar outputs for normal and noisy images (Figure 3), this could have other explanations besides language prior reliance. The connection between confidence alignment and reduced language prior dependency lacks a strong theoretical foundation.\nThe empirical results are strong and consistent across various settings\n\nUpdate: the last part was clarified during the rebuttal period.", "methods_and_evaluation_criteria": "The proposed method is well-motivated from the problem observation and appropriate for addressing multi-task fine-tuning challenges. The evaluation across multiple benchmarks like COCO-CAP and IconQA, and , with both open-response and fixed-choice tasks, provides a comprehensive assessment of the approach.", "theoretical_claims": "The paper doesn't present formal theoretical proofs but offers explanations for why NRCA should work. The rationale connecting confidence alignment to improved visual representation is reasonable but would benefit from a more rigorous formulation.\n\nUpdate: The author rebuttal addressed my concern.", "experimental_designs_or_analyses": "The experimental design is sound and comprehensive. The authors test across:\n\nMultiple MLLM architectures (VILA, LLaVA)\nDifferent model sizes (3B, 7B)\nVarious fine-tuning epochs (E=3, E=5)\nDifferent batch sizes (B=16, B=24, B=32)\nMultiple benchmark datasets\n\nThe ablation studies isolate contributions of key components. The authors also demonstrate robustness to hyperparameter settings and analyze the method's resource efficiency compared to alternatives.", "relation_to_broader_scientific_literature": "The work builds effectively on existing literature in MLLM fine-tuning, particularly addressing overfitting challenges. The authors appropriately contextualize their work within related methods like parameter-efficient fine-tuning and regularization approaches.", "essential_references_not_discussed": "Some important references that might improve the paper's contextualization:\n\nLiterature on modality bias in multimodal models that could provide alternative explanations for their observations\n\nOther recent work specifically on vision-language alignment during fine-tuning that could offer complementary insights on overfitting prevention\n\nUpdate: the author responded during the rebuttal period and cited meaningful comparable literature in the field.", "other_strengths_and_weaknesses": "Strengths:\n\nNovel identification of the open-response overfitting problem in multi-task MLLM fine-tuning\nSimple yet effective confidence alignment approach that's architecture-agnostic\nComprehensive empirical evaluation across diverse settings\nPractical resource efficiency compared to alternative methods\n\nWeaknesses:\n\nThe theoretical connection between language priors and open-response overfitting needs stronger justification\nSurprising effectiveness of random mask tuning (which achieves competitive results) isn't thoroughly analyzed", "other_comments_or_suggestions": "- Consider expanding the analysis of why random mask tuning performs so well\nThe paper would benefit from more explicit evaluation of visual attention patterns before and after applying NRCA", "questions_for_authors": "How do you explain the strong performance of random mask tuning as it relates to language prior issue with open-ended q-a geneartion? Does this suggest the problem might not be specifically about language priors but perhaps more general optimization challenges?\n\nThe gains on fixed-choice tasks are sometimes modest or even negative. Could you explain why enforcing confidence alignment might occasionally harm performance on these tasks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pXG8rD8N9B", "reviewer": "Reviewer_SJh5", "summary": "In this work, the author focuses on Multimodal Large Language Models tuning. Specifically, the authors propose a method called Noise Resilient Confidence Alignment (NRCA) that aims to alleviate the issue of overfitting, particularly in open-response tasks during multi-tasks tuning. approach emphasizes maintaining consistent prediction patterns in MLLMs by enhancing the integration of visual cues alongside language priors. Through experimental evaluations, they show that their method improves model robustness by aligning token prediction confidence towards the normal visual branch and reduces the over-reliance on language priors. The authors conduct extensive experiments on popular model architectures, validating the effectiveness of their approach across various multi-task downstream settings, including tasks like image captioning and visual question answering .", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer SJh5:\n\nThank you for your thoughtful review and for raising key concerns regarding our work. We aim to address your concerns in our detailed responses below, hoping to provide clarity and demonstrate the effectiveness of our proposed approach.\n\n**Q1: Conceptual Discussion** (Other Strengths And Weaknesses)\n\nA1: In the context of Multimodal Large Language Model (MLLM) tuning on downstream tasks, empirical evidence shows a tendency toward overfitting on the target distribution. This is primarily due to the mismatch between the model large-scale capacity and the limited size of domain-specific samples. Existing investigations either introduce stiff penalty regularization terms or design partial update masks. However, such approaches typically focus on single-task adaptation and apply uniform anti-overfitting objectives across all examples, without accounting for task-specific differences. Under multi-task settings, different task types often exhibit inconsistent fitting behaviors. Our analysis reveals that open-response tasks are particularly prone to overfitting, largely due to their heightened dependence on textual modalities. To address this, we propose a novel approach that mitigates overfitting on the target distribution by enforcing confidence alignment from noisy visual inputs to their corresponding clean visual patterns. We will provide a detailed conceptual explanation for the proposed approach in our revised manuscript to highlight our novelty. Thanks for your valuable suggestions!\n\n**Q2: Figure 3  Explanation** (Other Strengths And Weaknesses)\n\nA2: Figure 3 illustrates the performance under both normal and noisy visual input for different methods. Existing methods show minimal performance gap regarding visual quality, which reflects the restricted affect from the visual branch and presents a high reliance on the textual prior information to generate the corresponding output response. We will provide a detailed explanation for Figure 3 in our revised manuscript. Thank you for your helpful suggestion.\n\n\n**Q3: Dataset Construction** (Other Strengths And Weaknesses)\n\nA3: For fixed-choice tasks (ScienceQA and IconQA), we use the textual prompt: “Answer with the option’s letter from the given choices directly.” For open-response tasks (Flickr30k and COCO-Cap), we collect five manually written instructions and randomly sample one as the prompt for each caption, as follows:\n- \"Describe the image as simply as possible with a sentence or phrase\"\n- \"Give a brief summary of what you see\"\n- \"Provide a short description of the image\"\n- \"Write a short description for the image\"\n- \"Briefly describe the content of the image\"\n\nWe will provide a clear description of the dataset construction procedure in our revised manuscript. We will release the corresponding training and test JSON files in the final version. Thank you for your valuable suggestions!", "claims_and_evidence": "Yes. The claims and evidence are clear and convincing.", "methods_and_evaluation_criteria": "The NRCA method is designed to alleviate open-response overfitting during MLLM fine-tuning by encouraging prediction consistency between distorted and normal visual patterns. CIDER and Top-1 Accuracy are used to evaluate performance on downstream tasks. Comparisons with baseline methods and ablation studies further validate NRCA’s effectiveness.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "Yes. The authors use common evaluation metrics such as CIDER and Top-1 Accuracy, depending on the task. They perform experiments on popular Multimodal Large Language Models (LLaVA and VILA) using datasets like Flickr30k, COCO-Cap, ScienceQA, and IconQA.", "relation_to_broader_scientific_literature": "The author introduces an interesting problem that during multi-task tuning, open-response tasks exhibit overfitting behavior, a phenomenon that has not been thoroughly explored in existing research.  Proposed method consider encourage prediction consistency to alleviate the reliance on language prior.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Pros: \n\n-\tThe paper introduces an innovative problem of overfitting in open-response tasks during multi-task fine-tuning, a gap not previously explored in existing research. This adds a valuable contribution to the field.\n-\tThe paper includes a thorough set of experiments across a range of datasets and downstream tasks, such as image captioning and visual question answering, demonstrating the effectiveness of NRCA. The use of popular benchmark datasets like Flickr30k and COCO-Cap strengthens the reliability and generalizability of the results.\n-\tThe NRCA method is a unique approach to controlling overfitting. By encouraging prediction consistency between distorted and normal visual patterns, NRCA offers a more interpretable mechanism for preventing overfitting, compared to traditional fine-tuning methods.\n-\tThe paper includes a notation table and algorithm description in the supplementary material, making it easier for other researchers to reproduce the results and understand the underlying methodology.\n\nCons:\n\n-\tConceptual Discussion: The authors should further claim the different for your methods with existing methods. Why they can not solve the problem mentioned in the paper to highlight the contribution.\n-\tThe explanation for Figure 3 is unclear. Does it means your method alleviat the reliance on language prior?\n-\tDataset Construction. As for the MLLM tasks, the authors should report the textual prompt for different tasks for better illustration.", "other_comments_or_suggestions": "Refer to weakness.", "questions_for_authors": "Please refer to Other Strengths And Weaknesses parts.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sWNvjtZkwp", "reviewer": "Reviewer_XwCL", "summary": "This paper proposes Noise Resilient Confidence Alignment (NRCA) to reduce overfitting in open-response tasks during multi-task fine-tuning of Multimodal Large Language Models NRCA enhances performance on tasks like image captioning and visual question answering, outperforming traditional fine-tuning methods. Experiments and comparisons validate its effectiveness and compatibility with other optimization strategies.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer XwCL:\n\nThank you very much for your valuable comments and constructive feedback. Below, we carefully address each of your concerns point-by-point, providing detailed explanations and additional evidence to clarify our approach and validate its effectiveness.\n\n**Q1: Limitation Discussion** (Other Strengths And Weaknesses & Questions For Authors)\n\nA1: Open-response and fixed-choice tasks reflect two complementary aspects of Multimodal Large Language Model (MLLM) capabilities: the ability to generate free-form responses to open-ended questions, and the ability to select correct answers from predefined options. These two task types are widely recognized and are also included in the pre-training datasets construction. Solely tuning on fixed-choice tasks would degrade into the single task optimization and limits model application scope. In contrast, open-response tasks are crucial for unlocking MLLM ability to generate free-form expressions, which go beyond the limitations of traditional classification. We have provided a discussion in Page 5. We will provide a detailed limitation discussion in our revised manuscript. Thanks for your valuable suggestions!\n\n**Q2: Typos Errors** (Other Strengths And Weaknesses & Questions For Authors)\n\nA2: Thanks for your tips. We will fix the typos and check grammar and markup issues in our revised manuscript！\n\n**Q3: Rationale for Normal Branch Detach** (Other Strengths And Weaknesses & Questions For Authors)\n\nA3: In Eq. 5, we aim to enhance the visual behavioral effect to improve prediction robustness and reduce reliance on language priors. To achieve this, we encourage the noisy prediction confidence $\\tilde{\\mathcal{I}}$ to align with the normal prediction confidence $\\mathcal{I}$, enforcing a one-way alignment constraint. Therefore, we detach the normal branch to prevent it from being influenced by the noisy branch during optimization. We will provide a detailed rationale discussion in our paper. Thanks.\n\n**Q4: Clear Keyword Definition** (Other Comments Or Suggestions)\n\nA4: Open-Response task means the task of generating a response to a given question such as image captioning dataset. Fixed-Choice task denotes the task of selecting one of several options from the candidate list, e.g., visual question-answering dataset. We will clarify the definition in our revised manuscript. Thanks for your suggestion!", "claims_and_evidence": "The claims are well-supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The authors respectively measure the multi downstream tasks performance and further utilize the average performance to evaluate the overall performance. Authors conducts experiments on four downstream datasets into two task types: image captioning (open-response) and visual question-answering (fixed-choice) to formulate the multi-task setting.", "theoretical_claims": "The authors analyze token prediction confidence in Eq.7 and argue that open-response tasks tend to reach overly confident predictions due to an overreliance on inherent memorization.", "experimental_designs_or_analyses": "The experimental results are thorough and well-rounded, as the authors conduct a variety of experiments comparing different methods across multiple downstream tasks.", "relation_to_broader_scientific_literature": "This paper presents a simple yet effective approach to alleviate the overfitting issue of the open-response task during the multi-task tuning process. It addresses a novel research problem within the field of MLLM fine-tuning.", "essential_references_not_discussed": "Authors could add the discussion with the multi-task learning which shares the similar research objective with this paper.", "other_strengths_and_weaknesses": "For strengths:\n1.\tNovel Problem Focus: Identifies the under - explored issue of open - response overfitting in MLLM multi - task tuning, which is crucial as MLLMs are increasingly used for diverse tasks.\n2.\tEffective Solution: Proposes NRCA, a method that uses Gaussian - perturbed visual inputs and confidence alignment. It effectively reduces overfitting and improves performance, as shown by experiments on multiple datasets and MLLM architectures.\n3.\tComprehensive Testing: Conducts wide-ranging experiments with two MLLM architectures LLaVA and VILA and four downstream datasets across open-response and fixed-choice tasks. Ablation studies further analyze the method's components.\n4.\tProposed method is relatively resource-efficient. It doesn't need to store pre - trained MLLM weights, reducing computational burden. Furthermore, the method's hyper-parameters work well across different MLLM models\n\nFor weakness:\n1.\tApplication Limitation NRCA is designed for alleviate open-response overfitting and when there are only the fixed-choice tasks, its effectiveness is less obvious, restricting its universal application. It encourages the further discussion on limitation.\n2.\tTypos Errors. In Table 3, not retaining two decimal places. Please carefully check for grammar and markup issues\n3.\tThe rationale for Eq.5 to detach the normal branch lacks the explanation. The authors should discuss the reason behind it.", "other_comments_or_suggestions": "The paper is written in a complicated manner with many key words such as open-response and fixed choice. Provide the clear definition would be better for paper readability.", "questions_for_authors": "The authors should add more detailed limitation discussion and check the typo error. The rationale behind the proposed operation should add relative discussion.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses the increasingly relevant issue of overfitting in open-response tasks during the multi-task fine-tuning of Multimodal Large Language Models (MLLMs). This work proposes the “Noise Resilient Confidence Alignment” (NRCA) method, which augments the visual branch through Gaussian perturbations and enforces token-level confidence alignment between noisy and normal visual inputs. The central claim is that this approach mitigates excessive reliance on language priors, thereby enhancing the robustness of MLLM predictions. Extensive experiments on diverse tasks (image captioning vs. visual question-answering) and multiple MLLM architectures provide empirical evidence supporting the method’s effectiveness. Based on the positive evaluations and the authors’ rebuttal addressing most concerns, the meta-review recommends acceptance of this submission.
Decision: Accept (poster)