Language‑Bias‑Resilient Visual Question Answering via Adaptive Multi‑Margin Collaborative Debiasing
Authors: Huanjia Zhu
Shuyuan Zheng
Yishu Liu
Sudong Cai
Bingzhi Chen
Venue: NeurIPS 2025 poster ()
Keywords: Language biases
Multi-margin
Debiasing
Frequency-aware
Confidence-aware
Difficulty-aware
Primary area: applications
TL;DR: Investigate how language bias originates, why margin mechanisms are effective, and propose a novel Multi-Margin Collaborative Debiasing (MMCD) framework

Abstract: Language bias in Visual Question Answering (VQA) arises when models exploit spurious statistical correlations between question templates and answers, particularly in out-of-distribution scenarios, thereby neglecting essential visual cues and compromising genuine multimodal reasoning. Despite numerous efforts to enhance the robustness of VQA models, a principled understanding of how such bias originates and influences model behavior remains underdeveloped. In this paper, we address this gap through a comprehensive empirical and theoretical analysis, revealing that modality-specific gradient imbalances, which originate from the inherent heterogeneity of multimodal data, lead to skewed feature fusion and biased classifier weights. To alleviate these issues, we propose a novel Multi-Margin Collaborative Debiasing (MMCD) framework that adaptively integrates frequency-, confidence-, and difficulty-aware angular margins with a dynamic difficulty-aware contrastive learning mechanism, to dynamically reshape decision boundaries. Extensive experiments across multiple challenging VQA benchmarks confirm the consistent superiority of our proposed MMCD over state-of-the-art baselines in combating language bias.

Language-Bias-Resillen Visual Question Answering via Adaptive Multi-Margin Collaborative Debiasing
Huanjia Zhu
Shuyuan Zheng The University of Osaka zheng0ist. osaka-u. ac. jp
Beijing Institute of Technology, Zhuhai bvyih3@gmail. com
Yishu Liu
Sudong Cai*
Harbin Institute of Technology, Shenzhen liuyishu@stu, ext..ddu.cmm
Beijing Institute of Technology. Zhuhai caisudong ai ai@gmail. com
Bingzhi Chen
Beijing Institute of Technology. Zhuhai chenbingzhi @bitt edu. cn
Abstract
Language bias in Visual Question Answering (VQA) arises when models exploit spurious statistical correlations between question templates and answers, partic- ularly in out-of-distribution scenarios, thereby neglecting essential visual cues and compromising genuine multimodal reasoning. Despite numerous efforts to enhance the robustness of VQA models, a principled understanding of how such bias originates and influences model behavior remains underdeveloped. In this paper, address this gap through comprehensive empirical and theoretical analysis, revealing that modality gradient imbalances, which originate from the inherent heterogeneity of multimodal data, lead to skewed feature fusion and biased classifier weights. To alleviate these issues, we propose a novel Multi- Margin Collaborative Debiasing (MMCD) framework which adaptively integrates frequency- confidence-awand aware. and difficulty-awar angular margins with dynamic, difficulty- contrastive learning mechanism to reshape decision boundaries under biased training conditions. Extensive experiments across multiple challenging VQA benchmarks confirm the consistent superiority of our proposed MMCD over state-of-the-an baselines in combating language bias.
Introduction
Visual Question Answering (VQA) has emerged as a challenging task that blends computer vision and natural language processing to provide answers to natural language questions about images. The core difficulty of VQA models lies in their ability to reason multimodally, combining visual information from images with language patterns in questions. Recent progress [36
38 in deep learning has enhanced the capabilities of VQA models. However, studies have shown that networks
29 42
still suffer from language bias [15 6 9 17 4 29 36). where the model learns spurious correlations between questions and answers. This bias occurs when models overly rely on common patterns in questions and answers, neglecting crucial visual information. These models often perform well
"Corresponding authors: Bingzhi Chen and Sudong Cai. "Code available at httt:://gitubb..ommbby///////// -IP-:0022-IPPMMMMMMMM
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

0.78
Value
Similaity
405
Updta 4
600
UpDas
0.14
MMCCD
605
15
45
Epodb
Epochs
(d) Figure 1: (a) The gradient norms across modalities differ considerably, reflecting an imbalance in how learning signals are propagated. (b) This imbalance leads to pondality-ppccinng optimization deviations, where the question modality of baseline disproportionately accumulates gradient updates, thereby amplifying its influence. (c) As a result, the fused representation becomes skewed, with
(a)
(b)
(c)
question features occupying dominant share of the multimodal space and suppressing contributions from visual features. (d) Furthermore, the classifier weights reveal directional bias: the singular value spectrum is highly uneven, suggesting that the model primarily aligns with directions that capture biased cues while overlooking secondary directions that encode meaningful information.
in standard in-distribution (ID) evaluation settings 15 but fail to generalize to ofitr--disttibuttin (OOD) data [ 13 21]. where the distribution of answers differs from the training set.
To address this challenge. existing approaches can be grouped into three principal paradigms: ensemble-based [32
45], and feature 47 methods. Ensemble- mnseeble-based approaches [32 6 12 augment the base model
28 T7
T8
29 36). augmentation-basse
9 25
space-based [16
with auxiliary bias estimators, such as question-om only branches, to identify and subtract spurious corre-
45] or inject negative via heuristic rules rebalance answer distributions. Feature space-based methods impose angular margins [16 4 47 on the hyperspherical embedding to enforce class separability and
lations. Augmentation-baan based techniques synthesize counterfactual data [9] 25
samples [5 4
suppress dominant priors. Despite notable progress in bias reduction question remains unexplored: How are language biases formed?
29 36 a fundamental
Understanding the genesis of language bias is essential for advancing debiasing techniques. In this work, we conduct comprehensive investigation into bias formation. First, we identify modality gradient optimization deviation (see Fig. [(b] Section where the image modality is under-optimized and the question modality is over-optimized. Second, we observe a feature fusion component deviation (see Fig. [(e], Section 2.2). in which question features dominate the joint representation and image features are marginalized. These phenomena culminate in directional deviation of the classifier weights (see Fig. [(d] Section 2.2). amplifying primary (question-driven) directions and attenuating secondary (vision-driven) axes.
To mitigate language bias, we examine the efficacy marginn-aseed objectives, which refine model training by manipulating the angle between normalized feature and classifier weight vectors on the unit hypersphere. Despite their empirical success across diverse domains (5) 14 27
37
g 22 (4
16 4 47]. fundamental question remains unanswered: Why does margin mechanisms work?
Grounded in our analysis of bias, we show that margin penalties effectively balance modality gradient optimization (see Section 3. homogenize fused feature components (see Section 3.2), and equalize classifier weight directions (see Section 3.3) counteracting the skew induced by data heterogeneity.
Motivated by these findings, we propose an adaptive Multi- Multi-Margin Collaborative Debiasing paradigm called "MMCD", which aims to reshape discriminative class boundaries. Specifically, the proposed MMCD incorporates two well-established mechanisms, i.e., Multi-grained Adaptive Margins (MAM) and Difficulty Contrastive Learning (DCL). Specifically, MAM augments class discriminabil- ity through three adaptive angular penalties: trequency-aware confidence-aware, and difficulty-aware margins. 1) Frequency-awarr margins counteract class imbalance by assigning larger angular offsets to underrepresented answers. 2) Confidence-awai aware margins leverage logits as a proxy for sample uncertainty, dynamically tightening boundaries around ambiguous instances. 3) Difficulty-wwan margins employ an instance difficulty estimator constructed from per-sample learning speed and classification margins to further calibrate penalties at the sample level. Furthermore, DCL empowers the network to facilitate intra-class compactness and inter-class separation by incorporating the introduced difficulty model into the supervised contrastive framework. To the best of our knowledge, this is the first attempt to investigate the formation mechanisms of language bias in VQA.

---

2 How are Language Biases Formed?
Robust VQA has attracted extensive research interest in recent years, yielding rich suite of debiasing techniques. Yet, the origin of language bias itself remains uncharted. In this section, we undertake systematic investigation, combining empirical measurements with theoretical insights trace the full trajectory by which spurious question-answer priors become entrenched in VQA models.
To begin with, we first establish some task-specific preliminaries. Given image a question q E Q. the goal of VQA is to predict an answer a E A by optimizing a mapping f V x Q RC, where f(v, q) is the logits for candidate answers, C |-A] is the number of candidate answers. Standard VQA frameworks [17] 4 47 typically adopt a four-stage pipeline: (1) image encoder Ce. (2) question encoder <g. (3) multimodal fusion module y which fuses unimodal features to generate joint representations R. (4) classifier e which maps R to answer logits with learnable weights W. The VQA problem is formulated as:
V and
f(e, q) = ((g(ee(v),
(I)
Cross-Entropy Loss.
The VQA model is trained by minimizing the Cross-Entropy (CE) loss:
IAI
exp(Ji)
LCE
ag log
(2)
j== 1 exp(ff)
where is the logit for the i-th answer. We adopt UpDn [2] as our baseline. The feature fusion strategy g is the Hadamard product o thus R R2 @ Re, where Ry and R, are question features and image features, respectively.
2.1 Modality Gradient Optimization Deviation
Generally, models store the critical information for unimodal features captured from training data in their encoder weights. We denote the weights of ey and e, Wy and W.. respectively. We calculate the gradient of encoder weights with respect to the loss L:
OL OR RRg dL V Vw, L OR 8Rg OW4
dL JR OR. OC
= aR RR) ETy O..).ET
Vw.C=
= о R.y) E.
ÖR aR, ow. OR
where Fig and F, is the question embeddings and image embeddings, separately. The corresponding Frobenius norms are:
Oc
Oc 0R Roll2
Oc
Oc 3 11
|F
- 12.
aw. || =1
(4)
aw,
After experimental exploration, we found that Eylla ||Eøll2 and UR Roll2 110h + Rgy 12 _BC
which yields || aw, || aw. (see Fig. ((b).. In other words, modalities gradient optimization deviation arises from inherent data heterogeneity. Specifically, question tokens are encoded into 300-dimensional GloVe embeddings (31] and aggregated via single-layer GRU (T]. whereas visual inputs are represented by 36 object vectors of dimension 2048 extracted by a Faster R-CNN (33 Consequently. E, < |lEll 22 and S. O R. >> properties of the data is the culprit behind the imbalance in modal optimization.
&C O R2 This difference in the inherent TEE
Crucially, gradient deviation can not directly explain language bias, since the final predictions depend on fused representations R and classifier weights W. To uncover the true bias formation, we next analyze (I) the proportion of unimodal components in R and (2) the directional deviation of W.
2.2 Feature Fusion Bias and Classifier Deviation
Fusion feature component deviation. In the previous subsection, we identified the modality gradi- ent optimization deviation. Then we naturally ponder the question: Which modality predominates the fusion feature? Since the loss is essentially computed using fusion feature, this composition deviation determines the model's reliance visual versus textual cues prediction time (26).
Inspired by [19, given R, Ry. and R., we perform Singular Value Decomposition (SVD) and obtain the right-singular unitary matrices VR. VR,, and VR.- We assess the similarities of the subspace
3

---

spanned by the top-i singular vectors in VR and that of the top-j singular vectors of VR, and VR... respectively. We compute the normalized subspace similarity based on the Grassmann distance as:
|R 1 VasjT R,/RR.
(VR. VR,/Ro. 2.j)
E [0,
(5)
min(i.j)
Here, vb(-) ranges from 0 to 1, where indicates a complete overlap of subspaces and O signifies total separation. V and V represents the top-i and top-j column vectors of VR and /R./R.. RR/R,
respectively. As shown in Fig. [(c] we observed an important phenomenon: At the start of training, the question-subspace similarity is substantially lower than that of the image subspace, but by the mid- and late-training stages it exceeds the image-subspace similarity. These findings reveal that the fused features are disproportionately governed by the question features, with visual features playing a subordinate role. Such composition deviation causes the model to assign uneven importance to modality-speciff information when learning from fused features.
Classifier weight direction deviation. In general, the model stores key information about the classification task captured from the training data in its final classification weights. We perform SVD on W to extract corresponding singular values E. As depicted in Fig. [[d] the singular values appear significantly different. The top singular value greatly exceeds the remaining values, and the bottom singular value is almost O. This implies that the classifier predominantly captures feature correlations along a primary axis, neglecting orthogonal (secondary) directions that may encode complementary information. To elucidate which features are emphasized, we consider the gradient of Lee for W:
WwLCE (p R)
(6)
where
exp(ft 2 exp(fy) Here, the update for the correct class aligns exactly with the fused representation R. Since R is dominated by the question component (see above), the weight updates encode question-driven priors, while visual-semantic cues in secondary directions are largely ignored.
Why Does the Margin Mechanism Work?
Normalized CE loss for hypersphere embedding. To optimize the instance representation space, 47 project features onto a unit hypersphere by T.2- normalizing classifier weights W and joint representations R. In light of this, the posterior probability is determined by the angle 6 between W; and R2,, and the answer feature space converted from the Euclidean space to the angular space. The logit f; for each representation R, is redefined as:
prior works 27 16
f. W. R; ||] R4 ||s cos 0; cos fg
where |M&I| = 1, |R, = 1, is scaling factor for more stable computation. The bias term is viewed as zero for simplicity. Thus, the joint representations R are distributed on a hypersphere with radius The standard CE loss is transformed into normalized CE loss:
LAI
exp(s cos 0:)
LNCE Σ log
(8)
Σ
|AI Σ}=
exp(s cos 0,)
Rigorously, we theoretically analyze the rationale for spherical space learning and demonstrate its advantages in the supplementary materials. Recent studies 4 47 have focused on optimizing the instance spacing in inverse cosine space by adding a margin m to the clamp angle 0:
IA
exp(s cos(O: O. + m2))
LMARGIN
a; log
Σ A
exp(s cos(f; (8, + m2))
3.1 Modality Gradient Optimization Balance
We analyze how the margin term modulates encoder gradients by comparing the gradients of Lce and LMARGIN concerning the fused feature R:
VRLcE = WT (p a),
(10)
V R LMAAGG = sW ((p' - a. 3 C), where C = cos m + cot 0 sin m,
(11)

---

where p' ex
E exply)) cos(O; m.). The margin mechanism thus (1) introduces the scale
to amplify gradient magnitude, (2) perturbs the predicted probabilities y' to reweight class-wise errors, and (3) applies an adaptive coefficient C' along each logit dimension. As each margin my and angle 0, are independently specified, C differentially scales the alignment of VR with the question subvector Rg versus the visual subvector R. Consequently, the margin term rebalances modality-specifte gradient contributions and reduces the gradient deviation (see Fig. [(b)).
3.2 Fusion Feature Component Uniformity
Fig. [[c] illustrates that the question-modalit contribution within the fused representation grows sharply and ultimately surpasses the image-modalin contribution, resulting from disproportionate gradient updates favoring the question stream. In the previous subsection, demonstrate that the margin mechanism counteracts this imbalance by harmonizing modality gradient magnitudes, thereby producing more uniform multimodal feature composition.
3.3 Classifier Weight Direction Equalization
The margin m mitigates directional bias in W by promoting a more uniform singular spectrum. The gradient of Vw LMARGIN with respect to W is:
W s((p'< a) C)R.
(12)
The first term of C, cos m [-1, IJ, uniformly scales updates along R. attenuating the dominant (question-driven) component more strongly because of its larger magnitude. We then consider the second term of C, cot 0-sin m. Defining the unit vector W, [[. to W, we geometrically decompose R. into W and L4:
w and 21 as unit vector orthogonal
R = (WT RW) + (R (WTRR)) = cos 0 W + sin U.
(13)
The second term of C, cot sin m yields:
cot 0 sin R cos? 0
sin W + cos 0 sin U.
(14)
sin 0
Here, sin B sin m W balances the principal component adaptively, while cos 0- sin m U injects corrective push into the orthogonal subspace. As a result, primary singular values decrease and secondary singular values increase, equalizing the classifier's learned directions and enhancing its ability to capture complementary cues. We provide a more rigorous theoretical proof in the supplementary materials.
cos"
Methodology
4.1 Multi-Grained Adaptive Margins
Inspired by margin learning [6 4. our MAM mechanism aims to address the challenge of chaotic class boundaries posed by imbalanced data. By considering answer frequency and evaluating instance difficulty from coarse-grained and fine-grained perspectives, MAM enhances intra-class compactness and inter-class separation, thus refining discriminative and robust feature space. Specifically, MAM integrates three components: frequency-aware, confidence-aware, and difficulty- awwaee margins.
Frequency-aware margins. As mentioned in [6 imposing larger margin penalties on minority classes is crucial for driving their representations closer the respective class centers. Conversely, majority classes, which naturally have robust representation, benefit from smaller margin penalties. Similar to 16 4 the frequency-aware margins are defined as:
n2. +
gt m
(15)
Σイ
where mn! the frequency of answer a; with question type ql. n, the occurrence of answer a; with gt. e is a hyperparameter for avoiding computational overflow. Elasticface [5] demonstrates that
5

---

Prequency- based
0
m
02
MAM
Margins
Woar gin
Mogoriy Cless
Minonty Cless
Image Encoder
Adaptivve Maarins
Majurtty Class
Minucit
Classifier Cy
Malli yruut
WA
Antwerciees W6R8".v)
S
B
Ry
Nose
Sesseaaann Adkettive M
Confideuce
MM Margins
Pusion Module
Clossitie
MMCD
Hypersperre Space
Prediction Distciraton
Lce
Lpc
Contrastive Leaming
Dilicully dMAEE
Questian:
Rqg
Clessifiar
Diffiicllty based
Margins
How many waves are in the picture?
Spadd
Question Encoder
Compress
Duci
Samples/Hard samples of class B with predicted class
Compress
Class center
sumples
O Hard sumples
Class boundary
Figure 2: Illustration of our MMCD for combating language bias. Up: MaatiiGrrained Adaptive Margins rectify instance to reshape robust class boundaries. Bottom: Difficult-and aeffirult-aware aware Contrastive Learning improves intra-class compactness and inter-class separation, carving discriminative feature space. Right: Multi-Grained Adaptive Margins are driven by frequency and instance difficulty.
fixed margins fail to adapt to the dynamic inter- and intra-class variances in real-world data, thereby impairing the model's discriminability and generalization. Following [4, we incorporate random Gaussian noise into frequency-awer aware margins m freqi
m
((n o)),
(16)
where N is the Gaussian distribution with standard deviation a, and a a hyperparameter.
Confidence-awarc margins. As mentioned above, sample difficulty significantly affects the dis- criminative decision margin and class separability. A simple yet effective measure of difficulty the prediction logits. We incorporate auxiliary branches dedicated solely the question and image modal- ities. These branches promote multimodal integration by deliberately introducing controlled modality bias. This strategy not only boosts ID performance but also prevents excessive bias correction.
Specifically, we introduce a question-only classifier og and an image-only classifier Cp. The corre- sponding logits fg and fo are:
fa(a) - (e)()))),
- ccoo)))(
(17)
Recognizing the challenges of imbalanced multimodal learning [39 40 inspired by 30) we leverage
the posterior distributions 8q and 8, as the weights for unimodal logits Jg and fv, respectively.
Sy = softmax(W, (q) + 2) [gt].
softmax (v) + 2 (ot].
(18)
where W2 and W. are the parameters of cy and c respectively, gt refers to the index of the ground truth class of the sample. We denote , f(v, q) as the weight applied to the multimodal logits f. and Ti as the temperature, which is hyperparameter. Based on these definitions, the weighted hybrid confidence f. and confidence-wwand margins moonf are formulated follows:
fm 81 8..f+ Sq
+
Meonf cootmax(ffm/ fm/TL),
(19)
85 +
Bq
+ 8
Difficulty-aware margins. Although logits can simply and intuitively reflect sample difficulty, their static nature limits the fine-grained mining of intrinsic sample difficulty and makes difficult to modulate stubborn decision boundaries. Inspired by [4], we develop a fine-graing grained difficulty

---

model that evaluates instance difficulty from two perspectives: a) Learning Rate: akin to human learning, where easy samples are learned quickly, and b) Classification Margins: reflecting relative confidence, where smaller margins indicate closer proximity to the decision boundary. Specifically, given an instance representation Ri, its difficulty Di,t after t iterations is estimated as:
0u4,1 +
mu4, +
Di,
(1 a1- mli,t +
(20)
04e +
learning speed
clasiilication margins
where vu;, and mug, denote the prediction variation and marginal gap on the unlearning direction after / iterations respectively, vli, and ml,,; denote the prediction variation and marginal gap on the learning direction separately, a is a hyperparameter to balance the contribution of learning speed and classification margins. e prior parameter control the sensitivity of Da. for the variation of predictions and prevent division by zero. A larger Da, indicates that the instance is difficult to learn.
We apply Shannnnn (JS) divergence to quantify learning speed. Specifically, denote Pi, as 1 iteration. The distance Vi, the prediction distribution of instance R2 at t iteration and P at between Pi,-1 and pi, is defined as:
(P( + (p.rllllga.)
(21)
2
where Us (Pt,1-1 + Pt,) KL is Kullback-Leibler (KL. divergence. We denote p/,, as the probability of class j of R, at the / iteration. Obviously, p.. V < or 1i,-- > 0,j 1 C.j. * gt indicates unlearning, and p.t p > 0 or pl., - < 0,j = 1........ C, gt indicates learning, is the number of candidate answers. Therefore, vu,, and ele, can be defined as:
ougi B oui,t1 + (1 - 3) vullee whit = 1BB.BBll.+- + (1 B) vli
vul, min(p!. P, j=1jfot
Σ
max(p', - p1 O)v,,
(22)
uh, max(pf. pi.-1 (gt +
Σ min(y), 11,,-1,0)05,11
j=
which satisfy that v. vu',, + ol [-] denotes the index operator. Pi,o 1/C² for all instances. B is a hyperparameter used to weight historical and real-time information, thus preserving historical trends while being sensitive to short-term changes. Furthermore, innovatively quantify instance difficulty through classification margins. Specifically, the classification margins m,, is defined as:
mi = 1p pl,cl j= Fgj gt.
(23)
where |.| denotes the absolute value operation. Apparently, gt - vl. > 0 indicates learning, p.. ph. indicates unlearning. Therefore, mui, and mli, can be defined as:
muiji 3 -mii,e11 (1- B) -mul,er mlia B -mi + +1 B) mll...
mul,e = log( Σ exp(m),)) JE
mlk, = log(
1
Σ exp(mm
(24)
JEN
where V is the set of class j satisfying pi,, - > and the set of class satisfying - < O.
The difficulty-awarr margins maify are detined as:
maiff 1 softmax(Di.) (D..),
(25)
Margin loss formulation. Ultimately, the various margin terms are aggregated in a cohesive manner form the multi-grained adaptive margins MMAM:
TMMAM = 171 freq.
TMMM [gt] = (1 - A11 .MMMM [gt] + X1 Meon f [gt].
(26)
TMMAM[9 [gt] = =(1 (1-X2) -MMMM (g) + X2 . maifs epoch 2 w,
MMMAM = - -MMAM-

---

Table 1: Accuracy comparisons with other methods the VQA-CP v2 and VQA-CP v1 datasets.
Datasets
VQA-CP v2
VQA-CP vl
Methods
All
Y/N
Num
Others
All
Y/N
Num
Others
UpDn
CVPR'M 18
39.74
42.27
11.93
46.05
37.96
42.79
12.41
42.53
RUBi
NeurlPS' 19
47.11
68.65
20.28
43.18
LMH 12
EMNLP'19 19
52.15
70.29
44.10
44.86
55.73
78.59
24.68
45,47
GGE-iter iter
ICCV'21
57.12
87.35
26.16
49.77
59.82
85.52
28.93
46.67
Ada 16
IJCAI'21
54.02
70.83
49.00
46.29
61.20
91.17
41.34
39.38
COB 20
W3
57.53
88.36
28.81
49.27
60.98
87.41
32.02
46.34
GENB 10
CVPR 23
59.15
88.03
40.05
49.25
62.74
86.18
43.85
47.03
GGD 18
TPAMI'23
59.37
88.23
38.11
49.82
CVIV 29
TMM'24
60.08
88.85
40.77
50,30
PWVQA 36
TMM'24
59.06
88.26
52.89
45.45
MMCD
Ours
61.34
88.93
55.68
48.44
63.62
90.72
52.67
41.08
Table Performance our approach with different network architectures
Methods
Y/N
Num
Other
Increased f
SAN
26. 60.12
35.34
11.34
24.70
33.24
SAN+MMCD
86.97
54.62
47.56
S-MRL
38.46
42.85
12.81
43.20
22.23
S-MRL+MMCD
60.69
88.40
55.44
47.61
LXMERT
47.49
22.24
56.52
18.29
LXMERT+MMCD 66.95
91.79
63.28
54.39
where denotes specific epoch that marks the end of the warm-up stage. The mMaM added to the angle 0; as margin penalty:
A
exp(s cos((, mMAM))
MMMM
a; log
(27)
A
cos((0 + mmMM))
4.2 Difficulty-aware Contrastive Learning
We further propose the DCL mechanism that integrates our instance difficulty model into supervised contrastive paradigm [23]. which dynamically emphasizes hard samples by infficulty-adaatiii weight- ing, effectively enhancing intra-class compactness and inter-class separation to form discriminative feature space. Specifically, we consider mini-batch B {(x1, (I2,, a2),.... (z)s), {1s))} of Ln-normalized joint representations R and corresponding answers a;. For each anchor feature æ; with difficulty Dj.t at iteration, the positive set P, = {i E B a = a,. + j} contains indices of all non-anchor samples with identical answers, and the negative set N, { B a, # aj includes indices of samples with different answers. The DCL loss formulated as:
log (pp(Dp.)) exp(x kp/T22) EnEN, exp(Dn,)) exp(x.) xn. xn/T2)
LOCL
Σ JEB
(28)
PEP;
where temperature 72 is set to 1.0.
4.3 Training and Optimization
Based the above analyses, the comprehensive training objective of the proposed MMCD approach encompasses a combination various loss functions, i.c..
fCE + LMAM + Ag A3 epoch < w, LCF + LMAM + XaLDCr, epoch w,
LTOTAL =
(29)
where LsupCon is the standard supervised contrastive loss.

---

Table 3: Ablation experiments for different modules of the MMCD model on VQA-CP v2.
Frequency-aware
Confidence-aware Margins
Difficult--aware Margins
Methods
DCL
All
Margins
Bascline
39.74
Variant-I
59.44
Variant-II
59.74
Variant-III
61.09
Variant-IV
41.09
MMCD (Ours)
61.34
Accuracy (%)
61 61-60.27
61.34
60.85
Accuracy (%)
61
62
61.23
61.34
61.14
Accuracy (%)
61
62
60/70
61:34
60.86
60
60.82
60.90
60
60.87
60.77
60
60.92
60.80
59
59
59
58
58
58
0.1
2.0 (a)
0.5
1.0 1.5
2.5
7.5
X x 10
(b) X2 x 100
(c)
As
Figure 3: Comparison of Accuracy on the VQA-CP v2 dataset with different parameter configurations.
5
Experiments
5.1 Datasets Implementation Details
We select various OOD benchmarks assess the robustness of models against real-world biases, such as VQA-CP v2, VQA-CP v1 []] All experiments adopt the standard evaluation metric (3]. Further details the experimental setup and implementation can be found in the supplementary materials.
5.2 Comparisons with State-of-the-Arts
As shown in Table we report both the overall accuracy and the per-category performance across question types, including "yes/no", "number", and "other". Compared to the second-best method, MMCD achieves gains of 1.26% and 0.72% overall accuracy on the VQA-CP v2 and VQA-CP v1 datasets, respectively. Notably, MMCD achieves state-of-the-ard performance "number" categories, which are typically more susceptible to language priors. In particular, MMCD yields substantial 2.79% improvement in the "number" category VQA-CP v2, demonstrating its strong ability to mitigate language bias and enhance reasoning over numerical questions.
the "yes/no" and
5.3 Extensive Experiments with Different Architectures
We further evaluate the generalizability and robustness of MMCD across additional architectures, including SAN [43], S-MRL [6]. and LXMERT (34). As shown in Table the MMCD approach consistently outperforms the corresponding baselines, demonstrating strong adaptability and model- agnostic performance across diverse network designs. In particular, applying MMCD to LXMERT, widely adopted vision- languae pre-trained model commonly used in various multimodal downstream tasks, yields a notable 8..99% performance improvement, further highlighting its effectiveness in enhancing broad range of model families.
5.4 Ablation Studies.
To assess the impact of each component in the proposed MMCD method, we perform series of ablation experiments with various variations. The comparison results of the ablation study are shown in Table Specifically, Variant-I outperforms baseline by 18.02%. The substantial performance enhancement demonstrates
critical function in mitigating language biases resulting from class imbalance by utilizing prior information. Furthermore, by incorporating confidence-awan margins, Variant-II achieves 0.3% performance gains compared Variant-I, suggesting that the simple multimodal logits strategy effectively utilizes the inherent sample complexity and results in the

---

How wares picture?
Does this planes tal. colors?
Olber
.00%
i
Other
50%
53%
Tora gt Tex yi
Tato, ct Test
Raselime: 2.
Oune
GT:1
Baseline:
Ours:
GT: No
Figure 4: Visualization results of MMCD in robust reasoning and bias mitigation.
development of robust and discriminative feature spaces. With the integration of Difficulty-awan margins, Variant-III combines into a complete multi-grained adaptive margins mechanism, making significant contribution to shaping more structured and organized spherical representation space. Subsequently, Variant-IV emphasizes the pivotal role of the DCL mechanism in enhancing intra- class compactness and inter-class separation. Eventually, performance, demonstrating the effectiveness and availability of the designed components. More ablation analysis can be found in the supplementary materials.
full MMCD model achieves the best
5.5 Parameter Analysis.
As shown in Fig. we systematically evaluate MMCD under comprehensive range of hyperpa- rameter settings. Our study focuses on three key hyperparameters: A and X2 in Eqn. (26). and X in Eqn. (29.. Across all combinations, accuracy fluctuates by more than 1.07%. Such minimal variance highlights MMCD's robustness to hyperparameter selection, significantly reducing the need for exhaustive tuning. Moreover, this stability suggests that MMCD can be readily transferred new VQA benchmarks or application domains without extensive reconfiguration. Overall, the insensitivity to parameter settings not only simplifies deployment but also confirms MMCD's strong generalization capability in mitigating language bias. More parameter analysis can be found in the supplementary materials.
5.6 Qualitative Analysis.
As shown in Fig. 4 the MMCD method not only accurately localizes the correct but also exhibits exceptional performance in removing bias. By enforcing a well-structured feature space, our approach facilitates the learning of highly discriminative features and the extraction of high-quality multimodal information. This structured representation enhances the model's ability to identify and leverage key visual cues, which is critical for robust performance in VQA tasks.
Conclusion
In this paper, we investigated the origin of language bias in VQA and elucidated why margin-based mechanisms effectively mitigate it. Empirical evidence shows that multimodal data heterogeneity induces gradient optimization imbalances, leading to biased feature fusion and classifier weight devi- ations. We provide theoretical support from both gradient and spectral perspectives, demonstrating how margin- based objectives counteract these effects. Building on these insights, we propose MMCD, an adaptive multi-margin framework that incorporates sample frequency and difficulty to reshape decision boundaries and enhance feature discrimination via diffcuuly-awarr contrastive learning. Extensive experiments confirm the superior robustness MMCD, with potential implications for broader challenges such shortcut learning, long-tail recognition, and class imbalance.
Acknowledgments
This work was supported in part by the Guangdong Basic and Applied Basic Research Foundation in part by the National Natural Science Foundation
(Nos. 2025A 1555151501110 51010022,, of China (No. 62302172), and in part by the JSPS KAKENHI (No. JP25K21207) and JST CREST (No. JPMJCR22M2).
10

---

References
[1] A. Agrawal, Batra, D. Parikh, and A. Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 4971-4980, 2018
[2] Anderson, X. He, C. Buchler, D. Teney, and M. Johnson. Bottom-up and top-down attention for image captioning and visual question answering, In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 6077 2018.
[3] S. Antol, A. Agrawal, Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages -435-2433, 2015.
[4] A. Basu, S. Addepalli, and R. V. Babu. RmIvqa: A margin loss approach for visual question answering with language biases. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11671-11680, 2023.
[5] F. Boutros, N. Damer, F. Kirchbuchner, and A. Kuijper. Elasticface: Elastic margin loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1578-1581 2022.
[6] R. Cadene, C. Dancette, H. Ben-younes, M. Cord, and D. Parikh. Rubi: Reducing unimodal biases visual question answering. In Pruceedings of the Advances Neural Information Processing Systems (NeurIPS), volume 32, 2019.
[7] L. Cai, H. Fang, N. Xu, and B. Ren. Counterfactual causal-effect intervention for interpretable medical visual question answering. IEEE Transactions on Medical Imaging (TMI), 2024.
[8] K. Cao, C. Wei, A. Gaidon, N. Arechiga, and T, Ma. Learning imbalanced datasets with bubellddistriution-vvaeee margin loss. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
[9] L. Chen, X. Yan, J. Xiao, H. Zhang. S. Pu, and Y. Zhuang. Counterfactual samples synthesizing for robust visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10800-10809, 2020.
[10] J. W. Cho, D.-J Kim, H. Ryu, and I. S. Kweon. Generative bias for robust visual question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1 11681-1166 1690, 2023.
[11] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406 1078.1078 2014.
[12] C. Clark, M. Yatskar, and L. Zettlemoyer. Don't 't take the easy way out: Ensemble based methods for avoiding known dataset biases. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.
[13] C. Dancette, R. Cadene, D. Teney, and M. Cord. Beyond question- based biases: Assessing mul- timodal shorteut learning in visual question answering. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1574-1583, 2021.
[14] J. Deng, J. Guo, N. Xue, and S. Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages
2019.
[15] Y. Goyal, T. Khot, D. Summers-Suue D. Batra, and D. Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6904-6913, 2017.
[16) Y. Guo, L. Nie, Z. Cheng. F. Ji, J. Zhang, and A. Del Bimbo. Adavqa: Overcoming language priors with adapted margin cosine loss. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pages 708-714, 2021.

---

[17] X. Han, S. Wang, and C. Su. Greedy gradient ensemble for robust visual question answering. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1584-15933 2021.
[18] X. Han, S. Wang, C. Su, Q. Huang, and Q. Tian. General greedy de-bias learning. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 45:1-17, 2023,
[19] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. Lora: Low-rank adaptation of large language models. In Proceedings of the International Conference on Learning Representations (ICLR), volume 1, page 3, 2022.
[20] A. Jha, B. Patro, L. Van Gool, and T. Tuytelaars. Barlow constrained optimization for visual question answering. Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1084-1093, 2023.
[21] C. Kervadec, G. Antipov, M. Baccouche, and C. Wolf. Roses are red, violets are blue... but should vqa expect them to? In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 2776-2786 2021.
[22] S. Khan, M. Hayat, S. W. Zamir, J. Shen, and L. Shao. Striking the right balance with uncertainty, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 103-112, 2019.
[23] P. Khosla, Teterwak, C. Wang. A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan. Supervised contrastive learning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 18661-1863, 2020.
[24] A. Li, W. Huang. X. Lan, J. Feng, Z. Li, and L. Wang. Boosting few-shot learning with adaptive margin loss. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 2257-11588 2020.
[25] Z. Liang, W. Jiang. Hu, and J. Zhu. Learning to contrast the counterfactual samples for robust visual question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3285-3292, 2020.
[26] T.-Y. Lin, P. Goyal, Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2980-2988, 2017.
[27] W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 212-220, 2017.
[28] Y. Niu, K. Tang, H. Zhang, Z. Lu, X.-S. Hua, and J.-R. Wen. Counterfactual vqa: A cause-effect look at language bias. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages
2021.
[29] Y. Pan, J. Liu, Jin, and Z. Li. Unbiased visual question answering by leveraging instrumental variable. IEEE Transactions on Multimedia (TMM), 26:6648 (666666866662,, 2024.
[30] X. Peng, Y. Wei, A. Deng, D. Wang, and Hu. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8238-8244 2022.
[31] J. Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, 2014.
[32] S. Ramakrishnan, A. Agrawal, and S. Lec. Overcoming language priors in visual question answering with adversarial regularization. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), volume 31, 2018.
[33] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), volume 28, 2015.
12

---

[34] H. Tan and M. Bansal. Lxmert: Learning modastmy encoder representations from trans- formers. arXiv preprint arXiv: (et.:::99080799999 07490, 2019.
[35] D. Teney, E. Abbasnejad. K. Kafle, R. Shrestha, C. Kanan, and A. Van Den Hengel. On the value of out-of-diitributiin testing: An example of goodhart's law. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 33:407-411 2020.
[36] A. Vosoughi, S. Deng, S. Zhang, Y. Tian, C. Xu, and J. Luo. Cross modality bias in visual question answering: A causal view with possible worlds vqa. IEEE Transactions Multimedia (TMM), 2024.
[37] H. Wang, Y. Wang, Z. Zhou, X. Ji, D. Gong, J. Zhou, Z. Li, and W. Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5265- 5274,, 2018.
(38] Q Wang, Y. Yu, Y. Yuan, R. Mao, and T. Zhou. Videorft: Incentivizing video reasoning capability in milms via reinforced fine-tuning. arXiv preprint 1rx.i:2500./24344 2025.
[39] Y. Wei, R. Feng, Z. Wang, and D. Hu. Enhancing multimodal cooperation via sample- modality valuation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 27338-27341 2024.
[40] Y. Wei, S. Li, R. Feng, and D. Hu. Diagnosing and re-learning for balanced multimodal learning. In Proceedings of the European Conference on Computer Vision (ECCV), pages 71-86. Springer, 2024.
[41] Z. Wen, G. Xu, M. Tan, Q. Wu, and Q. Wu. Debiased visual question answering from feature and sample perspectives. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 3784-3796, 2021.
[42] J. Xie, Y. Cai, J. Chen, R. Xu, J. Wang, and Q. Li. mhowledge-augmentt visual question answering with natural language explanation. IEEE Transactions on Image Processing (TIP), 33:2652-26665 2024.
[43] Z. Yang, X. He, J. Gao, L. Deng, and A. Smola. Stacked attention networks for image question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 21-29, 2016.
[44] S. Yu, Guo. R. Zhang, Y. Fan, Z. Wang. and X. Cheng. A re-balancing strategy for class- imbalanced classification based on instance difficulty. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 70-79, 2022.
[45] C. Zhan, P. Peng, H. Zhang, H. Sun, C. Shang, Chen, H. Wang. Wang, and H. Wang. Debiasing medical visual question answering via counterfactual training. In Medical Image Computing and Computer Assisted Intervention (MICCAI), pages 382-393. Springer, 2023.
[46] Z. Zhou, J. Yao, F. Hong, Y. Zhang, B. Han, and Y. Wang. Combating representation learning disparity with geometrie harmonization. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 20394 20408, 2023.
[47] J. Zhu, Y. Liu, H. Zhu, H. Lin, Y. Jiang, Z. Zhang, and B. Chen. Combating visual question answering hallucinations via robust multi-space co-debias learning. In Proceedings of the ACM International Conference on Multimedia (MM), pages 955-964, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction accurately summarize the key contributions and findings of the paper, and they align with the theoretical and experimental results presented.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
This paper discusses the limitations of the work in the supplementary materials.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiittinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: The assumptions required for our theoretical results are described in the main paper, and proofs are provided in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide detailed experimental settings in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
The code will be released once the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ "cipps.c/puullc//guu /ct.://Suudeeooeeouuummmciii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide detailed experimental settings the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: This paper does not contain statistical experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
16

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes] We provided a description of the platform and hardware used for the experi- ments.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We adhere to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential positive social impacts of the work done in the supplementary materials.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
The research involves only publicly available datasets and standard models, posing no significant misuse risks, thus no specific safeguards were necessary.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have cited the relevant papers.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.paaersstt
package should be provided. For popular datasets, paperswi thcode. cow/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification: The paper does not introduce any new assets, thus documentation for such is not applicable.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects, thus the inclusion of participant instructions, screenshots, and compensation details is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
19

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to LLM policy for what should or should not be described.
20


Reviews:
[{"review_id": "EDpvfCLhTi", "reviewer": "Reviewer_GGBZ", "summary": "This paper tackles the persistent issue of language bias in Visual Question Answering (VQA), where models tend to rely on superficial linguistic correlations rather than genuine multimodal reasoning. The authors conduct a principled analysis to uncover how such biases arise and introduce a novel mitigation framework: Multi-Margin Collaborative Debiasing (MMCD). The paper makes a substantial conceptual and empirical contribution to the VQA field by clarifying the origins of language bias and proposing a theoretically grounded approach to address it.", "questions": "1.How sensitive is MMCD to the choice of hyperparameters (e.g., λ₁, λ₂, σ in the margin formulations), and how does this affect its generalizability across different datasets or domains?\n2.Given that the difficulty-aware module relies on handcrafted metrics like learning speed and classification margin, have the authors considered replacing or complementing this with a learnable, data-driven difficulty estimator?\n3.Can the proposed MMCD framework be effectively adapted to or integrated with large-scale pre-trained multimodal models (e.g., BLIP-2, Flamingo, or LLaVA), and if so, what challenges are anticipated in such a transfer?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n1. Theoretical Insight into Bias Formation\nThe paper offers a principled and detailed analysis of how language bias emerges in VQA models, linking it to modality-specific gradient imbalances, feature fusion asymmetries, and classifier weight directionality. This goes beyond empirical fixes and provides a deeper understanding of the underlying mechanisms—something that has been largely underexplored in prior work.\n2. Novel and Integrated Debiasing Framework (MMCD)\nDespite the presence of missing data during training or inference, SHIFT maintains competitive or superior predictive performance (as measured by c-index) compared to strong baselines. The model’s generalizability across cohorts (TCGA, CPTAC, UCI) suggests it is not overfitted and can scale across populations and institutions without manual adjustment.\n3. Strong Empirical Results and Generalization Across Architectures\nMMCD demonstrates state-of-the-art performance on challenging datasets such as VQA-CP v2 and v1, particularly in bias-prone categories (e.g., \"yes/no\", \"number\"). The method is also shown to be model-agnostic, achieving consistent improvements across multiple backbone architectures, including SAN, S-MRL, and LXMERT.\n\nWeaknesses：\n1．The proposed MMCD framework introduces multiple auxiliary components—such as adaptive margins, difficulty tracking, and contrastive learning—which significantly increase the training complexity and computational cost. This may limit the practical applicability of the method, especially in resource-constrained or real-time scenarios.\n2. While the paper provides a thorough treatment of language bias, it does not explicitly address other forms of bias prevalent in VQA, such as visual shortcut bias, question-type bias, or multimodal co-adaptation. This narrow focus may limit the generality and robustness of the proposed method when applied to broader or more complex bias scenarios.", "was_revised": true, "final_justification": "My concerns have been addressed", "rebuttal": "Thanks for your insightful comments.\n\n**Q1: Clarification of Computational Cost.**\n\nIn our work, ***all experimental configurations conform to standard VQA practice and entail a modest, operationally acceptable computational footprint.*** Specifically, the proposed MAM and DCL operate exclusively during training as loss-level regularizers that shape the representation space, without invoking any auxiliary branches or contrastive operations. In the test phase, the model follows the standard VQA forward path without invoking any auxiliary branches or contrastive operations. Consequently, inference-time latency, FLOPs, and memory usage remain effectively unchanged relative to the baseline, ensuring predictable and efficient deployment. As shown in Table 1, the training FLOPs increase from only marginally 0.18G (COB) to 0.19G (MMCD) while the parameter count is listed as 35.18M (COB) vs. 35.9M (MMCD). ***Compared with the existing baselines, MMCD delivers robustness gains with negligible inference-time overhead and an acceptable training-time cost.***\n\n\nTable 1: Comparasion of FLOPs and Parameters on VQA-CP v2.\n| Methods | Ref. | FLOPs(G) | Parameters(M) |\n| -------- | -------- | -------- | -------- |\n| UpDn | CVPR'18 | 0.18 | 12.94 |\n| RMLVQA | CVPR'23 | 0.17 | 18.88 |\n| GGD | TPAMI'23 | 0.19 | 33.98 |\n| COB | WACV'23 | 0.16 | 35.18 |\n| PHOH | AAAI'25 | 0.31 | 18.88 |\n| **MMCD** | Ours | 0.19 | 35.90 |\n\n\n**Q2: The Robustness of the Method.** \n\nIn Table 2, we present a comprehensive evaluation of the proposed MMCD method on the VQA-CE dataset, which emphasizes **challenging multimodal bias patterns** in VQA. MMCD achieves consistently higher accuracy across the biased test splits, outperforming baseline bias-mitigation techniques by a substantial margin. Experimental results confirm that MMCD yields stable, unbiased predictions under diverse bias scenarios. **These findings underscore the effectiveness and robustness of MMCD in addressing multimodal bias, surpassing existing methods designed for other bias types.**\n\nTable 2: Accuracy comparison on VQA-CE dataset.\n| Methods | Ref. | Overall | Counter | Easy |\n| -------- | -------- | -------- | -------- | -------- |\n| CSS | CVPR'20 | 53.55 | 34.36 | 62.08 |\n| GENB | CVPR'23 | 57.87 | 34.80 | 68.15 |\n| RMLVQA | CVPR'23 | 58.08 | 35.01 | 68.21 |\n| CVIV | TMM'24 | - | 36.12 | - |\n| PHOH | AAAI'25 | 59.10 | 36.21 | 68.31 |\n| MMCD | Ours | **67.86** | **38.69** | **80.83** |\n\n\n**Q3: Parameter Sensitivity.** \n\nIn Table 3, we systematically evaluate MMCD under a comprehensive range of hyperparameter settings. **Across all combinations, accuracy fluctuates by no more than 1.07%.** Such minimal variance highlights MMCD's robustness to hyperparameter selection, significantly reducing the need for exhaustive tuning. Moreover, this stability suggests that MMCD can be readily transferred to new VQA benchmarks or application domains without extensive reconfiguration. **Overall, the insensitivity to parameter settings not only simplifies deployment but also confirms MMCD's strong generalization capability in mitigating multimodal bias.**\n\nTable 3: Comparison of Accuracy on the VQA-CP v2 dataset with different parameter configurations.\n|λ_1| 0.01 | 0.05 | 0.1 | 0.15 | 0.2 |\n| -------- | -------- | -------- | -------- | -------- | -------- |\n|Accuracy(%)| 60.27 | 60.82 | **61.34** | 60.90 | 60.85 |\n\n|λ_2| 0.01 | 0.025 | 0.05 | 0.075 | 0.1 |\n| -------- | -------- | -------- | -------- | -------- | -------- |\n|Accuracy(%)| 61.23 | 60.87 | **61.34** | 60.77 | 61.14 |\n\n|α| 0.4 | 0.5 | 0.6 | 0.7 | 0.8 |\n| -------- | -------- | -------- | -------- | -------- | -------- |\n|Accuracy(%)| 61.11 | 60.95 | **61.34** | 60.73 | 60.82 |\n\n\n**Q4: Difficulty Estimator.**\n\nInspired by your suggestion, we propose a learnable weighted class-prototype network as a sample-difficulty estimator. Specifically, we define difficulty as the distance between each sample and its corresponding class prototype. During training, this difficulty score adaptively weights the loss, while each sample’s loss in turn updates the prototype. As a result, easy samples are gradually drawn closer to their prototypes and harder samples are repelled further away. This strategy yields rapid, accurate difficulty estimation with very low computational and memory overhead. We will implement and evaluate this approach in future work, recognizing that fast, precise difficulty estimation remains an open and compelling challenge.\n\n\n**Q5: Integration with MLLM.** \n\nTo evaluate MMCD's compatibility with state-of-the-art multimodal architectures, we incorporated it into the LXMERT framework. Table 4 summarizes the comparative results: MMCD-enhanced LXMERT achieves a 18.29% absolute increase in overall accuracy on the VQA-CP v2 benchmark. **These preliminary findings demonstrate that our approach can be seamlessly adapted to large-scale transformers without modifying their core attention or fusion modules.** We expect that integrating MMCD into next-generation architectures  (e.g., BLIP-2, Flamingo, or LLaVA) will yield equal or superior benefits, owing to their richer cross-modal representations. Due to resource and scheduling constraints, we reserve these evaluations for future work, where we will also explore optimal insertion points and fine-tuning protocols to maximize gains across diverse VQA datasets.\n\n**Table 4: Performance of our approach with different network architecture.**\n\n| Methods | All | Y/N | Num | Other | Increased |\n| -------- | -------- | -------- | -------- | -------- | -------- |\n| LXMERT | 48.66 | 47.49 | 22.24 | **56.52** | - |\n| LXMERT + MMCD | **66.95** | **91.79** | **63.28** | 54.39 | **18.29** |\n\n\nThanks! **We hope these clarifications satisfactorily resolve your questions and meet with your positive appraisal.**"}, {"review_id": "MGdrOpPvd7", "reviewer": "Reviewer_GABY", "summary": "This paper focuses on the issue of language bias in Visual Question Answering (VQA). The authors first conduct a thorough theoretical investigation into the origin of language bias, identifying modality-specific gradient imbalances, biased feature fusion, and classifier weight deviations as key factors. To tackle these issues, they propose a Multi-Margin Collaborative Debiasing (MMCD) framework, which consists of Multi-grained Adaptive Margins (MAM), including frequency-aware, confidence-aware, and difficulty-aware margins. And together with a Difficulty-aware Contrastive Learning (DCL) module. Extensive experiments on OOD benchmarks demonstrate that MMCD significantly outperforms previous state-of-the-art methods, showcasing its robustness and effectiveness in mitigating language bias.", "questions": "1. The MMCD hyperparameters (λ₁, λ₂, λ₃, α, β, and w) appear to play critical roles in the overall performance. However, it remains unclear how sensitive the method is to these hyperparameters, and how the specific combination was determined during experimentation. Can the method maintain robust performance across different datasets without extensive hyperparameter tuning?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n1. The paper conducts a comprehensive theoretical and empirical study into the origin of language bias in VQA, identifying three key factors, which is useful to the task and future research.\n2. The proposed framework is well motivated by prior observations and insights. It integrates multi-granularity margin design (answer frequency, model confidence, and sample difficulty) with contrastive learning in a principled way.\n3. The method consistently outperforms baselines on OOD benchmarks and exhibits strong generalization across diverse backbone architectures.\n\nWeakness\n1. The adaptive margin computation (particularly the sample-specific difficulty computation) likely adds complexity, but no analysis of training time or memory cost compared to baselines. This could also limit the scalability to larger datasets due to computational overhead.\n2. The narrow focus on specific types of bias and the reliance on dataset statistics may significantly hinder real-world deployment.", "was_revised": true, "final_justification": "My concerns have been addressed", "rebuttal": "Thanks for your valuable comments.\n\n**Q1: Clarification of Computational Cost.** \n\nIn our work, ***all experimental settings adhere to standard VQA protocols and incur only modest computational overhead.*** During training, our MAM and DCL act solely as loss‐level regularizers to shape the feature space—no additional network branches or contrastive modules are introduced. At inference, the model follows the conventional VQA forward pass without invoking any auxiliary components, leaving latency, FLOPs, and memory usage unchanged relative to the baseline. As summarized in Table 1, training FLOPs increase marginally from 0.18 G (COB) to 0.19 G (MMCD), and parameters grow from 35.18 M to 35.9 M. **Thus, compared to existing baselines, MMCD achieves significant robustness improvements with negligible inference‐time overhead and an acceptable increase in training cost.**\n\n\n**Table 1: Comparasion of FLOPs and Parameters on VQA-CP v2.**\n| Methods | Ref. | FLOPs(G) | Parameters(M) |\n| -------- | -------- | -------- | -------- |\n| UpDn | CVPR'18 | 0.18 | 12.94 |\n| RMLVQA | CVPR'23 | 0.17 | 18.88 |\n| GGD | TPAMI'23 | 0.19 | 33.98 |\n| COB | WACV'23 | 0.16 | 35.18 |\n| PHOH | AAAI'25 | 0.31 | 18.88 |\n| **MMCD** | Ours | 0.19 | 35.90 |\n\n**Q2: The Robustness of the Method.** \n\nTable 2 reports a detailed assessment of MMCD on the VQA-CE benchmark, which highlights **challenging multimodal bias patterns** in VQA. MMCD consistently outperforms baseline bias-mitigation methods across all biased test splits, delivering markedly higher accuracy. **These results demonstrate that MMCD produces stable, unbiased predictions under varied bias conditions, underscoring its superior effectiveness and robustness compared to existing approaches targeting other bias types.**\n\n**Table 2: Accuracy comparison on VQA-CE dataset.**\n| Methods | Ref. | Overall | Counter | Easy |\n| -------- | -------- | -------- | -------- | -------- |\n| CSS | CVPR'20 | 53.55 | 34.36 | 62.08 |\n| GENB | CVPR'23 | 57.87 | 34.80 | 68.15 |\n| RMLVQA | CVPR'23 | 58.08 | 35.01 | 68.21 |\n| CVIV | TMM'24 | - | 36.12 | - |\n| PHOH | AAAI'25 | 59.10 | 36.21 | 68.31 |\n| MMCD | Ours | **67.86** | **38.69** | **80.83** |\n\n**Q3: Parameter Sensitivity.** \n\n**Table 3 presents MMCD's performance across a wide spectrum of hyperparameter settings, showing that accuracy varies by at most 1.07%.** This negligible fluctuation underscores MMCD’s robustness to hyperparameter choices, thereby minimizing the need for extensive tuning. Furthermore, such stability indicates that MMCD can be seamlessly applied to new VQA benchmarks or domains without significant reconfiguration. **Overall, this insensitivity to parameter settings both streamlines deployment and validates MMCD's strong generalization in mitigating multimodal bias.**\n\n\n**Table 3: Comparison of Accuracy on the VQA-CP v2 dataset with different parameter configurations.**\n|λ_1| 0.01 | 0.05 | 0.1 | 0.15 | 0.2 |\n| -------- | -------- | -------- | -------- | -------- | -------- |\n|Accuracy(%)| 60.27 | 60.82 | **61.34** | 60.90 | 60.85 |\n\n|λ_2| 0.01 | 0.025 | 0.05 | 0.075 | 0.1 |\n| -------- | -------- | -------- | -------- | -------- | -------- |\n|Accuracy(%)| 61.23 | 60.87 | **61.34** | 60.77 | 61.14 |\n\n|α| 0.4 | 0.5 | 0.6 | 0.7 | 0.8 |\n| -------- | -------- | -------- | -------- | -------- | -------- |\n|Accuracy(%)| 61.11 | 60.95 | **61.34** | 60.73 | 60.82 |\n\nFinally, **we sincerely hope the above clarification would enhance the recognition and rating score of our work.**"}, {"review_id": "ukzyFHJT7Y", "reviewer": "Reviewer_UEGk", "summary": "The paper presents an empirical and theoretical investigation into why VQA models develop language biases. The authors find that skewed gradient norms between the visual and question encoders lead to question-dominated fused features and classifier weights. They also explain why margin-based methods can help alleviate this issue. Building on these insights, the authors propose a debiasing method, and experiments involving different architectures demonstrate improvements over existing approaches.", "questions": "Please see the questions raise in the Weaknesses section.", "limitations": "The limitations are discussed in Appendix K.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The paper offers an in-depth exploration combining experiments and mathematical derivation to uncover how biases arise in VQA models.\n\n2. Experiments demonstrate that the proposed method is consistently effective across different architectures and setups.\n\n\n## Weaknesses\n\n1. The Method section is difficult to follow. It would benefit from a high-level roadmap: a brief overview of the intuition and purpose behind each component before diving into the technical details.\n\n2. The caption for Figure 1 is confusing. Perhaps the current (c) is redundant, and the two (d)’s should instead be labeled (c) and (d). Additionally, the caption is not very informative—it would be helpful to briefly describe the elements shown in each plot.\n\n3. The ablation study on the three components of Multi-Grained Adaptive Margins is limited (it currently appears in the appendix but would be better placed in the main paper). Instead of adding each component individually to the baseline, it would be more informative to remove one component at a time from the full method to assess each component’s marginal contribution.\n\n4. Can the authors clarify how many hyperparameters the method has in total? From Figures 3 and 6, it appears there are six, which would make tuning very challenging in practice. Moreover, the plots seem to show that performance is quite sensitive to even a single hyperparameter changing from its optimal value, let alone simultaneous variations in multiple parameters.\n\n\n5. As a non-expert in this specific topic, I cannot fully evaluate whether the baselines, comparisons, and experimental setups cover the state of the art; I welcome perspectives from other reviewers.", "was_revised": true, "final_justification": "I am overall satisfied with the rebuttal and have adjusted my score accordingly.", "rebuttal": "Thanks for your discerning comments.\n\n**Q1: Motivation of Technology.** \n\nUnfortunately, we are not allowed to provide a high-level roadmap to showcase the intuition and purpose behind each component. We provide the following overview. **Margin learning effectively mitigates the chaotic class‐boundary issues induced by data imbalance.** Our multi-grained adaptive margins comprise coarse-grained margins and fine-grained margins by considering answer frequency and instance difficulty, respectively. **The frequency-aware margins are driven by the inherent properties of imbalanced data.** Minority classes rely on larger margin penalties to push towards the center of the class, while the robust features of most classes benefit from smaller margin penalties. By utilizing prior data information, appropriate boundary penalties can be applied at a coarse-grained level. **The confidence-aware margins are based on the difficulty of the sample.** The difficulty of the sample significantly affects the formation of discriminative decision boundaries and strong class separability. Predictive logits can easily and effectively evaluate sample difficulty. **Considering that logits cannot finely mine inherent instance difficulty, the difficulty-aware margins further construct a fine-grained difficulty model to analyze sample difficulty and enhance reasoning ability and robustness.** Overall, these complementary margins dynamically sculpt clear and robust decision boundaries across multiple granularities.\n\n**Q2: Confusion about the Caption.** \n\nWe have reviewed the caption for Figure 1 and corrected it as pointed out. The current (b) and (c) are duplicated. The first (d) should be labeled (c). The revised caption is now included in the manuscript. We provide a brief description as follows. \n(a) Comparison of gradient norms between modalities: in the baseline UpDn model, the question‐mode gradient norm exceeds that of the image mode, exacerbating language bias; MMCD balances these norms.\n(b) Evolution of gradient bias during training: UpDn maintains a higher question‐mode gradient than image‐mode gradient, whereas MMCD progressively reduces this discrepancy.\n(c) Modality–fusion feature similarity: UpDn shows a larger similarity for question features versus fusion features and a smaller one for image features; MMCD equalizes these similarities across modalities.\n(d) Classifier singular value distribution: UpDn’s top singular values dominate, indicating directional bias that captures only primary features.\n\n**Q3: More Ablation Study.** \n\nWe conducted an ablation study by sequentially removing each component of the multi‐grained adaptive margin. As summarized in Table 1, omitting any single margin degrades overall accuracy, confirming their complementary roles. In particular, the frequency-aware margins have the greatest impact: its removal incurs a 7.26% drop in accuracy, underscoring the importance of leveraging class‐frequency priors to mitigate language bias. Furthermore, performance on “yes/no” and “other” questions remains largely unaffected, demonstrating MMCD’s consistent reasoning ability, whereas accuracy for “number” questions declines markedly, indicating heightened sensitivity to numerical responses.\n\nTable 1: Ablation experiments for different modules of the MMCD model on the VQA-CP v2 dataset. We remove each component one by one from the complete method to evaluate the corresponding marginal contribution.\n| Methods | All | Y/N | Num | Others |\n| -------- | -------- | -------- | -------- | -------- |\n| MMCD | **61.34**| 88.93 | **55.68** | 48.44 |\n| -Frequency-aware Margins | 54.08 | 86.43 | 18.88 | 46.79 |\n| -Confidence-aware Margins | 60.44 | **89.51** | 51.07 | 47.77 |\n| -Difficulty-aware Margins | 61.22 | 89.12 | 54.16 | **48.54** |\n| Baseline | 41.42 | 46.83 | 12.96 | 46.40 |\n\n**Q4: Parameter Sensitivity.** \n\n**Table 2 reports MMCD’s performance over a wide range of hyperparameter combinations, revealing that accuracy varies by no more than 1.07%.** This negligible fluctuation demonstrates MMCD’s strong resilience to hyperparameter selection, thereby minimizing the need for extensive tuning. Moreover, such stability indicates that MMCD can be deployed across new VQA benchmarks and domains with little to no reconfiguration. **In summary, MMCD’s insensitivity to parameter settings both streamlines practical deployment and validates its robust generalization in mitigating multimodal bias.**\n\nTable 2: Comparison of Accuracy on the VQA-CP v2 dataset with different parameter configurations.\n|λ_1| 0.01 | 0.05 | 0.1 | 0.15 | 0.2 |\n| -------- | -------- | -------- | -------- | -------- | -------- |\n|Accuracy(%)| 60.27 | 60.82 | **61.34** | 60.90 | 60.85 |\n\n|λ_2| 0.01 | 0.025 | 0.05 | 0.075 | 0.1 |\n| -------- | -------- | -------- | -------- | -------- | -------- |\n|Accuracy(%)| 61.23 | 60.87 | **61.34** | 60.77 | 61.14 |\n\n|α| 0.4 | 0.5 | 0.6 | 0.7 | 0.8 |\n| -------- | -------- | -------- | -------- | -------- | -------- |\n|Accuracy(%)| 61.11 | 60.95 | **61.34** | 60.73 | 60.82 |\n\nFinally, **we hope these clarifications enhance our work’s visibility and positively impact its evaluation scores.**"}, {"review_id": "gytlAIIgk9", "reviewer": "Reviewer_qcZ8", "summary": "This paper tackles performance degradation in Visual Question Answering (VQA) caused by language bias. Through theoretical and empirical analysis, the authors identify modality-specific gradient optimization deviation, which results in feature fusion skewness and classifier weight imbalance. To mitigate this, they propose MMCD (Multi-Margin Collaborative Debiasing), a framework that reshapes decision boundaries under biased training via three adaptive angular margins (frequency, confidence, and difficulty-aware) and difficulty-aware contrastive learning mechanism. Experiments on multiple benchmarks demonstrate the effectiveness of MMCD.", "questions": "Pls refer to Strengths and weaknesses.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\nS1: Comprehensive and in-depth analysis: Through a combination of empirical and theoretical analyses, the authors systematically explore the formation mechanism of language bias from modal gradient optimization deviation, feature fusion deviation, and classifier weight direction deviation. Besides, the authors also provide theoretical support from the gradient and spectral perspectives as to why the margin-based mechanism can mitigate the bias, and the analysis is thorough.\nS2: The proposed MMCD framework innovatively integrates frequency-aware, confidence-aware, and difficulty-aware angular margins, as well as a dynamic, difficulty-aware contrast learning mechanism.Experiments show that the framework outperforms the existing baseline in several challenging VQA benchmarks, effectively countering language bias.\nS3: The writing is well-organized and easy to follow.\n\n\tWeaknesses:\nW1: MMCD introduces multiple branches (e.g., unimodal classifiers) and a difficulty estimation mechanism that bring additional calculation expense, and it is recommended to add an experimental analysis of the computational efficiency (inference time, runtime memory usage) to assess the feasibility of real-world deployment.\nW2: MAM mechanism relies on priors of question types and answer distributions, is it generalizable under different data distributions? It is recommended to conduct experiments on GQA/ScienceQA to verify the generalization performance of the method.\nW3: With the introduction of additional modules (FAM, CAM, DAM, DCL) in MMCD, several hyperparameters need to be tuned during the training process, so is there any difficulty in selecting ,combining and optimizing the hyperparameters with different network architectures and datasets?", "was_revised": true, "final_justification": "I have read the authors’ rebuttal and the other reviewers’ comments, the authors have addressed most of my concerns, and thus I consist of my original score.", "rebuttal": "Thanks for your perceptive comments.\n\n**Q1: Clarification of Computational Cost.** \n\n***All experiments adhere to standard VQA protocols while incurring only minimal computational overhead.*** During training, our MAM and DCL function solely as loss-level regularizers—no extra branches or contrastive modules are introduced. At inference, the model executes the identical forward pass as the baseline, so latency, FLOPs, and memory usage remain unchanged. As Table 1 shows, training FLOPs increase marginally from 0.18 G to 0.19 G, and the parameter count from 35.18 M to 35.9 M. **Compared to existing baselines, MMCD delivers significant robustness gains with negligible inference-time overhead and a modest training-time cost.**\n\nTable 1: Comparasion of FLOPs and Parameters on VQA-CP v2.\n| Methods | Ref. | FLOPs(G) | Parameters(M) |\n| -------- | -------- | -------- | -------- |\n| UpDn | CVPR'18 | 0.18 | 12.94 |\n| RMLVQA | CVPR'23 | 0.17 | 18.88 |\n| GGD | TPAMI'23 | 0.19 | 33.98 |\n| COB | WACV'23 | 0.16 | 35.18 |\n| PHOH | AAAI'25 | 0.31 | 18.88 |\n| **MMCD** | Ours | 0.19 | 35.90 |\n\n**Q2: Generalization Performance of the Method.** \n\nTo further assess generalization, we applied MMCD to the GQA dataset, which emphasizes compositional reasoning and visual grounding. **As shown in Table 2, MMCD achieves an overall accuracy of 75.1%, exceeding the runner‐up by 14.8%. This substantial gain demonstrates that our adaptive multi‐grained margins effectively transfer to tasks requiring intricate reasoning under diverse distributions.**\n\nImportantly, our experimental suite spans both language‐bias benchmarks (VQA-CP v2, where prior answer statistics dominate) and multimodal‐bias benchmarks (VQA-CE). In each scenario, MMCD attains state‐of‐the‐art results, illustrating its capacity to harmonize modality contributions and mitigate spurious correlations. **These findings confirm that MMCD is not tailored to a single bias type or dataset but provides a universal framework for robust VQA across changing domains and bias patterns.**\n\nTable 2: Accuracy comparisons with other methods on GQA.\n| Methods | Ref. | Accuracy |\n| -------- | -------- | -------- |\n| UpDn     | CVPR'18 | 49.7 |\n| COB | WACV'23 | 42.1 |\n| MSCD | MM'24 | 60.3 |\n| **MMCD** | Ours | **75.1** |\n\n**Q3: Parameter Sensitivity.** \n\n**Table 3 reports MMCD’s performance across a comprehensive hyperparameter grid, where accuracy varies by at most 1.07%.** This negligible fluctuation demonstrates MMCD’s strong resilience to hyperparameter settings, greatly reducing the need for exhaustive tuning. Furthermore, such stability implies MMCD can be deployed on new VQA benchmarks or specialized domains with minimal reconfiguration. **In summary, MMCD’s parameter insensitivity not only streamlines practical deployment but also confirms its robust generalization for mitigating multimodal bias.**\n\nTable 3: Comparison of Accuracy on the VQA-CP v2 dataset with different parameter configurations.\n|λ_1| 0.01 | 0.05 | 0.1 | 0.15 | 0.2 |\n| -------- | -------- | -------- | -------- | -------- | -------- |\n|Accuracy(%)| 60.27 | 60.82 | **61.34** | 60.90 | 60.85 |\n\n|λ_2| 0.01 | 0.025 | 0.05 | 0.075 | 0.1 |\n| -------- | -------- | -------- | -------- | -------- | -------- |\n|Accuracy(%)| 61.23 | 60.87 | **61.34** | 60.77 | 61.14 |\n\n|α| 0.4 | 0.5 | 0.6 | 0.7 | 0.8 |\n| -------- | -------- | -------- | -------- | -------- | -------- |\n|Accuracy(%)| 61.11 | 60.95 | **61.34** | 60.73 | 60.82 |\n\nFinally, **we hope this clarification strengthens the impact of our work and improves its evaluation.**"}]

Author rebuttal:

Meta-review: This work studied language bias in Visual Question Answering (VQA), where models over-rely on language bias rather than multimodal reasoning. The primary motivation is a lack of principled understanding of bias formation. This work conducts theoretical and empirical analyses, identifying modality-specific gradient imbalances as the core challenge. This work further proposed Multi-Margin Collaborative Debiasing (MMCD) framework with a difficulty-aware contrastive learning mechanism to dynamically determine decision boundaries. Experiments on multiple VQA benchmarks validated the effectiveness and robustness of the proposed method.

The main strengths are (1) the theoretical contributions for language bias in VQA in aspects with modality-specific gradient imbalances, skewed feature fusion, and classifier weight deviations, (2) the novelty of the proposed Multi-Margin Collaborative Debiasing (MMCD) in a principled manner, (3) strong experimental results on VQA benchmarks. The main weaknesses were (1) the complexity and computation due to multiple auxiliary modules and computational efficiency analysis, (2) generalizability to broader bias scenarios beyond language bias in VQA. After rebuttal, the reviewers' concerns have been addressed, and this work received four 5 ratings. The AC agreed with reviewers and recommended accepting this paper.
Decision: Accept (poster)