Title: Language‑Bias‑Resilient Visual Question Answering via Adaptive Multi‑Margin Collaborative Debiasing
Venue: NeurIPS 2025
Accepted: yes
Keywords: Language biases

Multi-margin

Debiasing

Frequency-aware

Confidence-aware

Difficulty-aware
Abstract: Language bias in Visual Question Answering (VQA) arises when models exploit spurious statistical correlations between question templates and answers, particularly in out-of-distribution scenarios, thereby neglecting essential visual cues and compromising genuine multimodal reasoning. Despite numerous efforts to enhance the robustness of VQA models, a principled understanding of how such bias originates and influences model behavior remains underdeveloped. In this paper, we address this gap through a comprehensive empirical and theoretical analysis, revealing that modality-specific gradient imbalances, which originate from the inherent heterogeneity of multimodal data, lead to skewed feature fusion and biased classifier weights. To alleviate these issues, we propose a novel Multi-Margin Collaborative Debiasing (MMCD) framework that adaptively integrates frequency-, confidence-, and difficulty-aware angular margins with a dynamic difficulty-aware contrastive learning mechanism, to dynamically reshape decision boundaries. Extensive experiments across multiple challenging VQA benchmarks confirm the consistent superiority of our proposed MMCD over state-of-the-art baselines in combating language bias.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles performance degradation in Visual Question Answering (VQA) caused by language bias. Through theoretical and empirical analysis, the authors identify modality-specific gradient optimization deviation, which results in feature fusion skewness and classifier weight imbalance. To mitigate this, they propose MMCD (Multi-Margin Collaborative Debiasing), a framework that reshapes decision boundaries under biased training via three adaptive angular margins (frequency, confidence, and difficulty-aware) and difficulty-aware contrastive learning mechanism. Experiments on multiple benchmarks demonstrate the effectiveness of MMCD.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nS1: Comprehensive and in-depth analysis: Through a combination of empirical and theoretical analyses, the authors systematically explore the formation mechanism of language bias from modal gradient optimization deviation, feature fusion deviation, and classifier weight direction deviation. Besides, the authors also provide theoretical support from the gradient and spectral perspectives as to why the margin-based mechanism can mitigate the bias, and the analysis is thorough.\nS2: The proposed MMCD framework innovatively integrates frequency-aware, confidence-aware, and difficulty-aware angular margins, as well as a dynamic, difficulty-aware contrast learning mechanism.Experiments show that the framework outperforms the existing baseline in several challenging VQA benchmarks, effectively countering language bias.\nS3: The writing is well-organized and easy to follow.\n\n\tWeaknesses:\nW1: MMCD introduces multiple branches (e.g., unimodal classifiers) and a difficulty estimation mechanism that bring additional calculation expense, and it is recommended to add an experimental analysis of the computational efficiency (inference time, runtime memory usage) to assess the feasibility of real-world deployment.\nW2: MAM mechanism relies on priors of question types and answer distributions, is it generalizable under different data distributions? It is recommended to conduct experiments on GQA/ScienceQA to verify the generalization performance of the method.\nW3: With the introduction of additional modules (FAM, CAM, DAM, DCL) in MMCD, several hyperparameters need to be tuned during the training process, so is there any difficulty in selecting ,combining and optimizing the hyperparameters with different network architectures and datasets?\n\nQUESTIONS:\nPls refer to Strengths and weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an empirical and theoretical investigation into why VQA models develop language biases. The authors find that skewed gradient norms between the visual and question encoders lead to question-dominated fused features and classifier weights. They also explain why margin-based methods can help alleviate this issue. Building on these insights, the authors propose a debiasing method, and experiments involving different architectures demonstrate improvements over existing approaches.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The paper offers an in-depth exploration combining experiments and mathematical derivation to uncover how biases arise in VQA models.\n\n2. Experiments demonstrate that the proposed method is consistently effective across different architectures and setups.\n\n\n## Weaknesses\n\n1. The Method section is difficult to follow. It would benefit from a high-level roadmap: a brief overview of the intuition and purpose behind each component before diving into the technical details.\n\n2. The caption for Figure 1 is confusing. Perhaps the current (c) is redundant, and the two (d)’s should instead be labeled (c) and (d). Additionally, the caption is not very informative—it would be helpful to briefly describe the elements shown in each plot.\n\n3. The ablation study on the three components of Multi-Grained Adaptive Margins is limited (it currently appears in the appendix but would be better placed in the main paper). Instead of adding each component individually to the baseline, it would be more informative to remove one component at a time from the full method to assess each component’s marginal contribution.\n\n4. Can the authors clarify how many hyperparameters the method has in total? From Figures 3 and 6, it appears there are six, which would make tuning very challenging in practice. Moreover, the plots seem to show that performance is quite sensitive to even a single hyperparameter changing from its optimal value, let alone simultaneous variations in multiple parameters.\n\n\n5. As a non-expert in this specific topic, I cannot fully evaluate whether the baselines, comparisons, and experimental setups cover the state of the art; I welcome perspectives from other reviewers.\n\nQUESTIONS:\nPlease see the questions raise in the Weaknesses section.\n\nLIMITATIONS:\nThe limitations are discussed in Appendix K.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the issue of language bias in Visual Question Answering (VQA). The authors first conduct a thorough theoretical investigation into the origin of language bias, identifying modality-specific gradient imbalances, biased feature fusion, and classifier weight deviations as key factors. To tackle these issues, they propose a Multi-Margin Collaborative Debiasing (MMCD) framework, which consists of Multi-grained Adaptive Margins (MAM), including frequency-aware, confidence-aware, and difficulty-aware margins. And together with a Difficulty-aware Contrastive Learning (DCL) module. Extensive experiments on OOD benchmarks demonstrate that MMCD significantly outperforms previous state-of-the-art methods, showcasing its robustness and effectiveness in mitigating language bias.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1. The paper conducts a comprehensive theoretical and empirical study into the origin of language bias in VQA, identifying three key factors, which is useful to the task and future research.\n2. The proposed framework is well motivated by prior observations and insights. It integrates multi-granularity margin design (answer frequency, model confidence, and sample difficulty) with contrastive learning in a principled way.\n3. The method consistently outperforms baselines on OOD benchmarks and exhibits strong generalization across diverse backbone architectures.\n\nWeakness\n1. The adaptive margin computation (particularly the sample-specific difficulty computation) likely adds complexity, but no analysis of training time or memory cost compared to baselines. This could also limit the scalability to larger datasets due to computational overhead.\n2. The narrow focus on specific types of bias and the reliance on dataset statistics may significantly hinder real-world deployment.\n\nQUESTIONS:\n1. The MMCD hyperparameters (λ₁, λ₂, λ₃, α, β, and w) appear to play critical roles in the overall performance. However, it remains unclear how sensitive the method is to these hyperparameters, and how the specific combination was determined during experimentation. Can the method maintain robust performance across different datasets without extensive hyperparameter tuning?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the persistent issue of language bias in Visual Question Answering (VQA), where models tend to rely on superficial linguistic correlations rather than genuine multimodal reasoning. The authors conduct a principled analysis to uncover how such biases arise and introduce a novel mitigation framework: Multi-Margin Collaborative Debiasing (MMCD). The paper makes a substantial conceptual and empirical contribution to the VQA field by clarifying the origins of language bias and proposing a theoretically grounded approach to address it.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n1. Theoretical Insight into Bias Formation\nThe paper offers a principled and detailed analysis of how language bias emerges in VQA models, linking it to modality-specific gradient imbalances, feature fusion asymmetries, and classifier weight directionality. This goes beyond empirical fixes and provides a deeper understanding of the underlying mechanisms—something that has been largely underexplored in prior work.\n2. Novel and Integrated Debiasing Framework (MMCD)\nDespite the presence of missing data during training or inference, SHIFT maintains competitive or superior predictive performance (as measured by c-index) compared to strong baselines. The model’s generalizability across cohorts (TCGA, CPTAC, UCI) suggests it is not overfitted and can scale across populations and institutions without manual adjustment.\n3. Strong Empirical Results and Generalization Across Architectures\nMMCD demonstrates state-of-the-art performance on challenging datasets such as VQA-CP v2 and v1, particularly in bias-prone categories (e.g., \"yes/no\", \"number\"). The method is also shown to be model-agnostic, achieving consistent improvements across multiple backbone architectures, including SAN, S-MRL, and LXMERT.\n\nWeaknesses：\n1．The proposed MMCD framework introduces multiple auxiliary components—such as adaptive margins, difficulty tracking, and contrastive learning—which significantly increase the training complexity and computational cost. This may limit the practical applicability of the method, especially in resource-constrained or real-time scenarios.\n2. While the paper provides a thorough treatment of language bias, it does not explicitly address other forms of bias prevalent in VQA, such as visual shortcut bias, question-type bias, or multimodal co-adaptation. This narrow focus may limit the generality and robustness of the proposed method when applied to broader or more complex bias scenarios.\n\nQUESTIONS:\n1.How sensitive is MMCD to the choice of hyperparameters (e.g., λ₁, λ₂, σ in the margin formulations), and how does this affect its generalizability across different datasets or domains?\n2.Given that the difficulty-aware module relies on handcrafted metrics like learning speed and classification margin, have the authors considered replacing or complementing this with a learnable, data-driven difficulty estimator?\n3.Can the proposed MMCD framework be effectively adapted to or integrated with large-scale pre-trained multimodal models (e.g., BLIP-2, Flamingo, or LLaVA), and if so, what challenges are anticipated in such a transfer?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-22"}