Title: Adaptive Logit Adjustment for Debiasing Multimodal Language Models
Venue: NeurIPS 2025
Accepted: no
Keywords: Large Multimodal Model

Fairness

Image-to-Text

Logit Adjustment
Abstract: Vision-Language Models (VLMs) and Large Multimodal Models (LMMs) have significantly advanced image-to-text generation tasks such as image captioning and visual question answering (VQA). 
However, these models often exhibit biases, including attribute misalignment between the generated text and the input image, or the reinforcement of harmful stereotypes.
Existing debiasing techniques primarily focus on modifying representations at the encoder or decoder level, which can degrade model performance and may be susceptible to bias reintroduction from external sources. In this work, we propose Adaptive Logit Adjustment (ALA) for Bias Alignment and Neutralization, a post-hoc debiasing method that operates directly on logits during autoregressive text generation. Unlike prior approaches that modify internal representations, ALA selectively adjusts token probabilities to mitigate biases without distorting essential model outputs. Our approach leverages external classifiers to measure bias misalignment between image and text, applies gradient-based importance analysis to identify bias-inducing tokens, and dynamically refines token probabilities to reduce undesired biases. 
We evaluate ALA on image captioning and various VQA tasks, demonstrating its effectiveness in mitigating bias while maintaining contextual accuracy. Notably, our approach is applicable to various multimodal architectures in a model-agnostic manner, including VLMs and LMMs, across different tasks that involve autoregressive text generation. Our results show that logit-based debiasing offers a flexible and efficient alternative to existing encoder- and embedding-centric approaches, providing a more practical solution for building fairer multimodal AI systems.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Adaptive Logit Adjustment (ALA), a novel post-hoc debiasing method for Vision-Language Models (VLMs) and Large Multimodal Models (LMMs). ALA operates directly on the output logits during autoregressive text generation. It leverages external classifiers to detect bias misalignment and uses gradient-based analysis to identify and suppress bias-inducing tokens.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The proposed method addresses a critical problem in multimodal AI: reducing harmful biases while preserving task performance.\n\n- The paper is well-structured and easy to follow, with a clear motivation, methodology, and empirical validation across multiple datasets and tasks.\n\nWeaknesses\n- The performance improvements of ALA over existing baseline methods appear to be marginal in some of the reported results, which raises questions about its practical advantage.\n\n- It is unclear why ALA-BA and ALA-N need to be treated as separate variants. If bias alignment and neutrality cannot be achieved simultaneously within a unified framework, the practical utility of the method may be significantly limited.\n\nQUESTIONS:\n1. Why do ALA-BA and ALA-N underperform compared to the baseline or other comparable methods in most cases in Tables 2 and 3? Does this suggest that ALA-BA and ALA-N may have limited practical value?\n\n2. In Tables 2 to 4, could the authors consider averaging across multiple metrics to more clearly demonstrate the performance advantages of ALA-BA and ALA-N?\n\n3. Could the authors provide visualizations of captioning and VQA results on specific samples for different methods, instead of relying solely on quantitative metrics?\n\n4. What is the rationale for designing ALA-BA and ALA-N as separate methods? To improve practicality, is it possible to integrate the two approaches into a unified framework?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
In this work, the authors propose Adaptive Logit Adjustment (ALA) for Bias Alignment and Neutralization, a post-hoc debiasing method that operates directly on logits during autoregressive text generation. Unlike prior approaches that modify internal representations, ALA selectively adjusts token probabilities to mitigate biases without distorting essential model outputs. The proposed approach leverages external classifiers to measure bias misalignment between image and text, applies gradient-based importance analysis to identify bias-inducing tokens, and dynamically refines token probabilities to reduce undesired biases. The authors evaluate ALA on image captioning and various VQA tasks, demonstrating its effectiveness in mitigating bias while maintaining contextual accuracy.

STRENGTHS AND WEAKNESSES:
Strengths:
1. In this paper，the authors propose Adaptive Logit Adjustment (ALA) for Bias Alignment and Neutralization, a post-hoc debiasing method that operates directly on logits during autoregressive text generation.

Weakness:
1. From the Table 2, I find the results of the proposed method are not satisfactory enough
2. The authors claim that the ALA can also mitigate biases introduced by external sources such as RAG, making it suitable for a wide range of applications. But I can't find experiments to verfy that point
3. In my view, only adjusting logit value is a delaying tactic and cannot fundamentally eliminate the bias, as the bias still exists in the model parameters. Without modifying the parameters, it is impossible to completely eliminate the bias

QUESTIONS:
NA

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

comments: SUMMARY:
This paper proposes a post-hoc debiasing method called ALA for vision-language models (VLMs). ALA operates directly at the logit level in autoregressive models to align and neutralize bias. The method incorporates an adaptive selection mechanism to adjust token probabilities and mitigate biased outputs. Experiments across multiple tasks demonstrate the effectiveness of the proposed approach.

STRENGTHS AND WEAKNESSES:
Pros

- The paper is well-written and clearly structured. The motivation is solid and addresses an important and timely challenge in vision-language modeling—mitigating bias in generated outputs.

- The proposed method is simple yet effective. By operating directly on the logits, it introduces a practical post-hoc approach to debiasing. Experimental results support the method’s effectiveness across different VLM tasks.

Cons

- The paper lacks qualitative examples comparing debiased outputs with baseline methods. Including such examples in the main paper would help the reader better understand the practical impact of ALA.

- Minor issue: while ablation studies are included in the supplementary material, the absence of such results in the main paper reduces its self-containedness.

QUESTIONS:
- Can the authors provide qualitative examples (including failure cases) to compare the debiasing effects of ALA with other baseline methods? This would help visualize the benefits and limitations of the proposed approach.

- According to Algorithm 1, the performance of ALA seems to rely heavily on the quality of the pretrained image encoder. How does the debiasing performance vary with different encoders? An empirical or qualitative discussion would be valuable.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of debiasing in multimodal LLMs (or LMMs in the paper). Because modifying or tuning intermediate features could lead to side effects, e.g., overall degraded performance, this paper proposes a method to mitigate these biases by adjusting the output logits of the LLM. This involves using an image classifier to output \"sensitive attribute\" signals, and trying to align the output of a text classifier such that the two signals are of similar value.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- Adjusting logits instead of internal features of the model (via retraining) can be beneficial not just because of training costs, but also because it doesn’t risk catastrophic forgetting of LLMs.\n\n## Weaknesses\n\n### Issues with clarity\n\n- “Sensitive-attribute signal” is not well defined. It is simply stated to have a range of [-1, 1], but it is not clear what it aims to measure. Judging by Figure 2, this signal seems to be akin to a binary classifier, where a score closer to -1 means the image depicts one value of the attribute (e.g., “female”) and a score of 1 means the image depicts another value of the attribute (e.g., “man”).\n    - Then, it is not clear how this would work for attributes with multiple values, such as race.\n- Building on the point above, it is not clear how to define “sensitive attributes” as well. For the purpose of this paper, the sensitive attributes seem to be defined entirely manually, to detect whether an image may contain gender or racial cues.\n- There is a lack of qualitative results that exhibit the debiased outputs by applying ALA-BA and ALA-N. It would be especially helpful to observe the adjustments made via ALA-N on the gender/racial attributes.\n\n### Fundamental issues with methodology\n\n- f^text and f^image need to be trained on an external dataset, to detect for sensitive attributes. However, the classifiers f^image and f^text that form the foundation of this approach may themselves be subject to severe biases, depending on their training procedures and data. There is no reason to assume these classifiers are free from the very biases they are meant to detect and correct, especially if they are trained on less data.\n- This methodology places the determination of what constitutes \"bias\" entirely in human hands (through the manual selection of which attributes to target and training of these classifiers). This creates a fundamental paradox: we are essentially imposing our own human biases in hopes to “eliminate bias”. Rather than achieving true debiasing, this approach forces particular human perspectives on what should be considered biased behavior. This strikes me as antithetical to the goals of debiasing research, which should aim to reduce arbitrary human prejudices rather than systematically embed them into our models.\n\nQUESTIONS:\nHow is f^image and f^text trained, and what exactly are \"sensitive-attribute signals\"? L192 states that these models are trained on two datasets, but it is not explicitly clear how these models are trained to achieve the desired functionality.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 2, "CLARITY": 1, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-07-02"}