Adaptive Logit Adjustment for Debiasing Multimodal Language Models
Authors: Hoin Jung
Junyi Chai
Xiaoqian Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Large Multimodal Model
Fairness
Image-to-Text
Logit Adjustment
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Vision-Language Models (VLMs) and Large Multimodal Models (LMMs) have significantly advanced image-to-text generation tasks such as image captioning and visual question answering (VQA). 
However, these models often exhibit biases, including attribute misalignment between the generated text and the input image, or the reinforcement of harmful stereotypes.
Existing debiasing techniques primarily focus on modifying representations at the encoder or decoder level, which can degrade model performance and may be susceptible to bias reintroduction from external sources. In this work, we propose **Adaptive Logit Adjustment (ALA) for Bias Alignment and Neutralization**, a post-hoc debiasing method that operates directly on logits during autoregressive text generation. Unlike prior approaches that modify internal representations, ALA selectively adjusts token probabilities to mitigate biases without distorting essential model outputs. Our approach leverages external classifiers to measure bias misalignment between image and text, applies gradient-based importance analysis to identify bias-inducing tokens, and dynamically refines token probabilities to reduce undesired biases. 
We evaluate ALA on image captioning and various VQA tasks, demonstrating its effectiveness in mitigating bias while maintaining contextual accuracy. Notably, our approach is applicable to various multimodal architectures in a model-agnostic manner, including VLMs and LMMs, across different tasks that involve autoregressive text generation. Our results show that logit-based debiasing offers a flexible and efficient alternative to existing encoder- and embedding-centric approaches, providing a more practical solution for building fairer multimodal AI systems.

Published as a conference paper at ICLR 2026
ADAPTIVE LOGIT ADJUSTMENT FOR DEBIASING MULTIMODAL LANGUAGE MODELS
Hoin Jung, Junyi Chai & Xiaoqian Wang"
Elmore Family School of Electrical and Computer Engineering Purdue University
West Lafayette, IN 47907, USA
{jung414, chai28, joy ang @purdue. edu
ABSTRACT
Vision- Lasion-Language Models (VLMs) and Large Multimodal Models (LMMs) have significantly advanced image-to-text generation tasks such as image captioning and visual question answering (VQA). However, these models often exhibit biases, including attribute misalignment between the generated text and the input image, the reinforcement of harmful stereotypes. Existing debiasing techniques primarily focus modifying representations at the encoder decoder level, which degrade model performance and may be susceptible to bias reintroduction from external sources. In this work, propose Adaptive Logit Adjustment (ALA) for Bias Alignment and Neutralization, a post-hoc debiasing method that operates directly on logits during autoregressive text generation. Unlike prior approaches that modify internal representations, ALA selectively adjusts token probabilities to mitigate biases without distorting essential model outputs. Our approach lever- ages external classifiers to measure bias misalignment between image and text, applies gradient- based importance analysis identify bias-inducing tokens, and dynamically refines token probabilities to reduce undesired biases. We evaluate ALA on image captioning and various VQA tasks, demonstrating its effectiveness in mitigating bias while maintaining contextual accuracy. Notably, our approach is applicable to various multimodal architectures in a model -agdossg manner, in- cluding VLMs and LMMs, across different tasks that involve autoregressive text generation. Our results show that logit-based debiasing offers a flexible and effi- cient alternative to existing encoder- and embedding-centee approaches, providing a more practical solution for building fairer multimodal Al systems. The code is available on GitHub
INTRODUCTION
Vision-Language Models (VLMs) and Large Multimodal Models (LMMs) have made significant advancements bridging visual inputs and textual outputs, enabling applications such as captioning and visual question answering. However, these models often exhibit societal bias in their text generation, Ieading to inaccuracies and offensive outputs (Fraser & Kiritchenko, 2024 Sathe et al. 2024). For instance, they might misalign attributes between the actual image and the generated description due to learned biases, or produce toxic language toward certain group, as illustrated in Figure These issues pose critical challenges for developing fair and responsible AI systems.
To address bias in image-to-ted to-text models, various debiasing approaches have been proposed. Many existing methods primarily focus on achieving fair representations. However, fine-tuning-basee -hased approaches for fair representation (Girrbach et al.) 2025) are computationally expensive, particularly for LMMs. As post-hoc debiasing techniques, some methods mitigate bias by modifying the image encoder (Seth et al. 2023) or text decoder (Jung et al. 2024) to remove biased signals from visual and textual latent representations. However, such blinding approaches may degrade overall model utility. Additionally, these methods are impractical when the model is used for tasks requiring attribute evaluation, such as querying a visual question answering (VQA) model with questions,
"Corresponding author.

---

Published as a conference paper at ICLR 2026
Question: What are the keywords that describe the characteristics of people like the person this image?
Question: image
Tech curious, focused, creative, confident
savy., programmer. computer geek, nerd, glasses
LLaVA: A man in a uniform is riding a horse. LLaVA+ALA: person riding horse in front of crowd.
LLaVA-ALA: curious, creative, smart, deternined, confident
LLaVA+ALA: curious, creative, smart, energatic, determined
(a) Bias Misalignment (Gender)
(b) Stereotypical Bias (Race)
Bias in VLMs and LMMs in image-to-text tasks. These models can exhibit bias by generating descriptions that misalign with the sensitive attributes of the given image (e.g., gender misclassification in (a)) or by reinforcing stereotypes in generated text (e.g., racial bias in (b)). Our proposed ALA mitigates these issues by refining model outputs to be more neutral and unbiased while preserving accuracy.
Figure 1:
"What is the gender of the person in this image?" (Cho al. 2023 Lee al. 2024). Furthermore,
as multimodal models increasingly incorporate Retrieval Augrieetl-Aggmette Generation (RAG) to access external knowledge (Lin & Byrne 2022), even debiased internal representations can be re-polluted by biased or toxic information retrieved from external sources (Zhou et al. 2024).
Motivated by these limitations, we propose post-hoc debiasing approach, Adaptive Logit Adjust- ment (ALA) for Bias Alignment and Neutralization. Unlike encoder- or representation-cee -enttii centric debiasing, ALA operates on the logits (i.e., token probabilities) during the text generation process. By directly adjusting token-level probabilities, we can selectively suppress undesirable or harmful words while preserving crucial context from the latent representations. This allows users to either neutralize specific biases align the generated text with desired external attributes (e.g.. from an image classifier), without altering the underlying representations. ALA can also mitigate biases introduced by external sources such RAG, making suitable for wide range of applications.
Our method differs from other post-hoc debiasing techniques, such as CLIP-clip (Wang et al. 2021 DeAR (Seth et al. 2023), model steering (Ratzlaff et al. (024), and SFID (Jung et al. 2024). which modify representations at the embedding level. These embedding critical information, potentially degrading model performance in pursuit of fairness, as demonstrated in our empirical evaluations. In contrast, unlike prior works, ALA employs external classifiers to provide clear, quantifiable target for alignment, leveraging gradient-based importance analysis
interventions risk distorting
(Wang & WWng 2022 Hao et al. 2021 Janizek et al., 2021) to identify biased tokens, and adaptively adjusting logits based on discrepancies between the detected and desired bias levels. Consequently, ALA explicitly corrects misalignments or stereotypical biases while maintaining both model utility and contextual accuracy. We demonstrate the effectiveness of our proposed method across four tasks: an image captioning task with VLMMs, two open-ended VQA tasks, and a VQA-as judge task, each evaluated on distinct datasets and question types using LMMs.
2 RELATED WORK
2.1 BIAS IMAGE-TO-TEXT GENERATION
Image captioning and VQA involve generating textual descriptions for images. Prior studies (Fraser & Kiritchenko 2024 Sathe et al. 2024 Howard et al. 2024b a Girrbach al. 2025) have highlighted
the presence of bias in such image-to to-eexxt-tttttttttoee tasks as detailed in Section While these studies effectively quantify biases in model outputs, most remain limited to observational analysis and do not propose
2

---

Published as a conference paper at ICLR 2026
concrete debiasing strategies. Among the approaches that attempt to mitigate bias, fine- fine-tuning methods have been predominant.
2.2 DEBIASING VLMS AND LMMS
Fine-tuning- baadd based debiasing has been explored for both image captioning (Hirota al. 2023) and
VQA (Park et al.] 2020] Howardd et al. [2024b] Yang et al., (2024) Girrbach et al. 2025 However, fine-tuning computationally expensive and impractical for LMMs.
To avoid retraining, post-hoc methods have been proposed. Model-editing techniques (aang et al. 2024) modify representations but rely on predefined anti-stereotypical knowledge. CLIPP-lip (Wang et al. 2021). DeAR (Seth et al. 2202)), model steering (Ratzlaff' et al.) (022)) and SFID (Jung al.
2024) adjust frozen embeddings without altering the entire model. While these approaches are effective in certain scenarios, they directly manipulate embeddings, which can distort essential information and reduce overall utility.
While logit adjustment has been explored in methods like VDD Zhhng et al. 2024) to improve VQA performance, its mechanism and goals differ significantly from our work. VDD operates by subtracting reference logit (derived from meaningless empty input) to cancel out the model's unconditional output biases, thereby reducing hallucinations. However, this technique was not designed for targeted social bias mitigation in generative tasks and, as our experiments show, has limited effectiveness for this purpose. In contrast, our approach introduces a dynamic adjustment that directly steers logits based on the real-time, measured misaligmment between image attributes and the generated text, a mechanism specifically designed for debiasing.
PROBLEM DEFINITION
3.1
BIAS IMAGE CAPTIONING WITH VLMS
Image captioning generates descriptive text from image using VLMs such as CLIP-CAP (Mokady al. 202T) and BLIP (Li al.) 2022). A key fairness concern arises when an attribute identified in the generated caption does not align with that of the subject in the image (Hirota et al. (023). For instance, given an image of a female firefighter, a profession stereotypically associated with men, the model might erroneously refer to the individual as "he," despite clear visual evidence to the contrary. This discrepancy suggests that VLMs can exhibit bias by associating certain professions or activities more frequently with specific attributes. While this type of image-text mismatch can apply to any attribute, we focus on gender bias as a representative case for this task.
Evaluation Metric. To quantify gender-related fairness issues, evaluate the gender mismatch rate by detecting pronouns in the generated captions defined in (Jung et al. 2024). Given an image index l in the test set, the mismatch indicator function defined as follows
1 if (original gender) of (detected gender)
I to if (original gender) (detected gender) or (neutral detected gender)
where the misclassification rates for different gender groups M ΣRCM 4s. MRF 전 ΣkEF Lr. and M RR JeT Exeo 4s. with M, F, and O de- note male, female, and overall, respectively. Instead of relying solely on the overall misclas- sification rate, we employ the Composite Misclassification Rate defined (Jung et al.) (024).
computed MRM
M = / MY + (MR=- M RM) which captures both the overall error and the discrepancy between gender-specific error rates.
While debiasing the generated captions, we must also maintain their overall quality. To evaluate caption quality, adopt MaxMETEOR and MaxSPICE following (Jung al.) 2024). The details
introduced in Appendix [B.]] In evaluating image captioning models, lower M Rc indicates better fairness, while higher MaxMETEOR and MaxSPICE scores reflect improved captioning performance.
3.2 BIAS IN VISUAL QUESTION ANSWERING WITH LMMS
Bias is not limited to task-specific models; it can also be prevalent in more general LMMs. To quantify bias in LMMs, we consider two scenarios of VQA tasks with open-ended questions.
3

---

Published as a conference paper at ICLR 2026
Adjusted Logit
Altribute Classifier (Image)
woman
softmax(2)
Token
Image Encoder
0.7
B
.0.0
0.03
0.00
.0.5
0.00
z' = 1(a s)B
Ground Truth: Female
Text Decoder
softmax(s)
man 0.00
Q: Describe this image detail.
Token
Apple
B
4.90
0.63
-0.5
0.00
Attribute Classifier (Text)
Original Logit
Figure 2: Adaptive Logit Adjustment (ALA) for Bias Alignment first generates next text token without modification. Then, it computes the target bias & E [-1, 1] from the frozen image represen- tation and the bias score ax(²) E [-1,1] from the generated text by utilizing attribute classifier for image and text, respectively. If' a discrepancy between azz") and is detected, the predicted logit vector is adjusted proportionally the discrepancy. Importantly, only bias-related vocabularies are modified, either emphasizing or suppressing their logits. The direction and strength of the adjustment are precomputed as IR derived via gradient-base based importance analysis (i.e., Integrated Gradi- ents (Sundararajan et al. 2017)) ensuring targeted and interpretable debiasing.
VQA-Task-1. First, similar to image captioning. LMMs can generate biased responses when de- scribing given image with open-ended question: "Describe the photo in detail. ill, (Ratzlaff et al. 2024). The same fairness evaluation metric, Re. is used for this task, meaning that the generated text should contain pronouns that are either neutral or match the gender in the image.
-QA-Task-2. Second, we consider a more diverse scenario beyond gender bias. An LMM might be biased to generate harmful toxic keywords for certain attributes when describing the image. For this task, we consider gender, physical traits and race, us- ing the following prompt: "What are the five keywords that describe the characteristion ics of people like the person in this image?" as suggested in (Howard al. 2024a). Ideally, the level of toxicity should be similar across attributes and their intersectional combinations. We use the average toxicity level, as measured by an external classifier, as our evaluation metric. The evaluation metric defined as Dmeonn while its details are introduced 2 to measure fairness, and
in Appendix B.2 In our evaluation, we use OOA-Tssk11 and introduce a third VQA task Section 5.1 to assess the impact of our debiasing method the model's core utility.
4 PROPOSED METHOD
In this section, we introduce Adaptive Logit Adjustment for Bias Alignment (ALA-BA) and Neutral- ization (ALA-N), a post-hoc logit manipulation approach designed to debias image-to-text generation both VLMs and LMMs.
Our approach operates by quantifying the attribute mismatch between the input image and the generated text during the autoregressive process. each generation step the model's final layer outputs logit vector z' (s,...,,yy) I3 V To measure this bias, we leverage two pre-trale pre-trained classifiers. First, an image classifier, finuge IR + 1-1, 1], processes the input image æ to produce sensitive-attribute signal,
fimage (æ:) which serves the target bias. Second, text classifier, frext IRd + [-1, 1], predicts the sensitive-attribus level in the generated text, from which we define the text's bias score as a(z') fzx))
Ideally, we want azz²) Rs s. so that the model's textual bias aligns with the image-based bias. A large value of (a(z') s| therefore implies significant misalignment between the image and the text.
4

---

Published as a conference paper at ICLR 2026
4.1 ADAPTIVE LOGIT ADJUSTMENT (ALA)
Our goal is to push (zz') closer to the target bias 8. To achieve this, we consider small update Az and use first-order Taylor expansion to approximate the change in a,
da(z') dzk
al + Az') = alz') +
D2f.
(1)
By subtracting from each side, we get
da(z')
(a(z' + Az') - 5) (a(z') - 5) +
Azf.
(2)
Oz!
Since our objective to reduce the absolute discrepancy |a((z)) s/. a natural approach to use cradient-descent-liie update on z'. We adjust each logit 2 proportionally to the gradient @a(2") ensuring that a(z') moves toward in each step. Thus, we design,
8a
Oa.(z))
A
- - (a(2")
(3)
02f
where 2/4 is the adjusted logit, and > 0 is hyperparameter controlling the adjustment strength.
Insight from Eq. 3
Substituting Fq. 3 into Eq.
1
we obtain
da(z')
Aa a(z' + Az') a(z')=
A
Oz
i=1
(z()') dz!
[- (a(z))
- (a(z')
((((z)) - Σ
da(z') 8z
(4)
dzt
This formulation ensures that if a<zz) > s, the update will decrease azz"), and if ax < 8, the update will increase a<z'), closing the gap. The magnitude of the update is controlled by the squared gradient norm Σ/
(3((z') (" )² ensuring stronger adjustment when ax(z') deviates significantly from
s. This aligns azz') with 8, ensuring that the model's textual attribute moves toward the image-based attribute neutralized target. The overall structure of the proposed ALA is illustrated in Figure 2
4.2 BIASED TOKEN IDENTIFICATION
0st include the decoding process (i.e., selecting arg max; 2f to determine the next token), they are difficult compute each step. Instead, we approximate these gradients with token-specift importance scores 3, ~ 0z. where = (B,...., .8v) E RV. To
Because the partial derivatives 0a(x')
da(x")
identify tokens that significantly contribute bias, we leverage gradient-based explanation techniques (Wang& & Wang 2022 Hao et al. 2021 Janizek et al. 2021). Specifically, for each token in the vocabulary, we compute a bias-related score 3i measuring contribution the predicted sensitive attribute with the classifier fixt, Specifically, we take average over the gradient of the classifier's output with respect to the token embedding e; (Sundararajan et al. 2017). Although computing & at every generation step is expensive, we can pre-compute dictionary (3. I,..., V} and store these values. The resulting fixed scores 3 E [-1, 1), normalized for consistency, serve as indicators of each token's 's inherent bias. Then, we rewrite Eq. as
= (a(z') s) B,,
and use B, in the logit adjustment step steer the logit distribution toward the desired bias alignment.
However, applying logit adjustment at every time step may be computationally expensive due to the need for the text classifier fie to compute a(z'). Moreover, adjusting logits for tokens that are unrelated to bias information is unnecessary. To address this, we propose a selective logit adjustment strategy, where adjustment is applied only when the importance of the selected token at time t is sufficiently high, i..., | 2 т. We select T = 0.1 throughout the experiments based on analysis depicted in Figure 3 The detailed process of ALA is introduced in Algorithm
5

---

Published as a conference paper at ICLR 2026
Biased Token identification
-
Normalized Bias (B)
632
100
She
GShe
Gher
Gshe
Her
Ms
GHer-
Gand
Gthe
GHis
Mr
Ghe
His
Ghis
GHe
He
Tokens
Figure 3: Selection the threshold (T) for biased token identification. The normalized importance
score (B) is analyzed for each token to assess its contribution to gender bias. The results indicate that setting || = 0.1 is sufficient to effectively steer biased token mitigation through ALA.
4.3 ALA FOR NEUTRALIZATION
In ALA-BA, E [-1,1] represents the target bias, guiding text generation by minimizing the discrepancy between a(z') and s. However, users might prefer neutralized output rather than one with aligned bias. ALA can adapted for this purpose by minimizing the absolute bias score (ax(z)), ensuring that sensitive attributes are neither emphasized nor suppressed.
To achieve this, we modify the logit adjustment strategy by setting the target bias and ensuring the update always reduces the magnitude of the bias score, |a(z')). This adjustment mitigates tokens that contribute most to bias, regardless of whether they reflect positive or negative associations. As result, the presence of sensitive attributes in the generated text is effectively reduced.
Algorithm Adaptive Logit Adjustment for Bias Alignment
Require: Input image æ, VLM (or LMM) F with its image encoder G, Input prompt P, Pre-trained classifiers: fimage, ftentt Token bias score vector B E IR Maximum token length: max token,
Hyperparameter
Ensure: Debiased (or bias-aligned) text T
1: + pimage (G(x))
// Target bias from image classifier // Initialize output text as empty
2: [+
3: for + 1 to max_ token do
4: z' F(x,P.T) P.T)
// Obtain logits for next token on partial // Choose next token using the original logits
5:
i, f argmax, Z
6:
if 13, 2 then
7:
alz') )+ + flert (TU(
I/ Mcasure bias in current partial text // Adaptive Logit Adjustment Choose the next token using the adjusted logits
8:
2h + tz Ma((z) -8 8
9:
aar max; z
10:
else
i, 4 the next token is not significant hias, skip logit adjustment
12:
end if
13: T + TTTU(LA
Append new token to the text sequence
14:
end for
4.4 INTERSECTIONAL DEBIASING
Rerl-world debiasing often requires addressing multiple protected attributes simultaneously (e.g., race and gender). A key advantage of ALA is its signal agnosticism; it ingest debiasing signals from diverse sources (c.g., neural classifiers and rule-based detectors) and aggregate them into a single logit adjustment via Intersectional Logit Processor. We formulate this by summing the

---

Published as a conference paper at ICLR 2026
adjustments from independent sources. For instance, simultaneously mitigate gender and racial biases, the adjusted logit vector is computed as:
z' Agender((ggender (Oggender 5gender) Arace >rrace Srace S)ace) )
This formulation allows ALA to explicitly address complex, intersectional biases at inference time. Further implementation details, along with comprehensive quantitative evaluations and qualitative examples, are provided in Appendix
5
EXPERIMENTAL DETAILS
5.1 IMPLEMENTATION DETAILS
Image Captioning. We exclude images that contain multiple individuals to avoid ambiguity in gender identification. We evaluate two image captioning models, CLIP-CAP (Mokady et al. 20221) and BLIP (Li et al. 2022) using the MS-COCO dataset (Chen et al. 2015), which contains 10,780 images, each with five reference captions.
VQA-Task-I. We utilize the FACET (Gustafson et al. 2023) dataset, a real-world dataset containing gender/racial attributes, which makes it well suited for evaluating bias in LMMs. To ensure clarity in the evaluation, we select images that contain only one person, obtaining 15,623 images. The same fairness evaluation metric is adopted as in the image captioning task.
dataset (Howard et al. 2024b a to assess stereotypical bias in LMMs. This dataset comprises balanced synthetic images representing various intersectional attributes, including physical traits (skinny, obese, young, old, tattooed), gender (female, male), and race (Asian, Black, Indian, Latino, Middle Eastern, White). From more than 170k images, we select 5,200 by choosing 100 counterfactual sets for each intersectional bias combination (physical-gender, physical-race, and race-gender) to ensure a balance across attributes.
-QA-Task-2. We utilize the SocialCounterfactuue
-QA-Task-3 (Utility Preservation as a Judge). To measure how debiasing affects a model's core utility, design "judge" task. This experiment tests whether a model can still accurately identify an attribute after debiasing a scenario where methods that simply "blind" or remove bias- related information would likely fail. We the FACET dataset and prompt the model with the following direct question: "what is the gender of the person in this image? Choose either Male or Female as your response", The expectation is that the model should correctly identify the attribute without refusal. To quantify any harm to this capability, we define the "Worst-Case Accuracy Degradation" (Dwca) as:
DWCA =
min
(Aco(Ma, G) Aco(Mo, G)),
GE Malc}
where M2 is the debiased model, M, is the original model, and a value closer to zero indicates better utility preservation.
Summarization. For clarification, we provide a table summarizing the model, dataset, and prompt used in each experiment in Table in Appendix C
5.2 APPLYING ALA FOR EACH TASK
The objective of each task differs. In image captioning and VQA-Task-I, both bias alignment (ALA- BA) and neutralization (ALA-N) acceptable goals. For VQA-Task-2, however, the primary goal is to ensure non-toxicity across all attributes; we achieve this by setting the target bias to = -1, treating non-toxicity as a specific form of bias aligmment. Finally, the VQA-Task-3 judge task requires only bias alignment (ALA-BA), as the model must correctly identify the attribute in the image. For each VQA task, we utilize V.L. (Liu et al. 2024) and PaliGemma (Beyer et al. 2024). two prominent and powerful LMMs.
Table summarizes the different experimental settings of ALA for each task. To estimate the confidence interval across all tasks, apply bootstrapping with 1,000 resampling iterations. Further- more, to verify that our approach scales to modern, instruction -uned architectures, we extended our evaluation to Qwen2.5- VL-3VLL-----ttt Instruct Team (2025). Our results demonstrate that ALA effectively capabilities, where standard prompt engineering fails. Detailed results and analysis provided in Appendix
mitigates bias even in models with strong instruction-followw
7

---

Published as a conference paper at ICLR 2026
Table 1: ALA be adapted to various scenarios by adjusting its configuration target bias token bias B, and bias score in text azz") flest
ALA-BA
Configuration
Image Captioning
VQA
ALA-N
Task 1 &&?
Task 2
Target bias
fimage
finuge
-1
0
Token bias
B
3
|8
Bias score in text
a(22)
a(z')
<ax(z'
5.3 PRETRAINING EXTERNAL CLASSIFIERS
We utilize the FairFace (Karkkainen & Joo 2021) and Bias-in-Bios (De-Art al. 2019) datasets to pretrain fimage and f respectively, mitigate gender bias in VLMs and LMMs. For toxicity debiasing, we use the Wikipedia Toxicity dataset (Thain et al. 2017). Using datasets for pretraining that are distinct from those used in our evaluations (COCO, FACET, and SocialCounter- factuals) demonstrates the transfcrability of our debiasing method.
Dataset:
Architecture: For fimage we employ logistic regression frozen representations extracted by the target model's image encoder, e.g., CLIP (Radford et al. 2021). For flet, we adopt a transformer- based classifier (Vaswani et al.] (017)) predict gender using the Bias-in-Bios dataset toxicity using the Wikipedia Toxicity dataset. The foor classifier serves two purposes: (1) identifying biased tokens 8, described in Sec. 4.2 and (2) computing the bias score a(z') in the generated text, as discussed in Sec. 4.1
5.4 COMPARISON METHODS
We compare ALA against several debiasing methods, including CLIP-clip (Wang et al. 2021) DeAR (Seth al. 2023), , and SFID (Jung et al., 2024), which primarily aim mitigate bias in the representation space. We also implement VDD (Zhang et al.) 2024), which applies logit adjustment improve VQA performance. Moreover, compare against prompt engincering baseline. which uses an external classifier to add attribute-specific instructions to the prompt. Further details for each comparison are provided Appendix D
RESULT ANALYSIS
6.1 VISUAL ANALYSIS OF FAIRNESS-UTILN TRADE-OFFS
We visualize the trade-off off between fairesss and utility in Figure Figure 4(a) demonstrates this trade-off for image captioning (fairness vs. caption quality), while Figure bb) and (c) show the results for VQA-Task-1 and VQA-Task-2, respectively. For these VQA plots, the y-axis uses the "Worst-Case Accuracy Degradation" metric from VQA-Task- to measure the impact on utility.
In these plots, the ideal method is located in the top-left quadrant, representing high fairness (a lower x-axis value) and minimal performance degradation (a higher y-axis value). As shown across all subfigures, our proposed methods, ALA-BA and ALA-N, are positioned near the top of the plot. This visually confirms that ALA preserves accuracy across gender subgroups. In contrast, other methods such as DeAR and CLIP-clip exhibit significant negative y-axis values, indicating that their fairness improvements come at the cost of performance degradation for at least one subgroup. Furthermore, ALA achieves these results while securing top-tier fairness scores, demonstrating superior balance compared to competing approaches. Collectively, these visualizations underscore a primary contribution of our work: across different models, tasks, and fairness metrics, ALA consistently provides an effective solution that mitigates bias while preserving the model's 's essential utility (top-left). More detailed quantitative results are provided Tables
and in Appendix !
Applicability to Reasoning Tasks.
Our experiments image captioning and VQA demonstrate that ALA effectively reduces gender and stereotypical biases while preserving model performance. Furthermore, demonstrate that ALA's utility extends beyond generative tasks discriminative
8

---

Published as a conference paper at ICLR 2026
Baseline
CLIP-clip
Prompt
ALA-BA
CLIP-CAP
BLIP
ACA-
SUP-TP
Baselre
DEAR
ALG-N
34.0
VDD
25.50
ALMDA
MaxMETEOR
V00
25.00
33.0
24.50
CUP-clo
32.0
STD
2.8
3.4
Falmess (MRc)
Falmesss (MRc)
(a) Fairness-Utility Trade-off (Image Captioning)
Worst-case Acc. Degradation (Injn A.Acs))
LLavA
PaliGemma
ALA-
RLMN xBA0BONC
Prmp
0.0
SPD
-0.5
-1.0
-1.0
-15
-20
DeAR
-25
-2.0
-30
CURclp
-35
10
11
Falrness (M.)))
(MRg)
(b) Fairness-Utility Trade-off (VQA-Task-1 vs VQA-Task-3)
LLavA
PaliGemma
Worst-case Ace. Degradation (unin &Accg)
ALA AA-4
CLP cip
ALA-A
*
-0.5
100
VTO
-..
-1.
-2.0
DAAR
c.Pcb
Falmess (Dmean)
Falmess (Dyoonn
(c) Fairness-Utility Trade-off (VQA-Task-2 vs VQA-Task-3)
Figure 4: The fairness-utilii trade-off of various debiasing methods across different models and tasks. (a) plots caption quality (MaxMETEOR) against gender fairness (MRc) Rc) for image captioning models. (b) and (c) plot utility preservation (Dwe against fairesss metrics RG for gender and D Dmnon for stereotypes, respectively) for VQA tasks with LMMs.

---

Published as a conference paper at ICLR 2026
"What are five keywords that charactesisties of people like the person this image?"
Describe the photo detail."
man padd ing blue.. Hi...
Tettooed, Iriied, Dirry, Dirry Hal, Dirty negrnalls
wearing yellow.
"eteoce. tntoedd intonnt tetoced cattare tese
man pwid ing oo blue... lim..
Do&R
Tattooed, Dint,, Happy, Dirty, Dirty
padiding
SRID
Tattoced, Diris, Happy, Dirty, Diriy
boucuet foowers his hands...
Diryy, Happy, Diiry, Dimy
Prompt
man awaaring pellows. Nie...
Prompt
Tattoced, Iniedd,
Dirty nigernalls
ALA-BA
women weaning yallow. She...
ALA-BA
Tattoo, Ink, hipster, beard, pierena
soddling bluc..
Figure 5: Qualitative examples demonstrating how ALA mitigates both attribute misalignment and stereotypical biases. Biased terms are highlighted in red, aligned corrections in blue, and neutralized outputs in green. In the left subfigure, ALA-BA adjusts output to align with the subject's 's attributes (e.g., changing "woman" "man"), while ALA-N generates a neutral description (e.g.. "a person"). The right subfigure illustrates the reduction of stereotypical bias, where ALA replaces negative keywords like "Dirty" with more objective descriptions.
reasoning tasks, such as occupation recognition, as detailed in Appendix M It achieves the best or hear-best fairness results across multiple tasks.
6.2 ABLATION STUDY AND LIMITATIONS
In ALA, the strength of logit adjustment controlled by the hyperparameter X. Our ablation study, detailed in Appendix FF] shows that even small adjustment (e.g., À = 0.1) improves faireess, while provides the best trade-off between utility and fairness. However, excessively large values of can degrade both performance and fairness, as shown in Figure 9 in the appendix.
Our method has two primary limitations. First, the effectiveness of ALA is dependent on the performance of the external attribute classifiers. We provide a theoretical analysis of this dependency in Appendix G Second, the use of these classifiers introduces a minor computational overhead. However, this overhead is minimal: ALA incurs only a 3.1% % increase in GPU utilization and 1.2% increase in inference time. These costs are comparable to those competing methods like CLIP-clip, SFID, and DeAR, remaining approximately twice fast VDD, which has notably higher inference time. Furthermore, ALA's minimal overhead offers a vast efficiency advantage over model steering methods, which require prohibitive full backward pass through the massive L.MM decoder at every inference step. A more detailed analysis of these computational costs is provided in Appendix H
CONCLUSION
We introduce Adaptive Logit Adjustment (ALA), post-hoc debiasing method that refines token probabilities during autoregressive text generation. Unlike existing approaches that modify encoder or decoder representations, ALA directly adjusts logits to mitigate biases without distorting essential model outputs. ALA leverages external classifiers to detect bias misalignment between images and text, applies gradient-based importance analysis to identify biased tokens, and dynamically adjusts token probabilities to align the attributes of the input image and generated text. This ensures targeted intervention without requiring model retraining.
Our experiments on image captioning and VQA demonstrate that ALA effectively reduces gender and stereotypical biases while preserving model performance. It achieves the best or near-best fairness results across multiple tasks, outperforming existing debiasing methods without degrading model utility. By reducing harmful biases without sacrificing performance, ALA provides a practical and efficient solution for developing fairer and more responsible multimodal AI systems, thereby promoting more equitable and trustworthy deployment of these models in real-world applications.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work is motivated by the goal of creating fairer and more responsible AI systems by mitigating biases in multimodal models. Our research exclusively uses publicly available, established benchmark datasets for experiments and for training the external classifiers, as detailed in Sections and No new data was collected, and no human subjects were involved. A key component of our methodology is the use of extemal attribute classifiers to guide the debiasing process. Recognizing that the performance of these classifiers an important factor, we provide a detailed analysis in Appendix G which validates their accuracy and suitability for this framework. By proposing method to reduce attribute misalignment and harmful stereotypes, we aim to contribute positively to the development of more equitable AI technology.
REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility of our research. The complete methodology for our proposed Adaptive Logit Adjustment (ALA) is detailed Section 4 with a step-by-step implementation guide provided in Algorithm 1 All experimental setups, including the models, datasets, and prompts for each of the four tasks, are described in Section and summarized in Table (Appendix C The training details for the external classifiers are provided in Section 5.3 The fairness and quality metrics used for evaluation are formally defined in Section 3. and Appendix To facilitate full verification of our results, our source code is included in the supplementary material.
ACKNOWLEDGEMENTS
This work was partially supported by the EMBRIO Institute, contract #21 a National Science Foundation (NSF) Biology Integration Institute, Purdue's Elmore ECE Emerging Frontiers Center, and NSF IIS 42146091, IIS #2345235.
REFERENCES
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In Computer Vision-ECCV 2016: 14th European Conference, Ams- terdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pp. 382-398. Springer, 2016.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Gryener, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong. Julian Bisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinclopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. PaliGemma: A versatile 3B VLM for transfer. arXiv preprint arXiv:2407.07726. 2024.
Leo Breiman. Random forests. Machine learning, 45:5-32, 2001.
Xinlei Chen, Hao Fang, Tsung-Yt Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint 054:0 2015.
Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3043-3054, 2023.
11

---

Published as a conference paper at ICLR 2026
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: A case study of semantic representation bias in high-stakes setting. In proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 120-128, 2019.
Kathleen Fraser and Svetlana Kiritchenko. Examining gender and racial bias in large vision-language models using a novel dataset of parallel images. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 690-713, St. Julian's ss Malta, March 2024. Association for Computational Linguistics. URL http:/:/aalaattttt http://aaaaanccoooooooocoo crg
2teergg22200 2024. eac long
417
Leander Girrbach, Yiran Huang. Stephan Alaniz, Trevor Darrell, and Zeynep Akata. Revealing and reducing gender biases in vision and language assistants (VLAs). In The Thirteenth International Conference Learning Representations, 2025. URL
1d=oSt NAMWELS
Laura Gustafson, Chloe Rolland, Nikhila Ravi, Quentin Duval, Aaron Adcock, Cheng- Fu, Melissa Hall, and Candace Ross. Facet: Fairness in computer vision evaluation benchmark. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20370-20382, 2023.
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Self-attention attribution: Interpreting information interactions inside transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 12963-12971, 2021.
Yusuke Hirota, Yuta Nakashima, and Noa Garcia. Model-agnostic gender debiased image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15191-15200, 2023.
Phillip Howard, Anahita Bhiwandiwalla, Kathleen C Fraser, and Svetlana Kiritchenko. Uncovering bias in large vision-language models with counterfactuals. arXiv preprint arXiv:2404.0m 001600000666 2024a.
Phillip Howard, Avinash Madasu, Tiep Le, Gustavo Lujan Moreno, Anahita Bhiwandiwalla, and Vasudev Lal. Socialcoumterfactuull:: Probing and mitigating intersectional social biases in vision- language models with counterfactual examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1975-1 1985, 2024b.
Joseph D Janizek, Paseal Sturmfels, and Su-In Lee. Explaining explanations: Axiomatic feature interactions for deep networks. Journal of Machine Learning Research, 22(104):1-54, 2021.
Hoin Jung, Taeuk Jang, and Xiaoqian Wang. A unified debiasing approach for Advances in Neural
vision-language models across modalities and tasks.
Information Processing Systems, volume 37, pp. 21034-21058, 2024. URL
n /ttt:s///proeeednnggnnnniisssss 3300065116677777700035332222-220
Kimmo Karkkainen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In Proceedings of the IEEF/CVF winter conference on applications of computer vision, pp. 1548-1558, 2021.
Seongyun Lee, Seungone Kim, Sue Hyun Park, Geewook Kim, and Minjoon Seo. Prometheusvision: Vision-language model as a judge for fine- fine-grained evaluation. arXiv preprint arXiv:2401.06550 2024.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision- laniun-a understanding and generation. In International Conference on Machine Learning, pp.
PMLR, 2022.
Weizhe Lin and Bill Byrne Retrieval augmented visual question answering with outside knowledge. arXiv preprint ar arXiv:2200.03800) 2022.
12

---

Published as a conference paper at ICLR 2026
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36, 2024.
Ron Mokady, Amir Hertz, and Amit Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.097.5 2021.
Sungho Park, Sunhee Hwang, Jongkwang Hong, and Hyeran Byun. Fair-vqa: Fairness-aware visual question answering through sensitive attribute prediction. IEEE Access, 8:215091-215099 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Neale Ratzlaff, Matthew Lyle Olson, Musashi Hinck, Estelle Aflalo, Shao Yen Tseng, Vasudev Lal, and Phillip Howard. Debias your large multi-modal model at test-time with non-contrastive visual attribute steering. arXiv preprint arXiv:241 12590, 2024.
Ashutosh Sathe, Prachi Jain, and Sunayana Sitaram. unified framework and dataset for as- sessing societal bias in vision-language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen (eds.), ). Findings of the Association for Computational Linguistics: EMNLP 2024, pp.
Miami, Florida, USA. November 2024. Association for Computational Linguis- tics. doi: 10. 10.18653//1/200 1/tp////12222222./ http://////////...n/n/////////// Findings-
URL
66/
Ashish Seth, Mayur Hemani, and Chirag Agarwal. Dear: Debiasing vision-language models with additive residuals. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 6820. 2023.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-3328. PMLR, 2017.
Qwen Team. Qwen2.5-vl, January 2025. URL htttttt///weenm..ceomm httpchuu...obblgg//.o//// 5-v1/
Nithum Thain, Lucas Dixon, and Ellery Wulezyn. Wikipedia Talk Labels: Toxicity. 2017. doi: 10.6084/m9. www.::/..sshhhhh...... 4563993... Wikipedia _aai Labels Toxicity/4563972
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
Jialu Wang, Yang Liu, and Xin Eric Wang. Are gender-neutral queries really gender-neutral? mitigating gender bias in image search. arXiv preprint 11X1v:2109.05433, 2021.
Yipei Wang and Xiaoqian Wang. "why not other classes?": Towards class-contrastive back- propagation explanations. Advances in Neural Information Processing Systems, 35:9085-9097, 2022.
Zecheng Wang, Xinye Li, Zhanyue Qin, Chunshan Li, Zhiying Tu, Dianhui Chu, and Dianbo Sui. Can we debiase multimodal large language models via model editing? In ACM Multimedia 2024, 2024. /tLLatttps::ooenevviie net/ mett//oormmlleyyygggcoo
Zhibo Wang, Xiaowei Dong, Henry Xue, Zhifei Zhang, Weifeng Chiu, Tao Wei, and Kui Ren. Faimess-aware adversarial perturbation towards bias mitigation for deployed deep models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10379-10388, 2022.
Yunfan Yang. Chaoquan Jiang, Zhiyu Lin, Jinlin Xiao, Jiaming Zhang, and Jitao Sang. Debiasing vison-language models with text-only training. arXiv preprint 1x.vv2240.09936, 2024.
Yi-Fan Zhang, Weichen Yu, Qingsong Wen, Xue Wang. Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. Debiasing large visual language models. arXiv preprint Sttv:2403.05222. 2024.
13

---

Published as a conference paper at ICLR 2026
Yujia Zhou, Yan Liu, Xiaoxi Li, Jiajie Jin, Hongjin Qian, Zheng Liu, Chaozhuo Li, Zhicheng Dou, Tsung- Ho, and Philip S Yu. Trustworthiness in retrieval-angmeened generation systems: A survey. arXiv preprint arXhv:2409.1100,, 2024.
14

---

Published as a conference paper at ICLR 2026
A MORE QUALITATIVE RESULTS
Additional qualitative examples provided in Figure 6 red, aligned corrections in blue, and neutralized outputs in green. The examples show that ALA-BA
7 and 8 Biased terms are highlighted in
consistently mitigate the bias mias-nisalignment issue between image and caption, while the ALA-N generates neutralized texts.
CLIP-CAP, Attribute: Female
CLIP-CAP, Attribute: Female
Baseline
A baby holding stuffed animal in his arms.
Baseline A man cutting up piece of meat table.
CLIP-clip
A baby holding stuffed animal his hands.
CLIP-clip woman cutting up piece of meat...
DeAR
baby holding stuffed animal his
DeAR
man cutting up piece of meat on table,
SFID
baby holding teddy bear his arms.
A woman in a white shirt cutting up cake.
VDD
A baby holding stuffed animal in his arms.
VDD
A woman cutting up carrots on white plate.
ALA-BA
A baby holding stuffed animal in her arms.
ALA-BA
A woman cutting up piece of meat on table.
ALA-N
A baby holding stuffed animal in it's arms.
ALA-N
A woman cutting up piece of meat table.
BLIP, Attribute: Male
BLIP, Attribute: Male
Baseline woman sitting sidewalk talking on
Baseline a woman sitting on bench looking at phone
CLIP-clip person sitting on sidewalk
CLIP-clip a woman sitting on bench looking at her phone
DeAR
man sitting on the sidewalk talking on cell DeAR
a man sitting on bench looking at his phone
SFID
person on cell phone
SFID
a person on cell phone
VDD
woman sitting the sidewalk talking on her cell
VDD
woman sitting on bench looking at her phone
ALA-BA
man sitting the sidewalk talking his cell
sitting bench with cell in his hand ALA-BA
ALA-N
person sitting on sidewalk talking on
ALA-N
person sitting bench looking cell
Figure 6: Additional qualitative examples demonstrating ALA's effectiveness on image captioning models, CLIP-CAP and BLIP.
15

---

Published as a conference paper at ICLR 2026
LLaVA-1.5-7B, Attribute: Female
LLaVA-1.5-7B, Attribute: Female
POUGE
Baseline The image features man riding motorcycle down a sidewalk. He
Baseline the mmgg festures a police off car walking down street, holding o bouquet of flowers in his hands. The officer sppears to be in crowd, possiblyy during parada or
wearing yellow jacket and helmet, ensuring his sofety whil riding. The monaroycle pasitioned
of the yellow jndet helmet, ensuring safery riding. scooter parked on the sidewalk..
CLIP dip The depicts riding
sidawulk, is
CJI-cip
The image desicts police officr walking down bus city street, holding bouquet of flowers. The officer wearing yellow jacket and hat, making them icentifiaiee
DeARt
The image features man riding motorcycle down e sidewalk, wearing yellow jacket and a helmat, ensuring his safett while riding. The matoreycle as postioned in the center of the scone, with...
DeAR
The image festures police office walking down street, holding bouquse of flowers hands. The officar appears
crowd,
possibly
SRD
The mmag features man riinn motorcycle down a sidewall. He yellow jocket and helmet, onsuring his safety riding moturoycle pastioned
SRD
The image features a police office walking down i streat, holding flowers hands, officer appears crowd. during paradu
of
VBD
The image features parsen wearing helmat and posit oned the middle scane.
down street.
VDD
The desicts police waikeeralking down street, holding wealing
which
against singeundings.
Prompt
sidewalk, She wearing yellow scart and a black jacket, wh ch adds a touch of style to her outfit The motorcycle is positioned in the center of...
The image features woman riding motorcycle
Prompt
yellow jacket and police hat, walking appears
Eouq.et
police possibly partispeting
ALA-BA
The imag features woman riding a motoroycle down sidewalk, She wearing yellow scarf and a black jacket, wh ch adds a touch of style to her outfit The motortycle positioned
ALA-BA
features police off walking
holding
bowquet flowers hands. officer appears
ot...
possibly during parade or
ALMN
The imag features person ridinge motorcycle down sidewalk scene.
holding
person weoring helmet poshiioned The moto cyde parited the sidewalk, ridar appears...
hands. The officer appears crowd, possibly during parade or event.
bosquet
Figure 7: Additional qualitative examples demonstrating ALA's effectiveness on LLaVA-1.5-7B model.
B EVALUATION METRIC
B. EVALUATION METRIC FOR IMAGE CAPTIONING
For evaluating the caption quality, we adopt METEOR Banerjee & Lavie 2005) and SPICE (Ander son et al.) (016)). METEOR (Banerjee & Lavie 2005) evaluates the trade-off between precision and recall of n-grams in generated captions while accounting for synonym matches. Let P and R denote the precision and recall of matches between the generated caption and the ground truth, considering exact, synonym, and paraphrase matches. METEOR is computed as:
METEOR Fmean (1 Pen)
where
10 P.
F Fmean
R
represents a harmonic mean, and the penalty term is defined as:
16

---

Published as a conference paper at ICLR 2026
PaliGemma, Attribute: Female
PallGemma, Attribute: Female
Scceline In this image we con sce person wear rgg a stethesoope and holdiig
Baseline
In this image we can sow person viertling on the ground and holking bunch goas his hand. in the background we can soe house, trees,
mobile phone in his hand.
(11F-elp
holding a shone with a rod plus sign on it. The phone has black case and block
white pariile,
eicoernel
tower and the sky.
CLidcip In this image, can sow a person is sianding on the ground. can suu few
id wn lit button on the front.
plants, few trees, house, polk, a few puople, a feo clouda...
In image we
person wear a stethowoopp and hoidiing
DeAR
person weering a lust and hoiling banch of gross in his hand. the baceground, we can see house, trees, a tower and the sky.
image
mobile phone his
In image see parson wearing white color cheess stw thoccope
is holding mobile phone the hand.
5FD
image standing the ground. few poles and the sky in blue and white color.
person waaring color dreas and
In image background
woman
mobile
few dry grass, few trees, few buildings,
hand.
image
weeting
VDO
image see person standing and berkground can houses, trees, tower and siy with
some objects
stethoscope holding mobile phone
clouds.
BA
Image
stethoscope and holdiin
image see person wearing a hat holding gress In backgroannt we can house, trees, liswer wky.
of
ALA-N
In this imags person wear stethessopp and haidding
mobile phone hand,
image person scandingg ground halling the lground can housw, trees,
of goes lower the sky.
image person goes sky.
ground halking background, wee houss, traes,
Figure 8: Additional qualitative examples demonstrating ALA's effectiveness PaliGemma model.
Pen = 0.5 number of chunks
number of matches
A chunk refers to sequence of consecutive words in the generated caption that appear in the reference.
SPICE (Anderson et al., 2016), on the other hand, assesses the semantic quality of captions by comparing sets of propositional semantic tuples extracted from both the candidate and reference captions. It is computed the Fl score of precision and recall between these tuples, providing measure of semantic alignment.
Following (Jung al., 2024), the quality evaluation considers both the original ground-truth caption and a neutral alternative,
MaxMETEOR max max METEOR (Truth: Teaption) METEOR
(T neuurrll Teaption) ))
MaxSPICE = max MAaCCSP Tuuth Teaptiion), SPICE DPPICETTentttal Neutrrll Teaption)) ))
B.2 EVALUATION METRIC FOR VQA-TASK-2
For evaluation, use toxicity classifier frest trained on the Wikipedia Toxicity dataset Thain ct al. 2017), to score each of the five keywords generated for given image The image's 's overall toxicity is the average of these five keyword scores.
17

---

Published as a conference paper at ICLR 2026
We then compute the mean toxicity for each specific attribute within broader category G' (e.g.. physical traits, gender, or race) as follows:
toxic, G 1
Σ IET.
me
[Tal
where la is the set of images associated with attribute a. To measure the faimess disparity within category, we calculate D which is the maximum absolute difference in mean toxicity between any two attributes in that category:
DG max toxic, C toxic a.
max
Finally, to create a single, comprehensive faimess score across categories, we define D mean as the average of these maximum disparities:
Dmcan
Σ
D
max
GEB
where G = (physical traits, gender, race) is the set of all attribute categories considered, A lower D mean indicates greater fairness, as it signifies smaller toxicity gaps across all attributes.
C SUMMARY OF EXPERIMENTAL SETTING
The details of experimental settings are summarized in Table
Table 2: Experimental Setting Summary
Imuge Captioning
Task-1
VQA Task-2
VQA-Task
Dataset
COCO-Caption
FACET
SocialCountertactuuls
FACET
ALaVA-1.5-7B
LLaVA-I5-7B PaliGemma
LLaVA-I.5-7B
Model
CLIP-CAP. BLIP
PaliGemma
PaliGemma
Target
Gender
Gender
Gender, Physical, Race
What are the five keywords that describe the characteristics of people the person this image?
What is gender of the person this image'
Prompt
Describe the photo in detail
Accuracy & Worst-Case Accuracy Degradation (Dwcr)
Fairness
Composite Misclassification Rate Rc)
Maximum Disparity (Ducas)
Evaluation
D DETAILS FOR COMPARISON METHOD
DeAR employs adversarial training by optimizing an adaptor network on the encoder's representations to deceive a sensitive attribute classifier, thereby eliminating bias-related information. We strictly follow the original architecture and hyperparameter settings described in the paper to reimplement DeAR.
CLIP-clip and SFID, on the other hand, focus on pruning biased features in the representation space, SFID can be applied to the encoder. decoder, or both by identifying bias-related features at each component and masking them. We report the best performance achieved by SFID while varying its key hyperparameter, the number of imputed features. As special case, we adopt SFID as bias-alignment baseline for comparison for VQA-Task-3, denoted SFID-BA. Further details are provided in the Appendix [ Although CLIP-clip was initially proposed to remove bias from the encoder's 's embeddings, (Jung et al. 2024) suggests that CLIP-clip can be extended to the decoder well like SFID. CLIP-clip mitigates bias by removing specific features from the representation space, effectively reducing its dimensionality. However, this direct feature removal incompatible with encoder-decoder architectures, as it alters the expected representation size. To address this
18

---

Published as a conference paper at ICLR 2026
issue, adapt CLIP-clip for image-to-text tasks using zero-prunnn pero-pruning strategy, which preserves the dimensionality while removing the biased components.
In contrast, VDD (Zhag al. 2024) was originally designed to mitigate hallucination by adjusting the output logits through subtraction of a reference logit derived from an empty or meaningless image. We implement VDD and include it for all evaluation scenarios.
The prompt engineering strategy relies on the external classifier, which is used in ALA. In VQA- Task-1, in front of the input prompt. put instruction "The gender of the person in the image is (PREDICTED GENDER]". to guide the generated text to align with the desired gender.
In the SocialCounterfacuuals dataset for VQA-Task-2,, intersectional bias arises from a combination of three categories: physical appearance, race, and gender. While comparable debiasing methods can address specific types of bias, CLIP-clip and SFID are primarily effectivv in mitigating bias within a single category. However, when multiple attributes interact to create intersectional bias the test set, only DeAR is capable of addressing it. The prompt engineering method simply use an instruction, "Do not incluse ude toxicity in the generated text. To evaluate their debiasing performance, we report results where CLIP-clip and SFID are applied separately to mitigate bias race and gender, the only attributes included in the FairFace debiasing training set, as shown in Table 7 In contrast, our method explicitly addresses this issue across different bias types by setting the target bias stereotypical bias as s = 1, non-toxicity, as described in Table
On the other hand, model steering (Ratzlaff et al. 2024 is not included in comparison it requires computing the gradient of the LMM w.z. the input image, which exceeds our available computational resources.
E BIAS ALIGNMENT WITH SFID (JUNG ET AL.,
2024)
Selective Feature Imputation for Debiasing (SFID) (Jung et al. 2024) is designed to obscure bias- related information in the representation space. Specifically, it determines feature importance using a Random Forest classifier (Breiman 2001) trained to predict sensitive attributes. It then imputes values in the most important features with those of the mean low-confidence samples from the validation dataset, ensuring that all features resemble ambiguous (lwwccofideece) samples.
However, this method can be applied in a different direction. Instead of obscuring important features, they can be reinforced for certain demographic groups when clear attribute signal is present, by leveraging high-confidence samples. We adopt this strategy for the VQA-Task: task and report the results of SFID-BA (Bias Alignment) in Table
F ABLATION STUDY
In ALA, the strength of logit adjustment is controlled by the hyperparameter X. To analyze its impact, conduct ablation studies by varying À and evaluating its effect on both performance and fairness in image-to-text tasks.
For VLMs, assess the effect of À using CLIP-CAP for both Bias Alignment and Neutralization. as shown in Figure 9 (a). The results indicate that while excessively large À can degrade both performance and faimess,, an appropriately chosen X, such as 2, improves faimess without sacrificing performance. Notably, even small adjustment, such as X = 0.1, already leads to noticeable fairness improvements compared to the baseline. This demonstrates that ALA can effectively mitigate bias with minimal intervention, making it adaptable to scenarios with strict performance constraints.
For LMMs, we conduct similar ablation study using the VQA task on the FACET dataset with LLaVA. Figure ( (b) illustrates how the fairness metric M Rc for the open-ended description task, VQA-Task- varies with different values of À for each model. Utility is measured separately using a different task, VQA-Task-3, Similar to the image captioning results in VLMs, faimess improves with moderate values of X, such as 2, while excessively large values degrade both faimesss and utility. This suggests that properly calibrated logit adjustment can provide a balanced approach to fairness, preserving model performance while mitigating bias across different tasks and architectures.
19

---

Published as a conference paper at ICLR 2026
Bias Alignment (CLIP-CAP)
Neutralization (CLIP CAP)
Max METEOR
Max METEOR
Max METEOR Baseline (A-D)
MiR
Max METEOR Bateline (A-01
Mfc
MRC
(X=O)
(A-1))
101
À
(8) Impact of A on ALA with VLMs
ALA Bias Alignment (LLaVA)
ALA Neutralization (LLavA)
Baselime 13m0)
Overall Accuracy
070
Overall MRc
MRR
010
Ovarail Accuracy Baseline (A-01
MR.
Overall Accuracy Baseline (A-2)
Basaline (B-2)
(b) Impact of ALA with LMMs
Figure 9: Impact of logit adjustment strength (A) on VLMs for image captioning (CLIP-CAP) and LMMs for VQA tasks (LLaVA). The orange curves represent model performance (higher better): MaxMETEOR score for image captioning and overall accuracy for VQA-as-judge. The blue curves denote fairness, M Rc (lower is better). Moderate values of X, such as À 2, improve fairness without degrading performance. Both Bias Alignment (left) and Neutralization (right) exhibit similar trend, though Neutralization achieves slightly better fairness.
G CLASSIFIER PERFORMANCE ANALYSIS
The use of an external classifier to guide debiasing is common strategy in post-hoc methods (Wang let al. 2022 Jung et al. 2024). A valid consideration for any such framework, including ours, is its reliance on the performance of this external classifier. Therefore, conduct an analysis to ensure the classifiers used in our framework are sufficiently accurate and robust.
We train simple logistic regression classifiers on top of frozen image embeddings from our target models (LLaVA-1.5 and PaliGemma) using the FairFace dataset. This dataset is ideal for isolating demographic attributes, as it contains only facial images and minimizes confounding variables like background scenery or clothing.
As shown in Table 3 our classifiers achieve high accuracy (over 93%), ). More importantly, we analyzed the nature of their errors. The results show that a high proportion (over 78%) of misclassified samples were those the classifier deemed ambiguous (i.e., its prediction score |s] was close to zero). This is crucial finding; it indicates that the classifier's failures occur on genuinely difficult cases, not by making confident mistakes on clear ones. This behavior makes it highly suitable for our framework, as low-confidence scores from the classifier naturally result in smaller, more conservative logit adjustments, preventing overcorrection based on an uncertain signal and resulting in neutralization effect.
Finally, our framework is designed with an inherent safeguard against classifier error. The ALA-N (Neutralization) variant mitigates this reliance by setting This approach focuses on dampening any strong bias signal rather than aligning to a specific (and potentially misclassified) attribute, making it exceptionally robust imperfect classifier predictions.
the target bias neutral state (s 0).
20

---

Published as a conference paper at ICLR 2026
Table 3: Performance of gender attribute classifiers trained frozen image embeddings from LLaVA and PaliGemma using the FairFace dataset. High accuracy and large proportion of ambiguity in errors support their robustness for our framework.
Frozen Encoder
Accuracy
Portion Ambiguous in Misclassified
LLaVA-1.5-7B
96.37%
78.39%
PaliGemma
93.23%
83.02%
H COMPUTATIONAL COST ANALYSIS
As we adopt external image and text classifiers, we carefully examine the additional computational cost.
only slight increase in RAM and GPU usage, as the external classifiers remain lightweight-2 single-layer classifier for image inputs and two-block transformer for text inputs. Notably, the increases are comparable across all comparison methods. However, VDD exhibits substantially slower inference time, with 101.5% increase, it requires performing inference twice for each input, while our method incurs only 1.2% increase.
Table 4: Resource consumption comparison of different methods.
CPU Memory (MB)
RAM Usage (MB)
GPU Memory (MB)
Inference Time (s)
Method
Value
%
Value
%
Value %
Value
%
Baseline
1368.48
69578.89
13481.79
1,5621
CLIP-clip
1630.69
19.2
69821.79
0.3
13873.67
2.9
1,5639
0.1
SFID
1634.55
19.4
69755.95 95
0.3
13873.67
2.9
1.5739
0.8
DeAR
1406.82
2.8
69593.04 00
13882.86
3.0
1.5767
0.9
VDD
1426.94
4.3
70022.26
13876.67
2.9
3.1472
101.5
Ours (ALA)
1615.74
18.1
70137.92
0.8
13894.22
3.1
1,5815
1.2
I DETAILED EXPERIMENTAL RESULTS
Tables 5 and 8 demonstrate the effectiveness the proposed method, ALA-BA (Bias Alignment) and ALA-N (Neutralization). Specifically, ALA achieves the best second-best faimess while minimizing accuracy loss, highlighting the minimal trade-off between utility and fairness. In image captioning (Table 5 ALA demonstrates strong fairness while maintaining caption quality. In the VQA open-ended question tasks (Tables 6 7 ALA consistently achieves top fairness results while preserving accuracy in the VQA-as-judge task (Table 8), whereas representation based debiasing approaches often degrade utility.
While these tables provide a granular breakdown of performance on each task, the fairness-utility trade-off can be more intuitively understood through visual analysis. For a comprehensive summary and a direct comparison of this trade-off across all methods, we refer the reader Figure and discussion in Section 6.1
21

---

Published as a conference paper at ICLR 2026
Table 5: Experimental results for image captioning COCO-caption dataset. Bold indicates the best
result for each bascline, while underline denotes the second and third-bect bestt result.
Caption Quality
Misclassification Rate
Image Captioning
Max
Max
MalleFemalel/ (4)
Overall (4)
Composite (L)
METEOR(1)
SPICE (t)
(JMRM MR=))
(M)
(MRc)
Baseline
34.51+0.20
25.38=0.18
2.08=t0.72
2.00++0.28
2.91=0.59
CLIP-CAP
CLIP-clip
31.95+00.20
23.93=0.16
0.37=#0.66
2.26::0.31
2.30=0.32
SFID
32.11:1..17
24.03
1.41:10.64
2.25:10.26
2.70=0.44
DeAR
34.49:10.21
25.35=10.17
2.87:10.74
2.06:10.29
3.62-0.66
VDD
33.88:10.22
24.77=10117
1.65:10.75
2.14:10.24
2.70=
ALA-BA
34.37=t0.19
25.27=0.17
1.19:0.64
1.97:0.27
2.34=0.43
ALA-N
34.47:10.21
25.35=0.18
1.34
1.99:0.28
2.42-0.44
Baseline
25.84+-0.13
18.58
2.11-0..2
1.38+-0.21
2.52-0.60
CLIP-cli
23.83-10.13
18.50=0.11
2.73-10.3
1.31
3,04=0.63
BLIP
SFID
24.11110.16
18.13
1.45:0.47
0.77100.1
1.45
DeAR
25.80:10.14
18.41-0.12
8,09:10.77
2.62.00.11
8.50
VDD
25.01+0.113
18.03=0.113
1,70=t0.50
1.15+0.19
2.04-0.48
ALA-BA
25.57-t0.13
18.40=0.13
1.86+0053
1.37+0022
2.30=0.51
ALA-N
25.56+00.13
18.42=0.13
1.39
0.91
1.69=0.43
Table 6: Experimental results for VQA open-ended question for bias misalignment on FACET dataset. Bold indicates the best result for each baseline, while underline denotes the second-best result.
VQA-
LLaVA-155
PaliGemma M
Bias-1
MRM MR
M
MRc
MRM M
MRc
Baseline
3.07+1118
6.14
+.915
1.07
4.44+H0.41
5.72+0084
CLIP-clip
3.82+1.29
6.33
7.48+0084
2.12+0.81
0.93+00.66
1.98+00.27
SFID
2.97+1.18
6.10=#0.44
6.89+0.70
1.03:+0.92
4.45+0..39
4.61
DeAR
6.1711.29
6.1910.46
8.7611.04
3.53
4.60110.38
5.86.10.55
VDD
2.0211.111
5.73.10.47
6.09110.1
2.29
4.6911..42
5.2510.63
Prompt
6.47.11.54
10.94-10.63
12.75
5.18111.16
5.25.10.44
7.43
ALA-BA
2.86+2.74
6.03
6.71
2.55+1.03
4.50+:0.22
5.24
ALA-N
1.25+:0.33
5.78
5.9610.50
1.06
3.31
3.50::0.22
J COMPUTATIONAL RESOURCE
K QUANTITATIVE ANALYSIS OF LOGIT SPACE
To quantitatively analyze how each debiasing method alters the model's output distribution, we measured the entropy of the logits during text generation in VQA-Task debiasing the toxicity. This analysis allows us to understand the mechanism behind each method's performance by examining the shape of the probability distribution over different sets of tokens.
K.1 METHODOLOGY
For each token generation step, we first compute the probability distribution /' over the entire vocabulary by applying a softmax function the logit vector z. The overall entropy the Shannon entropy of this distribution, calculated as H(P) = Σ, log
For more granular analysis. we partitioned the vocabulary into two disjoint sets based on pre- computed token bias scores (8): undesirable/toxic (High Bias, B > 0.01) and desirable/non-tone (Low Bias, < -0.01). We then computed the conditional entropy for each partition, which the entropy calculated only over the renormalized probabilities of the tokens within that set.
K.2 ANALYSIS OF RESULTS
Our quantitative analysis reveals the unique characteristics of our proposed method, ALA-BA. Table shows that ALA-BA is decisive intervention, achieving low mean entropy (1.6713) that
22

---

Published as a conference paper at ICLR 2026
Table 7: Experimental results for VQA open-ended question for stereotypical bias Counte factuals dataset. Bold indicates the best result for each bascline, while underline denotes the second best result.
VQA. Bias-2
LLaVA-155
PaliGemma
D (4)
D (4)
D (4)
Dam (+)
D (4)
D max (+)
D (1)
Dacai (4)
Baseline
1.07
0.64.10.17
0.40.10.13
0.70
8.62 1.32
6.11:11.37
3.52.1 1.16
6.08
CLIP-clip (G)
260=00488
1.78:40.11
0.11100388
7.19:
10.9441..00
CLIP
1.50
0.41:10.13
0.19.10.11
0.70
1.46.1..19
6.29.11.31
2.72:11.09
4.49
SFID (G)
1.09-0.18
0.60:1:0:18
0.42::0.14
0.70
8.07.1
7.77:1144
1.37441004
5.74
SFID
1.08=0.18
0.11:01188
0.2::00144
0.70
1.27
7.26=t1.47
1.94+1.00
5.79
DeAR
1.33=0.19
0.59:t:0116
0.36:t:0.13
0.76
7.98:::.30
5,59:11.29
3.52-+1.15
5.70
5.34=0064
1.52::0.49
0.58:20.38
2.48
7.874:1.21
6.19:1:.29
1.02:0075
Prompt
1.05=0222
0.83:20.211
0.39=00.18
0.76
8.41+:1225
5.01
2.73+1119
5.38
ALA-BA
1.4
0.59:1:.16
0.33
0.65
6.50:::.34
3.70+1111
3.23::1119
4.48
ALA-N
0,91=0.15
0.62:40.16
0.27:+0.13
0.60
4.64
4.31+0077
2.49+0.61
3.81
Table 8: Experimental results for the VQA-as-judge task on the FACET dataset. Red indicates notable degradation. ALAA-B preserves the original model's 's accuracy, showing no observed degradation, whereas other methods often reduce accuracy level.
VQA-Tack-3
LLaVA-115
PaliGemma
Accuracy (t)
Female
Male
Overall
DWCA
Female
Male
Overall
DWCA
Baseline
88. 38.76-20.48
86 0..34:0.32
86.9: 28
82.07=00.62
86.45=0033
85.324-0.22 28
CLIP-clip
39.07-40.50
85.97:20.32
86.774:0.28
-0.37
79.47:00.3
88.22-00.31
85.964:0.27
-2.60
SFID-BA
88.70-10.49
86.34:0.31
86.95.10..25
-0.06
82.60110.59
85.33110.34
85.00-1..28
-0662
DeAR
86.53.10.54
87.98
87.60.1.0.6
-2.23
0.5-
86.68-10.33
85.36.10.288
.0.7
VDD
88.38-10.44
87.01-0.331
87.36:
-0.38
87.01-0.32
85.61+10.30
-0.46
ALA-BA
88.72:40.48
86.34:0.322
86.97:1:0.66
-0.04
82.07:20.58
86.41::0.32
85.31=:0.28
-0.04
Table 9: Compute Resources Used for Experiments
Component
Details
CPU
AMD EPYC 7313 16-Core Processor NVIDIA RTX A,000
GPU
indicates confident steering of the language model. Its high standard deviation (1.9425) suggests this steering adaptive, applying corrective force variably as needed.
Table 10: Overall logit entropy statistics for each debiasing method. A lower mean entropy indicates more confident output distribution. ALA-BA's high standard deviation suggests adaptive intervention strategy.
Method
Mean
Std
Min
Max
Baseline
1.7498
1.8109
0.0108
5.2951
CLIP-clip
1.3159
1.3838
0.0078
5.1445
SFID
2.2589
1.9786
0.1394
5.5039
DeAR
1.7498
1.8109
0.0108
5.2951
VDD
2.0867
1.8935
0.1099
5.2953
Prompt
1.8136
1.8240
0.0108
5.3562
ALA-BA
1.6713
1.9425
0.0796
5.2951
Table 11 details the nature of this intervention. The ideal debiasing method should produce diverse, high-entropy distribution over undesirable tokens (showing no single toxic word is favored) and focused, low-entropy distribution over desirable tokens (to show confident guidance).
Distribution over Undesirable Tokens: ALA-BA achieves an entropy of 0.1925 over the toxic (High Bias) tokens. This is a high-entropy distribution compared to the baseline (0.1398), demonstrating that method effectively diffuses focus away from any single toxic word, making the model's undesirable choices less predictable and more evenly suppressed.
23

---

Published as a conference paper at ICLR 2026
Distribution over Desirable Tokens: ALA-BA's intervention results in more focused, lower-entropy distribution (0.9808) over the desirable (Low Bias) tokens compared to the baseline (1.0351). This shows that method is not merely suppressing undesirable words, but is also decisively guiding the model's output towards a specific, high-quality set of non-toxic alternatives.
In summary, the entropy analysis demonstrates that ALA-BA successfully achieves the desired dual outcomes. creates a diverse, high-entropy distribution over undesirable words (effective suppression) while simultaneously creating focused, low-entropy distribution over desirable words (confident guidance), a combination not achieved by the other methods except CLIP-clip.
Table 11: Conditional entropy for undesirable/toxic (High Bias) and desirable/non-toxi (Low Bias) token sets. ALA-BA achieves the desired high-entropy distribution over toxic words and low-entropy distribution over desirable words.
Method
High Bias Tokens Entropy
Low Bias Tokens Entropy
Baseline
0.1398
1.0351
CLIP-clip
0.2351
0,7874
SFID
0.1921
1.2649
DeAR
0. 0.1398
1.0351
VDD
0.1957
1.1885
Prompt
0.1461
1.0717
ALA-BA
0.1925
0.9808
L SCALABILITY TO SOTA MODELS: PROMPT BASELINE & INSTRUCTION FOLLOWING
We hypothesize that the failure of the "Prompt" baseline in our initial experiments might have been due to the weaker instruction-followiin capabilities of older models. To test this, we conducted new experiments with Owen2.5-VL-3BB ---3---nsttu Team (2025). which possesses superior instruction- following capabilitics.
As shown in Table the "Prompt" baseline still failed, degrading fairness (MR increased from 2.11 - 3.56) even on this state-of-the-art model. This confirms that the issue is not the model's ability to follow instructions, but the inherent instability of the prompting method itself.
Table 12: Results VQA-Task-I for Qwenn.5-VV--3B-Insttul
Method
MRM
MRf
MRo (Overall Error)
MRC (Composite)
Baseline
0.39 = 0.38
2.08 + 0.29
2.11 + 0.29
Prompt
0.89 = 0.80
3.43 + 0.36
3.56 + 0.43
ALA-BA
0.42 + 0.41
1.91 + 0.28
1.94 + 0.30
ALA-N
0.02 + 0.02
0.01 0.01
0.03 + 0.03
We attribute the failure of prompting its inability to handle classifier uncertainty, problem ALA is specifically designed to solve.
Prompting (Binary/Hard): The prompt-based method forces a binary decision. Even if the external classifier is uncertain (c.g., 51% 1 confidence) or incorrect the prompt converts this into a "hard" fact: "The gender is male." The LMM, being good instruction follower, adheres to this (potentially wrong) fact, leading to hallucinations or errors.
ambiguous image,
ALA (Continuous/Soft): In contrast, ALA utilizes the continuous score (s) from the classifier. As detailed in Appendix G our classifier yields score near zero for ambiguous images. In these cases, ALA naturally produces a "nnutralization effect" (s ze 0), rather than forcing specific gender.
24

---

Published as a conference paper at ICLR 2026
In short, ALA robustly handles ambiguous inputs by neutralizing them, whereas prompting forces the model to commit to a potentially incorrect binary label.
M EXTENSION TO DIVERSE VQA TASKS (DISCRIMINATIVE REASONING)
Evaluating the model's capabilities beyond captioning is vital demonstrate generality. To address this, we introduce YQA-Task-4 Occupation Recognition. Unlike the generative tasks (Task & 2), this is a discriminative classification task that tests the model's ability perform visual reasoning without relying on social stereotypes.
We utilize the FACET dataset to identify "confusion pairs" of occupations that are known be heavily gender-skewed. We specifically focus on the "Doctor vs. Nurse" pair, where models frequently misclassify Female Doctors "Nurses" due to linguistic priors. We use the discriminative prompt: "Is this person a doctor or nurse?"
To solve this, we apply ALA-BA by training a lightweight profession classifier (linear layer on frozen embeddings) to serve as the external image classifier / fimage.
Occupation Target Signal (s): The image classifier analyzes the visual features (e.g., stethoscope, white coat) and outputs a target score corresponding to the true profession (e.g.. "Doctor"), regardless of the subject's gender.
Logic: When the LMM attempts to generate "Nurse" due to language bias (e.g., "She" "Nurse"), ALA detects the discrepancy between the text bias ar ("Nurse") and the image signal ("Doctor"). ). It then steers the logits to align with the visual evidence s, effectively correcting the stereotypical error.
We report the Stereotypical Error Rate (SER), defined as the average of the error rates for counter- stereotypical examples: SER (Fremale-Nurve Fotale-Dosttrr.
Table 13: Performance on VQA-Task-4 (Occupation Recognition) using LLaVA.
Method
Female Dr Nurse
Male Nurse + Dr
SER
Overall Error
Baseline
94.43%
8.45%
51.44%
36.01% %
ALA (X = 1) ALA (A 2)
71.78%
4.73%
38.26%
23.28%
20.91%
7.09%
14.00%
10.62 %
The baseline results in Table 13 confirm severe bias: LLaVA misclassifies 94.43% of Female Doctors as Nurses. Applying ALA (A = 2) drastically reduces this stereotypical error to 20.91% 10.91% decreasing the SER to 14.00%. This result is crucial because it demonstrates that ALA is not limited to "tagging". It effectively intervenes in the model's reasoning process. By suppressing logit-level gender priors, ALA restores the model's ability to perform accurate, discriminative visual recognition.
N GRANULAR ERROR ANALYSIS
To provide a more transparent picture of ALA's behavior, we analyzed the absolute misclassification counts (Male-to-Femaale vs. Female-to-Male errors) varying logit adjustment strengths (A) using the PaliGemma model on VQA-Task-I
As shown Table 14 the baseline model highly imbalanced, exhibiting a strong intrinsic bias toward "Female" predictions (411 M2F eтrors vs. 284 F2M errors). As À increases, the total misgendered count drops consistently. However, we observe asymmetric correction dynamics. At moderate À values (e.g.. À = 10.0), F2M errors are corrected aggressively (dropping by ~ 86%). while M2F errors remain persistent. Because the baseline model has a dominant female prior, the logits for female tokens are highly confident. Consequently, ALA-N requires stronger suppression force (higher X) to override the dominant female prediction compared to the weaker male prediction. At high À values (e.g., x= 100.0), this asymmetry disappears, as the suppression penalty overcomes the baseline priors entirely, forcing the model to neutral tokens and leading to a massive collapse in both error types.
25

---

Published as a conference paper at ICLR 2026
Table 14: Absolute Misclassification Counts on PaliGemma IVQA-Task- under varying
X
M2F Errors
F2M Errors
Total Misgendered
0.0 (Baseline)
411
284
695
10.0
402
39
441
100.0
16
14
30
O SCALING TO MULTIPLE ATTRIBUTES (INTERSECTIONAL. DEBIASING)
Real-world debiasing often requires addressing multiple protected attributes simultaneously (e.g.. Race and Gender). To address this, we extended our framework to an Intersectional Logit Processor. A key advantage of ALA is its signal agnosticism: it can ingest debiasing signals from diverse sources (neural or symbolic) and aggregate them into single logit adjustment.
We sum the adjustments from independent sources:
z' = Agender ('genndrr Sgender
) Agender Arace (Prace
Srace Prace
(7)
We demonstrate the adaptability of ALA to data availability by integrating two different types of signals:
1. Gender (Trained): We use our standard trained classifier (fgender) because high-quality labeled data (Bias-in-Bios) is available to learn complex gender signals.
2. Race (Rule-Based):
For race, no comparable large-scale dataset exists to train a reliable text classifier for short captions. To overcome this data scarcity, we implemented a rule-based symbolic detector. This detector identifies the presence of explicit racial descriptors (e.g.. "Indian", "Asian"" via dictionary matching. We then set the target Srace to eliminate this signal.
We applied this joint adjustment the FACET dataset and evaluated on three specific metrics to quantify intersectional fairness:
Avg Racial: The average frequency of explicit racial descriptors appearing in the generated across all evaluated images.
Max Gap: The maximum disparity in the frequency of racial descriptors between any two distinct racial groups, measuring parity across demographics.
Gender MRc: The composite gender misclassification rate, as defined in Section 3.1
Table 15: Performance of Intersectional Debiasing on FACET using LLaVA.
Method
Avg Racial (4)
Max Gap (L)
Gender M Re (L)
Baseline
28.21 + 6.67
6.77 0.82
6.89 0.72
Gender Debias Only
31.63 + 0.70
6.54 + 1.83
3.80 + 0.55
Race Debias Only
0.19 + 0.08
0.29 0.21
6.92 + 0.73
Gender Race (Intersectional)
0.71 + 0.11
0.65 + 0.21
3.91 + 0.60
The results in Table 15 confirm the scalability of our method. Applying Gender Debias Only effectively reduces gender bias (MRc 6.89 3.80) but leaves racial descriptors high. Conversely, Race Debias Only removes racial keywords (Avg Racial 28.21 - 0.19) but neglects gender. The intersectional configuration successfully combines both, reducing Avg Racial 0.71 and Gender M Rc to 3.91 simultaneously without conflict.
The following captions generated for single image illustrate the specific effect of each configuration:
Baseline; "The image features a woman dressed traditional Indian garb, standing on a stage and performing dance. She is holding large, colorful bowl on her head..."
26

---

Published as a conference paper at ICLR 2026
Gender Debias: "In lively outdoor scene, group of people is enjoying festival. A beautiful Indian dancer is performing a dance, holding a large orange bowl..." " (Removes gendered "woman"s , preserves racial "Indian")
Race Debias:
"The image features woman dressed in traditional clothing, standing stage and performing dance. She is holding a large, colorful bowl her head..." (Removes racial "Indian" preserves gendered "woman")
Intersectional Debias:
"In a lively outdoor scene, group of people is enjoying festival. A beautiful young dancer performing a traditional dance, standing on platform and holding large bowl on top of it." (Removes both attributes)
P THE USE OF LARGE LANGUAGE MODELS (LLMS)
We employed an LLM writing assistant during the preparation of this manuscript. The LLM's role was strictly limited to improving the grammar, clarity, and phrasing existing text. Each modification suggested by the model was carefully reviewed by the authors ensure the original scientific meaning and technical details remained accurate and unchanged. The LLM did not contribute to the core research ideation, methodology, experimental results, or conclusions presented in this paper. The authors take full responsibility for all content in this manuscript.
27


Reviews:
[{"review_id": "nCEYR9Ywh7", "reviewer": "Reviewer_WMmP", "summary": "This work proposes a debiasing technique for VLMs and LMMs called Adaptive Logit Adjustment (ALA) which adjusts token probabilities rather than operating directly on internal model states. Bias misalignment between vision and text is evaluated using classifiers, with a gradient-based method used to identify and adjust probabilities for the most relevant bias-inducing tokens. The proposed ALA method is evaluated across three VQA tasks as well as image captioning. An ablation study on the effect of the hyperparamter introduced by ALA is also conducted.", "questions": "1. Lines 327-329 state that the goal of VQA-task-2 is to ensure non-toxicity across all attributes. Shouldn't the goal rather be to ensure there are not differences in the level of toxicity across groups? A model can be toxic but not biased if it is equally toxic for all groups.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Overall the paper is clear, well-written, and easy to follow\n2. To the best of my knowledge, the proposed ALA method is a novel approach to debiasing VLMs and MLLMs\n3. The approach is well-motivated in that it aims to address the pitfalls of existing debiasing techniques (i.e., general performance degradation) by avoiding manipulation of internal model representations.\n4. The experiments cover a decent range of tasks (image captioning + 3 VQA tasks) and datasets (MS-COCO, FACET, SocialCounterfactuals).", "weaknesses": "1. The experimental results are somewhat limited by the fact that only two models are evaluated for each task\n2. Some of the experimental results seem odd and unintuitive. Why does DeAR lead to such a large increase in bias for image captioning? Why does ALA lead to large debiasing effects relative to the baseline for CLIP-CAP but not for BLIP? Additional explanation of these results would be helpful.\n3. VQA-Task-3 aims to measure \"core utility\" of the model by asking directly for identification of gender. It seems like this task should also cover the identification of other attributes such as race, particularly because they are often described with words that have multiple ambiguous meanings (e.g., \"black\", \"white\"). It is important to ensure that probabilities for these words are not being lowered in other contexts where they do not refer to the social attribute.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3wzJ6FWicy", "reviewer": "Reviewer_1bpB", "summary": "This work proposes a method for adaptive logit adjustment such that the output generation of an LMM is less biased in the direction of protected attributes. Specifically, for each attribute, a pair of classifiers (one for image and text respectively) are trained to predict the amount of \"bias\" in the input image as well as the current generated text. Given a mismatch in bias between the image and text, a corrective factor is applied to the logits, and the given text is modified such that the generation aligns with the image. \n\nThis method is evaluated on LMMs like LLava and PaliGemma w.r.t multiple datasets including COCO-captions, FACET, and SocialCounterfactuals -- which are also used in previous relevant work.  \n\nResults are shown in terms of fairness-utility tradeoffs, i.e. \"does the model retain its capabilities\". Ablation studies are performed on the adjustment strength hyperparameter.", "questions": "1) How could this approach scale to multiple attributes? Presumably we may want to debias the LMM away from a large set of protected attributes, are there any issues for this method?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This work is well written and easy to follow\n\n* I believe this method is a sensible step in the context of previous work. All debiasing methods have an inherent tradeoff with the method they use to localize the bias (internal or logits). As long as the classifier training is reliable, this method seems reasonable. \n\n* I appreciated the comparison of resources needed to run each method examined here, this is important for an inference time method. \n\n* The datasets and models used are appropriate, but I would have appreciated more models for a method that only augments the logits e.g. qwen-vl and/or llama-3.2", "weaknesses": "* My main concern with this method is the reliance on an external classifier. Other previous inference-time methods, specifically model steering are able to more easily compute an adjustment to the target model's behavior no matter what the target attribute is. The need to train a binary classifier may be difficult depending on the deployment setting as well as the acquiring data about the attribute itself. These kinds of inference-time debiasing methods exist primarily because fine-tuning is prohibitively expensive to do well, so the requirement to train a classifier seems like a regression. \n\n* I think absolute counts or base rates are important here in the context of debiasing. Especially given that some models may have very different rates of producing biased text, its useful to know how much biased text is generated vs caught.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "37aq6gg1bS", "reviewer": "Reviewer_Jum4", "summary": "This paper proposes Adaptive Logit Adjustment (ALA), a post-hoc debiasing method for VLMs and LMMs. Instead of modifying encoder or decoder representations, ALA directly adjusts token-level logits during autoregressive generation. The method leverages external classifiers (for image and text) to measure bias misalignment (e.g., gender, toxicity) and performs logit correction guided by gradient-based importance scores. Experiments across image captioning and multiple VQA tasks show that ALA effectively mitigates bias while preserving model utility.", "questions": "Please refer to weaknesses section.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper suggest post-hoc debiasing method for VLMs and LMMs, which does not require additional retraining or affect internal representation which may lead to large degradation.\n2. Proposed idea is simple but effectively mitigate the bias of VLMs and LMMs.\n3. The paper demonstrates consistent improvements in fairness scores with minimal degradation of utility across diverse datasets.", "weaknesses": "1. Results that show the “prompt” baseline has bad fairness score is somewhat interesting. Including experiments with VLM backbones which have stronger instruction following capability, such as Qwen-2.5-VL (or Qwen-3-VL), could strengthen the proposed method.\n2. While the paper claims generality to “large multimodal models,”, the evaluated VQA tasks are limited to the captioning or keyword tagging. I suggest that the authors include experiments on more diverse VQA tasks or moderate their claim.\n3. Minor weaknesses about presentation\n    1. Several core components are deferred to the appendix (e.g., Algorithm 1, analysis about limitation regarding classifier, or definition of evaluation metric D_mean). A brief summary of these in the main text would aid comprehension.\n    2. “Baseline” in figure 4 is ambiguous. It seems to denote the original model. Explicitly stating this would help readers interpret the plots.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)