Label-Free Mitigation of Spurious Correlations in VLMs using Sparse Autoencoders
Authors: Bharat Chandra Yalavarthi
Nalini K. Ratha
Venu Govindaraju
Venue: ICLR 2026 Poster (2026)
Keywords: Spurious Correlations
Sparse Auto Encoders
Vision-Language Models
Interpretability
Primary area: interpretability and explainable AI
TL;DR: We propose an interpretable and label-free approach to remove spurious features from vision language model representations.

Abstract: Vision-Language Models (VLMs) have demonstrated impressive zero-shot capabilities across a wide range of tasks and domains. However, their performance is often compromised by learned spurious correlations, which can adversely affect downstream applications. Existing mitigation strategies typically depend on additional data, model retraining, labeled features or classes, domain-specific expertise, or external language models posing scalability and generalization challenges. In contrast, we introduce a fully interpretable, zero-shot method that requires no auxiliary data or external supervision named DIAL (Disentangle, Identify, And Label-free removal). Our approach begins by filtering the representations that might be disproportionately influenced by spurious features, using distributional analysis. We then apply a sparse autoencoder to disentangle the representations and identify the feature directions associated with spurious features. To mitigate their impact, we remove the subspace spanned by these spurious directions from the affected representations. 
Additionally, for cases where prior knowledge of spurious features in a dataset is unknown, we introduce DIAL+ which can detect and mitigate the spurious features. We validate our method through extensive experiments on widely used spurious correlation benchmarks. Results show that our approach consistently outperforms or matches existing baselines in terms of overall accuracy and worst-group performance, offering a scalable and interpretable solution to a persistent challenge in VLMs.

Published as a conference paper at ICLR 2026
LABEL-FREE MITIGATION OF SPURIOUS CORRELA- TIONS IN VLMs USING SPARSE AUTOENCODERS
Bharat Chandra Yalavarthi, Nalini Ratha, Venu Govindaraju University at Buffalo, Buffalo, NY, USA Syallvar@buufall. edv, nratha www.ffll..fl...ooooo sovind@buffalo.eddd
ABSTRACT
Vision-Language Models (VLMs) have demonstrated impressive zero-shot capa- bilities across wide range of tasks and domains. However, their performance often compromised by learned spurious correlations, which can adversely affect downstream applications. Existing mitigation strategies typically depend on addi- tional data, model retraining, labeled features or classes, domain-specific exper- tise, or external language models posing scalability and generalization challenges. In contrast, introduce fully interpretable, zero-shot method that requires no auxiliary data or external supervision named DIAL (Disentangle, Identify, And Label-free removal). Our approach begins by filtering the representations that might be disproportionately influenced by spurious features, using distributional analysis. We then apply sparse autoencoder disentangle the representations and identify the feature directions associated with spurious features. To mitigate their impact, we remove the subspace spanned by these spurious directions from the affected representations. Additionally, for cases where prior knowledge of spurious features in a dataset unknown, we introduce DIAL+ which can detect and mitigate the spurious features. We validate our method through extensive ex- periments on widely used spurious correlation benchmarks. Results show that our approach consistently outperforms matches existing baselines in terms of over- all accuracy and worst-grow group performance, offering a scalable and interpretable solution to persistent challenge in VLMs.
INTRODUCTION
Contrastive image-language models like CLIP have become foundational components in numer- ous applications, largely due their remarkable zero-shot generalization capabilities Radford al.) (2021) ; Cherti et al. (2023). By training on web-scale data, they eliminate the need for task-specific labeled datasets, enabling efficient and scalable solutions for a wide range of downstream tasks and generative pipelines Lu al. (2025); Zhu et al. (2025); Adila et al. (0224). However, despite strong aggregate performance, these vision- langune models (VLMs) often fail on specific demographic semantic groups, exhibiting performance far below the average Zhuu al. (2025); CChang et al. (2023a): Yang et af. (2023). This vulnerability stems from their tendency to learn spurious corre- lations relying on non-causal features that are coineidentally prevalent in the training data rather than the causal task-relevant attributes Li et al. (2025). A commonly cited example in the litera- ture when models rely imaging artifacts rather than causal disease features to make medical diagnoses Lu et al. 20255) ; Li et al. (2025). Figure 2 2]shows some examples of these spurious corre- lations visualized through a heatmap. As these spurious correlations may not hold in worl-world test data, the model's reliability and zero-shot promise are fundamentally undermined, raising serious concerns about fairness and robustness Varma al. 2024): CChang et al. (022b))
In recent times, growing body of work has sought to mitigate the spurious correlations in VLMs. Many works like Chuang et al. (2023b); Trager et al. (2023); Lauscher et al. (2020) have focused on the textual modality for debiasing, but do not address biases encoded in the visual representa- tions. Also, methods like Lauscher et al. (2020) require domain expertise or manual specification of debiasing textual prompts. Other prominent methods Yang al. (2023); Zhang & Ré (2022); Waan et al. (2023); (hu et al. al, (2025) require fine-tuning the model or access to class and/or spurious feature labels, which negates the primary zero-shot advantage of VLMss. Recently, few methods

---

Published as a conference paper at ICLR 2026
Zoro Shot Promots R
Zero-Shot Predictions
SAE
Cintrrad based Flaiiig
Orhegona Projection
Candidate Samples
Cnooder
Decoder
K-NN
did
8xl
Isd
Deblazed Embedding
J
Faskkre Viclore
Candidate selection
Decader Columree)
(eg: Male Female)
Ssurlous Featuures
Spurious Feature
Weighting and SoftMax
Spurious
Neuron Attribution
Subspace
Spurious Feature gsst
Through dotection by DIAL+
Figure I: Overview of our proposed method. DIAL takes in VLM image embeddings and spurious features of a given dataset (e.g.. "Male" and "Female" for the CelebA dataset). If DIAL+ used, then there is no requirement for these spurious feature descriptions. The entire method operates in
zero-shot setting without requiring training, external data, class labels, or spurious feature labels.
have emerged that operate in truly zero-shot setting Lu et al. (2025): ; Adila et al. (2024); ) C Chuang al. (2023b). However, they introduce their own set of challenges. For instance, TIE Lu et al. (2255) relies spurious feature labels for each sample to achieve optimal performance, which are often unavailable and expensive to acquire. Moreover, although it offers label-free variant (TTE*), both implementations practically depend on additional data to compute their scaling factors. Con- currently, methods like ROBOSHOT Adila et al. (2024) rely on Large Language Models (LLMs) to generate task- task-specific insights, introducing concerns about reliability, hallucination, and sensitivity to the choice of LLM Lu et al. (2025).
To address the challenges of the current methods in mitigating spurious correlations, propose an interpretable algorithm, DIAL (Disentangle, Identify, And Label-free removal), which works in a complete zero-shot setting without requiring training, additional data, or labels (both class labels and spurious feature labels). Our framework, when using DIAL requires two inputs: VLM embeddings of samples of a dataset and high-level description of spurious features affecting the dataset (e.g.. "Male", "Female" for CelebA). If DIAL+ is employed, it only requires VLM embeddings as it can detect the possible spurious features before mitigating them. Our mitigation method unfolds in three main steps. First, guided by the insight that samples affected by spurious features often deviate from their class centroids[ et (2025), we identify candidate set of potentially biased samples without class labels using zero-shot predictions as pseudo-labels. Second, we employ Sparse Autoencoders (SAEs) to project these embeddings into a disentangled feature space. Within this space, we intro- duce a technique to reliably identify the feature directions that encode the spurious features. Finally, we debias the identified samples by removing the spurious subspace via an orthogonal projection. We also provide a technique to select the optimal parameters for our debiasing process, namely the fraction of features (a) and the magnitude of subspace removal (A). The overview of our proposed approach is given in Figure
We conduct extensive experiments on five standard benchmark datasets, demonstrating the efficacy of our method compared to baselines. In summary, our contributions are:
We propose DIAL, fully zero-shot and interpretable framework designed to mitigate spu- rious correlations without requiring model training, additional data, class labels, or spurious feature annotations.
To address scenarios where spurious attributes unknown a priori, we introduce DIAL+, which autonomously detects and mitigates spurious correlations while maintaining perfor- mance comparable to DIAL.
2

---

Published as a conference paper at ICLR 2026
ISIC
COVID-19
CelebA
Waterbirds
Figure 2: This figure illustrates how a CLIP model relies on spurious correlations for zero-shot predictions. For the ISIC dataset, it focuses X-rays, it attends to a medical device rather than pneumonia indicators. On CelebA, it uses facial features instead of hair identify "Blond hair", and for Waterbirds, it relies the water background rather than the bird.
image artifact instead of the lesion. For chest
We develop novel technique to identify and isolate spurious feature subspaces directly from disentangled Sparse Autoencoder (SAE) representations
zero-shot setting.
We validate approach across multiple benchmarks and VLM backbones, demonstrat- ing that our method consistently outperforms or performs comparably to state-of-the-art baselines, while also estahlishing its efficacy in debiasing image retrieval.
RELATED WORK
Mitigation with training or labels: The problem of mitigating spurious correlations in deep learn-
ing models has been extensively studied, Techniques like (Saaaaa et al. (2019a); Liu et al. (2021)
Yao al. (2022); y Krueger et al. (202I): : Lu et al. (2024): Arjovsky et al. (2019 Idrissi et al. (2022)
Yang et al. (2023 Goyal et al. (2023): Zhang & Ré (2022) aim to remove the effect of spurious correlations through reweighting the training samples, finetuning, regularization, or disparate loss
functions. More recently Zhu et al.) (2025) proposed to train biased classifier to identify group labels and debias the classifier for VLMs. Li al. (2025) identifies the minority samples using their dispersed distribution, and learns transformation to a bias-invariant representation. Varma jet al. (2024) shows that using region-level information in the images during training helps VLMs to ignore spurious correlations. All these methods require some form of training/fine-tee -tneettunnnn.. labels, or access to the model parameters. In contrast, our method works completely in a zero-shot setting without needing any labels, fine-tuning, access model parameters.
Mitigation in zero-shot setting: Several of the recent works on mitigating spurious correlations in VLMs focused on doing in zero-shot setting. [e et al. (2023) proposes augment text prompts with parent and child from WordNet hierarchy to improve zero-shot generalization. Trager et al. 2023) uses the average of prompts, which made from combining class labels with spurious features to get debiased text prompts for each class. [Dehdashtian et al. (2024) uses reproducing ker- nel Hilbert spaces debias CLIP's image and text representations. Chuang et (203b)) proposes a closed-form method through a calibrated projection matrix to remove biased direction from CLIP embeddings. [u et al. (225)) mitigates spurious correlations by translating image embeddings along the direction of spurious vectors computed from text prompts. Its main algorithm needs access to spurious feature labels for each sample, so the authors also propose a variant that adapts when spu- rious feature labels are not present. Additionally, both variants of TIE require access to additional data to compute the scale parameter. Adiia et al. (2024) uses LLMs to generate insights on spuri- features, which are used to remove harmful components while keeping the useful ones. Unlike other zero-shot approaches, our method requires no auxiliary data for parameter tuning, no spurious feature labels, and no LLM for generating insights.
3

---

Published as a conference paper at ICLR 2026
Interpretable Methods for Mitigation: Some of the works have proposed using interpretability methods for mitigating spurious correlations. Wu et al. (2023) proposes an iterative framework that discovers numan-interpretable spurious concepts and intervenes training data to mitigate their influence. (Charraborty et al. (2024) uses explainability-hare
heatmaps for creating pseudo labels to retrain and improve robustness spurious features in an unsupervised manner. Karvonen et al. (2024) introduces a method to evaluate an SAE based on its capacity to mitigate spurious correlations. To do this, they train linear classifiers identify specific neurons correlated with known spurious attribute. The activations of these identified neurons are then ablated (i.e., zeroed out), and the resulting impact on model performance is measured. Unlike our approach, their method requires labeled training data and relies on activation zeroing rather than the removal of spurious subspace via orthogonal projection. Additionally, SAEs have been applied for concept erasure in diffusion models, TTia et al. (2025) finds unwanted concepts and deactivates them by modifying their activation with a temperature parameter. Recently, techniques to obtain contrastive sparse representations Wen et al. (2025) have been introduced, which could be used in combination with SAE for interpretability and mitigation applications.
METHODOLOGY
3.1 SETUP
Let D [(rr vi) == be dataset with labels ys E Y. A VLM uses image encoder & and text encoder Φ; map inputs into d-dimensional embedding space IRd.
For zero-shot classification, a set of class prompts (e.g.. "a photo (²) are tokenized and then embedded by the text encoder to produce set of class vectors {pe) where Pe = de (prompt.).
The probability that an image z; belongs to class e with temperature parameter r computed as:
Ply = c x;) softmax, ( - CosSim(@o(
The set of groups is defined as g x A, where y is the set of class labels and A is the set of spurious attributes. We measure robustness of VLM using three metrics: overall accuracy (Accavg), worst-group accuracy (Accwg), and the performance gap (AcCgap), defined as:
Accug
min Accg. gEg
AcCgap
Accaeg AcCag
The goal of our zero-shot mitigation strategy is to improve both Accavg and Accug, and minimize AcCgap, without requiring training or access to any labels.
3.2 FINDING SPURIOUS FEATURES
Our strategy is use an SAE to disentangle the VLM embeddings e; and isolate feature direc- tions corresponding to spurious attributes. An SAE decomposes an embedding into a sparse, linear combination of monosemantic features that are interpretable.
Given an embedding e E Rd, an SAE computes sparse feature activations 2 E IR' and a reconstructed embedding E d
z = =ct( ence + benc)
Wene + baee
Here, Wene E Rdxx is the encoder weight matrix, and the decoder matrix Waee E R contains the 1 disentangled feature vectors {f,}==1 as its columns. We refer to this set of vectors as the feature dictionary, F.
For each spurious attribute a E A (e.g., "male" or "female"), we identify a subset of feature vectors Ka C F that strongly correlate with it. To do this, we adapt the attribution score method from Karvonen et al. (2024) to a zero-shot setting. First, we use the VLM's zero-shot classification ability to partition the reconstructed embeddings {é) from dataset D into positive set P. (samples exhibiting attribute a) and a negative set N,. This is done using a prompt like "a photo of a a" and its negation.
4

---

Published as a conference paper at ICLR 2026
The attribution score S for each feature vector fy F with respect to attribute is then calculated as:
S(f, a) =
Σ 2.j VEP
Σ N. Zij
x CosSim(fJ, Ca)
PP
NNa
where zi.j is the activation of feature fj for sample i, and E. Φe (prompt,) is the text embedding of the spurious attribute itself. This score high when a feature's direction aligns with the attribute's semantic embedding and its activation is consistently higher for samples in the positive set.
Finally, to form the spurious feature set Ka, we select the top-k features that account for a fraction of the total attribution mass. We sort the features f, by (S(f),a) a)] in descending order (indexed by т) and choose smallest such that: Σラ=1 [5({x)):)) 2 Σj- [S(}. a)]
captures the primary directions in the embedding space associated with the spurious attribute a. The set UsEA Ka contains the feature vectors from every individual spurious feature set Ka
The resulting set Ka = .f
3.3 SPURIOUS FEATURE DETECTION:
To detect spurious features/concepts without relying on pre-defined attribute lists, we propose a data-driven detection method for DIAL+. This approach leverages the disentangled feature space of the SAE to isolate features that drive predictions in potentially biased samples.
Identification of Influential Concepts. First, we determine which disentangled concepts con- tribute decisively to the model's predictions. For given sample with reconstructed embedding é and sparse activations 2;, we simulate the ablation of each feature j. Let ê denote the recon- struction obtained when the activation of feature fj is set to zero:
e,-j = W Wecc(2 O (1 - 1,)) + bdec
where 1, is one-hot vector index We define the set of influential concepts I, for sample as the set of features whose removal alters the zero-shot prediction of the sample:
We define the local influential concepts I, and pool them to create global set Ipoot follows:
I = {1 E ((,,,.... (1,...,,)
argmax P(c êx) argmax P(c (. Ipoot = U 1=1
eEy
eEY
Candidate Sample Selection. Next, we identify the subset of samples in the dataset that are likely affected by spurious correlations. We employ the Candidate Selection Algorithm (Alg. I.. which detects these samples (based on class centroid and k-NN inconsistency). Let Scand denote the set of indices for the samples selected by the algorithm:
Scand = | i (1,...,mn A Algorithm I
returns True)
3. Extraction of Spurious Concepts. Finally, we identify the specific spurious features using the intersection of the pooled influential concepts (Ipoot) and the selected candidate samples (Soand). We compute the activation frequency My for each feature j Ipoot exclusively within the candidate set:
Pj Σ E
SES.ond
Features with high Vj represent concepts from the influential pool that are consistently active in causing samples to deviate toward incorrect class centroids or k-NN inconsistency. We select the top-k most commonly activated concepts based on v to form the final set of spurious concepts K.
3.4 MITIGATING SPURIOUS FEATURES
Given the identified set of spurious feature vectors K., aim to debias the reconstructed VLM embeddings é by removing their components that in the subspace spanned by these features. To account for noise in the feature selection process, first refine the spurious subspace by weighting
5

---

Published as a conference paper at ICLR 2026
each feature fy IC based on its alignment with the mean direction of the set. First, compute the mean vector m of the spurious features:
TeT E
Next, we compute vector of alignment scores E IR K where each element 8, corresponds to feature fj: 8j 3 CosSim(f ff, m) A weight vector then derived by applying the softmax function to these scores, where 3 is a temperature hyperparameter controlling sharpness: w = softmax(s)
To further denoise the set, we prune the features by setting weights that fall below speci- fied percentile to zero, yielding filtered set of feature vectors K S K. with corresponding non-zero weights. We then form a matrix V. whose columns are the weighted feature vectors {wrfi I.f ff E }. We perform QR decomposition on this matrix, V = QR, to obtain an orthonor- mal basis Q for the refined spurious subspace. The projection of é; onto this subspace is given by Cipruj = =Q" 6.
The final, debiased embedding Cr,ceea is obtained by subtracting this projection from the original embedding, scaled by a mitigation factor À E [o, 1]: Ci,ciean = é - Aes
This procedure removes information correlated with the identified spurious concepts while pre- serving other essential features of the original VLM embedding. We employ a targeted mitigation strategy, applying orthogonal projection to remove spurious features only from a subset of samples identified by our candidate selection algorithm (Alg- This algorithm designed to pinpoint sam- ples that are likely to be affected by spurious correlations, which often lead misclassifications. Operating in label-free, zero-shot setting, our approach builds on the insight from prior work Li al. (2025) that biased samples often lie far from their true class centroid. We approximate these class centroids by using the VLM's own zero-shot predictions as pseudo- To enhance the robustness of this selection against noise and outliers, we further refine the candidate set using a standard k-Nearest Neighbors (k-NN) algorithm.
Our framework has three key parameters: the number of neighbors for k-NN, the attribution mass threshold a, and the mitigation strength X. To select these values effectively, propose grid- search-based algorithm (Alg. 2) that optimizes a zero-shot score reflecting the alignment between sample embeddings and the identified spurious features.
EXPERIMENTS
4.1 DATASETS
Following prior work by Lu al. (2025) in zero-shot spurious correlation mitigation, five established benchmarks for evaluating our method. CelebA Liu et al. (2015). Waterbirds Koh al. (2021), FMOW Christie al. (2018) and two medical datasets ISIC et al. (2019), and COVID-19 Cohen et al. (2020). All datasets except FMOW have two classes and two associated spurious features, while FMOW has 62 classes with 5 spurious features. In accordance with the prior work Lu et al. (2025); Adila et al. (2024), we define groups as a combination of class label and spurious feature. For FMOW, procedure given in Wu et al. (202)). For zero-shot classification, we use the same text prompts used in our prior work and evaluate all the baselines with the same text prompts. For example, for the CelebA dataset, the zero-shot text prompts we use are "a photo of celebrity with dark hair" and "a photo of a celebrity with blonde hair".
the
define a group based on the spurious feature following the
4.2 BASELINES
We evaluate our proposed method against existing zero-shot mitigation methods, including TIE et al. 2025). ROBOSHOT[AGO et al. (2024), Ideal Words (Trager et (2023), Orth-Cali Chuang ct al. (202b)), and Perception CLIP An et al. (2024). We also include the zero-shot and GroupPrompt zero-shot performance as the baselines. As established by prior works Sagawa et al. (2019b), we compare on worst group accuracy (Accwg WG). average accuracy (AcCavg Acc), and gap between Acc and WG (AcCgap Gap). In the results, group the baselines into two groups, with meth- ods that require auxiliary information through either additional data, class/spurious feature labels, or L.M insights for mitigation. This group includes Perception CLIP An et al. (202)) ROBOSHOT

---

Published as a conference paper at ICLR 2026
Table 1: CelebA: Comparison our mitigation method with baselines in terms of zero-shot classi- fication. Note that DIAL requires priori list of spurious features, whereas DIAL+ automatically detects and mitigates them. Best performance is bolded, and the second best underlined.
Setting Requirements
CLIP VIT-B/32
CLIP VIT-L/14
Method
Additional Data
Class/Spurious
LLM
AVG (1)
WG(1)
Gap(L)
AVG (t)
WG(1)
Gap(L)
Feature Labels
PerceptionCLIP
80.32
76.46
3.86
81.41
78.70
2.71
ROBOSHOT
X
X
84.77
80.52
4.25
85.54
82.61
2.93
TIE
/
X
85.11
82.63
2.48
86.17
84.60
1.57
TIE>
/
X
85.1
82.63
2.48
86.17
84.60
1.57
Zero-Shot
X
X
X
84.27
78.89
5.38
81.20
73.35
7.85
GroupPrompt
X
X
80.38
74.90
5.48
77.86
68.94
8.92
Ideal Words
X
X
X
80.96
78.12
2.84
89.15
76.67
12.48
Orth-Cali
X
X
82.31
77.92
4.39
81.39
77.69
3.70
DIAL (Ours)
X
X
X 85.54
83,47
2.17
86.87
85.24
1.63
DIAL+ (Ours)
X
X
85.28
83.42
1.86
86.54
85.15
1.39
Comparison of our mitigation method with baselines in terms of zero-shot classification. Note that DIAL requires an priori list of spurious features, whereas DIAL+ auto- matically detects and mitigates them. Best performance is in bold, and the second best is underlined.
Table 2: Waterbirds:
Setting Requirements
CLIP ViT-B/32
CLIP ViT-L/14
Method
Additional
Class/Spurious
LLM
AVG (1)
WG(T)
Gap(L)
AVG (1)
WG(1)
Gap(L)
Data
Feature Labels
PerceptionCLIP
X
82.50
59.78
22.72
86.74
54.12
32.62
ROBOSHOT
X
71.92
54.41
17.51
64.43
45.17
19.26
TIE
/
x 79.82
71.35
8.47
84.12
78.82
5.30
TIEx
/
76.91 x
61.24
15.67
78.98
61.60
17.38
Zero-Shot
X
X
68.48
41.37
27.11
83.72
31.93
51.79
GroupPrompt
X
X
X
66.79
43.46
23.33
56.12
10.44
45.68
Ideal Words
X
X
X
79.20
60.28
18.92
87.67
64.17
23,50
Orth-Cali
X
XXX
X
69.19
54.99
14,20
86.31
58.56
27.75
DIAL (Ours)
X 71.88
$2.82
19.06
82.6
68.69
13.91
DIAL+ (Ours)
X
X
68.48
42.26
26.22
82.25
69.18
12.47
Adila et al. 2024), and VIE/TIE>Lu et al. 2025). The other group, which does not require any of these, proposed method, along with standard zero-shot, GroupPrompt classification, Ideal Words Trager et al. (2023), and Orth-Cali Chuang et al. (2023b). For a fair comparison, we divide the baseline methods into these two groups in the results.
4.3 BACKBONE MODELS
Following prior work Adila al. (004)) Lu et al. 2025 examine CLIP ViT-B/32 (OpenAD, and ViT-L/14 (Laion-2B) Radford Radford et al. (202)): Cherti et al. (2023) as backbones for Waterbirds and CelebA datasets. For the FMOW dataset, datasets ISIC and COVID-19, we use BiomedCL.IP Zhang et al. 2023b). For disentangling the representations,
ViT-L/14 (I (Laion-2B) model. For medical
the pre-trained Matryoksha Sparse Autoencoders (MSAE) Zaigrajew et al. (2025) for all the backbone models used in the experiments. Any other SAE trained for VLMs can also be used instead of MSAE. We have evaluated our method with additional backbones including EVA02-E- 14-plus, ViT-S0400M- 100MM--1-Sippp S-gLLII---3 Cherti et al. (0033 whose results are presented in the appendix.
4.4 RESULTS
CelebA and Waterbirds:

---

Published as a conference paper at ICLR 2026
Table FMOW: Comparison our mitigation method with baselines in terms of zero-shot classi- fication. Note that DIAL requires priori list of spurious features, whereas DIAL+ automatically detects and mitigates them. Best performance is bolded, and the second best underlined.
Setting Requirements
AVG (t) WG(1) Gap(L)
Method
Additional
Class/Spurious
LLM
Data Feature Labels
PerceptionCLIP
X
17.70
12.61
5.09
ROBOSHOT
X
19.79
10.88
8.91
TIE
/
/
X
26.62
20.19
6.43
TIE*
X
26.65
19.84
6.81
Zero-Shot
X
X
X
26.02
18.06
7.96
GroupPrompt
X
14.69
8.75
5,94
Ideal Words
20.21
11.14
Orth-Cali
X
26.11
19.45
6.66
DIAL (Ours)
X
X
X
26.09
19.90
6.19
DIAL+ (Ours)
X
26.23
19.24
6.99
Table Medical Datasets ISIC and COVID-19: Comparison of our mitigation method with base- lines in terms of zero-shot classification. Note that DIAL requires an priori list of spurious features, whereas DIAL+ automatically detects and mitigates them. Best performance is bolded, and the sec- ond best is underlined.
Setting Requirements
ISIC
COVID-19
Method
Additional
Class/Spurious Feature Labels
LLM
AVG (t)
WG(T)
Gap(L)
AVG (1)
WG(1) Gap(
Data
PerceptionCLiP ROBÓSHOT
X
52.74
41.55
11.19
56.87
48.84
8.03
X
X
59.84
53.30
6.54
53.10
32.75
20.35
TIE
x 69.90
65.87
4.03
62.50
52.17
10.33
TIEs
/
X
71.68 x
61.11
10.57
61.08
50.22
10.86
Zero-Shot
X
X 70.21
42.21
28.00
61.81
44.83
16.98
GroupPrompt
X
X
X
30.05
12.13
17.92
48.27
27.58
20.69
Ideal Words
X
X
X
53.07
41.42
11.65
56.84
23.53
33.31
Orth-Cali
X
X
X
72.54
21.43
51.11
51.72
44.83
6,89
DIAL (Ours)
X
X
70.71
68.42
2.29
48.28
12.83
X
DIAL+ (Ours)
X
X
68.93
65.45
3.48
61.11
48.28
12.83
On the CelebA dataset (results in Table I). our method demonstrates superior performance, par- ticularly with the ViT-B/32 backbone. surpasses all zero-shot baselines across all three metrics, even outperforming methods that require auxiliary data, spurious feature labels, the of LLMss. When using the stronger ViT-L/14 backbone, our approach continues to achieve the highest worst group accuracy, lowest performance gap, underscoring its robust efficacy in mitigating spurious correlations.
For the Waterbirds dataset (results in Table 2). using the ViT-L/14 backbone, our method yields sig- nificant improvements in worst group accuracy and effectively reduces the performance gap com- pared to the baselines. We hypothesize that the performance on this dataset influenced by the inherent complexity of the spurious attributes. The concepts "land background" and "water back- ground" are highly varied and complex, making challenging to fully capture the corresponding feature space using only a high-level semantic description. This ambiguity may impact the preci- sion of our attribution score calculation, explaining why some baselines perform better in certain configurations.
We next evaluate our method on the challenging FMOW dataset (results Table Owing to the complicated nature of the dataset, following prior work Lu et al. (2025), only the ViT-L/14 backbone. Our method improves over the baselines in our sub-group on the worst group accuracy while still maintaining comparable average accuracy.
FMOW:
8

---

Published as a conference paper at ICLR 2026
Top
Attribution Mass
Top
Attribution Mass
80.7%
73365
Worst- Cret-Grropp Accurncy
68.7%
Average Accuracy
43.3%
26.0% 26.1%
30
Waterfank
CARRA
Maurteck
Figure 3: Comparison spurious feature selection strategies.
Ablation
Orthogonal Projection
Ablation
Orthogonal Projection
85.89
82.8%
45.25
Worst-Croup ProttCrruup Accuraccy
68.7%
Averuge Accuracy
63.5%
10.9%
4.9%
2JP
Figure 4: Comparison of spurious feature removal techniques.
The results on the medical datasets are presented Table On the ISIC dataset, our method demonstrates a substantial improvement worst group accuracy and corre- sponding reduction in the performance gap compared to all baselines. Notably, our fully zero-shot approach surpasses even those methods that rely on auxiliary data additional labels for debias- ing. Similarly, for the COVID-19 dataset, our approach improves over baselines in worst-group performance; achieves this while maintaining a highly competitive average accuracy.
Medical Datasets:
4.5 DEBIASED RETRIEVAL
Beyond zero-shot classification, we evaluate the efficacy of our method in debiasing image re- trieval. Following the experimental protocol of et (2023b), we perform retrieval based on the cosine similarity between the query text and image embeddings from the FairFace benchmark Kärkkäinen & Joo (2019). To quantify fairness, employ the MaxSkew @ k metric [Geyik et al. (2019). which assesses the maximum logarithmic deviation between the observed frequency of sensitive attribute the top-k results and perfectly uniform distribution. We observe consistent reductions in MaxSkew scores across Age, Gender, and Ethnicity attributes compared to the original zero-shot baseline (ViT-L/14 trained on LAION-2B). These results (Table demonstrate that DIAL effectively mitigates bias within the embedding space, resulting in fairer retrieval outcomes.
Table 5: Evaluation of our framework with image retrieval task on FacrFace.
Sensitive Feature
Original (MaxSkew@ 1000) (L)
DIAL (MaxSkew 1000) (L)
Age
1.32
0.95
Gender
0.30
0.11
Ethnicity
0.61
0,32

---

Published as a conference paper at ICLR 2026
4.6 ABLATIONS
In this section, we justify the technical choices made in our framework through series of empirical studies. We focus on techniques to select the optimal spurious feature vectors and the removal of the spurious features. For the results reported in ablation studies, datasets CelebA, Waterbirds, and FMOW are used with ViT-L/14 as the backbone. Additional experiments concerning SAE selection, the relationship between SAE quality and performance, and further ablation studies are provided in the Appendix.
Selection through top i features vs attribution mass (a): We compare the difference between selecting the top k spurious feature directions and selecting a fraction of the attribution mass. When the proposed parameter search algorithm optimize
a, we that the latter provides better results as shown in Figure 3 This could be due to the varying representation of different features in the SAE. For example, specific concept like "color patch" might be represented with fewer feature vectors than "land background".
Orthogonal projection neuron ablation: Prior works have used both these techniques for con- cept removal. In our experiments (results shown in Figure 4 we find that orthogonal projection is much more effective at removing the spurious features than just ablating the corresponding activa- tions to zero. This may be attributed to orthogonal projection removing the entire spurious subspace, while ablating a specific set of neurons to zero may still leave some unidentified spurious feature vec- tors watering down the mitigation. On the other hand, orthogonal projection can affect non-spurious features they are very close to spurious features.
DISCUSSION
Modality and Scope: While this work focuses on mitigating spurious correlations in image em- beddings, our method is modality-agnostic and can be applied to any VLM embedding. Mitigating with image modality distinguishes our approach from most zero-shot baselines that primarily target the textual modality.
The choice of a pre-trained SAE, backbone feature extractor, and dataset can influence the optimal parameters (ax, X) for mitigation, as different settings yield varying levels of feature disentanglement. However, our zero-shot parameter search addresses this dependency by automatically identifying the optimal configuration. The algorithm optimizes towards embedding equidistance to spurious concepts, thereby reducing bias (for DIAL) or minimizing spurious sample coverage (as determined by Alg. [ for DIAL+)). We note that because our framework operates in a strict zero-shot, data-free regime, the search process relies on the hyperparameters governing candidate selection and the specified parameter search ranges. Future work could explore analytical solutions to further improve the performance of our framework and reduce these dependencies.
Parameterization:
Interpretable Mitigation: A key advantage of our method over prior work is its inherent trans- parency and interpretability. In high-stakes domains, this transparency is crucial for building trust and ensuring reliability. Our framework allows for a direct inspection of the mitigation process, providing a clear mechanism to diagnose the root causes of model failures and perform targeted debugging.
6 CONCLUSION
While VLMs possess remarkable zero-shot capabilities, they are often compromised by spurious correlations from web-scale data. We introduce a fully unsupervised, zero-shot method to mitigate these biases directly the embedding space. Using a pre-trained SAF, we disentangle features and remove identified spurious directions via orthogonal projection on image embeddings. We fur- ther extend this to detect and mitigate correlations without prior knowledge of spurious features. Crucially, our approach requires no additional data, training, labels, or external LLMs, distinguish- ing it from prior work. By targeting image embeddings rather than text, we provide a distinct debiasing alternative. Zero-shot classification experiments across five datasets with multiple back- bones, alongside image retrieval tasks, show our method matches or outperforms state-of-the. techniques. Future work could explore applications in unlearning and fairness.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Dyah Adila, Changho Shin, Linrong Cai, and Frederic Sala. Zero-soot robustification of zero- shot models. In The Twelfth International Conference on Learning Representations, 2024. URL
https: Settps://opentne http///pppeeeeeeeee/e/ee/e.eeeee net /ttr:/orrmm---feuoo------0-
Bang An, Sicheng Zhu, Michael-Andrei Panaitescu-Liess, Chaithanya Kumar Mummadi, and Furong Huang. More context, less distraction: zero-shot visual classification by inferring and conditioning on contextual attributes. The Twelfth International Conference on Learning Repre- sentations, 2024.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 000099997799331 2019.
Bart Bussmann, Patrick Leask, and Neel Nanda. Batchtopk sparse autoencoders. arXiv preprint arXiv:2412.06410,, 2024.
Rwiddhi Chakraborty, Adrian Sletten, and Michael C Kampffmeyer. Exmap: Leveraging explain- ability heatmaps for unsupervised group robustness spurious correlations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12017-12026, 2024.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language- image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2818-2829, 2023.
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6172- 6180, 2018
Ching-Yar Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka. Debias- ing vision-language models via biased prompts. arXiv preprint arXiv:2302.0mm 00070, 2023a.
Ching- Rao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka. Debias- ing dision-language models via biased prompts. arXiv preprint arXiv:2302.00077 2023b.
Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gut- man, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1902.033:
2019.
Joseph Paul Cohen, Paul Morrison, Lan Dao, Karsten Roth, Tim Q Duong, and Marzych Ghas- semi. Covid-19 image data collection: Prospective predictions are the future. arXiv preprint arXiv:2006. / 1988, 2020.
Sepehr Dehdashtian, Lan Wang, and Vishmu Naresh Boddeti. Fairerclip: Debiasing clip's zero-shot predictions using functions rkhss. arXiv preprint arXiv:2403. 15593, 2024.
Yunhao Gc, Jie Ren, Andrew Gallagher, Yuxiao Wang, Ming-Hsuan Yang, Hartwig Adam, Laurent Itti, Balaji Lakshminarayanan, and Jiaping Zhao. Improving zero-shot generalization and robust- ness of multi mooiii-ooal models. In Proceedings of the IEEE/CVF conference on computer vision and pultern recognition, pp. 11093-11101 2023.
Sahin Cem Geyik, Stuart Ambler, and Krishnaram Kenthapadi. Fairness-aware ranking in search & recommendation systems with application to linkedin talent search. In Proceedings of the 25th acm sigkdd international conference on knowledge discovery & data mining, pp. 2221-2231 2019.
Sachin Goyal, Ananya Kumar, Sankalp Garg, 7ico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19338-19347, 2023.
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive worst-group-teee group-aceuracy. In Conference on Causal Learning and Reasoning, pp. 336-351. PMLR, 2022.
11

---

Published as a conference paper at ICLR 2026
Kimmo Kärkkäinen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age. arXiv preprint arXiv:1908. 2019.
Adam Karvonen, Can Rager, Samuel Marks, and Neel Nanda. Evaluating sparse autoencoders on targeted concept erasure tasks, 2024.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of wn-the-wild distribution shifts. In International conference on machine learning, pp. 5637-5664. PMLR, 2021.
David Krueger, Ethan Caballero, Joemn- Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distiibttion generalization via risk extrapo- lation (rex). In International Conference Machine Learning, pp. 5815-5826 PMLR, 2021.
Anne Lauscher, Goran Glavaš, Simone Paolo Ponzetto, and Ivan Vulić. A general framework for implicit and explicit debiasing of distributional word vector spaces. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8131-8138, 2020.
Weiwei Li. Junzhuo Liu, Yuanyuan Ren, Yuchen Zheng, Yahao Liu, and Wen Li. Let samples speak: Mitigating spurious correlation by exploiting the clusterness of samples. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 15486-15496, 2025.
Evan Z Liu, Behzad Haghgoo, Annic S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6792. PMLR, 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference computer vision, pp. 3730-3733 2015.
Shenyu Lu, Junyi Chai, and Xiaoqian Wang. Neural collapse inspired debiased representation learn- ing for min-max fairesss. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 2048 2059, 2024.
Shenyu Lu, Junyi Chai, and Xiaoqian Wang. Mitigating spurious correlations in zero-shot multi- modal models. In The Thirteenth International Conference on Learning Representations, 2025.
Mateusz Pach, Shyamgopal Karthik, Quentin Bouniot, Serge Belongie, and Zeynep Akata. Sparse autoencoders learn monosemantic features in vision-language models, In The Thirty-ninth Annual Conference Neural Information Processing Systems, 2025. URL https:://ppereviieee
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PmLR, 2021.
Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprini arXiv:2407. 14435, 2024.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. arXiv preprint arviv: /9/ 1.08731, 2019a.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. arXiv preprint arXiv: 1.0831, 2019h.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings ACI, 2018.
12

---

Published as a conference paper at ICLR 2026
Zhihua Tian, Sirun Nan, Ming Xu, Shengfang Zhai, Wenjie Qu, Jian Liu, Ruoxi Jia, and Jiaheng Zhang. Sparse autoencoder as a zero-shot classifier for concept erasing in text-to-image diffusion models. arXiv preprint 210::::20030094440 2025.
Matthew Trager, Pramuditha Perera, Luca Zancato, Alessandro Achille, Parminder Bhatia, and Ste- fano Soatto. Linear spaces of meanings: compositional structures in vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15404, 2023.
Maya Varma, Jean-Benoit Delbrouck, Zhihong Chen, Akshay Chaudhari, and Curtis Langlotz. Ravl: Discovering and mitigating spurious correlations in fine-tuned vision-language models, Advances in Neural Information Processing Systems, 37:82235-82265 2024.
Zhengbo Wang. Jian Liang, Ran He, Nan Xu, Zilei Wang, and Tieniu Tan. Improving zero-shot generalization for clip with synthesized prompts. arXiv preprint arXiv:2307.0739) 2023.
Tiansheng Wen, Yifei Wang, Zequn Zeng, Zhong Peng, Yudi Su, Xinyang Liu, Bo Chen, Hong- wei Liu, Stefanie Jegelka, and Chenyu You. Beyond matryoshka: Revisiting sparse coding for adaptive representation. arXiv preprint arXiv:2503.01176, 2025.
Shirley Wu, Mert Yuksekgonul, Linjun Zhang, and James Zou. Discover and cure: Concept-aware mitigation of spurious correlation. In International Conference on Machine Learning, pp. 37765- 37786. PMLR, 2023.
Yu Yang, Besmira Nushi, Hamid Palangi, and Baharan Mirzasoleiman. Mitigating spurious relations in multi-modal models during fine-tuning, In International Conference on Machine Learning, pp. 39365-39379. PMLR, 2023.
Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Im- proving out-o--distriuution robustness via selective augmentation. In International Conference on Machine Learning, pp. 25407-25437. PMLR, 2022.
Vladimir Zaigrajew, Hubert Baniecki, and Przemyslaw Biecek. Interpreting clip with hierarchical sparse autoencoders. arXiv preprint arXiv:2500.20578, 2025.
Michael Zhang and Christopher Ré. Contrastive adapters for foundation model group robustness. Advances in Neural Information Processing Systems, 15:21682-21699 2022.
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, Matthew Lungren, Tristan Naumann, and Hoifung Poon. Large-scale domain-specifee pretraining for biomedical vision-language processing. 2023a. URL
https: http://wwwwr/aaab//3.....0
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Pre- ston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arX1v:2303.00915, 2023b.
Beier Zhu, Jiequan Cui, Hanwang Zhang, and Chi Zhang. Project-prober aggre-ag Efficient fine- tuning for group robustness. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 25487-25496, 2025.
13

---

Published as a conference paper at ICLR 2026
A REPRODUCIBILITY STATEMENT
The data pre-pun vre-proeessing techniques we used for these experiments are the default CLIP preprocess- ing transforms based on the backbone architecture Lu et al. (2025). All the results reported are on the test set of the datasets. MSAE models are trained with the settings mentioned in the GitHub repository of Zaigrajew et 2205 For non-medical settings, we used the datasets mentioned in the MSAE for training the SAES. For medical, we trained SAE with the BiomedCLIP embeddings of PMC-15M train Zhang al. (2023a). While the specific parameters used for our framework are presented, they can be derived by implementing the proposed parameter search along with the candidate selection algorithm.
B LLM USAGE: We used LLMs to polish the write-up after verifying its output content. We also used LLMs precisely for searching purposes find the relevant related works by prompting for related works based specific topic.
C IMPLEMENTATION DETAILS
MSAEs which trained for each backbone model and used across datasets. For non-medical datasets, off-the-shelf MSAEs Zaigrajew et al. (2025) which are trained using the CC3M [Sharma et al. 2018) dataset employed. For medical datasets, MSAEs trained using PMC-15M Zhang et al. (2023b). In all cases, the SAEs were trained using Reverse Weighting (RW) mode with an expansion factor of 8 for 30 epochs. The optimal hyperparameters for spurious feature selection (a), mitigation strength (A), k-NN neighborhood size (k:.), and text centroid weight (w) determined using our parameter search algorithm with specifie search ranges that vary based on the dataset and backbone capacity. The optimal parameter configurations utilized for DIAL to achieve our reported results are summarized in Table 6
Table 6: DIAL hyperparameter configurations used across datasets and backbones for candidate
selection and mitigation.
Dataset
Backbone
Ck
k+
Text Weight (w)
CelebA
ViT-B/32
0.3
0.5
10
ViT-L/14
0.3
0.6
10
Waterbirds
ViT-B/32
0.75
0.9
10
VIT-L/14
0.65
0.9
10
0
FMoW
VIT-L/14
0.7
0.6
25
1
COVID-19
BiomedCLIP
0.6
0.4
10
0
ISIC
BiomedCLIP
0.1
0.2
10
D ABLATIONS AND ANALYSIS:
We perform additional ablation studies to justify the design choices made in our framework and explain its effectiveness.
Candidate Selection: We apply our subspace removal only on candidates that could be dispro- portionately affected by spurious correlations. We find this selection to improve the overall perfor- mance. Table 7 compares our selective candidate selection approach against a global application of the subspace removal method.
14

---

Published as a conference paper at ICLR 2026
Table 7: Ablation study with and without candidate selection
Dataset - Model
Without candidate selection
With Candidate selection
CelebA VIT L/14
86,43/84.82
86.54/85.15
CelebA VIT B/32
84.85/81.67
85.28/83.42
Waterbirds ViT L/14
81.60/51 1.56
82.25/69. .18
Waterbirds ViT B/32
70.45/50.31
71.88/52.82 26.09/19.90
FMOW VTT L/14
26.04/19.55
Table 8: Ablation study for identifying SAE features.
Model Dataset
Mean Activation Diff
Cosine Similarity
Both
CelebA VIT L/14
3,3
86.87/85 24
CelebA VIT B/32
85.54/83.33
85.28/82.78
85.54/83.47
Waterbirds VIT L/14
82.61/68.85
74.95/50.29
82. 82.6/68 69
Waterbirds VIT B/32
71.85/52.68
70.69/48 12
71.88/52.82
ISIC
70.71/68.43
62.45/55.84
70.71/68.42
COVID-19
61.11/48.28
58.33/34.48
61.11/48.28
Feature Attribution: Motivated by prior work on identifying SAE features Karvonen et al. (2024).
we used both the mean activation difference of positive and negative sample activations and cosine similarity of the SAE feature with the text embedding. The ablation experiment (Table ( suggests that mean activation difference alone yields performance comparable to using both.
Analysis: In the four points below, we analyze the possible reasons for the efficacy of our frame- work.
1) Minimized Feature Interference via Disentanglement: Standard baselines often operate in dense, polysemantic embedding spaces. such spaces, removing spurious feature vector fre- quently degrades causal features due to feature superposition. By using the SAE latent space, we leverage a highly disentangled representation where feature vectors are nearly orthogonal. This or- thogonality allows surgically remove spurious features with minimal impact the semantic integrity causal features.
2) Selective Intervention using Candidate Selection: Even within the SAE latent space, perfect thogonality is not always achieved. Blanket removal features across all samples can inadvertently harm "clean" samples (those not relying spurious correlations). We explicitly identify samples that are disproportionately affected by spurious features (high activation of spurious components relative causal ones). We apply our removal intervention only to these identified samples. This preserves the integrity of samples that are already robust, preventing the performance degradation seen when applying global intervention (Table 7).
3) Subspace removal instead of feature ablation: Removing the entire spurious subspace, rather than simply ablating the corresponding feature activation zero, aids in removing unidentified spu- rious features that are highly aligned with that subspace. This results in a more effective elimination of spurious features, as demonstrated in Figure of the paper.
4) Feature selection through Attribution Mass: Since different backbone embeddings and SAEs exhibit varying activation patterns, using a fixed Top-K approach to select feature directions corre- sponding to spurious features was shown to be less effective than selection through attribution mass, as shown in Figure? of the paper.
E ANALYSIS ON SAE ARCHITECTURE AND DATA
In this section, we evaluate the robustness our framework across three dimensions: i) differ- ent SAE architectures (BatchT Bussmann al. (2024), JumpReLU Rajaaannaaan et al. (2024), ii) SAE quality metrics (reconstruction loss, sparsity, and decoder orthogonality). and in) the impact of training SAEs domain-specific debiasing datasets.
15

---

Published as a conference paper at ICLR 2026
CLIPZ.S
DIAL
Figure 5: Visualization of the DIAL shifting model focuses from spurious to causal features. In the CelebA hair color classification task, DIAL redirects attention from gender-correlated attributes toward relevant regions (hair, eyebrows, and facial hair). Similarly, in the Waterbirds dataset, the method shifts focus from the background environment to the bird itself.
E.
IMPACT OF SAE ARCHITECTURE
Table presents the performance of our framework when employing different SAE variants. We observe that MSAE achieves the best overall performance on average. This superiority is likely attributable the quality of MSAE measured by various evaluation metrics Zaigrajew et al. 2025).
Table 9: Evaluation of our framework using different SAEs. Average Accuracy/Worrtt Accu- racy reported.
Dataset
Batch SSPSAF
JumpRel.USAF
MSAE
CelcbA
87.35/84.44
87.74/84.44
86.87/85.24
Waterbirds
71.98/41.51
77.80/56.01
82.6/68.69
ISIC
69.57/58.73
72.16/66.67
70.71/68.42
COVID-19
46.52/20.69
58.33/34.48
61.11/48.28
E.2 SAE QUALITY Vs. MITIGATION PERFORMANCE
We analyze how intrinsic SAE properties, specifically reconstruction quality, sparsity, and decoder orthogonality, affect debiasing performance (Tables T0 and IT) Our analysis yields three key ob- servations:
Disentanglement Robustness: High disentanglement capability (measured here via de- coder orthogonality) persists in the evaluated SAEs even when reconstruction loss is rela- tively high.
2. Dataset Sensitivity: While framework remains effective across varying sparsity levels, the Waterbirds dataset is more sensitive to reconstruction degradation than CelebA.
3. General Improvement: Worst Group (WG) accuracy improves across all evaluated con- figurations.
From these results, infer that long SAE reconstructs well enough to preserve discrimina- tive features while maintaining high disentanglement, it can effectively used to mitigate spurious correlations.
16

---

Published as a conference paper at ICLR 2026
Table 10: Analysis of performance our method w.r.t SAE Quality on CelebA dataset
Recon. Loss (1)
Sparsity (f)
Decoder Orthogonality (1)
Avg/WG Ace (t)
0.44
0.72
0,0016
87.35/84.44
0.012
0.51
0.0014
86.41/84.71
0.019
0.79
0.0016
86.48/84.65
Table 11: Analysis of performance of our method w.r. SAE Quality Waterbirds dataset
Recon. Loss ())
Sparsity (t)
Decoder Orthogonality (1)
Avg/WG Acc (t)
0.44
0.72
0.0016
71.98/41.51
0.075
0.49
0,0013
75.2/49.31
0.019
0.79
0.0016
81.98/68.51
E.3 TRAINING ON DEBIASING DATASET
To assess the feasibility when large-scale pre-trained SAEs are unavailable, trained SAEs from scratch on the specific debiasing datasets: CelebA (~140k images) and Waterbirds (~4.6k images). Table 12 shows that these domain- domain-specifie SAEs successfully improve worst-group accuracy. The performance on CelebA is comparable to that of the pre-trained SAE. However, the Waterbirds- trained SAE performs slightly worse, likely due the limited training data size. These results demonstrate that training an SAE on the target dataset could be a feasible strategy for spurious correlation mitigation,, particularly when data volume is sufficient.
Table 12: Evaluation of our framework when SAE trained with debiasing dataset
Dataset
Original (Avg/WG) (t)
DIAL (Avg/WG) (t)
CelebA Waterbirds
81 $1.20/73.35
86.41/84.71
83.72/31.92
75.2/49.31
FMOW
26.05/18.
26.67/18.16
ADDITIONAL BACKBONES:
We have evaluated our framework additional backbones whose results are provided in Tables 13
Table 13: Evaluation of our framework with SigLIP (VIT:00400MM14--ggLPP----88
Dataset
Original (Avg/WG) (t)
DIAL (Avg/WG)
CelchA Waterhirds FMOW
82.51/79.11
84.32/82.02
80.92/61.37
80,54/66.16
34.53/25.18 18
34. 10/25.066
Table 14: Evaluation of our framework with EVA02-E-14-plus
Dataset
Original (Avg/WG) (t)
DIAL (Avg/WG) (t)
CelehA
84.78/80.54
87.76/86.52
Waterhirds
76.95/37.85
76.78/56.65
FMOW
29.62/15.97
29.30/16.70
17

---

Published as a conference paper at ICLR 2026
Table 15: Evaluation of our framework with ViT-H--44qqickkellu
Dataset
Original (Avg/WG) (t)
DIAL (Avg/WG) (t)
CelebA
83 23.80/80.00
83.81/81.59
Waterbirds
85.60/51.09
88.47/63.86
FMOW
30.21/19.44
30.57/19.90
G INTERPRETABILITY EVALUATION
To evaluate the interpretability of our framework, we utilize the Monosemanticity Score (MS) pro- posed by Pach al. (2025). We compute the MS score the validation sets of the SAEs. Specif- ically, we use the ImageNet validation set for the standard vision backbones (ViT-B/32, ViT-L/14) and the PMC-15M validation set for the medical backbone (BiomedCLIP), This ensures that the interpretability scores reflect the general quality and complexity of the features learned by the SAE, independent of the specific downstream tasks.
We computed the MS scores using the automated method from Pach et al. (2025) across all neurons
our trained SAEs. The average per-neuron MS scores are reported in Table 16
Table 16: Average Monosemanticity Scores (MS) for the SAEs used in our framework.
SAE Backbone Model
MS Score (Avg)
CLIP ViT-B/32
0.56
CLIP ViT-L/14
0.43
BiomedCLIP
0.42
To contextualize these results, we refer to the user study conducted by Paah et al.) 20255) which cali- brates MS scores against human perception. Their study establishes that MS score in the range of 0.400.5 corresponds to human alignment rate of approximately 65-70%. Our results (0.42-0.56) indicate that the features learned by the SAEs are largely monosemantic and align with human per- ception. Consequently, this semantic coherence enables a verifiable explainability pipeline because the underlying units represent interpretable concepts, and human experts can explicitly interpret the spurious feature directions identified by our model and inspect the mitigation process.
H ALGORITHMS
Algorithm Candidate Selection
Require:
1: E
{es (=))=== set image embeddings.
2: y {úr 1 set of pseudo-labek, from zero-shot predictions.
3: T {c-+t, }z): map of class labels to embeddings.
4: k: number of neighbors for k-NN.
5: w: text embedding weight.
Calculate hybrid centroids for each class
6: for each class E unique(Y) (Y) do
7: He t (1 --w w) Mean {er 0. = c}) + w T[c]]
8: end for
b Identify candidates based on centroid similarity or k-NN disagreements 9: Meentroid (argmax, CosSim(er,, (e) # Wel
MEnn < K-NN(e,, F, i=1
11: M 4- Moentroid V Mkan
Combine candidate sets
12: return M
18

---

Published as a conference paper at ICLR 2026
Algorithm 2 Optimal Debiasing Parameter Search
Require: E fer 1 the set of original VLM embeddings. tei, tez, text embeddings for oppos- ing spurious concepts (e.g., "male" and "female"). Sk, Sw, Sa, Sx search ranges for parameters W at, and X.
Ensure: ka, w; â, X, the optimal framework hyperparameters.
1: scorebest f- 00
2: for each k Sy and w Sw do
3: M < CANDIDATESELLECIONN..I..
> Algorithm Apply mask get the candidate subset
4: Eaub f- [[
5: if Essul = 0 then continue
6:
end if
7:
for each Sa do
8:
Q 4 I SPURIOUSSSUSSPACIEESSS a)
9:
if is not valid then continue
10:
end
Search for the best global À for this mask and subspace
12:
for each À E Sx do
13:
scorecarrent.patt
14:
for each sample e₁ Esuh do C,claan f- E; AQQ"e) eq)
> Apply orthogonal projection L2 normalization
16:
Ci,elam f- Normalize(
b Score measures neutrality between the two opposing concepts deurrent |CosSimmes,atemm fe1) -osSSimeei,llmnn (22)
18:
19:
SCOT Ecoreeurrent.paai SCOT Eourrent.pair + deurrent
20:
end for
21:
> Calculate the mean score for this parameter combination score carrennt pair current pair Esbbl
22:
23:
scor ecoreeurrett.pair < scorebest then
24:
scorCbest +- ecorfeurreet..pirr
25:
kx, Φ t- k, w
26:
à, + a, A end if
27:
28:
end for
end for
30:
end for
31: return k,, ú ā, Å
19


Reviews:
[{"review_id": "63jGKlsno7", "reviewer": "Reviewer_ddFq", "summary": "This paper proposes DIAL, a label-free and zero-shot method to mitigate spurious correlations in vision-language models (VLMs). The approach uses sparse autoencoders to disentangle image embeddings and identify feature directions associated with spurious attributes. These directions are then removed via orthogonal projection to produce debiased representations. The method is evaluated on several benchmark datasets and compared against existing zero-shot debiasing techniques.", "questions": "Can your method be extended to mitigate spurious correlations in text embeddings or multi-modal fusion layers?\n\nIs there any evidence that your projection-based debiasing improves downstream task performance beyond classification?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The method is fully zero-shot and does not require labeled data, retraining, or external models, which improves scalability.\n2. The use of sparse autoencoders for disentangling representations is well-motivated and contributes to interpretability.", "weaknesses": "1. Limited novelty: The core idea—removing spurious directions via projection—is conceptually similar to prior work. The use of sparse autoencoders is incremental and not fundamentally new in the context of representation disentanglement.\n\n2. Low practical impact: The spurious correlation issues addressed (e.g., background bias in Waterbirds, gender bias in CelebA) are well-known and have been extensively studied. The paper does not convincingly demonstrate that these issues remain critical in modern VLMs.\n\n3. Outdated model focus: The analysis centers on older VLMs like CLIP ViT-B. It remains unclear whether the same spurious correlation problems persist in newer models such as SigLIP, OpenCLIP, or multi-modal transformers trained with more diverse data.\n\n4. Assumption-heavy candidate selection: The method relies on pseudo-labels and centroid-based heuristics to identify biased samples, which may be unreliable in real-world settings or for more complex tasks.\n\n5. Lack of generalization evidence: The paper does not explore whether the proposed mitigation transfers across tasks (e.g., retrieval, captioning) or domains beyond the selected benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8QEKiTmi4D", "reviewer": "Reviewer_aziT", "summary": "VLMs often rely on spurious correlations, which can affect downstream tasks. The authors introduce DIAL, a zero-shot method that finds and mitigates spurious correlations. DIAL operates by (1) filtering representations that might be disproportionately influenced by spurious features, (2) applying a sparse autoencoder to disentangle the representations and identify feature directions associated with spurious feature, and (3) removing the subspace spanned by the spurious directions from the representations. Results across several benchmarks demonstrate the utility of the method.", "questions": "Questions are listed above under weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This work addresses an important problem - finding and mitigating spurious correlations learned by vision-language models\n- Results show performance improvements when compared to several baselines, suggesting utility of the approach. The method also works well across domains (i.e. general domain as well as medical domain).", "weaknesses": "- **Insufficient analysis:** Section 4.4 provides overall metrics across various datasets, but does not provide sufficient fine-grained analysis of results. Ablations are also limited. Ultimately, it is not clear to me *why* the method works better than baselines.\n- **Need for attribute labels:** The proposed method requires a set of candidate spurious attributes, which may not always be known ahead of time and might limit utility of the method in real-world settings.\n- **Choice of sparse autoencoder:** The authors consider one off-the-shelf pretrained sparse autoencoder for their analyses. How robust are the results to different choices of the autoencoder?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fr8eoLRlK2", "reviewer": "Reviewer_bU24", "summary": "The paper presents a novel, zero-shot method named DIAL to mitigate spurious correlations in Vision-Language Models. DIAL uses the VLM's own zero-shot predictions to create pseudo-labels for identifying samples likely affected by a known spurious attribute and leverages a pre-trained Sparse Autoencoder to decompose VLM embeddings to gain more disentangled, interpretable features that correspond to the spurious attribute.\nThe authors validate DIAL on five benchmark datasets using multiple VLM backbones. The results show that DIAL consistently improves worst-group accuracy over baselines.", "questions": "The paper's utilization of SAE with KNN for identifying useful sparse representations for downstream tasks is a technique that has been explored in prior literature. For example, the following papers should be discussed in the related work:\n\n[1] Tian, Zhihua, et al. \"Sparse autoencoder as a zero-shot classifier for concept erasing in text-to-image diffusion models.\" arXiv preprint arXiv:2503.09446 (2025).\n\n[2] Wen, Tiansheng, et al. \"Beyond matryoshka: Revisiting sparse coding for adaptive representation.\" arXiv preprint arXiv:2503.01776 (2025).", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Comprehensive Empirical Evaluation:** The experimental setup is thorough and convincing.\nThe use of five standard and diverse benchmark datasets, including challenging medical and real-world scenarios, demonstrates the method's broad applicability.\n\n2. **Clarity and Presentation**: The paper is well-written, logically structured, and easy to follow.", "weaknesses": "1. **Dependency on Pre-trained SAEs**: The method's effectiveness is largely affected by the quality of a pre-trained SAE for the given VLM backbone. The paper does not discuss the sensitivity of DIAL to the SAE's quality (e.g., degree of disentanglement, reconstruction error, sparsity level). If a high-quality, pre-trained SAE is not available for a particular VLM, the contribution of DIAL is limited.  Plus, I'm quite curious about which kinds of SAE models[1,2,3] are most suitable for spurious tasks. \n\n2. **Requirement of a Spurious Concept Description:** While label-free, the method still requires a user to provide a high-level textual description of the spurious attributes (e.g., \"Male\", \"Female\"). This assumes that the source of spurious correlation is known, which means that the method cannot discover unknown or hard-to-describe spurious features (e.g., a subtle imaging artifact without a common name). This limitation should be explicitly stated.\n\n3. **Potential for Negative Interference:**  The orthogonal projection forcefully removes any information in the direction of the spurious subspace. If a genuinely causal feature is closely aligned with a spurious one in the embedding space, this process could inadvertently harm model performance by removing useful information. I believe the paper should give a more detailed discussion or analysis of this potential failure mode.\n\n[1] BatchTopK SAE: Bussmann, Bart, Patrick Leask, and Neel Nanda. \"Batchtopk sparse autoencoders.\" arXiv preprint arXiv:2412.06410 (2024).\n\n[2] JumpReLU SAE: Rajamanoharan, Senthooran, et al. \"Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders.\" arXiv preprint arXiv:2407.14435 (2024).\n\n[3] SAE + Contrastive loss: Wen, Tiansheng, et al. \"Beyond matryoshka: Revisiting sparse coding for adaptive representation.\" arXiv preprint arXiv:2503.01776 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NsriL8sDJ2", "reviewer": "Reviewer_TSFp", "summary": "This paper presents DIAL, the motivation is to achieve robustness of zero-shot classification of VLM like CLIP. For the disentanglement, the author proposes the use of a spare autoencoder to decompose the attribution in the column space of the decoder. For identification, the author proposes the attribution score such that it can align the feature vector with the spurious concept without using spurious labels. To remove the spurious feature, the author first finds the space spanned by the spurious vector, then projects the visual embedding to this space and removes such components. The author conducts benchmark dataset evaluation on the Group robustness and overall performance. The results show it surpasses the existing SOTA, like TIE or Orth-Cali.", "questions": "(1) Related to weakness 3, I would like to see the dataset-specific SAE on the outcomes. \n\n(2) I don't quite get how you evaluate the explaniblity, as you mentioned this method is fully interpretable.\n\n(3) Figure 2 is a good motivation figure. I would also want to see the heatmap after applying this method.\n\n(4) I am also curious how the alignment between the spurious vector you found based on SAE and the spurious vector found by TIE using the spurious text prompt?\n\n(5) Can the method find the novel spurious feature? Like in ISIC, there are multiple spurious features. Can we apply DIAL to find and mitigate such spurious features?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I found this paper quite interesting.\n\n**1** It combines the recent trend of explainability in LLM that use a sparse autoencoder to disentangle the feature vector, and align with a post-hoc explanation to find spurious feature direction. \n\n**2** The paper is well presented, and the logical flow of the paper is good. The overall soundness of the paper is good for me. \n\n**3** The author conducted the benchmark evaluation, which is at least comparable with the existing work.", "weaknesses": "1. In Figure 4, I don't quite understand why the average Acc can be even lower than WGA  in FMOW. \n\n2. To my knowledge, sparse autoencoders (SAE) are often used in the explanation of the transformer encoder's FFN layer. I don't know how the performance would be that migrates the SAE to explain the latent representation. \n\n3. Correct me if I am wrong. I found the SAE uses pre-trained weights. Would there be any distribution shift to align with the spurious vector in the CLIP models? Why don't we train on the specific dataset?\n\n4. Equation at line 208, I think both parts could be sort of the attribution score. For the first term, it shows how the activation of the feature towards the positive spurious concept. For the second term, it shows how the feature vector aligns with the spurious text prompt. Then my question is, why do we need both terms multiplied together? Have you tried just using a single term in ablation? \n\n5. In line 239, why do we use the reconstructed embedding to remove the spurious vector, not the original embedding?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)