Bridging Explainability and Embeddings: BEE Aware of Spuriousness
Authors: Cristian Daniel Paduraru
Antonio Barbalau
Radu Filipescu
Andrei Liviu Nicolicioiu
Elena Burceanu
Venue: ICLR 2026 Poster (2026)
Keywords: spurious correlation
interpretability
clip
foundation models
Primary area: interpretability and explainable AI
TL;DR: An embedding space method for identifying spuriously correlated concepts in a dataset, based on foundation models' fine-tuned weights.

Abstract: Current methods for detecting spurious correlations rely on data splits or error patterns, leaving many harmful shortcuts invisible when counterexamples are absent. We introduce BEE (Bridging Explainability and Embeddings), a framework that shifts the focus from model predictions to the weight space and embedding geometry underlying decisions. By analyzing how fine-tuning perturbs pretrained representations, BEE uncovers spurious correlations that remain hidden from conventional evaluation pipelines. We use linear probing as a transparent diagnostic lens, revealing spurious features that not only persist after full fine-tuning but also transfer across diverse state-of-the-art models. Our experiments cover numerous datasets and domains: vision (Waterbirds, CelebA, ImageNet-1k), language (CivilComments, MIMIC-CXR medical notes), and multiple embedding families (CLIP, CLIP-DataComp.XL, mGTE, BLIP2, SigLIP2). 
BEE consistently exposes spurious correlations: from concepts that slash the ImageNet accuracy by up to 95\%, to clinical shortcuts in MIMIC-CXR notes that induce dangerous false negatives. Together, these results position BEE as a general and principled tool for diagnosing spurious correlations in weight space, enabling principled dataset auditing and more trustworthy foundation models. The source code is publicly available at https://github.com/bit-ml/bee.

Published as a conference paper at ICLR 2026
BRIDGING EXPLAINABILITY AND EMBEDDINGS: BEE AWARE OF SPURIOUSNESS
Cristian Daniel Páduraru!.
Antonio Barbalau
Radu Filipeecu 1.2
Andrei Liviu Nicolicioiu
Elena Burceanu
Bitdeffnner, Romania
University of Bucharest, Romania
Miila, Montreal, Canada
"University of Montreal, Canada
(cpaduraru, abarbalau, ebarceam)@@bttteeeedde....
andrei.nicolice iiclliicciiii quebec
ABSTRACT
Current methods for detecting spurious correlations rely on analyzing dataset statistics eπτоr patterns, leaving many harmful shortcuts invisible when coun- terexamples are absent. introduce BEE (Bridging Explainability and Embed- dings), framework that shifts the focus from model predictions the weight space, and to the embedding geometry underlying decisions. By analyzing how fine-tuning perturbs pretrained representations, BEE uncovers spurious correla- tions that remain hidden from conventional evaluation pipelines. We use linear probing as a transparent diagnostic lens, revealing spurious features that not only persist after full fine-tuning but also transfer across diverse state-of-the-arr mod- els. Our experiments cover numerous datasets and domains: vision (Waterbirds, CelebA, ImageNet- Ik), language (CivilComments, MIMIC-CXR medical notes), and multiple embedding families (CLIP, CLIP-DataCompXXL mGTE, BLIP2, SigLIP2). BEE consistently exposes spurious correlations: from concepts that slash the ImageNet accuracy by up to 95%, to clinical shortcuts in MIMIC-CXR notes that induce dangerous false negatives. Together, these results position BEE as general and principled tool for diagnosing spurious correlations weight space, enabling principled dataset auditing and more trustworthy foundation mod- els. Code publicly available HERE
PEDICTED
For Track
PREDCCED Artantant Solertrr
PRESACTED
PREDITED:
Draes
PREDCTED
Piene Dat
MEEECTED
Pundling Beg
PREZATTDD
Dack
Figure 1: Qualitative results with BEE for CLIP VIT-L/14 fine-tuned on ImageNet- Ik. Although (REAL) class is clearly depicted, adding object tied spurious concept (SC) flips the prediction to (PREDICTED) class absent from the image, leading to unexpected and unwanted behavior.
INTRODUCTION AND BACKGROUND
Deep neural networks, and especially fine-tuned versions of foundation models, are increasingly deployed in critical areas such as healthcare, finance, and criminal justice, where decisions based on spurious correlations (SCs) can have severe societal consequences (Angwin et al., 2016 Caliskan al. 2017). Even if pretrained model has been validated by the community, the dataset leveraged

---

Published as a conference paper at ICLR 2026
ARCHITECTURE
STEP 1: L-LINEAR LAYER TRAINING
Foundation Model M
Seppact
Training Data (Irages]
Encoder for Chosen Modallty (magel
Lineer Layer (W)
Shared embedding space
MC"enchh3) MinGolatih"1
LEGEND
W*
Concest entoobinng
Fire Truck?)
Cliss weigt
Concepts G
Text Encoder
Scope chay conopte
Trusk"
Class "Fire Truck?
STEP 2: SPURIOUS CORRELATIONS IDENTIFICATION
Step
concepts
Step Ranking each olass
2c: Dynamic thresholding
Concept oxtraction - Class-noutral Concepts
Rarkeed Correlated Concepts
Firofighter
Spuriously Correlated Concopts
Wheels Firefighter Ocean
Hydrant Extinguisher Roar Stop
Light Bur
Extingsssher
Hydrant
Firefighter
Hydrant
Class TFire Truck"
Class "Fire Truck"
Ocean
Class "Fire Truck"
Excingussher
Figure 2: Following BEE's steps for the ImageNet- Ik "Fire Truck" class. In Step 1, during training, the classification weights W drift from the initial class concept embedding W outside the scope of relevant concepts, towards spuriously correlated ones. In Step 2, our method filters out class- related concepts and, using an embedding-spac scoring system, ranks and automatically marks the highest-ranking class-neutral concepts as SCs.
in the fine-tuning process can, and often does, imprint the model with new SCs. As shown in Fig. I a CLIP model fine-tuned on ImageNet- k mislabels a "Peafowl" as a " "Fire Truck" " simply because a firefighter is present. This illustrates how fine-tuning can imprint hidden spuriousness in models.
Limitations of existing approaches Research on Spurious Correlations has largely relied two paradigms. Data-driven methods, such as SpLiCE (Bhalla et al.) 2024) and Lg (Zhao et al. 2024). decompose data into high-level concepts and flag those disproportionately associated with certain classes. While useful for dataset auditing, these methods cannot determine which correlations are actually learned by the model, and may overlook low- frequueeuccc but impactful SCs. Error-driven methods, such as B2T (Kim et al. 2024b), infer SCs from validation errors. This require that held- out sets contain counterexamples that reveal the model's shortcuts, an assumption that holds in subpopulation shift benchmarks like Waterbirds (Wah et al., (011b), CelebA (Liu et al.) 2015a), and CivilComments (Borkan et al. 2019). but is rarely fulfilled in more general setups. To this end. Firca et al. (2025) highlight the way dataset splits impact the generalization capabilities of the model. Separating content from environment improves robustness, as spurious cues can dominate learned representations beyond what validation errors reveal Smeu et al. (2022 2025). In contrast, our method identifies SCs learned during fine-tuning even without counterexamples les, and it is meant to complement robusteebss methods (such as GroupDRO (Sagawa et al. 2020)) which require annotations for the dataset's 's SCs. We distinguish previously discussed works from methods for SC- identification without external knowledge (Liu al. 2021a] Pezeshki et al. [2024] Zare & Nguyen 2024), that do not explicitly name the spurious correlations driving decision errors and only partition datasets into easy and hard samples. The utility of their partitions is limited to improving group robustness within subpopulation shift setups alone. We also argue that without explicit identification, mitigation remains opaque, and the underlying vulnerabilities may persist unseen.
Other partial alternatives counteerfactull-magee generation (Singla & Feizi 2021 Zhang al., 2024a) probes fragility but depends on large generative models to correct the biases or costly super- vision. Another approach to detecting spurious correlations is use models that are interpretable by design, such as Concept Bottleneck Models (CBMs) (Koh et al., (202)). These methods require a human expert to define the set of relevant concepts for each task, as well as providing a dataset with concept-level annotations in order to train the concept-extraction layer. To circumvent these limitations, Oikarinen et al. (2023) use concepts proposed by an LLM and obtain pseudo-labels for those concepts using a VLM. This intervention of CBMs on a models's architecture constrains its reasoning space down to the set of predetermined concepts, yielding drops in accuracy when com- pared to the unaltered models. Moreover, those models do not address the issue of SCs also being learned in the concept bottleneck layer, which questions their actual robustness in concept detection. Different from this line of works, never constrain the model in any way, shape or form. What
2

---

Published as a conference paper at ICLR 2026
aim uncover are SCs learned by general ofate-oftthe-ar models used in the industry. which
not explainable by design. Overall both approaches offer a different tradeoff between explainability and expressivity. A more detailed discussion of related work is provided in Appx. D
Our approach Prior work either focuses on mitigation without diagnosis, or on dataset-split level correlations without verifying what the models learn. In contrast, we target spurious correlations that models actually learn, even when no counterexamples exist in the training or validation sets to point out model misbehaviors. We introduce BEE (Bridging Explainability and Embeddings), weight-space framework that tracks how fine-tuning perturbs pretrained representations. As shown in Fig. [ classifier weights drift from their initial class embeddings toward spurious attributes. BEE exploits embedding geometry to rank class-neutral concepts outside the true class scope, exposing hidden drivers of biased decisions.
Our key contributions and findings are as follows:
1. A weight-space diagnostic for spurious correlations. We introduce BEE, the first framework to identify and name spurious correlations directly from the learned weights of classifier. Unlike prior error-based or data-based methods, BEE uncovers correlations even when no counterexamples exist in training or validation splits.
Evidence of persistence and transfer across models. We show that SCs identified by BEE not only artifacts of the studied classifier itself: they persist under full fine-tuning and transfer across diverse state-of-the-arr backbones, reducing the ImageNet tested class accuracy by up 95%.
3. Broad, multimodal applicability. We validate BEE across vision (Waterbirds, CelebA, ImageNet- 1k) and language (CivilComments, MIMIC-CXR notes) tasks, as well as across multi- ple embedding families (CLIP, CLIP- CLIP--ataComp.XX, mGTE, BLIP2, SigLIP2), proving relevance in large scope of generic and realistic setups.
OUR METHOD
For a standard classification task, we aim identify spurious correlations learned by model through training on a new dataset. By dataset's SCs we refer to class-independent concepts, whose presence in the samples greatly affects the class label distribution. By concepts refer to words or expressions with well-defined semantic content. Through SCs learned by classifier fo we refer to concepts causally unrelated with class, whose presence in the input significantly changes the distribution of class probabilities predicted by fe. We further classify them as positively correlated concepts w.r.t. to class k, if their presence the input increases the probability of so predicting the class k, and negatively correlated concepts if they decrease it.
Setup We start with dataset of samples U;) E (X, and construct the set of concepts e; Cau in textual form, that are present the training data. The details for building Call are provided in Sec. 3 Concept Extraction. We use foundation model M, capable of embedding both the input samples 2, and the concepts o; in aligned representations in R.
The main steps of our method (also revealed in Fig. 2 and in more detail in the Alg. 2. the following:
Initialization We train a linear layer on top of the embedding space from M. We initialize Wks the weights of class & in this layer, with the embedding of its corresponding class name, extracted by the model M:
w M (clas. & E K,
(I)
where K is the list of class names.
Step 1: Model training In the process of learning, the weights of each class le in the linear layer naturally shift from their original zero-shot initialization, M (class name;), towards w This drift is influenced by both class features and SCs, illustrated in Fig.
3

---

Published as a conference paper at ICLR 2026
Step 2: SCs identification To identify the learned SCs, we exploit the fact that the weights w/ and concept embeddings (ca) share the same embedding space. To this end, we take the following steps:
a. Filter out class-related concepts After extracting the concepts c; present in the dataset using existing tools, we filter out the concepts that are related to any actual class. This leaves only concepts that are causally unrelated to all classes, which we call class-neutral concepts. We argue that only these concepts are proper candidates for SCs, as they are not required nor useful for the robust recognition of a class. For example, a forest background is well correlated with species of landbirds, but we want to prevent the model from relying on this correlation, which unrelated to the class definition. The exact pipeline and tools for processing the concepts (extraction and filtering) are detailed in Sec. 3
b. Rank neass-n concepts For each class- neusll concept e; and class k. we rank the concepts based on their similarities with the learned class weights wiz using the following positive-SC score:
W M(c) WEI,K|
min w/ Mc.).
Intuitively, we want concepts similar to one class but not to all the others. Thus, for each class k, we select the concepts which starkly correlate to it, compared to all other classes. For the negatively correlated SCs, $K.5 we use the dissimilarity score: = (ca).
Smoothal (aateed)
Refeeence line
Other distances
Maximum distance
c. Dynamic thresholding We next keep only the highest-ranked SCs of each class, using dynamic threshold for the scores above. This allows automatically select the SCs for
Concept Indox tranked tht SC scorc)
each class. To smooth the curve of scores, we apply mean filter with window size r top of the ranked concepts. We denote the scores obtained this step with (Sis)) with p = --r+1 1, where is the total number of filtered concepts. We then select the top my ones, as positive SCs for class k where the my index is defined as:
Figure 3: The maximum distance between the reference line and the smoothed scores gives the threshold for our cut-off heuristic.
3k.1
5k.p
mg = [r/2] + arg max 88.1
8k.4
(3)
The intuition is that my represents the index where the curve of smoothed scores, 8k,i, deviates the most from the straight line connecting the points (1, 8k.1) and (p, 8k.p), as visually shown in Fig.
2.1 ALGORITHM
We present the pseudocode of proposed BEE approach in Alg. I We annotate the main steps presented in Sec. _ At line 1, we initialize the class weights of our linear probing layer with the text embeddings of the class names, the zero-shot classification weights of the foundation model M. We then fine-tune the layer (line 2), on the given dataset. At lines 3-4, we filter the list of concepts and compute the embeddings for the remaining class-ncutral concepts. The filtering can be performed by means of employing WordNet associations, Large Language Models, or both, as described in Sec. With the embeddings of the class-neutre concepts at hand, we proceed to determine the SCs for each class. At line 5, we initialize our set of spuriously correlated concepts with an empty set, and we compute the similarities between each selected concept and each class at line 6. Next, for each class k. we compute the set of scores for all concepts w.r.t. this class and store it in 8k (line 8). For each concept, its ranking score for the current class is the difference between its similarity to the current class and the smallest similarity with a different class. Lines 9- 1 formally implement the dynamic thresholding procedure, described in Sec. 2 Finally, we select the top concepts (above the computed threshold m)) and store them as spuriously correlated concepts for class k (lines 12-13).
4

---

Published as a conference paper at ICLR 2026
Algorithm BEE- Weight space Approach to detecting learned SPuriousness
Input: M foundation model with associated text encoder; (x,3)) Training set; (Keal: Vea!) Validation set; K list of class names; Caul list of all concepts; window for dynamic thresholding. Output: Identified positive SCs: B.
1: W +- (()
2: W < ERM (Wo, M, (X,)), (Kyal, Veal)
1. Model Training 2a. Filter concepts: LLM/WordNet
3: C (-Filter((aal Con)
4: C + M(C)
5: B 4-0 0
6: S < WT
2b. Rank class-ncutral concepts
7: for & 1, K do
8: Sk S min weijk Sky, for all j I, I,
D 2c. Dynamic thresholding
9:
3 = mean
p = C| -
11:
m, [r/2] arg max; [k,
- - 5k.
p-1
12:
by [ j S mpl
13: B (ks bR)
Positive SCs
14: end for
3 EXPERIMENTAL SETUP
Foundation models (FM) We used mGTE (gte-large-en--1.55 (Zhang al. 2024b)) for text embed- dings in CivilComments, and OpenAl CLIP ViT-L/14 for text and images otherwise.
Concept extraction For the image classification task, first use the GIT-Large (Wang al. 2022) captioning model (trained on MSCOCO (Lin et al.] 2014)) to obtain descriptions of the dataset's images. Next, we extract concepts from the captions (or directly from the text samples for the text classification task), using YAKE (Campos et al. 2020) keyword extractor, taking the top 256 n-grams for 3, 5. These keywords make up the set of concepts Call.
Concept filtering We use Llama-3. 1-8B-Instruct (Dubey et al. 2024) to remove class instances from the concepts extracted the previous step. We also apply a post-processing based on Word- Net (Miller) 1995) to catch obvious class instances that the LLM might miss. For each class we specify word used to search for synsets in WordNet (e.g. bird for Waterbirds) and then remove words that match with any of their hyponyms or hypernyms. The complete details are in Appx.
Training We train the linear layer on L2 normalized embeddings extracted by the FM, using Py- Torch's (Paszke et al. (017) AdamW (Loshchilov & Hutter 2019) optimizer with learning rate of 1e - 4. weight decay of Le and batch size 1024. We use the cross entropy loss with balanced class weights the objective. The weights of the layer normalized after each update and CLIP's temperature (r = 100) to scale the logits for each dataset and encoder. We use the validation set's class-balanes accuracy for model selection and early stopping. No explicit upper limit on the number epochs is set. For the GroupDRO experiment in Section 4.5 we use = le 2.
3.1 DATASETS
Waterbirds Sagawa et al. (2020) is a common dataset for generalization and mitigating spurious correlations.
is created from CUB Wah et al 2011a), by grouping species birds into two categories, land- birds and waterbirds, each one being spuriously correlated with the background, land, and water respectively.
CelebA (Lu al. 2015a) is large-scale collection of celebrity images (over 200, 000), widely used in computer vision research. For generalization context, the Litup et al. (2015b) consists of using the Blond Hair attribute the class label and the gender as the spurious feature.
5

---

Published as a conference paper at ICLR 2026
CivilComments (Borkan et al. 2019) is a large collection of 1.8 million online user comments. This dataset is used employed in NLP bias and fairness research concerning different social and ethnical groups.
ImageNet-Ik (Deng et al. 2009) is a larger-scale popular dataset for image classification (1000 classes, with approx. 1300 training samples and 50 validation samples per class).
MIMIC-CXR notes (Johnson et al. 2019), a corpus comprising medical notes from chest exami- nations, for binary classification of the presence absence of the pathological findings.
EVALUATING SPURIOUS CORRELATIONS
For a proper evaluation of our proposed SCs, we combine the concepts identified by BEE with different components.
In Sec. we use SC-enhanced prompts for zero-shot classification with Foundation Model, in some controlled setups, popular within the subpopulation shift literature. In Sec. 4.3 we expand to general, uncontrolled setups like ImageNet- Ik and less explored ones like MIMIC-CXR medical notes. We further generate samples exploiting the discovered SCs here. In Sec. 4.4 we present some qualita- tive analysis the SCs identified by BEE and competitors within the popular setups, underling their fundamental differences. In Sec. 4.5 we explore an extreme scenarios, lacking spurious correlation counterexamples. In Sec. Sec. we fur- ther validate BEE other embedding models. extended list of the extracted concepts can be found in Appx.
Table 1: SC-enhanced zero-shot prompts. Fol- lowing B2T. we inject SCs into zero-shot prompts, leveraging richer descriptions to im- prove classification. SCs identified by BEE sig- nificantly boost worst-group accuracy across im- age and text datasets.
Waterbirds
CelebA (Ace % f)
CivilComm (Ace % f) Worst Avg.
(Ace f) Worst Avg.
Zero-shot
Worst Avg.
Basic
35.2
84.2
72.8
87.7
33.1 80.2
B2T w SpLiCE
48.1
86.1
72.8
88.0
48.1
82.5
67.2
90.2
w Lg
46.1
85.9
50.6
87.2
w BEE
50.3
86.3
73.1
85.7
53.2
71.0
4.1 SPURIOUS-AWARE ZERO-SHOT PROMPTING
To further validate our identified spurious correlations, we follow Kim et al. (2024b) and evaluate them in the context of zero-shot classification task. We augment the initial, class-oriented prompt with the identified concepts through a minimal intervention (e.g. "a photo of {cls) in the {concept)' (see prompting details Appx. D. For each class we create prompt with each identified spurious correlation. When classifying an image, we take into account only the highest similarity among the prompts of class (zero-shot with max-pooling the SCs revealed by our method improve the worst group accuracy over the initial zero-shot baseline and other state-of-the-art solutions, in all the tested datasets. This highlights the relevance of the SCs automatically extracted by BEE. More ablation experiments can be found in Appx.
prompts). We show in Tab. [ how
4.2 SPURIOUS CORRELATIONS IN IMAGENET-IE 1K
Within this subsection, we apply our method in an uncontrolled, general setup. Specifically, we employ BEE to point out spurious correlations plaguing the decision-making process of OpenAl's CLIP VIT-L/14 fine-tuned on ImageNet. Within the ImageNet setup, the current state-of-the-an ap- proach, B2T, points out the SCs learned by the model by analyzing the mistakes the model makes when evaluated on the validation set. Different from B2T, our approach does not rely on the val- idation data to provide counterexamples able to expose the SCs, and is able to provide a list of SCs which exceeds the scope of the validation dataset. We provide extensive lists of SCs pointed out by our method in Appx. H Most the SCs pointed out by our method are previously untapped, opening up avenue for investigating ImageNet SCs.

---

Published as a conference paper at ICLR 2026
Table Results for three positively correlated SCs found using BEE for CLIP ViT-L/14 fine-tuned on ImageNet. We evaluate the model's capability to recognize depicted (correct) class before and after the introduction of an identified concept in the image. For each prompt, 1000 images are generated using FLUX.I-dew We observe throughout all considered scenarios, significant drop in the model's capacity to identify the correct class when the selected concept is involved and large increase in the likelihood of having the induced class predicted even though it is not illustrated.
Correct Class
Exploited SC (Induced Class)
Prompt
Samples Predicted As (%)
Correct Class Induced Class
peafowl
firemen
a photo of peafowl
100.0
0,0
(fire truck)
firemen and a peafowl
5.3 (-94.7)
93.4 (+93.4)
Mexican hairless
reading a newspaper (crossword)
a photo of Mexican hairless dog
47.5
0,0
a man reading newspaper in chair with Mexican hairless dog in his lap
0.9 (-46.6)
36.6 (+36.6)
dog
Bernese
shrimp
photo of Bernese Mountain Dog
99.8
0.0
Mountain
(American
shrimp and pasta near a Bernese
10.6 (-89.2)
37.2 (+37.2)
Dog
lobster)
Mountain Dog
Table 3: Accuracy of various convolutional and transformer- models trained on ImageNet- Ik, on the data generated for Tab. 2 with Fig. [ we note that the performance of these models is significantly affected, even though the correct class is illustrated right in front and center while the predicted class is absent from the generated images. An exhaustive list is presented Appx.
Prompt employed (correct class highlighted in bold and blue, SC in yellow)
Model
photo of a peafowl
firemen and peafowl
a photo of a Bernese Mountain Dog
shrimp and pasta near a Bernese Mountain Dog
alexnet efficientnet bl
100.0
4.6 (-95.4)
96.2
23.3 (-72.9)
100.0
42.6 (-57.4)
88.1
67.1 (-21.0)
regnet x x_32gf
100.0
66.1 (-33.9)
85.9
46.0 (-39.9)
resnet50
100.0
30.1 (-69.9)
73.9
54.5 (-19.4)
Resnex1101_32x40
100.0
66.6 (-33.4)
84.7
61.2 (-23.5)
squeczenetl_1
100.0
13.8 (-86.2)
91.2
46.1 (-45.1)
swin_b
100.0
81.5 (-18.5)
95.2
72.6 (-22.6)
vgg19_bn
100.0
35.9 (-64.1)
83.1
46.2 (-36.9)
vit_1_16
100.0
55.9 (-44.1)
95.3
76.0 (-19.3)
wide_resnet50_2
100.0
60.6 (-39.4)
95.7
63.9 (-31.8
Controlled SC validation via generative models We further invest the effort to generate and manually verify images in order open up this avenue and showcase previously undiscovered flaws in state-of-the-art models. To this end, we employ a quantized version of FLUX.1-der (Labs) 2024), and in order validate the impact of the SCs, we prompt the generative model to depict: (i) a chosen
(correct) class, (ii) the same class alongside a SC (that is not an ImageNet class) that we found to induce other (induced) class, like the prompts shown in Tab.
Quantifying the impact of spurious concepts The validation process is presented for three dis- tinct scenarios in Tab. 2 Each scenario is defined by correct class that is illustrated in the image, concept (object, property, or activity) that is not causally tied any class, and an absent class induced through the presence of the concept. We expect the classifier to predict this absent class, based on our scoring. We measure the impact of the concept by comparing the model's ability to predict the correct class before and after its introduction. We generate and manually ensure the compliance 1000 images for each scenario and we evaluate the model both in terms of accuracy and in terms of the frequency with which it predicts the induced class. Throughout all considered scenarios, we observe a significant drop in the model's capacity to identify the correct class when the SC factor is involved, with increased likelihood of having the induced class predicted, even though it is not illustrated in any way, shape or form in the image. We present smaller- scale tests for alternatives to the generative model FLUX, yielding similar results, in Appx. C
7

---

Published as a conference paper at ICLR 2026
Qualitative SCs examples, extracted on Waterbirds, CelebA and CivilComments datasets. See in orange concepts that are off-topic, person names, or too related to the semantic content of the class, and in blue new concepts, that were not identified before. BEE, w.r.t. others, focus on learned SCs, discovering many new spuriously correlated concepts (and expressions, marked with ...).
Table 4:
Waterbirds
CelebA
CivilComments
landbird
waterbird
blonde hair
non-blonde hair
offensive non-offensive
ocean, beach, surfer, boat,
forest, woods, Iree, branch
man, male
B2T
dock, water, lake
hairstyles, dolly, turban, actress, tennis, beard
hairstyles, visor,
bamboo, perched, rainforest
flying
amy, kate, fielder.
SpLiCE
cuff, rapper, cyclist
forest, woods, rainforest, tree branch, tree forcst.... bamboo....
beach, lake. water, scagull, pond
blonde hair, actress, model, woman long hair
man.... sunglasses, young man,
Lg
black hair, actor
swimming..., water,
hat..., man.... actor, person, dark, large, shirt
hypocrisy,
allowing, work, made, talk
BEE
ground, field, log, grass...
lake, flying.., boat, lifeguard, pond
troll, solly, hate
(ours)
Qualitative failures in ImageNet-1lk Within the same context, we present series of qualitative examples in Fig. [T] We emphasize that, even though throughout most of these samples, a single ImageNet- Ik class is clearly depicted, the model chooses to ignore it and label the image as completely different class, not illustrated at all the image, solely based on the presence of a non- ImageNet object. We underline, by means of the results presented in Tab. 2 that the model is not fooled by artifacts in the generated images to predict randomly. We test the performance of the models images featuring the correct class, without added objects. We observe this way that the model's 's performance on the generated data is on par with the original performance of the model on these classes, validating that the generated images are not out of distribution. Furthermore, we show that the rate at which the induced class is predicted increases significantly.
Generalization across state-of-the-art models. Although the analyzed model is strong, well- pretrained baseline, our experiments show that critical reasoning flaws can persist in production- ready models and remain undetected by standard held-out evaluation. We further evaluate the iden- tified SCs on a broad set of ImageNet-IK state-of-the-art models, with full results in Appx. K and selected results in Tab. [] Notably, even large transformer models such as ViT-L/16 are strongly affected by learned SCs, highlighting their generality and silent impact across modern architectures.
4.3 SPURIOUS CORRELATIONS IN MIMIC-CXR NOTES
To highlight the practical value of our approach and demonstrate BEE's utility in more specialized domains, we employ it on the clinical notes from the MIMIC-CRR dataset, using the "no finding" label as the classification target. BEE revealed concepts such as "chest examination" and "chest radiograph" " as being spuriously correlated with the "no finding" class. We find this to match with patterns in the training data: "finding" samples mention pathologies explicitly, whereas "no finding" samples often reference the examinations as showing no issues.
Adversarial example Given the sentence "The chest examination found signs of disease and the chest radiograph exam found the same". our GaE--bsed classifier incorrectly predicts it as "no finding" a serious error despite the explicit mention of disease.
Quantitative validation Adding the phrase "chest examination" to all samples does not change their label for the considered task, and yet leads to predictions more biased towards the "no finding" class, increasing its recall by 2.1% while decreasing the recall of the "finding" " class by 2.2%.
4.4 QUALITATIVE EXAMPLES
We present in Tab, 4 the concepts identified as spuriously correlated with each class by BEE and competitors. Notice how our method discovers many new concepts (in blue) when compared with
8

---

Published as a conference paper at ICLR 2026
Table 5: Learning in the context of perfect spurious correlations. the absence of samples that associate a class instance and concepts spuriously correlated with other classes, GroupDRO does not outperform the standard ERM. In contrast, our regularization based on the identified concepts consistently yields improvements (concerning worst group accuracy) over the considered baselines: ERM, GroupDRO, and regularization with random causally unrelated concepts (obtained after the filtering in Step2a).
Waterbirds
CelebA (Acc % f)
CivilComments (Acc % t)
(Acc % f)
Method
Worst
Avg.
Worst
Avg.
Worst
Avg.
ERM
43.2.15.7
72.712.2
9.6.11.0
58.2.10.4
18.6110.
49.9
GroupDRO
38.91554
71.2.12.0
8.1.0.3
60.311.0
18.710.4
50.2.10.5
Regularize w/ random SCs
46.6112.7
75.311.1
9.4110.0
61.41220
19.1
50.8
Regularize w/ Lg's s SCs
50.410.1
76.610.0
8.310.0
61.210..5
Regularize w/ BEE's SCs
57.9++03
79.8+0.1
10.4+0.5
62.0+
31.3++0.7
57.5+0.4
others. This is because our approach is fundamentally different, as relies on the decision-making process of the model being investigated, diverging from current techniques oriented validation set errors (B2T), or others that do data analysis over frozen concepts (SpLiCE, Lg). For CelebA-blome hair, B2T and BEE do not find any SCs. This turn out be an appropriate decision, since the presence of the feminine features do not incline the model towards one class or the other. See an exhaustive list of SCs revealed by BEE (ImageNet- 1k included) Appx.
4.5 TRAINING IN A FULLY SPURIOUS SETUP
We consider an extreme setting with no spurious-correlaton counterexamples by removing minority groups from standard robustness datasets (e.g., waterbirds on land). In this regime, GroupDRO- style methods are ineffective, as no underrepresented groups remain, matching ERM performance at best (Tab. 5). Such correlations robustness, we regularize linear probes using the identified SCs by constraining class weights to be equally distant from them, formulated an MSE between each class weight's similarity SC and the average similarity across classes, scaled by CLIP's temperature T.
naturally arise from dataset acquisition choices. To improve
]
1 N
Creg(h) (b)
Wh M(b) - -g
Σ W M(b)
(4)
with sg being the stop gradient operator. The final loss is L. LERM + O ΣvE feeg (b), where B is the set of selected concepts and 0.1. Tab. reports linear probing results in the no-counterexample setting. SC regularization reduces reliance on revealed SC's and improves worst- group accuracy, indicating more robust and better-aligned classification weights. For comparison, we replicate Lg's SC-identification process in this scenario.
4.6 CROSS-MODEL VALIDATION
Table 6: Zero-shot Waterbirds results with dif- ferent embeddings. BEE proves to be robust, improving worst-group accuracy across models, while keeping average accuracy stable.
We further validate BEE on other embedding models. In addition CLIP ViT-L/14, we also used CLIP ViT-L/14 DataCompXL, BLIP2, and SigLIP on the Waterbirds dataset. Using SCs identified by BEE, based on embeddings from each of these models, the zero-shot setting (Tab. 6) led to significant improvement in worst-group accuracy (while the average accu- racy stays in the same range). This highlights BEE's s effectiveness across diverse embedding models.
Zero-shot
Worst Acc. % Avg. Acc. %
Basic CLIP
35.2
84.2
BEE-CLIP CLIP
50.3
86.3
BEE-Date BEE-DataComp
45.5
85.6
BEE-BLIP2
54.8
86.0
BEE-SigL.P2
49.7
85.6

---

Published as a conference paper at ICLR 2026
4.7 HARDWARE AND COMPUTE TIME
We used an NVIDIA RTX 4090 for captioning and image generation and an NVIDIA RTX 2080 for linear probing. For ImageNet-Ik, the preprocessing takes 12 hours, making it the most expensive dataset to preprocess. This is incurred only once per dataset. Afterward, models can be evaluated efficiently, with linear probing taking minutes per encoder.
DISCUSSION: INSIGHTS AND LIMITATIONS
Below we outline several limitations of BEE, along with observations that help contextualize its applicability and scope (more in Appx. E:
Spurious Correlations:
Dataset-Induced or Coundation-Model-nndeeee Our empirical results suggest that the spurious correlations surfaced by BEE arise more from the downstream dataset than from the foundation model, though formally disentangling these factors remains an open challenge. BEE does not aim to separate these bias sources, but to reveal the shortcuts the fine-tuned model actually relies on. Empirical evidence. On datasets with known spurious attributes (e.g., CELEBA, WATERBIRDS), BEE reliably recovered the expected shortcuts, indicating strong datasct-level ef- fects. On IMAGENET, the concepts identified by BEE degraded performance across architectures trained independently of CLIP (e.g., EfficientNet, ResNet, VIT; see Tab, 3). This suggests that these correlations are learned during downstream training rather than inherited from frozen embeddings.
Are Linear Probes Sufficient for Spurious Correlation Analysis? At first, BEE approach ap- pear to be limited to linear or near-linear relationships. However, this reliance on linear probing is grounded in findings showing that such relationships are often sufficient for diagnosing spuriousness in the final representation layer. Prior work (Kirichenko et al. 2023) shows that spurious and core features tend be linearly separable in the final representation layer, supporting our methodologi- cal choice. Additionally, recent findings using Sparse Autoencoders (Huben et al. 2024 Barbalau al., 2025) demonstrate that many meaningful and disentangled features can be isolated within single linear layer. Based on these findings. linear probing offers an effective balance between expressiveness and interpretability for diagnosing spurious correlations.
Broader impact of BEE-SCs (apart from classification) The spurious correlations we detect of- ten reflect co-occurrences inherently present in the dataset. For example, in ImageNet, firefighters frequently appear together with fire trucks. Such co-occurrences are task-agnostic and remain in the data regardless of the specific downstream objective. This means that any model trained on the same dataset, whether for classification, VQA, another task, can implicitly learn these associations. For instance, a VQA model trained on such data might learn to answer yes when asked whether a fire truck is present whenever it sees firefighter, even if none is shown. Therefore, what BEE reveals root causes of shortcut learning that can propagate across tasks built on the same dataset. Although their exact effect depends the task, these patterns represent general dataset-level dependencies that are valuable to detect and understand heyond classification. Although we did not evaluate these spurious correlations on other tasks, their dataset-level nature means they could transfer to models trained on the same data, highlighting promising avenue for future work.
6 CONCLUSIONS
We introduced BEE, a weight-space framework for detecting and naming spurious correlations with- out relying on counterexamples. Our experiments across vision, language, and medicine show that the correlations uncovered by BEE persist across full fine-tuning and the method is generic for diverse foundation models. By exposing hidden shorteuts with interpretable signals, BEE comple- ments existing mitigation methods and provides practical tool for dataset auditing and building more trustworthy systems.
ACKNOWLEDGEMENT
The project was funded by the EU Horizon project ELIAS (No. 101120237).
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine Bias, There's software used across the country to predict future criminals. And it's biased against blacks., 2016.
Md Rifat Arefin, Yan Zhang, Aristide Baratin, Francesco Locatello, Irina Rish, Dianbo Liu, and Kenji Kawaguchi. Unsupervised concept discovery mitigates spurious correlations. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 4 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 1682-1688. PMLR, 21-27 Jul 2024. htttpp:://procedinggg. wwt.m//.wwweeeeeeeeeeeeeee
press vtsess/2335/aaaa
html
Antonio Barbalau, Cristian Daniel Päduraru, Teodor Poncu, Alexandru Tifrea, and Elena Burceanu. Rethinking sparse autoencoders: Select-and for faimess and control from encoder features alone, 2025. URL https: /atps:///axivvvi hwtp://ogga/bbb/aaae.a 10809
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), 2018.
Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio Calmon, and Himabindu Lakkaraju. Inter- preting clip with sparse linear concept embeddings (splice). Advances in Neural Information Processing Systems, 37:84298-843228 2024.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nu- anced Metrics for Measuring Unintended Bias with Real Data for Text Classification. CoRR, abs/1903.04566 2019, URL hRtp::////aaxii org abs/1903 04561
Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6333) 2017. doi: 10. 1226/ science.aal4230. URL https: wtww://wwww/w////tt///twwwww http:::wwwwwwwwwn.ee/eee scwence. orgeeerrg//ee
/2t///icieccc ence.
aa14230
Ricardo Campos, Vitor Mangaravite, Arian Pasquali, Alípio Jorge, Célia Nunes, and Adam Ja- towt. YAKE! Keyword extraction from single documents using multiple local features. In- formation Sciences, 509:257-289, 2020. ISSN 0020-0255. doi: http://dbo..or/100011//////
2019.09.013. URL httpss//www.scinncdiireee.. | wtt.ccoo/cccceceeee http:////eeeeeeeeeea/iie///////a
50020025519308582
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. IEEE Computer Society Conference on Computer Vision and Pattern Recognition CVPR, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.ne,, 2021. Id=YicbFdNTTy.
net. / forum?
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Liviu Nicolae Firca, Antonio Barbalau, Dan Oneata, and Elena Burceanu. Not all splits are equal: Rethinking attribute generalization across unrelated categories. CoRR, abs/2509. .6999,, 2025. doi: 10448550/ARXXVV2250006999 URL https:////doo http:///....gg......55/000////// 2509.
06998
Robert Geirhos, Jörn-Herrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 11:666667733 2020.
11

---

Published as a conference paper at ICLR 2026
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10.1109/CVPR.201..39
http:::::: //t:::/dooi.. org/ 1.....11111010 2016.99
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adver- sarial examples. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15262-15271, 2021.
Andrew Howard, Ruoming Pang, Hartwig Adam, Quoe V. Le, Mark Sandler, Bo Chen, Weijun Wang, Liang- MaingCChihh Chen, Mingxing Tan, Grace Chu, Vijay Vasudevan, and Yukun Zhu. Search- ing for MobileNetV3. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 1314-1324. IEEE, 2019. doi: 10. D0.1199/CCCV.211.00000
URL 11tp://////bb...1//11111///////0
201 001400
Gao Huang. Zhuang Liu, and Kilian Q Weinberger. Densely connected convolutional networks. CoRR, abs/1608.0699) 2016.
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features language models. In ICLR, 2024.
Forrest N. landola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <1mb model size. CoRR, abs/1602.07360, 2016. URL SttEp::/axxiv.ooggabb///SS 077.
Alistair Johnson, Tom Pollard, Seth Berkowitz, Nathaniel Greenbaum, Matthew Lungren, Chih-yynn Deng, Roger Mark, and Steven Horng. Minic-cxr, de-identified publicly available database of chest radiographs with free-text reports. Scientific Data, 2019.
Jeeyung Kim, Ze Wang, and Qiang Qiu. Constructing concept-bane based models to mitigate spurious correlations with minimal human effort. In ECCV, 2024a.
Younghyun Kim, Sangwoo Mo, Minkyu Kim, Kyungmin Lee, Jaeho Lee, and Jinwoo Shin. Dis- covering and Mitigating Visual Biases through Keyword Explanation. In CVPR, 2024b.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training sufficient for robustness to spurious correlations. In ICLR, 2023.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning, pp. 5338- PMLR, 2020.
Alex Krizhevsky.
One weird trick for parallelizing convolutional neural networks. (0ss/1444 2014. aaat::://aada..orr//bs//////
CoRR,
Black
Forest
Labs.
FLUX by Black Forest Labs. https: com/
black foresttoesttt.. labs/ flux 2024. Accessed on: 2024-11-13.
Weixin Liang Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y. Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In Advances in Newral Information Processing Systems 35: Annual Conference on Neural Information Pro- cessing Systems, NeurIPS, 2022.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In ECCV, 2014.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang and Chelsea Finn. Just train twice: Improving group robustness without training group information. In ICML. PMLR, 2021a.
12

---

Published as a conference paper at ICLR 2026
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer. Hierarchical vision transformer using shifted windows. 2021 IEEE/CVF International Conference Computer Vision, ICCV 2021, Montreal, QC, Canada. October 10- 17, 2021, pp. 9992-10002. IEHE, 2021b. doi: 10.1 1411099C/V49922.2000 100966.... URL https:
7/doi. org/1..rr//00 11t.:...1191CC/c1 Tt///CCCV442222 2021 00986
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.
A convnet for the 2020s. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA. June 18-24, 2022. pp. 119966--1976, IEEE, 2022. doi: 10. (109/CVPP52268822222..1111 URL https: / org 101////00//00110//PP.5/.3333.33
2022.
01167
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep Learning Face Attributes in the Wild. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 37330,378, 2015a. doi: 10. 10.1109/122222554424223
Ziwei Liu, Ping Luo, Xiaogang Wang. and Xiaoou Tang. Deep Learning Face Attributes in the Wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015b.
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization, 2019, URL https:
//arxiv.or /bbgg//b///// 11///11111101111
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design. In Vittorio Ferrari, Martial Hebert, Cristian Sminchis-
escu, and Yair Weiss (eds.), Computer Vision ECCV 2018 15th European Conference, Mu-
nich, Germany, September 8-14, 2018, Proceedings, Part XIV, volume 11218 of Lecture Notes in Computer Science, pp. 122-138. Springer, 2018. doi: 10. 10.0000/978-3-000-01266644 https: hopp:://di.. .rgg 10. 1007/978- 3-030 1-30-01264-9-- 9_8
URL
George A. Miller WordNet: a lexical database for English. Commun, ACM, 38(1):3994)9 nov 1995. ISSN 0001-0782. doi: 10.1145/219717:2199445 URL BeLLattpss///d1..or/11..111
219717.219749
Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng. Label-free concept bottle- neck models. arXiv preprint arXiv:2304. 06129, 2023.
OpenAl. Chatgpt (mar 14 version) [large language model].
com/
chat 2023.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS 2017 Workshop on Autodiff, 2017.
Mohammad Pezeshki, Diane Bouchacourt, Mark Ibrahim, Nicolas Ballas, Pascal Vincent, and David Lopez-Paz. Discovering Environments with XRM. In Forty-first International Conference on Machine Learming. 2024.
Joaquin Quiñonero-Candele Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence. Dataset Shift in Machine Learning. The MIT Press, 12 2008. ISBN 9780262255103. doi: 10.755 00755//mitpeess97788222777166 .0t.00000001.0.. 001.0000555000000000.......0.000
URL https: 010/10. 7551/
mi
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross B. Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2020, Seattle. WA. USA June 13-19, 2020, pp. 10425- 10433. Computer Vision Foundation / IEEE, 2020. doi: 10.1109/CVPR44260.2222 01044. URL ottpap/wwee...taaooooooeoeeoeoor htwp:rrre_eeeegrrrpaeeeee__eeeee 201PPR_22200__0 __000_paper__________________2._ html html Radosavevi c__ D__
CVPR 2020
13

---

Published as a conference paper at ICLR 2026
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust
neural networks. In International Conference on Learning Representations, 2020. URL https:
/aa.iiseeatt.ccccoll........ wwwp/oooooooo:.11.1... 21366621111
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. MobileNetV2: Inverted Residuals and Linear Bottlenecks. In 2018 IEEE Conference on Com- puter Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT. USA, June 18-22, 2018, pp-
1510-4520 Computer Vision Foundation / IEEE Computer Society, 2018. doi: 10.1 10.110R/ 2018.00474. URL U
http:tttppp/opeecccceeee Res dua Sasbaalsssss______e___a 201 p_ppppp...ttt.m
thecvf wtt.////mm//ccooccccoo wwwm///ootett... cappeent__v___r________ 8/html/
Sandl er @rMMabir___Miiiiie
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- localiza- tion. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 618-626. IEEE Computer Society, 2017. doi: 10.100//1C220 URL
https::/ddo wwr./orgg..... wt..o///...110///.... Nww.1109/TCV.201..2
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Truck Proceed- ings, 2015. URL http: http//aaavvvv/va www.oorg///.....oo 1 1409. 1556
Sahil Singla and Soheil Feizi. Salient imagenet: How to discover features in deep learning? In ICLR, 2021.
Stefan Smeu, Flena Burceanu, Andrei Liviu Nicolicioiu, and Emanuela Haller. Env-aware anomaly detection: Ignore style changes, stay true to content!, 2022. URL https://arxiv.or
abs/ / 2210. 2210.03103 03103
Stefan Smeu, Elena Burceanu, Emanuela Haller, and Andrei Liviu Nicolicioiu. Robust novelty detection through style-conscious feature ranking. In WACV, 2025.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp. 1-9. IEEE Computer Society, 2015. doi: 10. 10.11099CVPP.220559998544 URL
http://ooooorr11111111000 /VP.1////VVP//C CVPR. 2015. 7298594
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In 2016 IEEE Conference on Com- puter Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. pp. 2818-2826. IEEE Computer Society, 2016. doi: 10.1109/CVP.2206..300 URL https: 7/doi org 10. 1 1109/CVPR. 2016 2016.0006000 308
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Leurning, ICML 2019, 9-15 June 2019, Long Beach, Cali- fornia, USA, volume 97 of Proceedings of Machine Learning Research, pp. 6105-6114. PMLR, 2019. httL:tttpp/prrceediiie
mlr wwwpsppssssee.pp. / atnnnann 9tml
Mingxing Tan and Quoe V. Le. Efficientnet22: Smaller models and faster training. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 10096-10106. PMLR, 2021. URL httt:://procedings.nto ptess//v39/tan2la.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le. Mnasnet: Platform-aware neural architecture search for mobile. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 2820- 2828. Computer Vision Foundation IEEE, 2019. doi: 10.1109/CVPR..019.0000 Wt0.P...01900000 CVPR_220 2019/htm1/aan MnasNet_ http:/aaaaaaaaaaeeeaaaaaaaeooeo Plare Neuraa Architccture
URL http::/ wtw.eeeaaaccsss..........ee content
Gearchh_oorMobila CVPR_ 2019__aperr_____ppppppppp____ html
14

---

Published as a conference paper at ICLR 2026
Zhengzhong Tu, Hossein Talebi, Han Zhang. Feng Yang. Peyman Milanfar, Alan C. Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022: 17th European Conference, Tel Avix, Israel, October 23-27, 2022, Proceedings, Part XXIV, volume 13684 of Lecture Notes in Computer Science, pp. 1007 978 031 20053-
Springer, 2022. doi: 10.1007/ URL ottps:://oo..org/0... wtt.:o/ggg........
978-3-031-20053-3
/tt:://doi..oo
27
C. Wah, Branson, Welinder, Perona, and S. Belongie. Caltech-ucsd birds 200. Technical Report
California Institute of Technology, 20
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge J. Belongic. Dataset. hatpss//app..emaaiicccttt
The Maltech-UCSD 2irds-200-2011
orc:/CorrusTD:1611119 9123 201 1b. URL https: /ppi www.steeaatssciiieeeeeee www.ssmactccccccco.r...orr or
CorpusID:16119123
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. GIT: A Generative Image-to-text Transformer for Vision and Language. arXiv preprint arXiv:2205. 14100, 2022.
Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated resid- ual transformations for deep neural networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26. 2017, pp. 5987-5995. IEEE Computer Society, 2017. doi: 10.1109/CVPR.2017... 7.634. ht.tt::://// htt.::spp:/ooooooo////11
1169/CVPR. 2017. 634
Mert Yüksekgönüül, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. In ICLR, 2023.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Richard C. Wilson, Edwin R. Hancock, and William A. P. Smith (eds.), Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK, September 19-22, 2016. BMVA Press, 2016. URL https:// bmva aataa-rrchive. org. ht//://ww//2/1// http://///.//aaaes///aaaee////// /wt17///n/dxx..ee htmi
Samira Zare and Hien Van Nguyen. Frustratingly Easy Enviromment Discovery for Invariant Learn-
ing. Computer Sciences &amp; Mathematics Forum, 9(1), 2024. ISSN 2813-0324. doi: N0.3390/cms22022000902. URL https://///..........tt /www.
com/ 281 28131336 0324/9/1/2
Jianrui Zhang, Mu Cai, Tengyang Xie, and Yong Jae Lee. Countercurate: Enhancing physical and semantic visio-linguistic compositional reasoning via counterfactual examples. In ACL, 2024a.
Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang. Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, and Min Zhang. mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval. CoRR, abs/2407.19666, 2024b. doi: 1044850/ARXIV.2400.000 48550/arXiv. 2407. 1407.1966999 19669
htts:t::: //oii orq/10.
Yuhui Zhang, Jeff Z. HaoChen, Shih-Cheng Huang, Kuan-Chich Wang, James Zou, and Serena Yeung. Diagnosing and rectifying vision models using language. In The Eleventh International Conference on Learning Representations, ICLR, 2023a.
Yuhui Zhang, Jeff Z HaoChen, Shih-Cheng Huang, Kuan-Chies Wang, James Zou, and Serena Ye- ung. Diagnosing and rectifying vision models using language. arXiv preprint arXiv:2302.04266) 2023b.
Zaiying Zhao, Soichiro Kumano, and Toshihiko Yamasaki. Language-quided Detection and Mitiga- tion of Unknown Dataset Bias, 2024. URL
0288/ 02889
15

---

The datasets and Published as a conference paper at ICLR 2026
APPENDIX
A BROADER IMPACT STATEMENT
By systematically detecting a wide spectrum of spuriously correlated concepts. our work stands to enhance the reliability and trustworthiness of AI-driven decisions across various real-world contexts. BEE could help researchers and developers to address unintended consequences that arise when models latch onto misleading data associations, drawing attention to critical responsibilities tied to deploying Al at scale.
B SOFTWARE AND DATA
We make our BEE code in PyTorch Paszke et al. (2017) publicly available HERE pre-trained models used for BEE are already public.
C ALTERNATIVES TO THE GENERATIVE MODEL
ChatGPT-40 generator. We performed small-scale experiment with an alternative generative model, other than ILUX.I-dev from the main paper. Forty images were generated with ChatGPT- 4o (OpenAl 2023): half depicted only peafowl, and half depicted both a peafowl and firefighter, but no firetruck. The linear probe trained on top of CLIP achieved 100% accuracy on the first set, yet its accuracy dropped to 50% once the SC was introduced. This decline confirms that the evaluation results are not intrinsically dependent on the FL.UX generative model. Furthermore, in the cases where the model failed, it always because it predicted the fire truck class, though no fire truck was present the image, while the peafowl was realistically depicted and occupied a large portion of the image. The fact that the model achieves 100% accuracy when the SC is not involved, and the fact that it proceeds to misclassify the input image into the specific class suggested by the SC, suggest that these errors not due to artefacts in the generative models, but rather due actual flaws of the investigated model.
Natural images. If we avoid synthetic data altogether, scaling the evaluation becomes imprac- tical: each image must be located, licensed, and annotated by hand. To probe the phenomenon nevertheless, we performed focused Google News search and selected naturally occurring photos for the same SC. Our investigation of real images confirmed the same statistics presented for the ChatGPT-40 generator.
D DETAILED RELATED WORK
Machine learning methods naturally capture relevant factors needed to solve a task. However, mod- els might also capture shortcuts (Geirhos et al. 2020), correlations between non-essential features of the inputs and the label. These shortcuts represent spurious correlations, that don' hold in a more general setup (e.g. using water features classify waterbirds instead of focusing the birds' fea- tures), and should not be used for reliable generalization outside the training distribution, as they often lead degraded performance (Quiñonero-Camm et al. 2021).
al. 2008 Beery al. 2018 Hendrycks
SCs from error analysis Approaches like B2T Kim et al. (2024b) rely exclusively on the valida- tion samples, identifying which correlations between concepts are more prevalent the misclassi- fied examples. To catch SCs, B2T needs samples that oppose the strong correlations in the training set, thus leading to misclassification. To circumvent this limitation, DrML Zhhag et al. (003b)) manually builds a list of texts containing classes and concepts associations, that could potentially underline an SC. forwards each such textual association through a classifier, learned on the im- age modality, and keeps SCs the erroneously classified ones. In those approaches, the burden falls on the practitioners to come up with exhaustive samples or associations, failing to detect unex- pected SCs. Differently, BEE focuses analyzing the explicit learned weights of model, covering

---

Published as a conference paper at ICLR 2026
all trainset samples. We thus extract the spuriously correlated concepts directly from the weights, bypassing the need for an exhaustive validation set or correlations candidates.
SCs from train data analysis In SpLiCE (Bhalla et al. 2024 each image is decomposed into high-level textual concepts, searching next for concepts that frequent for certain class, but not for the others. LG Zhao et al. (2024) relies on LLMs to propose concepts potentially correlated with each class, using image captions. Next, it uses CLIP (Radford et al.) (02T) estimate class-specificity score for each concept, and highly scored concepts for a class w.r.t. the others are considered SCs. These methods focus on the concepts' occurrences per class, making them prone to missing low-frequency concepts, as their presence can be drowned when averaging scores over a large dataset. Moreover, the SCs found through data analysis could be harder to learn than the class itself, so they are not necessarily imprinted in the model. In contrast, learned SCs (including error analysis revealed ones) must always be addressed, as they are, by definition, proven impact the classifier. For this reason, BEE targets learned SCs by looking directly at the impact of the training set upon the model's weights.
Manual interpretation of correlations The method introduced in Singla & Feizi (202)) finds spurious features learned by model, but it requires humans to manually annotate whether an image region is causal or not for a class. While this ensures higher quality of the annotations, it also poses problems of scalability to large datasets. In contrast, BEE works fully automated, at scale, identifying SCs for each class in ImageNet-Ik.
SCs from subpopulation shift setup Other previous works (Pezeshki et al. 2024; Liu et al, a Arefin et al. 2024 Zare & Nguyen 2024) have focused on SC identification strietly within
the context of subpopulation shifts. The particularity of this setup is that the training and validation sets always contain subsets of samples that oppose the strong spurious correlations of the dataset. Most of these methods (Pezeshki et al.] 2024) Liu et al.] 2021 Zare & Nguyen| 2024) focus on first learning strongly biased classifier and then either separate the samples of each class into two groups (Pezeshki et al. 20244 Zare & Nguyen| (024) (one containing correctly classified examples and the other containing misclassified ones), or place higher weights on hard samples (Liu et al, 2021a), in order to balance the dataset. CoBalT Aretin et al. (2024) on the other hand uses an unsupervised method for object recognition and then samples the dataset examples such that all object types are uniformly distributed in each class. Their result contains heatmaps overlays on images, which can offer insights to guide further manual SCs identification. Some of the most commonly used datasets in this setup are Waterbirds (Sagawa et al. 2020), CelebA (Liu et al, 2015a) and CivilComments (Borkan et al. 2019).
Preventing the learning of SCs As the statistical correlation of attributes and classes lies at the root of learning SCs, breaking this correlation is an accessible way of preventing their learning. Assuming that the training set features all combination of classes and attributes, this can be achieved by balancing all the existing groups of samples, defined by the intersection of class and attribute labels. GroupDRO (Sagawa et al.) 2020) uses group-specific weights that are dynamically updated during training to balance them, and it is the approach most commonly taken by works that simply find dataset partitions (Pezeshki er al.) 2024 Zare & Nguyen 2024), and also by works that name the SCs, and then obtain pseudo-labels for those attributes (Kim al. 2024b Zhao et al. 2024). To judge the robustness of classifier, the worst-group accuracy metric is employed, which computes the accuracy on each individual group of samples and then reports their minimum. The worst- group accuracy of GroupDRO with ground truth attribute labels is usually viewed by the previously mentioned works as an upper bound on the performance that can be obtained.
Fairness It is important to note that the proposed method can be utilized to evaluate the fairness of a given dataset and that we do conduct benchmarking on the CivilComments dataset, which encompasses racial and religious concerns. However, it is critical to emphasize that our approach is neither designed to measure nor address issues of fairness. Instead, our method is specifically developed to examine whether a given dataset imparts a clear definition of the featured classes to model, namely, whether classifiers lear spurious correlations and confound class features with environmental features. Accordingly, our work is situated within the literature on subpopulation shift setups and assess the quality of our proposed approach within this framework. Evaluating
2

---

Published as a conference paper at ICLR 2026
approach on fairness benchmarks lies outside the scope of the current study, but may constitute a subject for subsequent research.
D.1 CONCEPT BOTTLENECK MODELS
Another approach to detecting spurious correlations would be use models that are interpretable by design, such as Concept Bottleneck Models (CBMs) (Koh et al. 2020). CBMs feature a special layer where each neuron's ' activations signals the presence or absence of specific concept within the input sample. This makes it easier to see which concepts are used by the model down the line and also allows a user to filter out the concepts that he may consider as irrelevant for the task at hand. On the downside, CBMs, proposed by Koh et al. (2020), require a human expert to define the set relevant concepts for each task and also concept-level annotations in a dataset in order to train the concept extraction layer. To circumvent these limitation, Oikarinen et al. (2023) use concepts proposed by GPT-3 and then obtain pseudo-labels for those concepts using a CLIP model. This intervention of CBMs set of predetermined concepts. yielding, compared to unaltered models, drops in accuracy of up to 4.97%, as reported by Oikarinen et al. (003) in Table Different from this line of works, we never constrain the model in any way. shape or form. What we aim to uncover are SCs learned by general state-of -otteeoff-h--trrr-tt thetherrt models used the industry, which are not explainable by design. Overall both approaches offer a different tradeoff between explainability and expressivity.
models's architecture constrains its reasoning space down to the
Oikarinen et al. (2023) Similar to our approach, the method proposed by Oikarinen al. (2023) can be applied in general setups, but with some caveats. They train concept layer, within which each neuron signals the presence or absence of a predetermined concept proposed by GPT-3, This intervention constrains the reasoning space of the model down to the set of predetermined concepts, yielding, compared to unaltered models, drops accuracy of up to 4.97%, as reported Table from Oikarinen al. (2023). Different from them, never constrain the model in any way, shape or form. Our analysis is completely post-hoc, allowing the model to reason in its natural embedding space while achieving the best performance it can on the given dataset.
Yüksekkönüül et al. (2023) Similar to our method, the approach proposed by Yükkekgönül et al. (2023) is post-hoe. That is, the authors train a CBM layer top of a pre-trained non-CBM model. Furthermore, their approach features a mechanism designed to reduce the performance loss usually seen in CBM models. In this respect their work differs significantly from prior CBM literature. Our work differs from this approach in two main respects pointed out directly by Yüksekgönül et al. (2023) in their Limitations and Conclusion section of their paper: Users should be careful about the concept dataset used to learn concepts, which can reflect various biases. While there are several such real-world tasks, is an open question if ftmannccosstruteel concept bottlenecks can solve larger-scale tasks(e.g. ImageNet level). First, in this direct quote, the authors point out that their method cannot find out the spurious correlations learned by their bottleneck layer during training. In contrast, our method is specifically designed to point out the spurious correlations learned by classifier during training. Second, while in their work, the possibility of applying their method to large-scale datasets such as ImageNet remained an open question, we have successfully applied our method on the ImageNet dataset.
Kim et al. (2024a) This work further improves upon the method proposed by Yüksekgönül et al. (2033) by means of introducing a mechanism meant to automatically obtain unbiased MLLM at- tribute annotations in order train the bottleneck layer without human supervision. In terms of differences, even if the MLLM annotations were to match the human level, the limitations pointed out by/ūūksekgööüi et al. (002) still remain. That is, the spurious correlations learned by the bottle- neck layer itself during training remain hidden. The annotation procedure ensures that the labels that are used are as accurate as possible, but it cannot determine what spurious correlations are present in the training dataset used for the bottleneck layer. In contrast, our method investigates the spurious correlations learned by our classifier during training.
E DISCUSSION: INSIGHTS AND LIMITATIONS (CONTINUATION)
BEE's reliance on external models (concept extraction, LLM and WordNet filtering) These components are used only for candidate generation and filtering, so their impact the core di-
3

---

Published as a conference paper at ICLR 2026
agnostic signal is limited. Potential biases could cause either excessively- Over-filterngg might remove valid spurious cues, yielding too few or trivial SCs, but this was not ob- served, BEE consistently surfaced novel and meaningful shortcuts (see ImageNet results). Under- filtering might admit class-related concepts and incorrectly flag them as spurious; however, our experiments did not show this effect at seale. Incorporating BEE-identified SCs consistently duced model robustness (Tab. 3 confirming their spurious nature. Overall, these heuristic steps support scalability, while the core diagnostic signal comes from the geometric alignment between class weights and concept embeddings, which is independent of auxiliary model biases.
cuufficieentfilterinn
Concepts Input features as SCs The learned SCs can be described by our method in relation with the predefined (large set of) concepts, but not directly w.r.t. the input features (e.g. Grad- CAM (Selvaraju al. 2017) like methods).
Trade-off between accuracy and fairness Removing SC reliance often reduces average accuracy, since SCs are predictive within biased datasets. This drop is widely recognized the robustness literature and we report both average and worst-group metrics to transparently reflect this trade-off.
Generalizability While our mitigation experiments focus on binary classification benchmarks, BEE detection itself is agnostic to task cardinality, evidenced by the ImageNet experiments.
Captioning model used for extracting concepts These models usually do not extract all the details in the images, so relying on them limits the concept space, that limits further discovering all SCs from the original images.
SCs from dataset (only) through the lens of Foundation Model While the Foundation Models are usually very robust ones, some SCs (specially those related to low-level- pixel-level infor- mation) can disappear in the siga-semantic embedding space of the foundation model, making impossible for BEE to detect such SCs.
Relying known hierarchies of concepts The method also relies known hierarchies of con- cepts (like WordNet) to filter out concepts related to the desired class. These hierarchies and the relations they provide thus limit the type of filtering that ensure.
F CONCEPT FILTERING RULES
In the filtering stage process list of n-grams extracted from the set of image captions or from the set of samples in the case of text dataset, and output a list of strings that not contain concepts related to the classes.
F.1 LLM-BASED FILTERING
We provide below the prompt used for filtering the list of concepts.
4

---

Published as a conference paper at ICLR 2026
Prompt used for concept filtering with LLMs
I will provide list of concepts and sequence of words. Your task is to remove any instance of the concepts from the given sequence. If no instance of any concept is present then you must return the sequence as is.
Here are a few examples:
Example I:
Concepts: [dogs and any specific species of dogs] Sequence: 'a golden retriever with bone
Answer: 'bone"
Example 2: Concepts: [clothing and anything related to their color] Sequence: 'a shiny black and white dress' Answer: 'shiny"
Example 3:
Concepts: [mentions of peopleš names]
Sequence: 'John is an assistant' Answer: 'assistant'
Example 4:
Concepts: [cats, horses, dolls, the sun and any specific species or types of these concepts]
Sequence: "A picture of the rising sun"
Answer: 'picture'
Now complete the following case, without thinking step by step or asking for anything else. Concepts: [(]]
Sequence: 'O'
Answer:
We next describe the rules used to process the response of the LLM:
We first locate the two apostrophes that are expected to be in the answer and keep only part of the answer that lies between them. A null string results if two apostrophes are not present the answer.
We ignore an n-gram if the answer of the LLM is null string if it contains words in the initial sequence. This can signal a hallucination or failure to adhere to the required output format. We used the word tokenizer from nltk in this step.
For n-grams where the LLM provides valid answer, stopwords. We use the set of english stopwords in nltk, as well as the words "next" and "many", which we found to be often produced by the chosen captioning model.
trim the starting and trailing
If the answer of the LLM only contained stopwords we discard it. Otherwise it added to the list of candidates.
F.2 WORDNET-BASED FILTERING
For this approach we use the folowing rules:
We first tokenize each n-gram into individual words with the word tokenizer in nltk.
We then lemmatize each word and mark it as class-related if its lemma appears in the class name (e.g.. "blonde" appears in "blonde hair" for CelebA) or is a hypernym or hyponym of the corresponding WordNet synsets for any class (e.g.. "pelican" is hyponym "bird"). Note that the user could define one or more WordNet synsets for a class is union of multiple synsets in WordNet, and fitting hypernym does not exist. E.g., class 0 in a dataset could represent venomous animals, which includes certain snake species, as well spiders, stingrays etc..
5

---

Published as a conference paper at ICLR 2026
Each n-gram with class-related words is added in the final list of candidates
For the other n-grams we remove all class-related words. If only stopwords remain, we discard them. Otherwise, the resulting string is added to the list of candidates.
At the end of the filtering stage we deduplicate the obtained list of candidates. To be noted, we can have a word with both its singular and plural form in the list of candidates. The lemmatization in WordNet-
filtering only used determine if the word is related to the classes or not, but raw words are used in subsequent steps. This is an intended behavior, because if only the plural form ever appears then we could gain additional information about the dataset- the entity always appears in group.
G LOSS CORRELATION WITH PRESENCE OF SPURIOUSLY CORRELATED CONCEPTS
In this experiment, we look at the correlations between the loss values and the concept-to-sampll similarities. We compare basic ERM with GroupDRO, applied on groups, that are obtained based on our revealed SCs (and further grouped using the B2T Kim et al. 2024b) partitioning strategy).
See in Fig. 4 how for GroupDRO, the loss-to-similarity correlation significantly decrease, revealing that the model is less prone to make mistakes on the samples containing SCs. The results show a reduction in correlation scores across all SCs, demonstrating that the revealed groups are relevant to the dataset's s underlying distribution, and can be effectively utilized with specific algorithms to mitigate the model's dependence on spurious correlations. Fig. 4 shows the Pearson correlation scores after epoch of training on Waterbirds, on a subset of all SCs.
H SCS AND TOP CLASS-NEUTRAL CONCEPTS
We present the exhaustive list of SCs found for the Waterbirds (Tab. 15 & T6 CelebA (Tab. 17 and CivilComments (Tab. 18 & 19) datasets. We also present top neutral concepts for ImageNet performed using WordNet relationships alone, without
classes. Concept filtering the intervention of a Large I Language Model. Accounting for the size of the dataset, we will publish the available data on repository upon acceptance, and will restrict the presentation within the context of the current format to a few classes for illustrative purposes in Tab, 28 21
ImageNet
Number of SC candidates After passing through both filtering stages (LLM-based and WordNet- based), we were left with the following number of candidates per dataset:
Waterbirds: 294
CelebA: 280
Comdationn (UIIM loss, simierly SC1
Conne anker (Grouptiio uos, simiurity to se))
0.2
0.0
Conturiens
Sene por
foree ww tem
borotte soaat roor
tasss of bantoos lovel
snowy toras
avea wwdaaaa
JMA DMAU
aNy CND RMAU
1000
Leetu Sogl
Figure 4: Correlation sample. loss, sample_to. bias similarity) under ERM/GDRO after one epoch of training on Waterbirds. Loss correlation w/ biases, ERM vs GroupDRO using groups created with the B2T partitioning method. It can be seen that, when training with ERM. loss value is highly correlated with the biases. In contrast, GroupDRO reduces the correlations, intuitively showing that biases discovered with our method are closely related to the ground truth groups of the dataset, being used as shortcuts by the model unless mitigated.

---

Published as a conference paper at ICLR 2026
Table Class names and prompts used in the zero-shot classification task.
Waterbirds
CelebA
CivilComments
class names
waterbird landbird
non-blonde hair blonde hair
non-offensive offensive
zero-shot prompt SC-enhanced prompt
a photo of {cls)
photo of person with {cls] photo of (SC)
{cls}
photo {els}
a/an [cls} comment about {SC}
in the {SC}
with fels)
Table 8: Ablation. Following the zero-shot SC-augmented prompting setup, we variate the cut-off threshold for considering causally-unrelatee concepts spuriously correlated and also try address the text-image modality gap.
Waterbirds
CelebA (Ace % 1)
CivilComm (Ace % f) Worst Avg.
(Ace % f)
Variations
Worst Avg.
Worst Avg.
top-30 candidates
46.3
86.1
66.2
86.4
46.9 71.1 66.2
top-20% candidates
46.1
86.0
64.8
86.7
48.8
modality gap: closed
48.7
85.9
72.7
85.3
BEE
* dynamic threshold
modality gap: open 50.3
86.3
73.1
85.7 53.2 71.0
CivilComments: 587
ImageNet: 368.170
I ZERO-SHOT PROMPTS
In Tab. we structured the class names used for initializing the initial weights of the linear layer, along with the prompt templates employed in the zero-shot classification experiments discussed in Sec. 4.1
ABLATIONS
J.1 THRESHOLDING STRATEGIES
We validate several BEE decisions in Tab. for zero-shot classification task, using prompts en- hanced with SCs. The number SCs per class turns be very important, taking too many adds noise to the prompts and lowers the performance. Nevertheless, dynamically choosing the threshold, as described Sec. Step2c.. proves to good strategy adapting the cut-off across classes. Following prior observations regarding the modalities gap between text and image embedding space (Liang et al. 2022 Zhang et al.] (203a),, we subtract half of the gap from the embeddings and re-normalize them, ending marginally lower performance w.r.t. not addressing the gap.
J.2 IMPACT OF THE LLM CHOSEN FOR FILTERING
To check the importance of the LLM used in the filtering stage have also employed GPT-3.5 Turbo for the filtering of concepts from the Waterbirds dataset. We provide a qualitative comparison of the responses from -lama-3.1-8B-Inae parison we repeat the zero-shot experiment presented in Table with the candidates obtained from GPT-3.5 Turbo. The results presented in Tab. 10
and GPT-3.5 Turbo in Tab. 9 For quantitative com-

---

Published as a conference paper at ICLR 2026
Table Ablation. Qualitative comparison of the filtering done by Llama-3.1-B--nstruul and GPT- 3.5 Turbo.
Original n-gram
Llama-3, output
GPT-3.5 Turbo output
bird flying over a body black bird sitting on top yellow and black bird standing bird in bamboo forest bird sitting on boat flying over the ocean black bird sitting
flying over a body sitting top standing
flying over a body sitting top yellow and black standing in bamboo forest sitting on a boat sitting
bamboo forest boat
flying over the ocean
sitting
Table 10: Ablation. Quantitative comparison of filtering done by Llama-3.1 and GPT-3.5. We use the candidates obtained in both cases for the SC-enhanced zero-shot classification experiment from Tab.
Waterbirds (Acc %
Zero-shot
Worst
Avg.
BEE w/ Llama-3.1 Filtering (in paper)
50.3
86.3
BEE w/ GPT-3.5 Filtering
49.5
84.5
J.3 ROLE OF NEGATIVE SCs
Negative SCs of a class hint at the conditions in which said class is less likely to be correctly iden- tified. As such, setting where generating and labeling new data points is feasible, the negative SCs indicate what concepts should be depicted near this class in order mend the SCs learned from the original dataset.
Binary Classification Tasks (CelebA, Waterbirds, CivilComments) In binary settings, positive SC for one class implicitly serves as a negative SC for the other. A positive SC of class 0 favours the prediction of class 0, while implicitly hindering the prediction of class 1. Positive and negative SCs in task are thus entangled they share the same underlying pool of concepts. This is also reflected in the symmetry between the positive-SC score of class and the negative-SC score for class for +
the binary classification setting 00 = $1,5- Proof:
Let a; w/ (cx) and by w T M(e)). We have that S0, a; min(ai, by) and 87.5
-b; min( 0,, -b.)
Using the fact that
min(a,, by) = a, + b, max(a;, by) = as + b, + min( - -a;, -b.)
we obtain the equality:
0, 0; min(a; b.) 0g b min( ai, -b:) = -b; min( as. -b.) 87,
The SC-enhanced zero-shot classification application from B2T uses the positive SCs for both classes to create more expressive prompts. That is, for class we use both its positive and nega- tive SCs. We repeated the experiment, using only the positive only the negative SCs identified by BEE on Waterbirds, leading to significant drops in performance (Tab. II
Intuitively, using only the positive SCs for class amplifies the SCs (we using prompts of wa- terbirds water environments and landbirds in land environments), while using only the negative ones reverses the SCs.
8

---

Published as a conference paper at ICLR 2026
Table 11: Ablation. the zero-shot SC-augmented prompting setup we use only the positive or only the negative SCs identified for each class.
Zero-Shot Prompting
Worst Acc %
Avg. Acc %
with All SCs (in paper)
50.3
86.3
only with Negative SCs
34.7
60.9
only with Positive SCs
16.7
67.9
K RESULTS OF STATE-OF-THE-ANE MODELS
We provide results using the same data and experimental setup used for Tables and for an exhaustive list of ImageNet classifiers, in Tables 12 13 and 14 Pre-trained models together with their respective weight sets are employed from the torchvision package.
Table 12: Accuracy of various convolutional and transformer-
models trained on ImageNet- Ik, the data generated for Tab. 2 As with Fig. 2 and Fig- [ we note that the performance of these models significantly affected, even though the correct class is clearly illustrated right front and center while, and the predicted class is absent from the generated images.
Prompt employed (correct class highlighted in bold, SC in red)
Model Weights
a photo of firemen and peafowl
photo of
shrimp and pasta near
peafowl
Bernese Mountain Dog
Bernese Mountain Dog
alexnet V1 Krizhevsky 2014
100.0
4.6
96.2
23.3
convnext_tiny V1 Liu ct al. (2022
100.0
77.2
94.2
69.8
convnext_smale VT
Lau et
2002
100.0
92.9
96.3
78.3
convnext base V1
al. 2022
100.0
83.5
99.3
78.0
convnext large
2022
100.0
88.2
99.6
81.9
densenet121 V1
Huang
2016
100.0
52.3
93.1
74.8
densenet161 V1
Huang el al.
2016
100.0
52.8
84.0
51.6
cfficientnet_b0- Huang et a
densenet201 V1
2016
100.0
49.7
86.1
80.8
& L.
2019)
100.0
64.5
99,2
92.9
cfficientnet.b V1
2019
100.0
42.6
88.1
67.1
efficientnet.b b1 V2
Tan & Lc
2019
100.0
84.2
99.9
69.5
efficientnet b2 V1
Tan & Le
2019
100.0
61.7
99.6
82.4
efficientnet b3 V1
Tan & Le
2019)
100.0
89.1
99.1
92.8
efficientnet b4 V1
&
2019
100.0
94.4
99.9
72.7
efficientnet.bb Tan
2019
100.0
82.9
99.4
86.3
cfficienttnet.6-
Tan & Le
2019
100.0
91.2
100.0
95.6
efficientnet.b7 V1
lan & Le
2019
100.0
88.3
99.9
90.3
efficientnet_v22s
VT
lan
(e22
100.0
98.3
99.2
88.1
efficientnet_v2_mm V
Tan Le (202)
100.0
95.1
99.8
94.3
efficientnet v2
Tan
2021
100.0
96.0
99.1
83.6
googlenet V1 Szegedy el al.
2013
100.0
45.4
95.2
$7.0
inception_v3. VI Szegedy
20166)
100.0
82.2
98.9
80.8
maxvit_1 V1 Tu
(2022)
100.0
91.7
99.7
85.3
mmasnet0_5- VT et al. (2019)
100.0
23.3
96.7
60.8
mmasnet0_755
lan et al
2019
100.0
36.9
98.4
71.1
mnasnet1 0 V1
Tan ct al
2019
100.0
32.0
89.4
74,9
mnasnetl_33 - V1
Tan al.
2019
100.0
63.9
91.7
75.6
mobilenet.ve v2 VI
Sandler
2018
100.0
26.2
84.0
45.0
mobilenet v2
Sandler
2018
100.0
54.1
98.6
62.8
mobilenet.v3_smaa
mobilenet.v3.nmm (2019)
V1 Howard
100.0
16.6
91.0
30.3
V1 Howard al. (2019)
100.0
25.4
94.2
23.1
mobilenet.v3.1am large
V2 Howard (0099
100.0
56.0
96.6
73.7

---

Published as a conference paper at ICLR 2026
Table 13:
Accuracy of various convolutional and transformer-bat models trained on ImageNet- Ik, on the data generated for Tab, [ As with Fig. [ we note that the performance of these models is significantly affected, even though the correct class is clearly illustrated right in front and center while, and the predicted class absent from the generated images.
Prompt employed (correct class highlighted in bold, SC red)
Model Weights
photo of firemen and peafowl
photo of
shrimp and pasta
peafowl
Bernese Mountain Dog
Bernese Mountain Dog
regnet_y_4000
Radosavovic et al.
2020
100.0
35.6
74.7
34.7
regnet_y_400me V2
Radosavovic et al
2020
100.0
72.1
97.7
87.9
y 800mf 8000m V1
Radosavovic et al.
2020
100.0
22.7
94.8
54.4
regnet 800mf V2
Kadosavovic et
2020
100.0
81.7
98.9
88.0
regnet.y_I-6gf
Radosavovic
2020
100.0
47.0
71.7
regnety-_I6ggT V2
Kadosavovic et al.
2020
100.0
88.2
96.4
74.1
regnet_y_3_2gf V1
Kadosavovic ct al.
20200
100.0
33.5
99.4
90.9
regnet.y_3_2gf V2
Radosavovic
20200
100.0
94.6
98.4
75.4
regnet.y 8gf V1
Radosavovic
2020
100.0
58.3
71.3
54.8
regnel ygf_y-_8g---
Radosavovic
100.0
98.2
67.5
regnet_y_16gi V1
Radosavovic et al.
2020
100.0
86.7
97.8
49.8
regnet.y-16ff V2
Radosavovic ei al.
2020
100.0
98.7
91.8
83.5
regnet.y-16ff-
SWAG.E2E.V
Radosavovic et al. (2020)
100.0
99.6
97.7
78.2
regnetyy_16gff
SWAG.LINEAR,
Radosavovic al. (2020)
100.0
78.3
100.0
90.7
regnet.y-322
V1
Radosavovic al. 2020
100.0
84.5
99.1
82.3
regnet_y322gf
Radosavovic al.
2020
100.0
98.0
99.5
79.6
regnet_y_22ff-
SWAG_E2E.yI
Radosavovic et al. (2020)
100.0
99.6
93.5
71.5
regnet.y_32gf-
SWAG.LINEAR_VI
Radosavovic
2020
100.0
100.0
85.6
regnet_y_128gf-
SWAG_E2E.VI
Radosavovic et al.
2002
100.0
99.6
54.4
67.8
regnet.y_12gff-
SWAG.LINEAR_VI Radosavovic
2020)
100.0
96.9
regnet.x_400mf
Radosavovic et al.
2020
100.0
37.4
82.3
29.0
regnetxx4400ff V2
Radosavovic et al
2020
100.0
50.7
99.5
81.7
regnet.x_800mf V1
Radosavovic et al.
2020)
100.0
25.4
77.0
52.6
rgget.x_800mff V2
Radosavovic et
(2020)
100.0
75.5
97.1
71.4
regnetx__1 _6gff
Kadosavovic el al.
2020
100.0
38.2
regnet._116pp L6gf- V2
Radosavovic el al.
2020
100.0
82.2
99.2
88.1
regnet 2..2gff V1
Radosavovic el
2020
100.0
45.1
62.1
69.4
regnet 2gf V2
Kadosavovic
2020
100.0
83.4
99.6
88.3
regnetx__gff V1
Radosavovic
2020
100.0
98.8
89.0
regnetx_8gff V2
Radosavovic el al.
2020
100.0
93.5
99.2
81.6
regnet_x_16ff V1
Kadosavovic ct al.
202()
100.0
48.7
86.6
54.5
regnet_x_16gf- V2
Radosavovic al.
2020)
100.0
93.4
97.9
88.1
regnet x 32gf
Radosavovic
2020
100.0
66.1
85.9
46.0
regnet x 32gf
Radosavovic et
2020
100.0
83.7
resnet18 V1
He el al.
2016)
100.0
36.8
84.3
41.9
resnet34 - VI
He et al
2016
100.0
32.9
54.1
26.1
resnet50 V1 He al.
2016
100.0
30.1
73.9
54.5
resnet50 He al.
100.0
80.1
99,7
88.4
resnet101 V
He et
2016
100.0
60.4
92.3
84.4
resnet101 V2
2016
100.0
93.4
98.8
87.4
resnet152. V1 He
2016
100.0
66.1
98.4
78.2
resnet152- V2
He
2016
100.0
93.1
98.5
92.5
resnext50 32x44 VI
Xie
100.0
74,8
resnext50 32x4d V2
Xie el
2017
100.0
80.3
98.5
88.0
resnext101_32xxd V
Xıe al
201
100.0
66.6
84.7
61.2
resmext101_32x8d V2
Xie
2017
100.0
90.3
99.4
85.2
resnext101.64x4d
V1
Xie
100.0
77.9
97.7
74.2
10

---

Published as a conference paper at ICLR 2026
Table 14: Accuracy of various convolutional and transformeer-basdd models trained on ImageNet-Ik, the data generated for Tab. 2 As with Fig. 2 and Fig. [ we note that the performance of these models is significantly affected, even though the correct class is clearly illustrated right in front and center while, and the predicted class is absent from the generated images.
Prompt employed (correct class highlighted in bold, SC red)
Model Weights
photo of firemen and photo of peafowl
shrimp and pasta near Bernese Mountain Dog
peafowl
Bernese Mountain Dog
shufflenct_y 1atpiennet___xx_____
Ma al.
2018
100.0
23.8
36.9
21.0
shufflenct v2
x 0
V et al.
2018
99.8
30.4
72.1
55.9
shulllenet. v
x
M1 et al.
2018)
100.0
41.2
97.9
52.5
shufflenet.v 2___000_____000
V1 al. 2018)
100.0
61.4
99.3
64.5
squeezenet] 0 V1
landola ct al.
2016
100.0
11.4
95.9
28.7
squeezenet1 V1
landola
2016
100.0
13.8
91.2
46.1
swin_t-
al.
b021b
100.0
72.7
81.6
swin_s- V1
Liu et al.
2021b
100.0
74.3
99.3
81.4
swin b- V1 Liu et al.
(2021b)
100.0
81.5
95.2
72.6
swin.v2.. v2.t VI al.
100.0
76.2
88.7
73.6
swin.v2. v2.s VI
100.0
85.7
74.4
swin_v2.b V1 Liu et al.
2021b
100.0
73.0
96.0
85.2
vgg 1 VI Simmnnyan & Zisserman (0055)
100.0
9.9
96.6
60.7
vgg 1 [_hn- Simonnyn & / Zasserman
2015
100.0
15.9
86.7
54.8
vg V1 Simonnynn & Zisserman
2015)
100.0
5.9
97.7
68.1
13 bn
Simoonynn & Zisserman (2015
100.0
15.5
87.0
13.0
16 V1
Simonnan & Zissermani 15
100.0
12.2
93.7
66.0
1gg16_bn VI Simonyan & Zisserian (2015)
100.0
22.6
96.1
70.7
vgg19. VI
Simonyan & Zisserman
100.0
26.6
98.5
40.8
vgg19_bn
Simonyan Zisserman 22015
100.0
35,9
83.1
46.2
vit_b_16 16-66-1 V1
Dosovitskiy et al. (021)
100.0
85.4
97.1
75.2
vit_b_16
SWAG F2E
Dosovitskiy et al. (2021)
100.0
88.9
59.1
vit_b.16
SWAG.LINEAR.
Dosovitskiy el
100.0
79.2
84.6
vit_b_32 32 V1
Dosovitskiy et al 2021)
100.0
56.1
96.0
86.0
1i V1
Dosovitskyy et al. 2021
100.0
55.9
95.3
76.0
1it_1_16
SWAG F2E VI
Dosovitskiy et al. 2021
100.0
92.1
100.0
93.3
vit vit.116.
SWAG SWAG.LINEAN LINELLINAARRN VI Dosovitskiy et al.
202
100.0
97.4
100.0
72.5
vit_I.32- - V1 Dosovitskiy et al. 2021
100.0
54.0
96.9
82.9
vit.h.14
SWAG Dosovitskiy et al. 2021
100.0
99.4
98.5
86.4
vit vit h.hh114 14
SWAG LINEAR V1 Dosovitskiy et al.
202
100.0
99.7
100.0
96.1
wide resnet50 2
V1
Zagoruyko & Komodakis (016
100.0
60.6
95.7
63.9
wide 15de_resnet50
V2 Zagoruyko Komodakis 2016
100.0
83.9
99.5
85.4
wide rew.etttttl 2-
V1
Zagoruyko Komodakis 2016)
100.0
69.3
72.8
wide resnet V-
Zagoruyko Komodakis (2016)
100.0
91.0
84,9
11

---

Published as a conference paper at ICLR 2026
Table 15: Top Waterbirds class-neutral concepts Table 16: Top Waterbirds class-neutral concepts for "landbird".
for "waterbird".
Landbird
Score
Waterbird
Score
forest floor
0.055562317
swimming the water
0.11482495
forest next to tree bamboo forest floor
0.053587496
water lily
0.10905403
0.05134508
boat in the water floating in the water water
0.1066975
forest of bamboo
0.04781 133
0.106155455
forest
0.047080815
0.106134474
snowy forest
0.044688 106
flying over the water standing in the water
0.10561061
ground field
0.043406844
0.10444009
0.043168187
sitting in the water body water water in front standing in water water and swimming
0.103776515
log
0.043052554
0.09977633
standing on forest floor
0.041143
0.0902465
grass covered tree branch a forest
0.040526748
0.086544394
0.039670765
0.08424729
forest with trees
0.03949821
0,07818577
tree in forest bamboo forest
0.039123535
standing on lake flying over the ocean
0.06565446
0.03876221
0.06509364
front bamboo
0.03738105
flying over pond boats
0.06463468
mountain
0.036155403
0.061231434
forest trees
0.03600967
lifeguard
0.06122452
flying through forest
0.035692394
flying over lake
0.060126305
platform
0,03565806
boat
0.0571931
standing in forest
0.034528017
pond
0.053261578
hill
0.03341371
Table 17: Top CelebA class-neutral concepts for "non-blonde".
Non-Blonde
Score
hat on and blue 0.13952243
hat on and man
0.13853341
man in the hat
0.13803285
man who made
0.13307464
man behind man with the hat
0.13247031
0.13186401
man is getting
0.12861347
actor
0.12726557
dark
0.12713176
man in blue
0.1269682
person
0.12541258
in the blue
0.124844133 134
man is not man
0.12308431
man
0.12290484
large
0.1228559
shirt in a dark
0.12170941
hat
0.121646166 66
close
0.12146461
man with the blue
0.12136656
man face
0.12130207
12

---

Published as a conference paper at ICLR 2026
Table 18: Top CivilComments class-neutral concepts for
"non-ow noon-offensive".
Non-offensive
Score
allowing
0.07341421
work
0.06982881
made
0.069063246
talk
0.06858361
none are needed
0.067236125
check
0.06664443
helping keep the present
0.06531584
policy
0.06339955
campaign
0.06333798
involved in the first place
0.063222766
Cottage
0.063149124
IDEA
0.06310266
stories
0.0625782
Table 19: Top CivilComments class- neutral concepts for "offensive".
job
0.06236595
allowed
0.062137783
latest news about the origin
0.062061936
giving others who have experienced
0.061925888
"Offensive" Top Concepts
Score
proposed
0.061897278
hypocrisy
0.046756804
one purpose
0.06122935
troll
0.035944045
starting
0.061154723
silly
0.029536605
small
0.061071455
hate
0.013704538
question
0.060854554
silly how do you study
0.00325954
practice
0.060740173
spite
0.002645433
raised
0.060681045
kid you have the absolute
0.001619577
entering
0.060585797
registered
0.060475767
beliefs
0.060165346
accept that they promoting
0.060070753
Security
0.059328556
new
0.059324086
subject
0.058983028
close
0.058632135
views
0.058573127
Hold
0.058341324
reality for change built at that parish
0.058261245
0.057885766
rest
0.057804525
historic
0.057656527
concept
0.057422698
people
0.057151675
passage seems to in reflection
0.05699992
attempt
0.056797385
13

---

Published as a conference paper at ICLR 2026
Table 20: Top ImageNet class-neutral concepts for "Crossword".
"Crossword" Top Concepts
Score
reading newspaper
0.30469692
Table 21: Top ImageNet class-neutras concepts for "Guacamole".
man reading a newspaper
0,29045385
crochet squares
0.28316277
sitting on newspaper newspaper sitting
0,27749887
0.27106437
"Guacamole" Top Concepts
Score
crochet squares square
0.2708223
newspaper that has the words
0.26934764
tomatoes and avocado
0.33948907
holding newspaper newspaper while sitting newspaper laying square with a few crochet 0,2461972
0.26375395
avocado
0.3125001
0,26288068
nachos
0.3076 188
0,25538272
ham and parsley
0.2960047
with avocado
0.2952027
square with a crochet
0,24225119
colorful mexican
0.28313732
square of crochet squares checkerboard
0,23986068
bacon and parsley
0.27773544
0,23797607
of avocado
0.2754345
crochet blanket in a square 0,2364017
side of salsa
0.27470407
square of square crochet on newspaper
0,23590976
peridot salsa
0.2734925
0,2357213
0.27124816
crochet square with a crochet 0,23569846
nachos with
0.27092364
crochet blanket with crochet
0.23438567
lime body
0.27088284
newspaper
0,23315597
cheese and parsley
0.26951400
crochet with a square crochet square newspaper sitting on square of crochet yarn square crochet crochet blanket with square crochet square sitting
0,23304509
pile of limes
0.26944226
0,23259673
peas and bacon
0.267 204
0,23098715
limes and limes
0.26682717
0,2291883
lime cut
0.2662533
0,22903368
nachos with and
0.26580203
0,22888878
pasta with peas tacos
0.2626204
0.22860557
0.2619322
crochet in square square with a single crochet free crochet newspaper with on newspaper crochet blanket checkered blanket square of crochet
0,22856355
pasta with ham and parsley
0.26183394
0.22752959
tomatoes and cilantro
0.26161948
0.22748157
0.22672665
0.2257084
0.22547376
0,22514643
0.22324148
Table 23: Top ImageNet class-neutral concepts for "Blllpoit Pen".
Table 22: Top ImageNet class-neutral concepts for "Bald Eagle".
"Ballpoint Pen" Top Concepts
Score
wearing a pilot
0.36252645
markers
0.36050144
"Bald Eagle" - Top Concepts
Score
sharpie
0.34563553
stylus
0.34399948
osprey flying
0.2888234
pair of eyeglasses notepad
0.32426757
osprey
0.27377927
0.3228797
row of american flags
0.26513425
calligraphy
0.3222967
group of american flags flying
0.25681199
paint and markers
0.32198787
american flag and american flag emu standing yellowstone national
0.23945728
close up of needle on straw and markers eyeglasses
0.32131955
0.23752406
0.3209229
0.23740456
0.32090995
0.31017447
dots crayons
0.30764964
0.30598855
14

---

Published as a conference paper at ICLR 2026
Table 25: Top ImageNet class-neutral concepts
Table 24: Top ImageNet class-neutral concepts for "Doormat". for "Coffeemaker".
"Doormat" Top Concepts
Score
"Coffeemaker" Top Concepts
Score
brick sidewalk
0.37059835
kettle sitting
0.43714887
laying gravel
0.34978455
with a kettle
0.4207235
laying on carpeted
0.34279323
thermos
0.40833473
crochet blanket
0.3421431
kettle
0,40526068
sitting carpeted
0.34104648
kettle sitting
0,3881535
laying on step
0,3400078
stovetop maker
0.37747166
crochet blanket in square
0.336428 14
kettle kettle
0.37671012
brick walkway carpeted floor on a brick sidewalk crochet blanket with crochet
0.3319164
kettle kettle kettle kettle
0.37649006
0.33067068
vases and vases
0.37567452
0.32878387
kettle kettle kettle
0.37547356
0.32599914 14
flask
0.37485123
floor with a welcome
0.32548892
kettle kettle kettle kettle kettle
0.37305972
square of crochet yam
0.32428077
large canister flasks
0.37186915
crochet blanket made
0.32336423
0.36940324
carpeted staircase
0.3227385
decorative vases sitting
0.3683877
dot blanket
0.3213501
kitchen aid milkshakes
0.36793774
mosaic floor
0.32125634
0,36605325
crocheted blanket
0,31886423
large pottery
0.36506
on blanket
0.3182252
set of kitchen
0.3650242
crochet blanket with a square
0.31753486
cookbook vases sitting
0.36469316
square of crochet squares crochet squares
0.3169018
0.3643943
0.3152317
standing in a doorway
0,3126963
Table 26: Top ImageNet class-neutral concepts Table 27: Top ImageNet class-neutral concepts for "Fraser".
for " "American Lobster".
"Eraser" Top Concepts
Score
"American Lobster" Top Concepts
Score
crayons
0.38733196
pasta with shrimp
0.3148532
chalk
0.37349075
adirondack sitting
0.31178916
graphite
0.36778685
large shrimp
0.3085378
crayon
0.36178917
shrimp and pasta
0.3082 1544
charcoal
0.351962
shrimp cooking
0.3032636
lip balm lip
0.35033816 16
large cast cooking
0,30282205
band aid cookie
0.34825876
pasta with shrimp and cheese
0.2984723
lip balm
0.3409983
adirondack
0.29169592
nose sticking
0,34016216
and mussels legs and other seafood close up of shrimp of pasta with shrimp seafood
0.2904386
sticking
0.33971623
0,28870153
markers
0.33940658
0.2871699
matchbox
0.33480892
0.2823377
wand
0,33415005
0.2808096
stylus
0.3318595
mussels
0.2801877 8713
band aid card
0.33174193
cast cooking
0.27951777 1774
toothbrush
0.33009088
shrimp and cheese shrimp
0.27841 187
office supplies
0.32956824
0.2726224
band aid flexible
0.32946587
15

---

Published as a conference paper at ICLR 2026
Table 28: Top ImageNet class-neutral concepts for "Fire Truck".
"Fire Truck" Top Concepts
Score
firefighter spraying
0.34155905
group of firefighters
0.3347583
firefighters
0.3305 1466
firefighter
0.31450543
firefighter wearing
0.2843979
hydrant spraying
0.26793447
firefighter wearing
0.26590723
firefighter cuts farmall parked dashboard with flames
0.2613345
0.2591076
0.25825307
flames painted
0.2540929
16


Reviews:
[{"review_id": "b5IfY9pkHI", "reviewer": "Reviewer_hqDR", "summary": "The paper introduces BEE (Bridging Explainability and Embeddings), a framework for identifying spurious correlations by analyzing model weight space and embedding geometry rather than relying solely on data splits or prediction errors. BEE examines how fine-tuning perturbs pretrained representations and uses linear probing as a transparent diagnostic tool to reveal hidden shortcuts. It demonstrates that many spurious features persist even after fine-tuning and can transfer across diverse state-of-the-art models and domains, including vision and language tasks. Experiments show that BEE effectively uncovers severe and transferable spurious correlations, establishing it as a principled framework for model auditing and trustworthiness.", "questions": "1. Section 4.5 (Experiment Setup) is unclear and requires further clarification. What is the exact setup for GroupDRO? Did you use the same CLIP-based encoder and train only the classifier? Were all worst-group samples removed from the original training set. And if so, what is the rationale behind this choice? In that case, labeling the baseline as “GroupDRO” may not be appropriate, as the setup differs from the original formulation. The authors should provide a detailed justification for this experimental design.\n\n2. In Table 5, the improvement on the CelebA dataset appears noticeably smaller compared to other datasets. Could the authors explain the reason behind this limited gain?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The authors showcase a wide range of use cases, including zero-shot classification, discovering spurious correlations within datasets, and applications to both text-based and image-based datasets.", "weaknesses": "1. The proposed method closely resembles the concepts of Label-free CBM [1] and Post-hoc CBM [2], both of which also utilize CLIP-based image /text encoders. In particular, Post-hoc CBM (see Table 10 in its Appendix) demonstrates a similar approach to identify biased concepts residing in the dataset. While the authors could emphasize the spurious concept discovery component as their main contribution, the process of using LLMs and captioning models to enumerate and filter potential concepts appears similar to LLaVA-CBM [3]. The paper should more clearly articulate its unique contribution and novelty relative to these existing works.\n\n2. What would happen if the encoder component were fine-tuned rather than kept frozen?\n\n3. The Method section requires more detail. Some crucial explanations are deferred to Section 3, making it difficult to follow the approach before reaching the experiments. Including a concrete example using a well-known dataset could substantially improve clarity.\n\n4. Section 4.5 (Experiment Setup) is difficult to understand. The authors should provide a clearer description, particularly regarding the setup and implementation details of GroupDRO.\n\n5. Since the performance of the image captioning model appears to play a crucial role, it would be helpful to show example outputs and explain how LLMs are used (e.g., prompts, output format, etc.). Have the authors experimented with different captioning or language models, and if so, how do the results vary?\n\n6. **Writing:** Some sentences are overly long and contain excessive commas, which makes them difficult to read. \n\n[1] Label-Free Concept Bottleneck Models, ICLR 2023\n\n[2] Post-hoc Concept Bottleneck Models, ICLR 2023\n\n[3] Constructing Concept-based Models to Mitigate Spurious Correlations with Minimal Human Effort, ECCV 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NRi03zb7RZ", "reviewer": "Reviewer_5jdF", "summary": "This manuscript introduces BEE (Bridging Explainability and Embeddings), a framework designed to diagnose spurious correlations learned during linear probing of frozen model. It uses linear probing to reveal spurious features that persist after full fine-tuning and transfer across diverse SOTA models. BEE is evaluated on vision (Waterbirds, CelebA, ImageNet-1k), language (CivilComments) and medical notes (MIMIC-CXR); and across embedding families (CLIP, CLIP-DataComp.XL, mGTE, BLIP2, SigLIP2). BEE surfaces concepts that can slash ImageNet accuracy by up to 95% and exposes clinical shortcuts in MIMIC-CXR dataset that cause false negatives.", "questions": "1. Could you please share epochs, batch sizes, learning rate, temperatures (for CLIP and non CLIP encoders), and hardware / compute used for each experiment?\n2. Do you actually use negative SCs anywhere? If yes, where and how are they incorporated? If not, please explain why and provide an example showing whether negatives are useful.\n3. Could you please provide the full procedure for building C_all?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Core idea is clear: both the linear head and concepts are in the same embedding space. Direct comparison of learned weights and concept embeddings constitutes a direct probe into the decision mechanisms of the classifier.\n2. Efficient as it eliminates the need for expensive backbone retraining or the construction of elaborate counterexample data splits.\n3. Empirical validation across multiple modalities / datasets and different foundational embedding families (e.g., CLIP and BLIP-2).\n4. ImageNet-1k generative experiment is good evidence: prompting generator with BEE surfaced concepts leads to large drops in accuracy and even to class swaps.", "weaknesses": "1. Missing implementation details for reproducibility about the construction of the concept pool (prompts used with Llama-3.1-8B-Instruct, the exact filtering and de-duplication rules applied with WordNet and the final vocabulary size for each dataset).\n2. BEE has a variable number of concept prompts per class. No details on whether the baselines like B2T / SpLiCE were given an equivalent prompt budget - difficult to ascertain if the observed gains are because of the higher quality of the BEE selected concepts or because of a larger query allowance.\n3. Meaning of reversed diagonal reference, specification of the smoothing window and the exact roles & sensitivity of the r and p are not clearly defined. Sensitivity or ablation study is needed to confirm the stability of the concept selection process.\n4. Missing empirical results / experiments scoring for negatively correlated concepts introduced in the paper.\n5. Suggest adding multi-seed results with CIs for all tables.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vr7M25rgZR", "reviewer": "Reviewer_Auj2", "summary": "This paper introduces BEE (Bridging Explainability and Embeddings), a novel framework for detecting spurious correlations in fine-tuned foundation models without requiring counterexamples or group annotations. \n\nThe method uses a transparent linear probing setup, identifies class-neutral concepts in the embedding space, and ranks them based on their alignment with classifier weights.\n\nExperiments cover a wide range of modalities (vision, language, healthcare) and embedding models (CLIP, BLIP2, mGTE, etc.).", "questions": "NA", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- a conceptually elegant and model-agnostic approach to uncover spurious correlations from classifier weights, which is both simple and broadly applicable.\n\n- tightly aligned with foundation models: BEE leverages their shared embedding spaces to analyze both classifier weights and textual concepts. This makes BEE natively compatible with large pre-trained models, including CLIP, BLIP2, mGTE, and others. \n\n- strong empirical results on diverse tasks and datasets.\n\n- the methods provides explicit concept-level explanations for model decisions, thus good interpretability.", "weaknesses": "- BEE operates entirely within the embedding space of large foundation models such as CLIP or mGTE. If the embedding model itself has already encoded biased or spurious associations, BEE may merely surface these existing biases, rather than revealing new or independent shortcuts learned during downstream training. This raises the question of whether BEE is diagnosing the fine-tuning process or simply interpreting the biases already present in the frozen embeddings.\n\n- BEE relies on linear probing to identify correlations between classifier weights and concept embeddings. As such, it is inherently limited to capturing linear or near-linear relationships. What will happen if BEE is applied on more complex spurious patterns? \n\n- BEE does not offer any mechanism for diagnosing or correcting the biases that may exist in the foundation model’s embedding space. This is a critical limitation, as the root causes of many spurious correlations may lie within the embedding model itself.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rUq73Ii5ng", "reviewer": "Reviewer_Qgvj", "summary": "This paper proposes BEE, a weight-space diagnostic to identify class-specifc spurious correlations learned during fine-tuning. Specifically, it can be divivided into two steps: (1) Initialize linear probe with zero-shot class embeddings from a foundation model and finetune the probe; (2) Rank class-neutral concepts by their similartiy in terms of class weights and thresholding to select SCs. The method is evauate across different modalities including vision (e.g., ImageNet) and language (e.g., CivilComments) with embeddings from different foundation models. It shows strong generalization ability across different tasks with spurious correlation settings.", "questions": "1. Could you provide the details of the computation overhead for the spurious correlation detection?\n\n2. Is there any broader impact of detecting such spurious correlation apart from classification? Will it affect downstream tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The author proposes a novel method with clear motivation. Rather than error- or data-driven SC doscovery, them operate the measurement in weight space. This makes spurious concept can be dicsovered from the geometry of embedding space, which is simple and aligned with human intuition.\n\n2. The experiments are evaluated on broad scope with multi-modal validation, demonstrating strong generalization. The pipeline is also simple and easy-to-understand, which can improve its practical usage.", "weaknesses": "1. The concept extraction, LLM filtering and dynamic thresholding the spurious concepts are heuristic and heavily relied on the other foundation model which can potentially also suffer from spurious correlation.\n\n2. The gemoetric view is novel, but the paper lacks theoretical analysis about when weight drift towards concept embeddings necessarily indicates spuriousness versus legitimate subclass structure.\n\n3. The defnition of class-neutral is ambiguous. The filtering relies on LLM heuristics. It's possible to assign the attributes relevant to class context (but not spurious) incorrectly as spurious concept. It will be better to discuss it with precison/recall of the class-neutral label.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)