Say My Name: a Model's Bias Discovery Framework
Authors: Massimiliano Ciranni
Luca Molinaro
Carlo Alberto Barbano
Attilio Fiandrotti
Vittorio Murino
Vito Paolo Pastore
Enzo Tartaglione
Venue: Accepted by TMLR ()

Abstract: Due to the broad applicability of deep learning to downstream tasks and end-to-end training capabilities in the last few years, increasingly more concerns about potential biases to specific, non-representative patterns have been raised. Many works focusing on unsupervised debiasing leverage the tendency of deep models to learn “easier” samples, for example by clustering the latent space to obtain bias pseudo-labels. However, their interpretation is not trivial as it does not provide semantic information about the bias features. To address this issue, we introduce “Say My Name” (SaMyNa), a tool to identify semantic biases within deep models. Unlike existing methods, our approach focuses on biases learned by the model, enhancing explainability through a text-based pipeline. Applicable during either training or post-hoc validation, our method can disentangle task-related information and propose itself as a tool to analyze biases. Evaluation on typical benchmarks demonstrates its effectiveness in detecting biases and even disclaiming them. When sided with a traditional debiasing approach for bias mitigation, it can achieve state-of-the-art performance while having the advantage of associating a semantic meaning to the discovered bias.

Published in Transactions on Machine Learning Research (11/2025)
Say My Name: a Model's Bias Discovery Framework
Massimiliano Ciranni"
MaLGa, DIBRIS, University of Genon, Italy
Luca Molinaro'
Lmolinaro@uniieo it
Computer Science Department, University of Turin, Italy
Carlo Alberto Barbano
carlo. bawbaaaobborrrrrero it
Computer Science Department, University of Turin, Italy
Attilio Fiandrotti
atiitio.fanndroti@@urrr it
Computer Science Department, University of Turin, Italy LTCI, Télécom Paris, Institut Polytechnique de Paris, Fronce
Vittorio Murino
vittorio.murinn@hittt
Istātuto Italiano di Tecnologia (ITT), Genon, Italy University of Verona, Italy
Vito Paolo Pastore
vito. vito. pwl.oppoooopptteoo @unigee. it
MaLCa, DIBRIS, University of Cenoa, Italy Isitudo Itaiono di Tecnologia (ITT), Cenoa, Italy
Enzo Tartaglione
LTCI, Télécom Paris, Institut Polytechniqqe de Paris, Prance
Reviewed OpenReview: https: openreview. net/ forum? id= Eutwi
Abstract
Due to the broad applicability of deep learning to downstream tasks and end-to-end training capabilities in the last few years, increasingly more concerns about potential biases to specific, nou-representative patterns have been raised. Many works focusing on unsupervised debiasing leverage the tendency of deep models to learn "easier" samples, for example, by clustering the latent space to obtain bias pseudo-labels. However, their interpretation is not trivial as it does not provide semantic information about the bias features. To address this issuc, we introduce "Say My Name" (SaMyNa), tool to identify semantically polential biases within deep models. Unlike existing methods, our approach focuses on concepts learned by the modol, enhancing explainability through a text-based pipeline. Applicable during cither training or post-hoc validation, our method can disentangle task-related information and propose itsell as a tool discover biases. Evaluation on typical benchmarks demonstrates its effectiveness in detecting biases and even disclaiming them. When sided with traditional debiasing approach for bias mitigation, it can achieve tate-of-thheart performance while having the advantage of associating a semantic meaning with the discovered bias. The code is available at httpss///ggttabb..ccc/////yaaaea
"Equally contributing first authors
'Equally contributing senior authors

---

Published in Transactions on Machine Learning Research (11/2025)
1 Introduction
In the past decade, advances in technology have made it possible to widely use deep learning (DL) techniques, greatly impacting the computer vision field. By al- lowing systems to be trained end-to-end, DL offers potentially fast model doployability to solve complex problems, leading to fast progress and changing how we perceive and analyze visual information. Today, deep learning is applied in various real-world scenarios, such self-driving cars (Zhang et al. 2022a), medical imaging (Yan et al. 2024 and augmented reality (Liu et al. 2020
Potential spurious features learned
Class landbirds
Class waterbirds
trea Forest stalks
sea ocean beach
Model under
analysis
These huge possibilities come with potential pitfalls. One challenge to face when deploying DL solutions lies in guarantccing that the model does not over- rely on specific patterns that are non-representalive of the real-world data distribution (Ming et al.) 2022 Izmailov et al., 2022). This is key for safety, fairness, and ethics (Tartaglione et al., 2023). To provide a sim- ple example, when deploying solutions in autonomous driving, simple tasks like pedestrian detection can be implicitly solvod by associating specific background elements (such as sidewalks or pedestrian crossings) with the presence of pedestrians. This reliance environmental cues can act as shortcuts for the net-
(a) Landbird.
(b) Waterbind.
Figure 1: Top: SaMyNa searches potential spurious features learned by a model, providing a ranked list of key Bottom: Example of bias heatmaps (red = correlated with bias, blue anti-correlated.
work Geiiros et al. 2020). leading to poor performance when the context changes, such as when a pedestrian is crossing a road without a marked crossing lane. DL models, if not discouraged, tend to rely on spurious correlations captured during training, especially when they are easier to learn than the actual semantic attributes (Nam et al.j 2020). A visual example can be found the bottom of Fig. where a model trained on waterbirds, a dataset presenting a strong spurious correlation between target labels (waterbird and landbird) and background, indeed focuses on the latter for making its predictions. A description of how these heatmaps are constructed can be found in Sec. A of the Appendix. We refer to these spurious correlations biases, and we say that the model that learned such shortcuts is biased towards them. In 2021, the European Commission introduced the Artificial Intelligence Act (AI Act) to regulate Al based on the potential risks it poses (Madicga 2021). . Like the General Data Protection Regulation (GDPR) (GDPR) 2016), the AI Act could set the next. few years a global standard. Ensuring that DL models avoid spurious biases that could affect their safety, trust, and accountability is not only essential for user safety but might soon also become a legal roquirement.
A massive recent effort has been conducted by the Computer Vision community to try to discourage the presence of biases. In nutshell, we can roughly distinguish three main research lines: (i) supervised, where the labels of the bias are provided (Barbano et al. 2023 Hong & Yang 2021); (ii) bias-tailored, where a hint on what the potential bias might be is provided prior to training, and an ad-hoc model is deployed to
capture it (Bahng et al. 2020b Wang et al. 2019); (iii) unsupervised, where biases are guessed directly
within the vanilla-trained model (Nam al. 2020 Nahon et al. 2023 Creager et al., 2021 Li et al. 2022 When deploying DL solution in the wild, the latter line of research appears to be the best fit, detailed
information about bias is almost surely missing. Although some solutions already exist in this context (Kim ct al.)
2024 Nam et al. 2020 Nahon et al. 2023 Ji et al. 2019), there is still gap in the literature related to the problem of namning specific bias alfecting a DL model, providing natural language descriptors that can be directly interpretable by a human. Existing solutions either start from a predefined set of attributes [Eyuboglu
ct al.] 2022 Wiles et al.) 2023) or explicitly require the availability of a validation set known to contain confficnni samples (Kim et al., (2024), and require to perform computationally expensive operations such captioning the entire validation set, which may be unavailable or made up of only aligned samples.
2

---

Published in Transactions on Machine Learning Research (11/2025)
Our method is effective by using only the training set, relaxing the assumptions of existing works (Kim et al. 2024), thus setting first step towards mining specific model biases in realistic scenarios. Unlike approaches that discover biases in the dataset, our focus is on naming the biases captured by the model under examination. We mine the specific features in common with these samples and we associate semantic (lextual) meaning to them. From this, an expert user (or prospectively a certifier software) can search and discriminate whether the learned feature is a bias or rather feature for the system (Fig. D Through "Say My Name" (SaMyNa), we aim providing tool that enhances explainability for the DL model's learned features, top of which, if necessary, any state-of-the-art debiasing approach can be used to sanitize the model.
Our contributions are here summarized at a glance.
We propose human-readable text-based pipeline for discovering and naming biases in DL models, which:
does not require a validation set, contrarily to existing approaches (Kim et al. 2024 and remains human-readable and interpretable at every step (Sec. 3.2
leverages the embedding space of text encoder to create learned class reprosentations that caplure potential biases of the model (Sec. 3.2.4);
can be applied both at training and inference time, with the former relying on a simple yet effective strategy for discovering potential biases of the model directly on the training set (Sec. 3.1
We validate our approach on established benchmarks, finding biases well-known by the community (Sec. 4.2.1p. Furthermore, we also test on ImageNet-A, distinguishing cases where a bias exists and where, on the contrary, gencralization issucs are not bias-related (Sec. 4.2.2). This demonstrates the broad applicability of SaMyNa even for more general DL model diagnosis.
After bias discovery (Sec. 3.1 and naming (Sec. 3.2) our method, when coupled with standard debiasing strategy, can attain debiasing results in line with the state-of ofetrr-atte-ttttt (Sec. 4.3) with the big advantage of assigning scmantic meaning to the discovered bias.
2 Related Works
The problem of bias and model debiasing has been widely explored in recent years, within three main frameworks, differing from how or if bias knowledge explored for mitigating model dependency on bias: supervised, taal-da,,, and unsupervised.
Supervised Debiasing. Supervised methods require explicit knowledge of the bias, generally in the form of labels, indicating whether sample presents a certain bias or not. One of the most typical approaches consists of training an explicit bias classifier, trained on the same representation space as the target classifier, in an adversarial way, forcing the encoder to extract unbiased representations (Alvi el al. 2018 Xie et al.) 2017). Alternatively, bias labels can be exploited to identify pre-defined groups, training model to minimize the worst-case training loss, thus pushing the model towards learning biasod samples (Sagawa* et al.) 2020 Another possibility is represented by regularization terms, which aim at achieving invariance to bias
features (Barbano et al. 2023 Tartaglione et al. 2021
TaisoTai Debiasing. Bias-tailored approaches usually rely on some kind of knowledge about the bias nature. For example, if the bias is textural, then custom architectures can be designed to be more sensitive to textural information. For example, (Bahng et al. (020aa propose ReBias, where custom texture bias-capturing model is designed using Ix1 convolutions. A similar approach is followed by (Hong & Yang 2021 where BagNet-18 (Brendel & Bethge 2019) is used as a bias-capturing model.
Unsupervised Debiasing. Differently from the previously described approaches, unsupervised debiasing methods do not assume any prior knowledge of the bias, facing realistic situation where bias unknown.
3

---

Published in Transactions on Machine Learning Research (11/2025)
(Nam et al. 2020) propose LfF, where a vanilla bias-capturing model is trained with focus on casier samples (bias-aligned), using the Generalized Cross-Entrops (GCE) loss (Zhang & Sabuncu 2018), while a debiased network is trained by giving more importance to the samples that the bias-capturing model struggles to discriminate. (Ji et al. 2019) propose an unsupervised clustering method that learns representations invariant to some unknown or "distractor" classes in the data, by employing over-clustering. A set of unsupervised methods relies on the assumption that bias-conflicting samples are likely to be misclassified by a biased model (Kim et al. 2022a). In (Liu et a 2021), model is trained for a few epochs and then used in inference on the training set, considering misclassified samples as bias-conflicting and vice versa. The debiasing is then performed by up-sampling the predicted bias-conflicting samples. In (Kim et al. 2022a), the training set is split into a fixed number of subsets, training a model on each of them. Then, the trained models are ensembled into a bias-commillee and the entire training set is fed to the committee, proposing that debiasing can be performed using a weighted ERM, where the weights are proportional to the number of models in the ensemble misclassifying a certain sample. Other lines of work, such as tackling training data containing noisy labels (Pleiss et al.] 2020), introduced Lechniques to identify problematic training samples through margin-like measures. Similarly to (Nahon et al., 2023), we are able to identify during the training of a vanilla model in which moment the bias is potentially best fitted by the model, with the advantage of working directly at the output of the same model instead of mining the information in its latent space. This comes both with computational advantages (given that the latent space is typically higher dimensional) and with better interpretability of the outcome, given that we work in the model's output space.
Bias Naming. Recently, methods exploiting natural language and vision-language models to identify and mitigate bias have been proposed (Eyuboglu et al.) 2022, Kim et al.) 2024). (Zhang et al.) 2023) introduce a method capable of determining subsets of images with similar attributes systematically misclassified by a model (i.e., error slices) and rectification method based on language. However, it starts from pre-defined set of attributes, thus hindering the possibility of discovering completely unknown and multiple biases. (Eyuboglu let al.) 2022) exploits a cross-modal embedding space to identify error slices, providing natural language predictions of the identified slices. (Wiles et al. 2023) propose a method for automatically determining a model's s failures, exploiting large-scale vision-language models and captioners to provide interpretable descriptors of such failures in natural language. In (Kim et al. 2024), the authors propose to extract class-wise keywords representative of bias, later used for model debiasing, exploiting group-DRO (Sagawa* * et al. 2020) the identified groups. In their work, CLIP score is defined using the similarity between extracted keywords and correctly and incorrectly misclassified samples (class-wise) and used to find the keywords associated with a bias. In (D'Inck et al. 2024). the authors use large-language models and lext prompts for bias discovery in text-to-image generative models. An alternative framework is proposed in (Guimard et al., 2025), where bias proposals are obtained relying on an LLM fed with task descriptions, including the textual name of the classes. text-to-image retrieval approach, based on captions generated from the identified proposals, is then exploited to obtain biased images from an external large-scale dataset, further employed to evaluate the target model. Differently from the previously cited works, introduce an unsupervised method for diagnosing a model dependency on bias for image classification tasks, which can either be performed during or after training and only exploits task-related knowledge to provide a transparent analysis of the potential hidden biases captured by the model.
3 Method
In this section, we present our proposed method SaMyNa to identify potential spurious correlations learned by the model under analysis (Sec. 3.2 Our method can perform model diagnosis either at training or at test time: for the first case, we also present an approach to identify at which iteration to mine a potential bias for the target model (Sec. 3.1
3.1 Mining Model's Biases
Consider a supervised classification sctup (having C' target classes), where we learn from a datasct D Dirain containing N input samples (Z1...., xN) X, each with a corresponding ground truth label (yi..... y. A deep neural network M, trained for iterations, produces an output distribution Φr. E RC over the C
4

---

Published in Transactions on Machine Learning Research (11/2025)
Samples subset selection
Samples captioning
Keywords
selection
Model under analysis
Multmodal LLM
Forest Vogelalion
Gcean
Keywords ranking
Learned class embedding
Text
Similarity
Figure Pipeline for SaMyNa. Given model, we can tell between either Dirain D which are the correctly (with green border) and the incorrectly (red border) classified samples. Amongst these, first perform sample subset selection looking at the latent space of the model under analysis and choosing through k-medoids the most representative samples for the learned class. Then, we employ a captioner to get a lextual description of these samples. From these descriptions we extract non-rare words as keywords, and, in parallel, working in the latent space of a text encoder, we extract the mean description for the learned classes, cleansed from common features within the dataset. We finally compare this representation with the embedding of the keywords, revealing learned correlations that extend beyond the target class signal.
classes, for each input a, (typically, the activation of the last layer a softmax). The network is trained match the ground truth label ya. by minimizing loss function such cross-entropy.
If any bias is present in the training set, however, the learning process could drive the model towards the selection of spurious features (Sagawa* et al., 2020 Nam et al. 2020 Bahng et al. 2020a), resulting in misclassification errors. Recent findings show that it is possible to identify the moment when the model best fits the bias in the training set (Nahon et al. 2023) We will formulate here the problem of identifying, at training time, when the trained model maximally fits a potential bias.
We say that M misclassifies the n-th sample at the I-th learning iteration Un * arg max( Focusing on this example, we know that by minimizing the loss function we aim incrcasing the value of the yv-th component of the model's output, denoted (t.n(y)) while decreasing all the others. To measure how far the model is from predicting the desired category, can define per-sample distance metric telling us how far the n-th sample is from being correctly classified:
de = max( (v,)())) ifarg max,, (9c..) # Un otherwise.
(1)
To give an intuition, the distanco-metric formalized equation I inspired by the Hinge Loss (Rosasco et al. 2004), and could actually be formulated directly as max(f vectors for labels. Here, for each sample, define the target class as the category for which the model yields the highest prediction confidence, regardless of its correctness. We then compute per-sample metric as the margin between the confidence in this target class and the confidence in the actual ground-truth class. This value quantifies the confidence gap the model must overcome to make a correct prediction. By definition, this metric is zero when a sample is correctly classified and positive otherwise.
as use one-hoo-eneoded
Intuitively, the higher de, is, the most M, confident misclassifying H, as specific class. We are interested in finding the iteration !* such that the model most confidently misclassifies pool of samples,
5

---

Published in Transactions on Machine Learning Research (11/2025)
in equation
t. arg max
Σ Id
(2)
Damiselass
Draai
where Dinisclsss is the set of misclassified training samples, and [Diniolass] is its cardinality. When reaching train 6*, the most informative samples are the misclassified ones: given that the model is most confident in misclassifyngg them, the model may have learned some spurious features deeply affecting its final predictions. Unlike prior works (Nahon et al. 2023) that speculate that misclassified samples embody a bias (with the goal of applying debiasing methods), our goal is to understand why these samples are misclassified, ultimately providing the end user of the system the possibility lo acknowledge the presence of a bias. For the model Mr Dinisclass we will split the training dataset in a pool of correctly classified samples Derain and misclassified train Examples of the vanilla model's output softmax distribution during training can be found in the Appendix (Sec. C.1
3.2 Bias Naming
We present here SaMyNa, our bias naming approach. Fig. proposes an overview of the main pipeline we used, consisting of the following steps:
1. Samples subset selection. Given that the objective of our proposed method is to identify biases learned by M, we are allowed to propose a subset of most representative samples for given target class (Sec. 3.2.1
2. Samples captioning. We then generate textual descriptions for the selected samples (Soc. 3.2.2
Keywordss selection. Starting from the captions, we construct a pool of keywords by applying a lightweight word frequency filter. Heavy filtering is done as last step to better account for synonyms (Sec. 3.2.3
4. Learned class embedding. Simultancously, we generate embeddings for the captions. We use these embeddings, along with the corresponding (arget and predicted classes, to calculate learned class embeddings that represent the biases of M present (Sec. 3.2.4
Keywords ranking. We then embed the keywords and rank them according to their similarity with the learned class embeddings. Finally, low-ranking keywords can be filtered out (Sec. 3.2.5
3.2.1 Samples Subset Selection
Given M, for a given target class c, we extract the pool of correctly classified samples Doorrect and the pool of samples misclassified class c, which call Dmisclass (c) Provided that M clusters both Doorrect (c) and Duisccaes (c) logether, our hypothesis is that these two share common set of features, behind which we might find bias. In the typical deployment scenario, the correctly classified examples are abundant, and, for instance, M may project them in a very narrow neighborhood of its latent space. We build on top of this observation, and we cluster correctly classified samples using the k-medoid algorithm to reduce the cardinality of correctly classified samples while maintaining a good coverage of the model's 's latent space. The choice of using k-medoid is natural, since it provides samples as output instead of centroids (which may be arbitrary points in feature space). As our long-range objective is to capture the set of features learned by the model to make its predictions, we also select. sample of examples misclassified them according to de .n. selecting the & ones that were misclassified with more confidence, We refer to the k samples resulting from running k-medoid on the correctly classified samples as Scorrect (c), while the pool of k samples misclassified as c denoted as Smisclass (c).
class by sorting
pleasse that have dropped the "train" subscript from the partitioning as this pipeline will work equally well also
when using a validation set..
6

---

Published in Transactions on Machine Learning Research (11/2025)
3.2.2 Samples Captioning
At this point, we will generate captions from the selected samples. To do this, we use a pre-trained multimodal large language model that takes as imput both a prompt and an image. We used the following generic captioning prompt for all the experiments: "Describe this image in detail. Pay particular attention to the subject and where they are. Describe as many details as possible".
We decided employ a large-scale image captioner in all our experiments, given that biases might hide in the subtle characteristics of the provided images (scc Sec. B of the Appendix).
3.2.3 Keywords Selection
From the captions obtained in the previous step (Sec. 3.2.2 we select candidate keywords. First, we perform some NLP standard processing, consisting of lower-case conversion, word-level tokenization (Bird et al.) 2009 and stop-words removal. Then, for each token w and class e, we count the number fw(c) of captions of class that contain w at least once. Lastly, given fmin between 0 and 1, we filter out the token w if lf (c)] is less than smin [[Soomeect()) (c)l + [Smisclass (c)]] The remaining tokens are aggregated into keywords proposal pool v
3.2.4 Learned Class Embedding
In parallel to keywords selection, we aim at having a representation for the learned class, disentangled from the specific domain M, which is trained to. To do this, we work in the embedding space of a pre- trained text encoder. From the generated captions we obtain, for each class c, the embedding matrices georrect (c) E (l (c)[x and Ermisclasscc (c) E (j]]m where Z is the dimensionality of the embedding vectors. These matrices are simply the results of stacking the embedding vectors coming from the text encoder applied to the various captions of the samples in Storrcttc)) (c) and gunisclass(c) (c). Our goal here is to calculate the embeddings (c) RR semantically representing the learned representation of the class c. Not only that, we would like to disentangle this from the features in common to all the classes learned from the model, given that M could be trained (and tested) to fit a specific domain, which in such specific case would not constitule a bias but rather a feature. For this, we first calculate the embedding E(e) for a specific learned class:
11 (c)
Σ (sminctess Exporrcc (c) + E. Exmisclass (c)
E(c) Zi=1
(3)
2 | Scorrect (c)l SSmlaclase (c)l
This equation computes the average of the means across all pairs of embeddings, where each pair consists of correctly classified example of class and an example that was misclassified as class We adopt this pairwise formulation instead of a simple average to improve robustness in scenarios with significant imbalance between correctly and incorrectly classified samples. This imbalance usually happens when & is greater than the number of misclassified examples for a given class. Additionally, because each pair includes embeddings from examples with different ground-truth classes, this approach helps discourage the encoding of the target class into the learned class embedding. E(c) will now contain all the common features of the class c. However, it will also contain some information shared in the entire dataset (for example, common characteristics of the different classes). From this, extract the embodding without the shared information from the datasct. through:
1 C Σ E())
(c)
The intuition of this approach originates from the arithmetic and semantic propertics of natural language latent spaces (Mikolov el al. 2013). To provide a realistic example, consider the task of gender recognition from facial pictures, in which the hair color is a spurious correlation. E(c) might contain features related concepts such as "blonde" and "face". As we are only interested in the former, computing E*(c) is an effective solution to filter out the shared information "face". We demonstrate the effectiveness of equation through ablation study in Sec. C.2.1 of the Appendix.
7

---

Published in Transactions on Machine Learning Research (11/2025)
3.2.5 Keywords Ranking
Now, we are ready to compare the embedding of each keyword with (c)c) using the cosine similarity:
(1b.,) c) = sim sim[t/embed (c),,
& E VV,
(5)
where gjembed is the embedding of the keyword in the same latent space used to calculate (c). This tells how much the concept is embodied by the proposed keywords. Based the ranking, we will obtain set of keywords that correlate with the learned class c, and others that become decorrelated as they embody some knowledge shared through all the classes (as filtered in equation 4 For this, we introduce a hyper-parameter fsim > 0 that thresholds the relevant keywords for the learned class c, based on the similarity score. The final ranking we obtain embodies the set of features that correlate with the learned class c, from which an end user of the system can deduce the presence of a bias. In some cases, the ranking may also contain keywords related to the target class, but such keywords are easily ignored by the end user.
4 Empirical Results
We provide here the main results obtained. We highlight that, for visualization purposes, all the figures contain up to the top nine keywords identified by SaMyNa: additional ablation studies and the full results are presented in Sec. C and H of the Appendix, respectively. For our experiments, we have employed an NVIDIA A5000 with 24GB of VRAM, except for the captioning step for which we have employed NVIDIA A100 equipped with 64GB of VRAM. The source code, attached to the submission, will be open-sourced upon acceptance of the article.
4.1 Setup
Models tested. We lested the most popular architectures benchmarked from the debiasing literalure: ResNet-18 for CelebA and BAR, and ResNet-50 for Waterbirds and ImageNet-A. All the models are pre- trained on ImageNet-1K, with architecture and weights provided by torchvision. On ImageNet-A, models are run only in inference, we are interested in mining biases already existing in the original pre-trained models, while for all the other experiments we apply the training procedure described in Sec. 3.1 For this step, we train with batch size of 128 and a learning rate of 0.001 for Waterbirds, as done in (Sagaaa et al. 2020); for CelebA, we use a batch size of 256 and a learning rate of 0.0001, following (Nam et al. 2020 For both, we employ SGD with Nesterov, set to 0.9. Finally, for BAR,, we employ a batch size of 256 and a learning rate of 0.001, with Adam as the optimizer (Kim et al. 2021).
Captioning. For the captioning model, we used LLaVA-NeXT Liu et al. 2024 in its 34B configuration, quantized in 8 bits. Before feeding our imput images to the image captioner, we apply the corresponding preprocessing transform provided by the huggingface library.
Class and keywords embedding. For this part, we used a MiniLM mode 3 from the sentence- transformers (Reimers & Gurevych 2019) library to generate 384-dimensional embeddings of the captions. The minimum frequency for the keywords fuin is 15%. For the correctly classified samples, we set 10
and Isim is 0.2.
Datasets. For our study, employ the following datasets: Waterbirds (Sagaaa et al. 2020 CelebA Liu et al. 2015), BAR (Nam et al. 2020), and ImageNet-A (Hendrycks et al. 2021
Waterbirds is an image dataset introduced in Sagawa* et al. (0200) to test the robust of optimization methods against distribution shifts. The associated task is to classify the habitat of bird species, divided into waterbirds and landbirds. In the training set, though, 95% of waterbirds are associated with a water background, and 95% of landbirds are set on land. Only 5% of the two classes' samples are presented with an opposite background. This results in potentially strong spurious relation between the target label and the background.
8

---

Published in Transactions on Machine Learning Research (11/2025)
Not blonde
Blonde
Landbirds
Waterbirds
forest-
orssn
foliage
bloade-
shore
tree
water
mascara
stailes
waves
branch
bost
makorpp
river
midifight-
lipetick
branches
lake-
vegetation
flying
male
eyeliner
0.2
0.3 s(k,c)
0.2
0.4
0.2
0.3
0.4
0.4
0.5
s(k,c)
*(k,c)
s(k.c)
(b)
Figure 3: Similarity scores for Waterbirds (a) and CelebA (b).
CelebA (Liu et al. 2015) is one of the most popular datasets of face images, depicting celebrity individuals with multiple samples per subject and equipped with extensive annolations regarding roughly 40 altributes, encompassing several face and style characteristics. Regardless of its popularity, it is notoriously affected by several biases (Nam et al., 2020; Barbano et al., 2202)). In this work, we analyze the task of classifying whether a person has blond hair or not, for which the attribute female not uniformly represented, but spuriously correlated with the blond class. A deep network trained with standard techniques exhibits strong bias in predicting subjects with female perceived gender as blond, while male appearing subjects are often classified as not blond.
Biased Action Recognition (BAR) is an action recognition dataset crafted by Nam et al. in Nam et al. 2020) The target classes of BAR (Climbing, Diving, Fishing, Racing, Throwing, Vaulting) present a strong bias towards the setting where they are performed. For instance, the large majority of samples from the class Racing is depicted in circuit track context, while in the test. set, which deep classification models fail to generalize. This dataset does not provide bias group annotations and does not provide a proper validation set, making it impossible to compute a Worst Group accuracy.
find many off-road settings on
Nel-A is a collection of 7,500 real-world images sharing the same category of a 200-class subset of ImageNet, onto which deep models systematically fail to output the correct prediction. Originally introducod in Hendrycks et al. (2021) for lesting adversarial model robustness, it is also commonly adopted in the context.
of model bias (Bahng et al. 2020b Kim et al. 2022b
4.2 Bias Discovery
Our empirical validation on bias discovery is here divided into naming the bias during training (Sec. 4.2.1 and naming it post-hoe, at inference (Sec. 4.2.2.. In Sec. 4.3 we provide description of how our keywords can be used to perform bias mitigation, and in Sec. 5.2 show how SaMyNa behaves in unbiased case.
4.2.1 Naming the Bias at Training Time
We begin by discussing the results of SaMyNa when applicd in model's training on Waterbirds, ColebA, and BAR.
Waterbirds. The barplot in Figure 3a shows the candidate bias keywords for Waterbirds, alongside their relative similarity value. We can observe how the obtained keywords are mainly related to the background information (forest and foliage for landbirds; ocean and beach for waterbirds). Most importantly, the top keywords display high similarity values, indicating a high correlation with their class targets. We deduce that the model suffers from bias about image backgrounds, which indeed is the case for Waterbirds. is also worth noticing how the top similarities differ among the two classes. We hypothesize two possible factors causing it: (i) model bias towards ocean is stronger than the towards forest, as it is constituted by
9

---

Published in Transactions on Machine Learning Research (11/2025)
Climbing
Diving
Throwing
climber
smia
potch
muserwater
hasehall
dimbing
diver
pllceer
climb
diving
rock
dive
mosusd
steep
player
aw0mm
degllie
Uhrow
tocky
xwinaning
athisia
Riggedaoss
wetruit
sport
11.4
((3,0) Fishing
a(k,c]
a(k,c]
Racing
Vaulting
bool
racing
vaulter
fashing
varating
river
coz
vachh
track
jump
staditen
mokaar
wider
pole
speeds
stbleticio
Wares
spred
suspended
underwsder
sepholt
diding
0.4
0:4
0.4 s(k,e)
x(4,c)
s(le.c]
Figure 4: Similarity scores for the BAR. dataset.
simpler visual patterns, casier to learn for the network; (ii) the landbirds class has a much larger population, thus allowing for the bias on this class to be averaged over more instances than the walerbirds case.
CelebA. In analyzing the possible biases of our vanilla model on CelebA (see Figure 3b), we find that the top-1 keyword for both classes represents a gender: male/man for class not blonde (with similarity ss 0.4), and woman for class blonde (similarity of As 0.4). Additionally, we find among the blonde class, keyword terms typically associated with gender stereotypes, such mascara, makeup, and lipstick, with moderately high similarity. This suggests that the vanilla model is not only biased regarding the classification task but also incorporates unfair features, arising from societal biases reflected in the training data.
BAR. Bar presents a more challenging situation, due to the complete absence of bias group annotations. Regardless, the output of our approach still provides insights about the biases captured by the model. the training class climbing, scenes where the subject is ascending on rocks are overly represented, and thus the vanilla model has wrongfully learned rely on the presence of rocky backgrounds. This is reflected by the top three keywords not related to the target class in Figure 4 (cliff, rock, and steep), all having high similarities. Similar considerations ions can be made on the other classes: pitch, baseball, pitcher, and batter suggest an even stronger bias for the class throwing towards specific sport, baseball, which is also the second most correlated keyword. The same can be said for scuba and underwater in the diving class. Keywords for other classes show slightly lower maxima. Still, we can measure relevant correlations with the presence of cars and circuit races for racing, as well as boats and water (e.g. rivers, ocean, lake) for fishing. Complete outputs of the keyword extraction process are available the Appendix.
4.2.2 Naming the Bias at Inference Time
To evaluate the capabilities of our method in describing potential biases at inference time, we design a dedicated experiment involving ImageNet-A. In particular, we are interested in finding specific model failures and extracing an interprelable set of keywords that can guide an expert practitioner to tackle them. With this aim, we first build an evaluation set as the union of the whole ImageNet-A A dataset with the samples in the validation set of ImageNet-1K sharing the same 200 categories of ImageNet-A. Then, we run the model in inference over this dataset, collecting Drorrett and missclass directly without any additional training. In not interested in finding dataset-wise biases, but rather in assessing the behavior of
this experiment,
10

---

Published in Transactions on Machine Learning Research (11/2025)
Crayfish
Rlinoceros Beetle
Stäck Insect
Cockroach
hand
Ulmbb
crsalure
mI0L
back
touch
Buprr
darkaniors
hest
crattooss:
plant
dark
branch:
plauts
Blatteeed
fond
lologge
grosehpper
garden
crustannem
sanly
stande
fiid
darker
0.2
0.3
0.4
0.2
0.3
0.4
0.2
0.3
0.4
0.2
0.3
0.4
=(k.)
=(k.e)
8(k,)
s(ke))
Figure 5: Similarity scores for the crayfish, rhinoceros beetle, stick insect and cockroach classes from ImageNet- A.
CelebA (Mair Color)
Watcrbirds
BAR
Method
Unsup-
in Biss-ld
Named
Average
Worst Group
Average
Worst Group
Average
LISA Yao et
aZ22
X
92.40
80.30
91.80
69.20
GroupDRO Sagara
2020
X
92.90 = 0.20
88.90 2.30
93.50 = 0.30
91.40 1: 1.10
George Solooni
20220
/
94.60
54.00 1: 1.90
95.70
76,20 2,00
JTT L/b ml 2021
X
88.10
81.50
1.70
89.30
83.80 + 1.20
08.53 +1.3++55 3.29
CNC Ziaang et
/
B32
32TO Kima <t
2026
x
4
93.20
90.10 0.90
91.50
90.70 0.50
68.54 0.86
47.70
LIF
Nam LaNN
al.
2000
V
8121
81.21 +: 1.38
91.20
78.00
02.96 2.76
Debr Li es dk
2012
V
84.00
52.00
4.70
69.58
2.92
SoMyNa [0075] GDRO
92.20
90.60
0.08
91.11
0.44
90.62
0.10
71. 26 1.46
Table 1: Performance of GDRO Debiasing top of our training-time bias naming pipeline. Column Unsup. indicates if the method uses ground truth bias information. Best results for unsupervised debiasing methods are highlighted in bold. No Val.Set in Bias-Id highlights if the method does not rely on validation set for inferring subgroups or bias-attributes (green tick, /) or it assumes having one (red cross, X). Bias Named indicates if the method extracts semantic names of found bias attributes. BAR lacks worst-group accuracy due the absence of bias annotations. Average refers the unbalanced test accuracy. These results on GDRO were computed by us.
our approach in specific and challenging roal-world scenarios. Hence, we derive a specific case study from the systematic model confusion obtained from its predictions, which could hide the presence of a possible model bias. Another analysis on ImageNet-A describing more general DL diagnosing features provided in Sec. 5.3 while here we limit the discussion to the model bias perspective.
Our key study (see Figure 5) involves subset of four classes: crayfish, rhinoceros beetle, stick insect, and cockroach. Samples (correctly or incorrectly) classified as crayfish are often placed in a setting depicting plates, tables, or people eating, thus the bias reflected by the keywords meal and food. Moreover, crab probably suggests the inability of the model to distinguish between the two closely related species. At the same time, for the classes stick insect and rhinoceros beetle, several samples show the creature being held in a person's hand, often in a vegetation sctting (hence the keywords hand, thumb, plants, foliage). From these observations, we validate the presence of possible bias among these classes, for which the source of error is not caused only by the fine-gra grained nature of these categories. An additional analysis involving Vision Transformers is provided in Sec. 5.1
4.3 Mitigating the Discovered Bias
Starting from the keywords that were extracted to describe potential bias affecting the classification model, we here validate if the attributes suggested by SaMyNa can be leveraged to improve our network's 's generalization capabilities.
Setup. After training-time time bias identification and naming (SaMyNa), we exploit a pre-trained CLIP model zero-shot classifier (Radford et al. 2021) to infer the alignment of each sample towards the
11

---

Published in Transactions on Machine Learning Research (11/2025)
Figure 6: Misclassified samples from crayfish (three left-most pictures), and from the insects classes (three right-most pictures). It can be seen how the contexts in which the objects of interest appear reflect the found key-keywords.
bins-keyword(s) of its target class not. As result, we obtain subgroup pseudo-labels, which can then be leveraged in state-of-the-art supervised debiasing algorithm (e.g. GroupDRO (Sagawa* et al., 2020 ). For each dataset, we employ the following approach: given training sample W)), and the set of keywords found for its class yi, we use CLIP's image and text encoders to obtain its embeddings Zimg and Eext- A bias label is then computed for each sample by just annotating if the zero-shot classification from CLIP corresponds to a bias-keyword from its class or not. Finally, we plug our set of psoudo-labels GroupDRO and measure the obtained lest performances. In this stage, we employ the same hyperparameters used in the GroupDRO original implementation for CelebA and Waterbirds. For BAR, we set the learning rate and weight decay to 5 10-4 and 10-3, respectively, with batch size of 128, Group adjustment is set to zero, and a is set to 0.5.
Results. Table outlines the obtained results. Here, we categorized existing work according to two main factors: (i) Unsupervised (Unsup.); i.e. if the method does not use ground truth bias information (V) it does (X); (ii) No Val.Set in Bias-ld, to better highlight whether bias information inference relies the usage of validation set with bias annotations. Our semantic bias discovery- based approach outperforms all the unsupervised methods. Compared to the only other method able to name biases (B2T+GDRO) our approach achieves slightly higher worst-group accuracy on average. Since our method uses GroupDRO with automatically inferred pseudo-labels, its upper bound is supervised GroupDRO with ground-truth bias labels. On CelebA, our method exceeds this upper bound due to noisy group annotations in the dataset; on Waterbirds, where labels are synthetic and accurate, we closely match it.
While +DT+ performs similarly to SaMyNa on CelebA and Waterbirds, its keywords describe the inverse of the bias (which is less interpretable), and on CelebA it detects only one of two spurious features- yet, for the binary case, this is sufficient for GroupDRO to work. Additionally, our mothod does not rely on any validation set (c.g. BAR docs not have one), and the amount of image captions we require is quite limited, thanks to the exemplar-mining step described in Section 3.2.1 whereas B2T directly captions the entire validation set (for an in-depth comparison with B2T refer Sec. D in the Appendix). This feature allows our Bias- Bias-Disy method to be employed when bias annotations if we extract held-out validation set from BAR and run G2T+ on it (see Tab. 20 in the Appendix for the extracted keywords), Notably, since both our method and B2T rely on GroupDRO for the final bias mitigation step, our advantage has to be imputed on a finer semantic bias discovery.
not present at all, as in BAR. Indeed,
measure average drop in Average Accuracy of 3% compared to SaMyNa.
5
Additional Analyses and Ablation Studies
5.1 Bias Discovery on Vision Transformer Models
We present here a study on two popular pre-trained Vision Transformers architectures: ViTb-16 and Swin-V2. Tab. reports the outcome of SaMyNa for the classes crayfish, rhinoceros beetle, stick insect, and cockroach of ImageNet-A. Despite the potential of generalization for these architectures, we are still able observe, although with different magnitudes, some biases. Regarding ViTb-16, the class crayfish is still associated with meal, and cockroach is associated with floor: interestingly, the impact of hand for the stick insect is heavily
12

---

Published in Transactions on Machine Learning Research (11/2025)
Tested architecture
Target
Keyword (value)
crayfish (0.50581), crab (0.48959), crustacean (0.37085), meal (0.35827),
Crugfish
food (0.27727), plate (0.25765), water (0.22597)
beetle (0.40866), tree (0.26703), forest (0.26600), trees (0.25613), branch (0.22754), spider (0.21616), insect (0.21564)
Rhinoceros Beetle
VrTb-16
foliage (0.40032), plants (0.36978), plant (0.36220), garden (0.33877), grass (0.29191), leaves (0.28791), trees (0.28398), forest (0.27841), flow- ers (0. (6.26615), tree (0.24779), field (0.22410), thumb (0.21469), hand (0.21322), green (0.21072), brown (0.20495)
Stick Insect
cockroach (0.31074), insects (0.27413), floor (0.26161). insect (0.25462), grasshopper (0.23879), glossy (0.23501), beetle (0.23324), darkcolored (0.23225)
Cockroach
lobster (0.60554), crab (0.49117), crayfish (0.44054), crustacean (0.40692), underwater (0.34094)
Crayfish
beetle (0.49412), butterfly (0.41977), insect (0.28676), insects (0.27957), wings (0.23552). forest (0.23534), trees (0.22520), nature (0.22299),
Swin-V2 B
Rhinoceros Beelle
grasshopper (0.21111), black (0.20005)
foliage (0.37396), plant (0.35905), garden (0.32784), leaves (0.28342), grass (0.27446), hand (0.27024), green (0.26462), trees (0.23665), touch (0.23344) lush (0.21824) forest (0.21634), stem (0.20281)
Stick Insect
Cockroach
cockroach (0.32120). floor (0.23421), uneven (0.20641)
Table 2: Testing pre-trained Vision Transformer architectures on crayfish, rhinoceros beetle, stick insect, and cockroach classes from ImageNet-A.
reduced compared to the ResNet model, while with the introduction of sliding windows, it goes back up for Swin-V2. In general, we notice that these architectures, although still suffering from bias, are less prone to it, probably due to finer training enhanced by larger parametrization combined with the self-attention mechanism they embody.
5.2 SaMyNa an Unbiased Dataset
We propose here study on a virtually balanced version of the Waterbirds datasct. Fig. 7 reports
Landbirds
Waterbirds
the results in a graphical form, while Tab. 27 the
Appendix reports the numerical values. While we should have priori removed the bias by balancing the dataset, resulting in general, massive reduc- tion of the similarity scores, we still observe some mild correlations arising, especially for the waterbirds class. Specifically, besides the seagull keyword ev- idencing a (potential) higher presence of seagulls in the data split, we still see some concepts like sea and midfli migflight correlating with the learned class. This is expected: given that these features are easy to learn, the model still captures them, but the low similarity score indicates that it does not hcavily roly on them. This shows that, despite balancing the dataset, some biased features can still permeate through the model, depending on how easy they are to capture. This further motivates our work, focusing on model debiasing rather than dataset debiasing. At the same time, the absence of highly correlated keywords for the landbird class (which naturally more variable) is somewhat. expected due to the removal of the original spurious correlation with the target.
900
midflight:
seagull
0.2
0.3 s(k.c)
0.2
0.4
s(k,e)
Figure 7: Ablation study Waterbirds, where we balance the two classes, a-priori removing the bias.
13

---

Published in Transactions on Machine Learning Research (11/2025)
5.3 Another Case Study for ImageNet- A: nails mushrooms
Besides the study provided in Sec. 1.2.2 of the main paper, we present here another case study the ImageNet-A datasct, comparing two other critical classes: nails and mushrooms. Indeed, also for this case, the tested ResNet-50 model presents a big error between these two classes: there a bias involved? Running our SaMyNa (Fig. 8) we observe that indeed there is big correlation towards certain concepts for the nails class; however, these hardly resemble biases, but rather features of the target class. Indeed, we can easily imagine that concepts like metal, frame, or rusted can be casily associated with the larget nails class. At this point, where is this big confusion arising from? The answer comes from visual inspection of the samples, wherein multiple cases the shape factor of the two classes is extremely similar (both show bulge lop and thinner body underneath), making the classification
Naila
Fungi
mesal
plants
frume:
rusted
wanden
Golinge
weathered
wall
black
orange
0.2
0.3
0.4
0.5
0.2
0.3
0.4
0.5
s(k,e)
s(k,c)
Figure 8: Similarity scores for the nails and fungi
classes from IeageNet-A.
task harder. In this case, we doduce that the model simply was unable to properly fit the two classes because of a lack of samples in the training set.
5.4 Human Feedback
To further validate whether the biases found by SaMyNa are meaningful, we have performed a human annotation study on CelebA, Waterbirds, and BAR, by surveying 20 participants. In our survey, show participants a set of images from each datasel, divided by target class (for example, blond and non-blond people, waterbirds and landbirds, and so on). We asked participants to provide sets of keywords that, in their opinion, represent a bias in the different groups. Then, we analyzed the keywords found by participants and we ranked them based on the number of occurrences. We report the results in Tab. 3 (for CelebA), Tab. (for waterbirds), and Tab. (for BAR). We report the top-5 results on SaMyNa, and all the keywords provided by the participants, which occurred least three times.
As can observe from the results, the output of SaMyNa is aligned with human annotations. For more information about the survey, see Sec. E in the Appendix.
Blond (Human)
Blond (SaMyNa)
Not blond (Human)
Not blond (SaMyNa)
Keyword Count
Keyword
Score
Keyword
Count
Keyword
Score
woman
20
woman
0.40
man
14
man
0.46
long hair
5
blonde
0.34
hat
5
male
0.39
white
4
mascara
0.25
short hair
4
white skin
4
makeup 0.25
smile
3
lipstick
0.22
Table 3: Comparison between human annotations and SaMyNa's output on the CelebA dataset.
5.5
Ablation on k
We present here the ablation on k, which selects the cardinality of aligned and conflicting samples per class. Fig. g reports the study for the most occurring kcywords the cases under exam, while the full results are later reported in Tab. 28 in the Appendix. In the general case, we observe that for lower values of the similarity score is in general lower, evidencing that the information extraction process is less accurate due to
14

---

Published in Transactions on Machine Learning Research (11/2025)
Landbird (Human)
Landbird (SaMyNa)
Waterbird (Human)
Waterbird (SaMyNa)
Keyword
Count
Keyword
Score
Keyword
Count
Keyword
Score
forest
forest
0.33
water
5
ocean
0.54
trees
foliage
0.33
grey
4
beach
0.43
green
tree
0.33
red
shore
0.39
stalks 0.31
water 0.39
branch 0.31
waves
0.33
Table 4: Comparison between human annotations and SaMyNa's output on the Waterbirds dataset.
Climbing (Human)
Climbing (SaMyNa)
Diving (Human)
Diving (SaMyNa)
Fishing (Human)
Fishing (SaMyNa)
Keyword
Count
Keyword
Score
Keyword
Count
Keyword
Score
Keyword
Count
Keyword
Score
rocks
g
climber
0.56
water
10
scuba
0.58
water
7
boat
0.
mountain
6
cliff
0.55
blue
5
underwater
0.56
fishing rod
4
fishing
belnet
6
climbing
0.52
diver
0.49
children
4
river
rock
4
climb
0.50
pool
diving
0.35
fish
4
ine
rock
0.42
man
dive
0.33
lake
3
lake
Racing (Humnn)
Racing Na)
Throwing (Human) Throwing (SaMyNa)
Vaulting (Human)
Vauiting (SaMyNa)
Keyword
Count
Keyword
Score
Keyword
Count Keyword
Score
Keyword
Count
Keyword
Score
cars
5
racing
man
pitch
0.55
sky
vaulter
0.58
car
G
cars
baseball
baseball
0.48
pole
4
vaulting
wheels
5
car
0.39
pitcher
0.48
4
vault
0.47
road
5
track 0.32
sport
4
batter
0.46
womnan
4
jump
0.43
stadium
0.29
mound
0.44
midair
0.38
Table 5: Comparison between human annotations and SaMyNa's output on the BAR dataset.
general lack of information (and variety). This trend is particularly evident for k == 1. Overall, we find that a fair compromise between performance and complexity is given by the intermediate = 10 for which the scores of bias-related keywords reach a high value with diminishing returns for higher values of k. We highlight that maintaining at bay reduces the number captions that need to be generated, which is a timo-consuming process when using large captioning models. Furthermore, the number of comparisons grows quadratically with k, but this is usually not problem since the captioning process much slower when k is reasonable value (for ls < 50 captioning is orders of magnitude slower than calculating the learned class embeddings).
5.5.1 Other Ablation Studies
We provide other ablation studies in Sec. C of the Appendix. In particular, we show an ablation on the bias mining step, an ablation on Eq. an experiment with different text embedders, and ablations on all hyperparameters.
5.5.2 Other use cases for SaMyNa
In the Appendix, we demonstrate how two additional SaMyNa features can further facilitate keyword interpretation for the end-user. We provide experimental techniques Sections and G that further automate this process.
5.6 Limitations and Biases from Pretraining
Our method assumes the usage of pre-trained captioning models to extract descriptions (and eventually keywords), potentially representing a bias the model under analysis. Even we validated our method both by critically evaluating SaMyNa's outputs and by designing a dedicated human evaluation study verify that human subjects' interpretations align with the biases highlighted by our algorithm, recognize that.
15

---

Published in Transactions on Machine Learning Research (11/2025)
k =
== 10
=
k= 50
Landbirds
Waterbirds
branches
midllihht
trees
branch
Lake
follinge
00690
tree
Water
0.0
0.2 c)
0.3
0.4
0.0
0.4
s(k,e)
Figure 9: Ablation study Waterbirds, where analyze the impact of k that selects the cohort of images to extract keywords from.
analyses cannot globally prevent the interference of unknown and undesired biases embodied in cases outside the ones we addressed in this paper. Pretrained captioning models (as every deep learning model over which direct control over data and training protocols is not available) may present biases themselves and, as such, exaggerate the presence of certain attributes or completely overlook them. While our results allow us to confidently exclude this for the examined benchmark datasets and employed models, this may be different for other use cases. Accordingly, it's worth underlining that we propose SaMyNa. as a supporting tool for human users, who hold the final decision on how interpret the algorithm's output. This is crucial, further rescarch is needed to fully understand and prevent the presence of biases in deep learning models, challenge shared by the whole community.
6 Conclusion
In this work, we presented "Say My Name" (SaMyNa), a tool designed identify and address biases within deep learning models semantically. Unlike similar methods that generate bias pseudo-labels without clear semantic information, SaMyNa offers text-based pipeline that enhances the explainability of the bias extraction process from the model.
Our approach, validated on well-known benchmarks, proved its effectiveness by both providing the woyw encoding the bias and assigning an interpretable score telling how much the model under analysis is biased to the found attribute. SaMyNa proposes itsolf not only as a post-hoc analysis tool: through its bias mining approach, it can determine the specific moment the model might be fitting a bias, for which there is principle no need for validation set mine and name the bias. SaMyNa's ambition is self-establish as a foundational tool in making deep learning models more transparent and fair, offering practical solutions for the scientific community and end-users alike.
Broader Impact Statement
This work has the polential for a positive societal impact by promoting fairness in AI systems. SaMyNa is designed to uncover and analyze biases in models, providing a clearer understanding of their decision decision-making procosses. By identifying and representing biases a semantic way, our approach, combined with state-of- the-art debiasing methods, can help mitigate unfairness and support the development of more transparent and equitable machine learning models.
16

---

Published in Transactions on Machine Learning Research (11/2025)
Acknowledgements
This work was supported in part by the French National Rescarch Agency (ANR) in the framework of the JCJC project "BANERA" under Grant ANR-24-CE23-4369, and in part by the HilPARIS Center on Data Analytics and Artificial Intelligence. We acknowledge ISCRA for awarding this project access to the LEONARDO supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CINECA (Italy).
References
Mohsan Alvi, Andrew Zisserman, and Christoffer Nellåker. Turning a blind eye: Explicit removal of biases and variation from deep neural network embeddings. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 0 0-0, 2018.
Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh. Learning de-biased representations with biased representations. In International Conference on Machine Learning (ICML), 2020a.
Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh. Learning de-biased representations with biased representations. In International Conference on Machine Learning, pp. 528 539. PMLR, 2020b.
Carlo Alberto Barbano, Benoit Dufumier, Enzo Tartaglione, Marco Grangetto, and Pictro Gori. Unbiasod supervised contrastive learning. In The Eleventh International Conference on Learning Represenlations, 2023. URT. httpp:://openrevieww. httppp///ppeeeeeeeeeeeeetrrrrrrr wwt.tt///oormmnoccccsssss11111
Steven Bird, Edward Loper, and Ewan Klein. Natural Language Processing with Python. O'Reilly Media Inc., 2009.
Wieland Brendel and Matthias Bethge. Approximating cnns with bag-of-local-feturee models works surpris- ingly well on imagenet. arXiv preprint Xxw::9044.00 2019.
Elliot Creager, Jörn-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In International Conference on Machine Learning, pp. 2189 2200. PMLR, 2021.
Moreno D' D'Incà, Elia Peruzzo, Massimiliano Mancini, Dejia Xu, Vidit Goel, Xingqian Xu, Zhangyang Wang, Humphrey Shi, and Nicu Sebe. Openbias: Open-set bias detection text-to-image generative models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 12225 12235, 2024.
Sabri Eyuboglu, Maya Varma, Khaled Saab, Jean-Benoit Delbrouck, Christopher Lee-Messor, Jared Dunnmon, James Zou, and Christopher Ré. Domino: Discovering systematic errors with cross-modal embeddings. arXiv preprint arxi::200.. 14960, 2022.
General Data Protection Regulation GDPR. General data protection regulation. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal dota and on the free movement of such data, and repealing Directive 95/46/EC, 2016.
Robert Geirhos, Jörn-Heurik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelligence, 2(11): 665-673, 2020.
Quentin Guimard, Moreno D'Tncå, Massimiliano Mancini, and Elisa. Ricci. Classifier-to-bias: Toward unsupervised automatic bias detection for visual classifiers. In Proceedings of the Computer Vision and Pollern Recognition Conference, pp. 15151 15161, 2025.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15262-15271, 2021.
17

---

Published in Transactions on Machine Learning Research (11/2025)
Youngkyu Hong and Eunho Yang. Unbiased classification through bias-contrastive and bias- bias-balanced learning. Advances in Newral Information Processing Systems, 11926449-244666 2021.
Pavel lamailov, Polina Kirichenko, Nate Gruver, and Andrew G Wilson. On feature learning the presence of spurious correlations. Advances in Neural Information Processing Systems, 35:38516 38532, 2022.
Xu Ji, Joao F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9865-9874. 2019.
Eungyeup Kim, Jihyoon Lee, and Jaegul Choo. Biaswap: Removing dataset bias with bias-tailored swapping augmentation. In Proceedings of the IEEE/CVE International Conference Compuler Vision, pp- 14992 15001, 2021.
Nayeong Kim, SEHYUN HWANG, Sungsoo Ahn, Jaesik Park, and Suha Kwak. Learning debiased classifier with biased committee. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Newral Information Processing Systems, volume 35, pp. 18403-18415. Cur- ran Associates, Inc., 2022a. URL httpps: mwwteediiinnnnnenn..e....ee neurips. cc/paper ftlles/aperr/0222/ELL. wtw.:/522228881112ff2 Paperr--reeeee----eeeeeeeeeeee
Nayeong Kim, Sehyun Hwang, Sungsoo Ahn, Jaesik Park, and Suha Kwak. Learning debiased classifier with biased committee. Advances in Neural Information Processing Systems, 35:18403 18415, 2022b.
Younghyun Kim, Sangwoo Mo, Minkyu Kim, Kyungmin Lee, Jaeho Lee, and Jinwoo Shin. Discovering and mitigating visual biases through keyword explanation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 11082-11092, 2024.
Zhiheng Li, Anthony Hoogs, and Chenliang Xu. Discover and mitigate unknown biases with debiasing alternate networks. In European Conference on Computer Vision, pp. 270-288. Springer, 2022.
Daquan Liu, Chengjiang Long, Hongpan Zhang, Hanning Yu, Xinzhi Dong, and Chunxia Xiao. Arshadowgan: Shadow generative adversarial network for augmented reality in single light scenes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8139 8148,, 2020.
Evan Z Liu, Behzad Haghgoo, Annie Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In Marina Mcila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Reseaveh, pp. 6781 6792. PMLR, 18 24 Jul 2021.
URL httpp://proceeddnng. mlr, ppress/v39//ii12222 html
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https: http:::://laaaa...ggiibb........ wwt.:ligttttbbbbbbbeeeeeee blog/
2024-01-30-Ilav--
Ziwci Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Tambiama Madiega. Artificial intelligence act. European Parliament: European Parliamentary Research
Service, 2021.
Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word repre- sentations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pp. 746-751, 2013.
Yifei Ming, Hang Yin, and Yixuan Li. On the impact of spurious correlation for out-of-distribution detection.
In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 10051-10059, 2022.
Ron Mokady, Amir Hertz, and Amit H. Bermano. Clipcap: Clip prefix for image captioning, 2021. URL https://arxin: ptrg/bbs/2111...
18

---

Published in Transactions on Machine Learning Research (11/2025)
Rémi Nahon, Van- Tam Nguyen, and Enzo Tartaglione. Mining bias-target alignment from voronoi cells. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 4946-4955, 2023.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. Advances in Newral Information Processing Systems, 33:20673 2020.
Geoff Pleiss, Tianyi Zhang, Ethan Flenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. Advances in Newral Information Processing Systems, 33:17044 17056, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL http://arxii..orrraa///////
Lorenzo Rosasco, Ernesto De Vito, Andrea Caponnetto, Michele Piana, and Alessandro Verri. Are loss functions all the same? Neural compulation, 16(5):1063-1076; 2004.
Shiori Sagawa*, Pang Wei Koh*, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020. URL https://openreviiet htt./forummid-ryyGuJJFFF...
Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine- grained robustanss in coarse-grained classification problems. Advances in Neural Information Processing Systems, 33:19339 33:19339-19852, 2020.
Enzo Tartaglione, Carlo Alberto Barbano, and Marco Grangetto. End: Entangling and discntangling deep representations for bias correction. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. -5508-13517, 2021.
Enzo Tartaglione, Francesca Gennari, Victor Quétu, and Marco Grangetto. Discntangling private classes through regularization. Neurocompulingg 554:126612, 2023.
Haohan Wang, Zexue He, and Eric P. Xing. Learning robust representations by projecting superficial statistics out. In International Conference on Learning Representations, 2019. URL htttps://openrevler. net/ forum?id=rJEjjoRR
Olivia Wiles, Isabela Albuquerque, and Sven Gowal. Discovering bugs in vision models using off-the-shelf image generation and captioning, 2023. URL http:://awxii.oooraa//////// 08831
Qizhe Xie, Zihang Dai, Yulun Du, E. Hovy, and Graham Neubig. Controllable invariance through adversarial feature learning. In NIPS, 2017.
Xiangyi Yan, Shanlin Sun, Kun Han, Thanh- Tung Lc, Haoyu Ma, Chenyu You, and Xiaolui Xic. After-sam: Adapting sam with axial fusion transformer for medical imaging segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 7975-797777 2024.
Huaxiu Yao, Yu Wang, Sai Li, Linjum Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving out-of-distribution robustness via selective augmentation. In International Conference Machine Learning,
pp.
PMLR, 2022.
Kunpeng Zhang, Liang Zhao, Chengxiang Dong, Lan Wu, and Liang Zheng. Ai-tp: Attention-based interaction-awant trajectory prediction for autonomous driving. IEEE Transcctions on Intelligent Vehicles, 8(1):73 83, 2022a.
19

---

Published in Transactions on Machine Learning Research (11/2025)
Michael Zhang, Nimit S Sohoni, Hongyang R Zhang, Chelsea Finn, and Christopher Re. Correct-n-contrast: contrastive approach for improving robustness to spurious correlations. In Kamalika Chaudhuri, Stelanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 26484-26516. PMLR, 17 23 Jul 2022b. URL ltttps://proceeddnngg mlr www.mllrrrr.... ptwssr.reessvvv22 20t22//hang22..2. .html
Yuhui Zhang, Jeff Z HaoChen, Shih-Cheng Huang, Kuan-Chieh Wang, James Zou, and Serena Yeung. Diagnosing and rectifying vision models using language. arXiv preprint arXiw:2302.04200 2023.
Zhilu Zhang and Mert R. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in Neural Information Processing Systems, 2015-Decembbe::7788887555 2018. ISSN 10495258. doi: 10.48550 /atp///axiiiiaa/// arxivv1805.07836 URL (ttp:://axiivoorr/a///8655..5..
20

---

Published in Transactions on Machine Learning Research (11/2025)
Appendix
A
Visual feedback
(a) Images classified as landhird: the bias feature is in the tree's branches.
(b) Images classified waterbind: the bias feature is the sea.
Figure 10: Bias heatmaps generated with SaMyNa using CLIP's vision encoder Radford et al. (2021 Waterbirds.
For visualization purposes only, SaMyNa can leverage visual encoder instead of a text encoder to identify the part of the image where the potential bias is located. To do this, we adopt the same strategy described in Sec. 3.2.4 to generate the learned class embeddings (c) using image embeddings generated with CLIP's visual encoder Radford et al. (2021 4 instead caption embeddings. We can then compute the cosine similarity between (c) and the embeddings of patches from the image we want to analyze. We then plot the similarity values feature is the tree for the landbird, while it is the sea for the waterbird: the model under exam does not focus on the birds but rather on the background, coherently with what we have observed in Sec. 4.2.1
heatmap overlay. Fig. 10al and Fig. 10b highlight (in red) that the most salient.
B Comparison of different captioners
We perform comparison between different captioners, notably with ClipCap (Mokady et al. 2021 it has been recently used by related works (Kim et al.] 2024). First of all, we are interested in evaluating whether ClipCap can be used to detect biases on CelebA, and how it compares to LLaVA-34b. For this, we report in Tab. c the percentage of captions that contain keywords related to CelebA's attributes. As can seen, ClipCap detects these concepts very rarely, except for "man", which detected 10% of the images. LLaVA-34b, on the other hand, detects these keywords much more frequently, for example dotects "man" in 22% of the images, while "woman" is detected in 71.5% of the images (totaling 93.5% detection for gender related keyword, compared to ClipCap's 11.47%). This is consequence of ClipCap's short captions, that cannot capture enough information from the images. ClipCap's captions are 9 words long average, while LLa VA 's are 137 words long on average. Finally, we show a qualitative comparison of ClipCap's and LaVA's 's captions in Tab. and Tab.
21

---

Published in Transactions on Machine Learning Research (11/2025)
Keywords
ClipCap
LLaVA 34b
old/oldest
0.04%
2.00%
middle-age/
haadlll-ogmmmidll--ggiee middle-aged
0.00%
4.50%
young/ youngest
0.86%
10.0%
blond plond/blonds /llodde
0.02%
49.0%
1.89%
58.0%
tic
0.05%
2.00%
eyeglasses
0.69%
3.00%
beard mustache makeup man/mal
1.70%
4.50%
0.25%
3.50%
1.27%
61.0%
9.26%
22.0%
woman/femaal
2.21%
71.5%
hat hat/hats
0.84%
2.00%
earring/ carring/earrings earrings necklace
0.02%
14.5%
0.02%
8.50%
Table 6: The table shows the percentage of captions that contain keywords related to CelebA attributes, both for ClipCap and LLaVA-34B. As can be seen, LLaVA-34B is able to capture much more attributes from CelebA's 's images. Thus, ClipCap is not suitable for detecting biases other than "man" on CelebA.
22

---

Published in Transactions on Machine Learning Research (11/2025)
Image
ClipCap
LLaVA-34B
actor with a weight of kg and a feet size of in favorite outfit & clothing style.
The image close-up portrait of a woman with fair skin and light-colored hair. Her hair is styled in sidc-swept manner, with some strands falling over her forehead. She has a neutral expres- sion on her face and is looking directly at the camera. The woman is wearing makeup, including eyeliner and lipstick, which accentuates her features. She has small carring visible in her left car. The background of the image is plain and does not provide any additional con- text about the location or setting, The lighting on the subject is soft and even, highlighting her facial features without creating harsh shadows. The style of the image suggests it could be from red carpet event formal photoshoot, given the subject's 's attire and the pro- fessional quality of the photograph. The image is a portrait-style photo- graph of a man. He appears to be middle-aged with short, dark hair and a fair complexion. The man is wearing a patterned shirt with horizontal stripes in various colors, including red, white, and black. He has a slight smile and is looking directly at the camera. The background is a solid, light blue color, providing a neutral backdrop that con- trasts with the subject. There are no visible texts distinguishing marks on the image that provide additional con- text or information about the location or the identity of the man. The style of the image is straight to be a standard headshot or portrait commonly used for identification pro- motional purposes.
actor was a popular actor and come- dian.
and seems
Table 7: Comparison of captions generated by ClipCap and VA-34B on CelebA. The images to caption were selected randomly.
23

---

Published in Transactions on Machine Learning Research (11/2025)
Image
CappCap
LLaVA-34B
actor is the face of beauty.
The image is close-up portrait of woman with a soft, natural appearance. She has long, wavy blonde hair that falls over her shoulders. Her facial features are symmetrical, with a straight nose, full lips, and clear skin, Her eyes are a light color, possibly blue green, and she is looking directly at the camera. with gentle expression. The woman's 's makeup is subtle, enhancing her natural beauty without appearing overly dra- matic. The background of the image is solid, neutral color, providing a clean and uncluttered backdrop that allows the focus to remain the woman. The lighting in the photograph is soft and even, which contributes to the gentle and approachable look of the subject. There are no visible texts or logos the image, and the style of the photo- graph suggests it could be used for riety of purposes, such headshot, beauty advertisement, or personal portrait. The image does not provide any additional context about the location or setting, as the focus is solcly on the woman's face and upper body.
professional
actor with her hair in bob.
The image close-up portrait of a woman with smiling expression. She has short, dark hair and is wearing makeup that includes dark eyeliner and cycshadow, as well as what appears to be a light foundation or powder on her face. Her lips are slightly parted, and she is looking directly the camera. The woman is wearing black garment that is not fully visible in the frame. The background is blurred, but it sug- gests an indoor setting with architec- tural features such as arches and what might be a stone or brick wall. The lighting on the subject is bright, high- lighting her features and the contours of her face. The style of the image is standard portrait with a focus on the subject's face and expression. There are no visible texts or logos in the image.
Table 8: Comparison of captions generated by ClipCap and LLaVA-34B on ColobA. The images to caption were selected randomly.
24

---

Published in Transactions on Machine Learning Research (11/2025)
C Ablation Study
C.1 Ablations on the Bias Mining step
Allgoed Conflicting
Allyned
Aligard Conflicting
Contieting
Randoms
Randon
Randors Couns
Dueably (80
Dranly
SaRtaax Outper
Beftmaa Ouippt
Softmax Ouripet
Figure 11: Output distributions Waterbirds target class from the ResNet-50 trained Waterbirds at early stages (epoch 1, left), extraction time t* (epoch 6, center) and in the final stage (epoch 10, right).
In this section, we provide visualizations on the output distributions on the Walerbirds targel when training a ResNet-50 on Waterbirds in the same setup as described in Sec. 4.1 Fig. II proposes visualizations of the output distributions for the bias-target aligned samples in blue (waterbirds and sea landscape), and bias-target conflicting samples in orange (waterbirds and ground landscape), in three different moments of the training: in the early stages (at t 1, on the left, where is the training epoch), at the chosen bias extraction time t" (t 6, at the center) and in the final stages (t = 10, the right). We recall that the entire bias extraction process happens directly on the training set Dirain and does not require the employment of a validation/ set.. We observe here that, at extraction time, there is an evident separation between bias-aligned and confiictn samples, and we are able to confidently isolate the most biased among the conflicting samples (the orange distribution having a larger population below the random guess threshold). This does not hold in case the extraction time is delayed, with the bias-conflicting distribution not exhibiting peak anymore.
C.2 Ablation on Bias Naming
C.2.1 Ablation of Eq.
To demonstrate the effectiveness Eq. 4 in filtering out the shared information across all dataset we did an ablation study on the CelebA dataset. Fig. shows the top-9 keywords for each class on CelebA when Eq. [ is removed from method. As be scen, the ranking is full of irrelevant keywords that can be used to describe just about any image in the dalaset, like portrait, photo, image, and headshot.
C.2.2 Ablation on Text Embedders
In this subsection, we are interested in testing SaMyNa with more diverse models for the textual embedding, in an attempt to check the generality of the proposed approach. We provide, in Tab. g the results obtained with five other popular textual embedding models. From our results, we can clearly see that when employing any of the tested models, we are able find back the two typical biases from waterbirds. All models produce similar outputs, except for DistilRoberta and NOMIC, that produce very few, albeit correct, keywords for Landbirds.
25

---

Published in Transactions on Machine Learning Research (11/2025)
Not blonde
Blonde
portrait
portrait
photograph
portraitstyle
photo
photograph
portraitstyle
photo
image
headshot
headshot
image
photography
photography
graphic-
makeup
complexion
complexion
0.2
0.4
0.6
0.2
0.4
0.6
s(k,c)
s(k,e)
Figure 12: Similarity scores for the CelebA dataset when ablating Eq- from method.
Embedding Model
Target
Keyword (value)
Landbirds
tree (0.22477)
DistilRoberta
occan (0.41353), shore (0.37904), beach (0.37184), boat (0.35631), water (0.28810), river (0.28312), waves (0. (0.28209). sandy (0.25046), seagull (0.23950), lake (0.22972), across (0.21785), foam (0.20274)
Waterbirds
tree (0.35688), forest (0.33959), trees (0.31249), foliage (0.27917), branches (0.25548), vegetation (0.25477), branch (0.25477), leaves (0.21215)
Landbirds
AL-MiniLM-L12---
ocean (0.40874), boat (0.35452), shore (0.31332), waves (0.27757), beach (0.27478), seagull (0.23625), lake (0.22088)
Waterbirds
tree (0.38503), forest (0.31970), trees (0.31858), foliage (0.26440),
Landbirds
branches (0.25911), branch (0.23637), bird (0.22693). bamboo (0.22661), perched (0.20324)
MPNET
waves (0.34030), ocean (0.33423), beach (0.31522), shore (0.26980), water (0.22304)
Waterbirds
Landbirds
forest (0.22448), branches (0.21085)
NOMIC
waves (0.30429). ocean (0.29857), beach (0.26858), water (0.23373), boat (0.21911), shore (0.21033), lake (0.20263)
Waterbirds
trees (0.32898), tree (0.32710), foliage (0.31355), forest (0.30978), branch (0.29175), branches (0.29141), vegetation (0. (0.28953), bamboo (0.27675). stalks (0.23657)
Landbirds
AIBERT
ocean (0.49492), beach (0.47047), waves (0.42517), water (0.38388), shore (0.35404), boat (0.29732), foam (0.26902), lake (0.26790) sandy (0.25054), surface (0.24005), midflight (0.21686), river (0.21444)
Waterbirds
Table 9: Ablation study on Waterbirds, where we analyze the impact of employing a diverse encoder for SaMyNa.
C.2.3 Ablation on fmin
We propose here, in Table 10 the results we obtain by varying the value of the hyperparameter fmin: i.c. the minimum fraction of captions from examples of the same predicted class in which keyword must appear that it be considered a possible output. As fmin increases, each set of selected keywords is
26

---

Published in Transactions on Machine Learning Research (11/2025)
subset of the previous one, with all keywords maintaining identical scores. When fuin is too high, one class will be wrongfully marked not holding bias (with smin = 0.60). On the other hand, when not employing filtering at all (fmin = 0), lot of more fine- inn-nnrined classes like coniferous or wilderness arise. We chose a low threshold (fmin = 0.15) to filter out just the least frequent words that may come from mistakes of the captioner or from details that appear in just few images. Filtering out too many words with this method may remove important synonyms from the ranking, so we avoid it.
Imin
Jmin
Juin 0.15
Landbinds
Waterbinde
Landbirals
Waterbirds
Landbirts
Waterbrinss
shrubs
0.35
sea
0.54
shrubs
ocean beach
forest
0.33
ocean
0.54
Torest
0.33
ocean
0.54
forest.
foliage
0.33
beach
0.43
twips
follage
tree
deciduous
0.33
0.33
0.32
0.33
harbor
aquatic
beach
ships
0.43
0.39
0.14
0.40
dociduous
foliage
0.32
shoreline
0.38
0.39
stalks
branch
tree
0.33
0.31
shore
waves
0.39
0.39
0.33
stalks
0.31
coastal
0.35
stalks
0.31
shore
0.39
trees
0.30
boat
0.29
branch
0.39
branch
0.31
0.33
branches
0.30
river
0.27
Jorested
0.31
shoreline
0,38
0.30
0.30
vegetation
0.27
midflight
0.24
trees
0.30
pier
0.37
branches
0.30
boat
0.29
lake
0.24
bushes
brunches
0.30
ship
constal
0.37
0.35
vegetation
grasses
0.24
0.27
midflight
river
0.27
0.24
flying
0.22
wooded
0.29
0.33
lake
0.24
vegetation
0.27
suilboat
0.32
flying
0.22
words
0.26
constline
0.32
watermark
0.20
grasses
sail
0.30
gromerr
0.23
0.30
wilderness
0.22
vessel
0.30
stalk
0.22
floating
0.29
contferous
0.22
boat
0.29
garden
speedboot
bark
0.20
wetsuit
0.29
grbblsh
0.20
pond
0.29
swimming
0.28
lagoon
bay
0.27
splash
0.27
river
0.27
surfboard
0.27
wake
0.26
surfer
0.26
glides
0.25
surfing
0.25
dock
0.25
pool
0.25
midfigght
0.24
0.24
lying
0.22
sand
0.21
watermark
0.20
fuin 0.30
fmin = 0.45
Smin 0.60
Landbirds
Waterbirds
Landbirds
Waterbirds
Landbirds Waterbirds
forest
0.33
ocean
0.54
trees
0.30
water
0.39
water 0.39
branch
0.31
water
0.39
trees
0.30
waves
0.33
Table Ablation study on Waterbirds, where we analyze the impact of the threshold for the frequency for the found keywords (fmin).
C.2.4 Ablation on tsim
In this ablation study, we provide an example of the keywords resulting from SaMyNa when not employing any thresholding the minimum similarity values for the found keywords. In Table 11 we provide the full
27

---

Published in Transactions on Machine Learning Research (11/2025)
output only for landbinds, due to the excessive length of the unfiltered output. However, this is not issue because SaMyNa, when applied to binary classification, possesses mathematical property that guarantees symmetrical keyword rankings for the two classes. Consequently, the keyword ranking for waterbirds is simply the inverse of the ranking for landbirds (the cosine similarity scores are the same but with opposite sign). Nevertheless, we provide the full list the resulting keywords as a lext file, available the supplementary materials zip archive (keywords keyyordssablationn _allaaaall_aaeeeee ptt.ttsimtxxtt.... .xt). From an analysis of the emerging keywords, we observe three interesting intervals of values. High similarity values indicate those concepts that correlate well specifically with the learned class and are those presented in the paper. Concepts whose similarity is close to zero are not correlated: indeed, we can find keywords like possibly, various, and open that are neutral concepts. Interestingly, we can also identify concepts like background and environment that are super-classes of the two biases. This confirms that SaMyNa works properly since it puts itself in the best spot to best discriminate the two biases. Finally, the third region for negatively correlated concepts (anti-corrclated), where we easily find the concepts correlated with the other class (waterbirds) given that we are in binary classification task.
C.2.5 Ablation fmin. feim. and k
In this ablation study, we show the compound effect of the Smin: fsim, and K hyperparameters. By sctting fiin = and fsim 1, we effectively disable filtering. We also set k 50 since it the maximum value of k tried, and will produce more captions, and, by extension, more keywords. Tab. 12 shows the results for both classes of Waterbirds. Due to space constraints, we show only the top keywords. The total number of keywords ranked for cach class is 1537 and the full result can be consulted in the supplementary matcrials zip archive (keywords_able ablation____..... all_ www.hpyrrrasa...... txt).
28

---

Published in Transactions on Machine Learning Research (11/2025)
Use of fism
Keyword (value)
forest (0.33454), foliage (0.33412); tree (0.32720), stalks (0.31389), branch (0.31242). trees (0.29919), branches (0.29649), vegetation (0.27349), leaves (0.19398), bamboo (0.16775),
brown (0.16085), small (0.15988), black (0.15947), stands (0.15171), green (0.15048), gray (0.14847), nature (0.1 (0.14642), yellow (0.14465), perched (0.14381) habitat (0.14101), posi- tioned (0.12863), corner (0. 12602,, position (0.12585), style (0.12585). standing (0.12558),
color (0. (0.12359), bird (0.12310), natural (0.12179), webbed (0.11827), naturalistic (0.11359),
lush (0.11342), darker (0.11082). dark (0.10989), colors (0.10947), features (0.10804),
slightly (0.10770), texts (0.10516), photography (0.10478), markings (0.10345), light- ing (0.10088), plumage (0.09747), shades (0.09703), setting (0.09479), camera (0.08995),
slender (0.08957); soft (0.08622), feathers (0.08274), beak (0.08256), pointed (0.08064),
facing (0.08016), center (0.07854), feet (0.07604), area (0.00895), vibrant (0.06787),
short (0.06720), frame (0.06472), depicts (0.06152). appears (0.06135), central (0.06021),
background (0.06013), ground (0.05912), creating (0.05908), photograph (0.05901), el-
ements (0. (0.05896), side (0.05860), located (0.05815), main (0.05688), subject (0.05661),
top (0. (0.05)37) snapshot (0.05501), looking (0.05431), marks (0.05423), focal (0.05365),
eyes (0. (0.05237), suggests (0.05027), composite (0.04975), eye (0.04772), lighter (0.04710),
presence (0.04675), create (0.04601), tall (0.04574), turned (0.04134), perspective (0.03960),
indicating (0.03954), calm (0.03905), predominantly (0.03879), suggesting (0.03870), path (0.03797). thin (0.03732). light (0.03667), moment (0.03628), left (0.03577). touch (0.03549), surrounding (0.03547), wide (0.08536), lower (0.03233), featuring (0.03144); distinctive (0.03032), landscape (0.02956), onto (0.02917), red (0.02857), consists (0.02824), foreground (0.02821), bottom (0.02585), scale (0.02508), sense (0.02173), context (0.02116), focus (0.02088), passibly (0.02039), capturing (0.01853), around (0.01757), legs (0.01733), visible
fsim = -1 (Landbirds)
(0.01702), day (0.01428), visually (0.01370), composition (0.01359), point (0.01284), scene
(0.01166). might (0.01119), making (0.01009). blue (0.01004), distinguishing (0.01002), reflecting (0.00721), (0.00713), amidst (0.00643), open (0.00581), objects (0.00573), pro- viding (0.00562), quality (0.00505), towards (0.00415), image (0.00329), taking (0.00250), captured (0.00087), buildings (-0.00146), viewer (-0.00204), peaceful (-0.00231), taken (-0.00423), wings (-0.00644), various (-0.00649), enviromment (-0.00728), movement (- 0.00924), view (-0.00953), head (-0.01000), scattered (-0.01045), provide (-0.01047). overall (-0.01051), fully (-0.01158), detail (-0.01264), captures (-0.01321). near (-0.01394), likely (-0.01408), tranquil (-0.01455), appealing (-0.01471), drawing (-0.01530), body (-0.01553), expanse (-0.01751), could (-0.01791), backdrop (-0.01856), large (-0.02118), immediately (-0.02224), white (-0.02302), right (-0.02431), adding (-0.02552). realistic (-0.03162), deep
(-0.03255), surroundings (-0.03269), otherwise (-0.03385), attention (-0.03709), beauti-
fully (-0.03766), relative (-0.03802), beautiful (-0.03860). sunny (-0.04031), tranquility (-0.04087), freedom (-0.04306), contrast (-0.04520), also (-0.04593), sky (-0.04655), across
(-0.04660), blend (-0.04802). extended (-0.04905). depth (-0.05085), juxtaposes (-0.05146), dense (-0.05250). adds (-0.06059), two (-0.07112), clouds (-0.07306), dynamic (-0.07317), spread (-0.07693), contrasting (-0.07756), surface (-0.07825), harmonious (-0.08196), serene (-0.08363), world (-0.10627), distance (-0.10900), moving (-0.11027), mix (-0.11326), filled
(-0.12270), soars (-0.13097), overcast (-0.14479), flight (-0.14884), sandy (-0.14945), clear
(-0.16600), seagull (-0.18060), foam (-0.19652), flying (-0.21832), lake (-0. (-0.24335), midflight
(-0.24404), river (-0.26848), boat (-0.29352), waves (-0.33462), water (-0.38571), shore
(-0.38639), beach (-0.43417), ocean (-0.53961)
Table 11: Ablation study on Waterbirds, where analyze the impact of the threshold for the similarity score (fsim) for the class landbirds.
29

---

Published in Transactions on Machine Learning Research (11/2025)
Class
Keyword (value)
tree (0.35786), forest (0.35347), shrubbery (0.34675), shrubs (0.34670), foliage (0.34207),
leaf (0.33083), twigs (0.32846). deciduous (0.32429). trees (0.32224), stalks (0.31635).
branch (0.31585), plants (0.31248), forested (0.31134), plant (0.30522), bushes (0.29749),
wooded (0.29437), branches (0.29422), planted (0.27602), woodpecker (0.26986), woods (0.20887), vegetation (0.26780), pine (0.26381), weeds (0.25234), leaves (0.23516), grasses (0.23446), flora (0.22909), owl (0.22551), twig (0.22138), garden (0.21504), bark (0.21230), stalk (0.20237), greenishbrown (0.20123), coniferous (0.20117), grayishbrown (0.19607), greenishgray (9.19218), wood (0.18801); greenery (0.18465), wilderness (0.18064), lily (0.17892), yellowishgreen (0.17775), greenish (0.16826), yellowishbroww (0.16707), porch (0.16707), stripe (0.16603), bamboo (0.16412), reddishbrown (0.16392), trunk (0.16291);
fence (0.16257), moss (0.15525), grayishblack (0.15461), flower (0.15428), brown (0.14858). claws (0.14673)) songbird (0.14653), greemishhlue (0.14383), squirrels (0.14337), gray- ish (0. (0.14330), trail (0.14072), green (0.13819), wooden (0.13784), striped (0.13633).
Landbirds
grass (0.13416), sparrow (3303)) nest (0.13205), pouch (0.12906), nesting (0.12292), smokestacks (0.12155), canopy (0.11981), species (0.11944), patterned (0.11719), gray (0.11595), grey (0.1 1442), flowers (0.11391) darkcolored (0.11316). bluebird (0.11249). curved (0.11216), row (0.10993), cracks (0. (0.10955), plaid (0.10909), wires (0.10906), line (0.10826). black (0.10768), cutting (0.10540), potted (0.10506), border (0.10473), crouched (0.10454). foot (0.10222), pole (0.10100), bird (0.10011), handlebars (0.09951), reeds (0.09944), bordered (0.09866), standup (0.09824), gannet (0.09747), yellow (0.09704),
crow (0.09696), shaded (0.09692), beige (0.09642), neck (0.09496), eagle (0.09413), corner (0.09216), gate (0.09152), blade (0.09038), blues (0.09002), ring (0.08994). lined (0.08993)
sea (0.58878), ocean (0.57950), beach (0.53000), beachgoers (0.48649), tide (0.48087), shore (0.47279), shoreline (0.46738), aquatic (0.46162), coastal (0. (0.44221), harbor (0.43874), maritime (0.42833), pier (0.41840), coastline (0.40540), marina (0.40293). submarine (0.40068), water (0.40000), ships (0.39669), ship (0.37869), waterfront (0.36526), waves (0.36266), boats (0.35816), beachwear (0.35474), lagoon (0.35376), bikini (0.34825), sub- merged (0.34601), naval (0.33164), floating (0.33144), pond (0.32841), wave (0.32569), bay (0.32533), wet (0.32244), wetauit (0.31642), swim (0.31542), sailboat (0.31461), crab (0.31139), boat (0.31050), sail (0.30284), wake (0.30191), bathing (0.30030), swimming (U.29998), sand (0.29526), lake (0.29238), fish (0:29191), river (0.28856). vessel (U.28363). dock (0.27980), surfboard (0.27760), pool (0.27661). splash (0.26832), sinking (0.26343). liquid (0.26182), watermark (0.26116), surfer (0.26046), midflight (0.25716), speedboat (0.25574), coral (0.24456), cranes (0.24285), docked (0.24119), surfing (0.24079), flows
Waterbirds
(0.23242), jetty (0.22524), flying (0.22223), sandy (0.21984), island (0.21959), lakeside
(0.21866), glides (0.21866), diving (0.21596), foam (0.21396). skyline (0.20477), horizon
(0.19709), seagull (0.19641), mirror (0.19333), mirrorlike (0.19307), stunning (0.19107); mirrors (8.18967), inland (8.18)22), pontvie (0.18885), soaring (0.18690), picturesque (U.18462). dive (U.18419), meal (U.18359), serene 18.15)55) paddleboarder (0.17829),
terfall (0.17474). flowing (0.17394), cloud (0.17373), panoramic (0.17271), paddleboard (U.16915). paddlchoarding (U. (0.16890). rocky (U.16798), clear (0.16612), gliding (0.16549).
surface (0.16489), ripples (0.16461), sunset (0.16234), wading (0.16154), bathed (0.15626), waterfowl (0. (5250),, multiple (0.15204) overcast (0.15028), whimsy (0.14977), paddle (0.14693), stream (0.14617), buoy (0.14510), dam (0.14358), reality (0.14326)
Table 12: Ablation study on Waterbirds, where we analyze simultaneously the impact of faims smins and K. Filtering is disabled by setting tsim = -1 and fmnn = O. We use k = 50 to generate more captions and, as a consequence, more keywords.
30

---

Published in Transactions on Machine Learning Research (11/2025)
Method
Captioning Time (LLaVA 34B)
SaMyNa (k=1)
17 minutes
SaMyNa (k-5)
86 minutes
SaMyNa (k=10)
3 hours
SaMyNa (k=25)
7 hours
SaMyNa (k=50)
14 hours
B2T
60 days
Table 13: Comparison of captioning runtime between SaMyNa (for different values of k) and B2T using LLa on CelebA. B2T must. caption the whole validation set of CelebA, which is composed of about. 19k images, while SaMyNa uses bias-mining to select sample of t C * 2 images, where C is the number of classes. By default, SaMyNa uses k = 10, for a total of 40 samples on CelebA.
Comparison with B2T
In this section, we provide more in-depth comparison with the B2T algorithm proposed by Kim al. (2024). B2T relevant to our work, as shares the same goal of extracting iuman-interpretable descriptions of potential biases affecting visual models. The key differences between SaMyNa. and B2T can be summarized as follows:
SaMyNa does not require validation set for bias discovery, in contrast to B2T;
SaMyNa can extract few candidate exemplars directly from the training set thanks to the Bias Mining step.
This represents a key aspect in unsupervised bias discovery and mitigation, as in a realistic scenario a validation set comprising conflicting samples is rarely available (like BAR or BFFHQ). Besides, B2T requires captioning the entire validation in order to extract relevant biases from conflicting samples. In contrast, SaMyNa is much more efficient and allows for the usage of larger and more accurate captioners such as LLa Va-34B (see Sec. B for a comparison between ClipCap and LLa VA-34B))).
Why B2T cannot leverage better captioners while SaMyNa can. The quality of extracted keywords directly depends on the quality of the captioner. B2T is forced to employ smaller and quicker captioners such as ClipCap, which, however, provides less accurate captions when compared to models such as LLa VA-34B. Tab. 13 shows the time required for SaMyNa and B2T on the CelebA dataset using LLaVA-34B on an NVIDIA A40 equipped with 48 GB, tested on batch size 5.
Why B2T requires full validation set and SaMyNa does not. To showcase that B2T requires large enough validation set to extract accurate keywords, we compare the keywords extracted by SaMyNa and B2T with varying sample sizes. The results are presented in Tab. 14 We highlight cases in which the relevant keyword was ranked higher than the other method. The results clearly show that our method, which leverages the bias mining step, is consistently more accurate than B2T in extracting the right keywords. Furthermore, keep in mind that B2T keywords need be inverted as reported the original paper (e.g. woman - man), thus for k=1, B2T actually predicts the opposite bias. This is straightforward for binary attribute such ColebA's gender, but not obvious when more than two biases present in the training set. For completeness of results, report the full ranking of both algorithms for all values of k, in Tab. 15 (k=1), Tab. 16 (k=5), Tab. 17 (k=10),
Tab. 18 (k=25), and Tab. 19 (k=50).
Why B2T cannot use a held-out validation set. One could ask why B2T is not applicable if a validation set is missing, because one could generate a held-out validation set from the training set. However, this is not as simple it scems: B2T on a held-out validation lack of a bias mining step that ensures that misclassified examples can be found in the validation set for all classes. Without stopping the training at the right epoch, the model will start to generalize and will make
is not guarantced
work bocause of the
31

---

Published in Transactions on Machine Learning Research (11/2025)
Class
Method
Expected
K=1
K=5
K=10
K=25
K=50
Blond
B2T
man
woman (6th)
N/A
N/A
N/A
N/A
SaMyNa
N/A
woman (6th)
woman (1st)
woman (5th)
woman (1st)
Not Blond
B2T
woman
N/A
woman (5th)
woman (3rd)
woman (5th)
woman (4th)
SaMyNa
man
male (Ist)
male (1st)
man (1st)
man (1st)
man (1st)
Comparison between SaMyNa and B2T using the same captioner (LLaVA-34B) and using an equally sized subsample of CelebA's 's validation set. The number of sample images is K * 4. For B2T selected K random images for the correctly classified, and K for the incorroctly classified examples of cach class. For SaMyNa, we use our subsampling algorithm. B2T's expected answer is the opposite of SaMyNa's 's. We show the position in the ranking of gender keywords. For both algorithms, the default filtering method is used, except for SaMyNa, where we don't filter the target class since it's also not filtered by B2T. As can be seen, SaMyNa detects the bias for "not blond" every time, while for the "not blond" class, it fails only for K = B2T, on the other hand, detects the bias for the "not blond" class times out of 5 with worse ranking positions than SaMyNa, while for the "blond" class, it never detects the bias, and for K 1, its answer is the opposite of the expected answer.
Table 14:
Blond (B2T)
Blond (SaMyNaa
Not bloud (B2T)
Not blond (SaMyNa)
Keyword
CLIP Score
Keyword
Cosine Similarity
Keyword
CLIP Score
Keyword
Cosine Similarity
wavy
1.844
earrings
0.27081
blonde hair
1.594
male
0.32069
fair complexion
1.781
makeup
0,26793
grass
0.24336
close-up
1.609
blonde
0.21713
subject
0.23409
close-up photograph
1.5625
lipstick
0.20272
environment
0.21606
neutral expression
1.375
eyeshadow
0.20149
camera
0.21347
1.078
capturing
0.21132
complexion
0.7344
mond
0.20288
lips slightly
0.4688
complexion and long
0.375
lips slightly parted hair
0.375
0.2812
Table 15: Full keyword rankings of B2T and SaMyNa for the results shown in Tab. 14 with K 1. As can be scen, while SaMyNa fails for the "blond" class, it detects keywords that still make sense like "makoup", "carrings", and "lipstick", which are usually associated with women. While B2T fails in worse way, since its answer is the polar opposite of the expected one (it. should answer male). B2T also fails for the "not blond" class.
fewer classification errors. To show this, we test B2T on BAR, which is a dataset that lacks a validation set. Furthermore, BAR interesting because has 6 classes. As previously noted, B2T generates keywords that represent the opposite of the bias, which is easy to interpret in the binary classification case (gender in CelebA), but it loses interpretability in datasets like BAR that have more than 2 classes. For the experiment, extract a held-out validation split by stratifying with respect to the known class population distributions. The held-out validation set is composed of 195 examples (10% of the training set). The results are shown in Tab. 20 As can be seen, the results for the "Climbing" class are missing because no misclassified examples were found in the held-out validation set, which proves our point. As highlighted in the table's caption, the results for the rest of the classes are mostly wrong, probably due to the low number of misclassified examples. Our method achieves good results, as shown in Tab. 26 Furthermore, we achieved these results without relying validation set and by using only 120 samples (K=10) opposed to the 195 examples extracted for B2T.
32

---

Published in Transactions on Machine Learning Research (11/2025)
Blond (B2T)
Blond (SaMyNa)
Not blond (B2T)
Not blond (SaMy
Keyword
CLIP Soore
Keyword
Cosine Similarity
Keyword
CLIP Score
Keyword
Cosine Similarity
provide additional context
0.5938
blonde
0.38934
wavy blonde hair
4.734
male
0.30272
provide
0.5625
makeup
0.34895
wavy blonde
3.922
0.27394
distinguishing
0.33196
blonde
3.719
shirt
0.20739
additional context
0.48444
lipstick
0.31817
blonde hair
3.64
context
0146888
eyeliner
0.31324
1.844
texts distinguishing marks
0.3125
woman
0.29510
wearing makeup
1.3125
0.2656
eyeshadow
0.24667
eyeliner
1.297
additional
0,1562
shadows
0,21538
eyess
1.125
provide additional
0.1406
hair
0,23935
makeup
1.078
style
0,03125
face
0.22412
camers
0,9062
head
0.20731
expression
0.703
styled
0.20134
long
0.578
directly
0.5156
wearing
0.2188
hair
0.0
Table 16: Full keyword rankings of B2T and SaMyNa for the results shown in Tab. 14 with K = 5.
Blond (B2T)
Blond (SaMyNa)
Not blond (B2T)
Not blond (SaMyNa)
Keyword
CLIP Score
Keyword Cosine Similarity
Keyword
CLIP Score
Keyword
Cosine Similarity
eyeliner
0.9688
woman
0.39947
wavy blonde hair
3.594
man male
0.45820
wearing makeup
0.4844
blonde
0.34301
blonde hair
2.938
0.39175
photograph
0.03125
mascara
0.24959
woman
makeup
0.24877
light-colored hair
1.359
lipstick 0.22272
close-up 0.01563
eyeliner
0.20065
Table 17: Full keyword rankings of B2T and SaMyNa for the results shown in Tab.
with 10.
Blond (B2T)
Blond (SaMyNa)
Not blond (B2T)
Not blond (SaMyNa)
Keyword
CLIP Score
Keyword
Cosine Similarity
Keyword
CLIP Score
Keyword
Cosine Similarity
visible texts
blonde
0.45956
wavy blonde hair
3.766
man
0.40512
close-up photograph
0.7656
makeup
0.32295
blonde
provide additional context
0.6406
muscara
0.32257
blonde hair
3.438
directly additional context.
0.5938
lipstick
0.30535
blonde hair styled
3.438
0.5156
0.30441
woman
1.594
portrait-style photograph
0.4062
eyeliner
0.24918
wearing makeup
0.797
fair face
0.4062
hair 0.22028
makeup
0.672
0.1875
eyeshadow
0.20621
fair skin
0.3906
eyes
0.2812
hair styled
0.2344
styled
0.2344
portrait
0.2188
Table 18: Full keyword rankings of B2T and SaMyNa for the results shown in Tab. 14 with K 25.
33

---

Published in Transactions on Machine Learning Research (11/2025)
Blond (B2T)
Blond (SaMyNa)
Not blond (B2T)
Not blond (SaMyNa)
Keyword
CLIP Snore
Keyword
Cosine Similarity
Keyword
CL.P Score
Keyword
Cosine Similarity
close-up photograph
0.8438
woman
0.42156
wavy blonde
4.0
man shirt
0.47463
visible texts
0.797
blonde
0.39872
blonde hair
3.5
0,20972
photograph
0.6875
masrara
0.29177 77
blonde
3.406
expresssoo
0.5
lipstick
0.29163
woman
1.375
additional context
0.4531
makeup
0.264433
wearing makeup
0.7188
provide additional context
0.4375
makeup
0.5625
visible
0.3594
close-up
0.1875
person
0,2812
eyeliner
0.1719
texts or distinguishing
0.2656
fair
0.1562
provide additional
0.25
hair
0.0781
distinguishing marks
0.1875
style
0,1875
wearing face
0.1400
0.03125
Table 19: Full keyword rankings of B2T and SaMyNa for the results shown in Tab. 14 with k 50.
34

---

Clinbing
Diving
Fishing
Racing
Throuiing
Vaulking
Published in Transaction tons on Machine Learning Research (11 (11 (202))
Keyword
CLIP Score
Keyword
CLIP Score
Keyword
CLI Score
Keyword
CLIP Seore
Keyword
Score
Keyword
CLP Score
emply (crashed)
N/A
soldiers jump
boy fishing
3.
1.656
frienda playing football
emply filtered)
N/A
trampoline swimners jump
4.72
beach
3.11
track
1.594
american football
7.188
4.61
hoy
1.828
mee
1.484
american foothall soun
6.547
young uaa jumping
fishing
0.4377
notoreycle
1:219
playing football
jumpe
3.969
fishing
0.375
practive
0.6406
foothall
5.17
mas jumping
3,969
lake
0.2812
leads the field
0.625
football team
5.08
pool
3.344
racecar driver leads
0.2344
friends playing
3.844
jump
3.328
driver leads
0.1719
young man throws
3.188
jurpe
3.047
driver drivs
0.1719
group of friends
2.914
water during samset
2,688
drives
0.09375
pass
2.64
sunset
game against american
swinmers
1.953
man throws
1.8125
swimming
1.75
ball
1.703
woman swinming
1.234
game
dog
0.953
throws ball
1.625
water
0.5312
team
1.3125
man
0.2344
young man
lake
0.1562
man
0.04688
Table 20: Results for B2T the BAR dataset. Since B2T requires validation set, extracted ourselves held-out validation split, stratifying with respect to the known class population distributions. Because of B2T's lack of bias mining step, no misclassified examples could be found for the "Climbing" class, leading to a crash of the algorithm. Furthermore, there are no results for the class "Vaulting" because B2T filtered all the keywords out due to negative CLIP Scores. For the remaining classes, keep in mind that B2T's output consists of keywords that represent the opposite of the bias, which means that for the class "Fishing" the key words "boy fishing", "beach", "fishing", "man fishing", and "lake" are wrong because they represent the actual bias or the class itself, and the opposite of the bias as expected from B2T. The same can be said for the class "Racing" and the keywords "track", "race", ctc. Overall, B2T's results on BAR appear to be wrong. For comparison, Tab. 26 shows SaMyNa's results BAR.

---

Published in Transactions on Machine Learning Research (11/2025)
Differences in keyword filtering An important novelty of SaMyNa is that it can find embedding vector that represents the bias of the model in a certain class. This embedding vector is found by doing arithmetic operations between the embeddings of the captions, as explained in the paper. This means that we solve the problem of synonyms. In B2T there is a heavy filtering step before the ranking of the keywords that uses the YAKE keyword eaction algorithm. YAKE does not take into account the semantics of words, which means that may filter out synonyms if they do not reach a certain frequency threshold individually. Conversely, our method does a very lightweight filtering of keywords before ranking, removing only very rare keywords that may result from captioning mistakes. After this lightweight filtering, work entirely in the embedding space of the text embedder, which can account for all the synonyms and construct an embedding vector that represents the bias itself semantically. Keyword embeddings are then compared to the bias embeddings and ranked according to cosine similarity. Our heavy filtering is done after ranking by setting a similarity threshold. Evidence of B2T's filtering being too heavy is found in the full keyword rankings for the class "not blond" of CclobA the supplementary material of B2T, where the keyword "woman" does not survive filtering and does not appear in the ranking. In particular, B2T selects the top 20 best keywords according to YAKE before ranking, while discard keywords that do not appear in at least 15% of the captions for a given class and then aggregate the surviving keywords in single pool of keywords that will be used for all classes.
Symmetrical keyword rankings. Additionally, our algorithm has the interesting mathematical property that for binary classification datasets, the ranking for one bias class is symmetrical with respect to the other class (because the two bias embedding vectors point in opposite directions, the cosine similarity will give opposite scores).
SaMyNa can be applied on images. In Sec. A we show the possibility of SaMyNa to work on other modalities without involving text. We use image embeddings instead of caption embeddings to produce the embedding vectors that represent the biases, and then we rank image patches instead of keywords according to the similarity of the patch the bias embedding and we display this as heatmap. This is a further novelty of SaMyNa with respect to B2T, showing that the underlying mechanism is fundamentally different (B2T only works with CLIP-like models and needs both images and text).
E Human Study Implementation Details
As for SaMyNa, we did not provide any guidance to the survey participants (in terms of bias features), so they did not have any prior knowlodge about the kind of biases in cach datasct. We report below the instructions provided to the participants in the survey:
What represents bias? In the ezamples we will show, you will encounter different kinds of images. Some of them portray people, others portray animals, and others show activities performed by humans. In this questionnaire, we refer to every possible recurring feature as bias. Everything that shows a high correlation with the group should be then included in your answer (e.g. do not limit yourself to well-known biases such as gender).
How to fill out the form. We will show you different groups of images. Each group has a common characteristic that we will highlight (hair color, type of bird, ac- lion name). For each group you must find up to 10 keywords, these keywords must refer to objects/featee /characteristicc or other elements of the images and must meet the following requirements:
The keyword must appear in MOST of the images of the group
The keyword must not be common to other groups in the same task (e.g. "person", "bird")
The keyword must not be the feature we highlighted about the group (if the group is about blond people, the keyword must not be "blond")
36

---

Published in Transactions on Machine Learning Research (11/2025)
F Automatic Procedure for Defining the sim Threshold
Even though we conceive SaMyNa as a supporting tool for which the final interpretation has to be performed by a human user, and that our filtering operations are intended to help tidying up our algorithm's output, we experimented with an automated way of finding per-class similarity thresholds (c) that are tailored to each specific model to be analyzed. The approach consists of obtaining a null distribution of cosine similarities by training the model under analysis on modified version of the dataset where the labels have been randomized. In this way, we make sure that the original correlations between data and target labels are destroyed. The obtained model, while practically useless for the downstream classification task, will not. be affected by particular biases as it has been trained on random task. At this stago, we compute mull distribution of cosine similarities by extracting the learned class embedding and the keywords for each target class, computing an empirical distribution of the similarities between each keyword and the learned class embodding. The maximum cosine similarity emerging from this step will be chosen as the threshold for filtering keywords of specific class.
We tested this alternative filtering approach on Waterbirds. A comparison between the obtained null distribution of cosine similarities, the distributions of the toy balanced version of Waterbirds (See sec. 5.2 and the original Waterbirds, can be found in Fig. 13
As can be seen, the balanced version Waterbirds has a similar distribution to the randomized version of Waterbirds, which confirms that both were able to produce unbiased outcomes. The obvious difference is that for the randomized waterbirds dataset, as previously stated, the model produced is useless, but constructing the dataset is easier. The original Waterbirds' distribution, on the other hand, shows longer tails and 70% higher standard deviation. We exploit this fact to set the threshold for class to be equal to the maximun cosine similarity score of the keywords for class obtained from the model trained on the randomized dataset.
These thresholds are then applied when generating the keyword rankings for the model trained on the original dataset. For the Waterbirds datasct, we obtained a threshold of 0.196 for the Landbird class, and a threshold of 0.320 for the Waterbird class. Comparing these thresholds with the scores shown in Tab. 22 you can see that the keywords that would be removed are "boat", "river", "midffight", "lake", and "flying" from the Watcrbirds class, and no keyword would be removed from the Landbirds class.
Even though this method may seem better than setting a predefined threshold, it has two main disadvantages:
It requires training the same model with the same procedure and hyperparameters on a randomized version of the dataset it was originally trained on, which is not always possible when using pre-trained models.
It requires twice the compute time, since the entire pipeline must be run two times.
Random Waterbirds (Class 0)
Balanced Waterbirds (Class 0)
Original Waterbirds (Class 0)
10a
10a
10a
Frequency
Frequency
Frequency
10a
10a
10a
100
10a
10a
-0.50
-0.25
0.00 0.25 0.50
0.50
-0.25 0.00 0.25 0.50
-0.50
-0.25 0.00 0.25 0.50
Cosine Similarity
Casine Similarity
Cosine Similarity
(a) Distribution of cosine similarities for the Landbirds class of the water- for the Landbirds class of the balanced birds dataset with randomized targets. waterbirds dataset in Sec. 5.2
(b) Distribution of cosine similarities
(c) Distribution of cosine similarities for the Landbirds class of the original waterbirds dataset.
Figure 13: Comparison of the distributions of cosine similarities for the Landbird class of different versions of the waterbirds datasct. The histograms for the Waterbirds class are not reported because they are symmetrical, see Sec. D As can be seen, the histograms from the random and balanced waterbirds version are similar in terms of variance (Std. Dev. 0.07 vs 0.075), while the original waterbirds dataset shows more variance (Std. Dev. 0.119), with longer distribution tails.
37

---

Published in Transactions on Machine Learning Research (11/2025)
G How to Distinguish Harmful Biases from Target Concepts
A rare but still possible situation may be the one where target classes are highly stereolyped but not necessarily biased. As an ideal example, one might think of the case where one of the target classes is the Zebra animal: it is reasonable to expect SaMyNa to extract keywords such as black-white, stripes. In this particular case, it would not be correct to state that these concept keywords represent harmful bias, but rather indicate that the model is relying a very narrow set of concepts that are sufficient to correctly characterize the target class of interest. This occurrence may still be considered as a form of bias, but evidently not an unwanted one.
Even though this case may be rarely encountered, we show in this section how it is possible to leverage SaMyNa's pipeline to detect whether a model has captured a form of benign bias or an actually harmful one.
To simulate this corner case, we utilize the Colored MNIST dataset from Nam et al. (2020), where target classes are digits, and the bias is injected in the form of digit color, thus introducing strong correlation between a specific color and particular class. Consequently, a naive model ends up classifying according to color instead of shape.
In this custom experiment, we select the pool of correctly classified and misclassified samples for each class (Storrect(c) (c), Smissdass (c)) as follows: Storrect() contains only samples belonging to class e and sharing the same digit color (e.g.; only red zeros for class zero). At the same time, (misselss(c) contains only samples from other classes having a different color (e.g., digits from 1 9, none of which are red). At this point, we may expect that all the keywords related to the correctly classified samples of certain class (Storredt (c)) would be strongly correlated with the actual larget and with the red concept, whereas samples from other classes present in Smiselass (c) would not share some common feature causing them to be systematically mispredicted as, in this case, zero
Hence, in practice, following the keyword extraction pipeline (Section 3.2.5 we obtain a set of potential bias keywords. At this point, whether a keyword is simply a target attribute and not a bias can be assessed by comparing two values: the average cosine similarity of the keyword's embedding with the caption embeddings of correctly classified samples, versus its average cosine similarity with the caption embeddings of misclassified samples.
In Table 21 we report the results of a comparison of such quantities when utilizing Colored MNIST as is (i.e., the biased case, 'Original' in Table 21p. or when we simulated the presence of a harmless bias ( ('Benign' in Table 21]. The difference between the two aforementioned quantities explicitly indicated in the two "(corr misclass)' columns, evidencing how in the Benign case this quantity is scnsibly higher, indicating that the keywords correlate much more with correct predictions than with the model's mistakes.
Class (sy
Deaign:
Original:
sim (cm/((((k)).. (coos(c))
stm(ccub(/ss)) gutrie (c)
(cor -misclass)
stm(emb((hw), [c))
stm///mb(kk)..g (xinin())))
Afcorr misclass)
red
0.4644
0,2580
0.2029
0.3978
0.2937
0.0021
orange
0,5088
0.1958
0.3120
0.4630
0.2975
0,1855
rw]
0. 9616
0.2580
0,2059
0.3614
0.3351
0.0263
yellow
0.5775
0.2533
0.2961
0.5497
0.3736
0.176L
3
green
0.5200
0.2901
0,2800
0.1815
0.3012
0.0803
gowen
0.52315
0.2810
0.2116
0-1530
0.1833
0.0127
5
blue
0. 0.1912
0.2687
0.2225
0.1714
0.1095
0.0701
0.5104
0.2687
0.2423
0.3821
0.0864
pupple
0.5412
0.2652
0.2530
0.5106
0.3074
0.1132
plak
0.5016
0.1765
0.9251
0.4750
0.3812
0.0988
9
pink
0.5033
0,2307
0,2732
0.4601
0.360S
0.0933
Table 21: Average difference in cosine similarity between caption embeddings of correctly classified samples and an output keyword, compared to the average difference in cosine similarity with respect to misclassified samples of a specific class. This quantity can be an indicator of the presence of a benign bias rather than a harmful one.
38

---

Published in Transactions on Machine Learning Research (11/2025)
H Outputs of SaMyNa
In this section, we provide the detailed output of our bias naming process for the experiments described in the main paper. We report the obtained keywords alongside their associated cosine similarity, in decreasing
order of value, for Waterbirds (Tab. 22 CelebA (Tab. 23). BAR (Tab. 26), IetgeNet-A (Tab. 24 and 25 on
a completely balanced version Waterbirds (Tab. 27 and for the ablation study on & (Tab. 28
Londbirds
Waterbirds
Keyword
Cosine Similarity
Keyword
Cosine Similarity
forest
0.335
ocean beach
0.540
foliage
0.334
0.434
tree
0.327
shore
0.386
stalks
0.314
water
0.386
branch
0.312
waves
0.335
trees
0.299
boat
0.294
branches
0.296
river
0,268
vegetation
0.273
midflight
0.244
lake
0.244
flying
0.218
Table 22: Cosine similarity scores for keywords associated with the Landbirds and Waterbirds classes.
Not blonde
Blonde
Keyword
Cosine Similarity
Keyword
Cosine Similarity
man male
0.45820
woman
0.39947
0.39175
blonde
0.34301
mascara
0.24959
makcup
0.24877
lipstick 0.22272
eyeliner
0.20065
Table 23: Similarity scores for the CelebA dataset.
39

---

Published in Transactions on Machine Learning Research (11/2025)
Crayfish
Rhinoceros Beetle
Keyword
Cosine Similarity
Keyword
Cosine Similarity
crab meal
0.39954
forest.
0.27621
0.31551
black
0,24731
crustacean
0.28440
branch
0.20823
food
0.23936
stands
0.20675
sandy
0.20224
Stick Insect
Cockroach
Keyword
Cosine Similarity
Keyword
Cosine Similarity
hand thumb
0.41610
insect creature
0.29116
0.39488
0.27834
finger
0.36924
darkcolored
0.27402
touch
0.33881
beetle
0,26163
plant
0.31364
dark
0.24713
plants
0.30637
flattened
0.24523
foliage
0.28583
grasshopper
0.24468
garden
0.27045
crustacean
0,22505
field
0.22898
darker
0.22500
leaves
0,21886
floor
0.21054
forest
0.21871
black
0.20142
holding
0.21059
grasshopper
0.20454
Table 24: Similarity scores for the crayfish, rhinoceros beetle, stick insect, and cockroach classes from ImageNet-
A.
Nails
Mushrooms
Keyword
Cosine Similarity
Keyword
Cosine Similarity
metal
0.37880
plants
0.25968
frame
0.27216
foliage
0.22674
rusted
0.25972
orange
0.20766
wooden
0.23891
weathered
0.23700
wall
0.22942
snake
0.20548
black 0.20246
Table 25: Similarity scores for the nails and mushrooms (fungi) classes from ImageNet-A.
40

---

Published in Transaction Lons on Machine Learning Research (11 (11 (2025)
Climbing
Diving
Fishing
Racing
Throwing
Vantting
Keyword
Cosine Similarity
Keyword
Casine Similarity
Keyword
Cosine Similarity
Keyword
Cosine Similarity
Keyword
Crsine Similarity
Keyword
Cosine Similarity
climber
0.55677
scuba
0. 0,58300
boat
0,13818
racing
0.42291
pitch
0,51658
vanuber
0.37871
0.54837
underwater
0,56077
fishing
0,425416
cars
0,42024
baseball
0,48251
vaulting
0.50903
climbing
diver
0.49010
0.42202
0.38508
pitcher
0.48111
vaul.
0,47589
climb
0.50305
diving
0.34910
0.39295
track
0.31835
batter
0.46329
jump
0.42575
rock
0.11596
dive
0.33087
lake
0.32910
stadium
0,28773
mound
0,44152
midair
0.37533
steep
0.364449
oceaun
0,28006
water
0.31149
race
0.27802
player
0.42568
pole
0.36865
usccul
0.32149
depths
0.27261
marriee
0.29890
spreds
0.26371 1
throw
0.37765
athleticiion
0.32692
rocky
0.30365
swimmming
0.26935
wavrs
0.25388
sprond
athlese
suspended
0.28935
ruggedness
0.25319
wetsuit
0,26063
underwater
ssphalt
0,26199
sport
0.36638
diving
0,27763
jagged
0.26699
marine
0,25820
wetssil
0.21951
football
0.253881
playing
0.36413
bar
0.26088
uscending
0.20509
water
0.22338
scatb
0.206444
compesition
0.23812
glave
0.36378
high
0.24619
pitch
0.23165
elbow
0,31375
fields
0.23763
Iuseball
0.22954
0.33142
arched
0.23102
batter
foothall
raised
0.22868
player
0.21 114
game
0.31113
athlete
0.22734
team
0.21087
bull
0.30598
dive
0.22547
pitcher
0.20611
team
0.29213
flipper
0.21940
arm athlesiciem
0.28914
skill
0.21579
0,27865
peak
0.20534
fields
0,25688
skill
0.23882
striking
0.22815
target
0,22690
focused
0.22104
0.21313
position
0.21137
action
0.21031
stadium
0.21019
amms
0,20977
positions
0,20790
pale
0.20539
activity
0.20472
casting
0,20145
stands
0,20009
Table 26: Similarity scores for the BAR dataset.

---

Published in Transaction ions on Machine Learning Research (11 (11 (202))
Landbirds
Waterbirds
Keyword
Cosine Similarity
Keyword
Cosine Similarity
sea midflight seagull
0.31901
0.23266
0.22417
Table 27: Ablation study on a completely balanced version of Waterbirds.
25
Landbirdd
Waterdirds
Londbirds
Woterfrins
Landbiide
Wattrrhrds
Landbirdss
Waterrörds
Londbbirds
Woterrånk
middii:: 0.18 = 0.12
water: 0.24 + 0.00
troe: 0.33 0.02
water: 0.38 0.02
forest: 0.36 0.02
oecan: 0.54 0.02
tree: 0.36 0.01
oecan: 0.58 + 0.01
troe: 0.38 0.02
consa: 0.56 0.01
seagall: 0:17
0.33 0.03
beach:
faliage: 0.01
0.34
beach: 0.51 0.02
froest: 0.01
beach: -
free: 0,07 0.10
fsh: 0.17 0.12
fobage: 0.31 # 0.02
scagull: 0.31 d: 0.03
branch: 0.34 #: 0.02
water: 0.37 0.02
folinge: 0.34 # 0.01
water: 0.39 # 0.02
folinge: 0.35 # 0.01
water: 0.37 + 0.02
boashonse: 0.17
beasches: 0.31
0.31 0.02
0.33
troes 0.33 0.01
0.30
waves: 0.34
seabirds: 0.15 = 10
0,30 0.01
0.27
branches: 0.32 0.02
0.35 0.01
beandh: 0.31 0.02
middlgght 0.23 0.02
0.01
lake: 0.27 0.02
ships 0.14 0.19
beanch. 0.30 + 0.02
midiflghtt. 0.27 0.01
stalks: 0.32 0.01
bost: 0.31 0.01
beanclos: 0.30 0.01
wavess 0.23 0.16
leaves: 0.24 0.01
middiight: 0.26 + 0.00
stalls::
flying:
0.24 0.01
fying:
shop: (.13 0.19
bamboo: 0.26 + 0.02
ocnam: (1.17 0.24
vegetation: 0.19 0.13
shore: 0.26 0.18
seagull: 0.14 + 0.10
sandy: 0.15 0.11
pier: 0.10 1 0.15
vegetation: 0.15 à 0.11
sea: 0.16 0.23
laaves 0.16 0.11
lake: 0.26 0.01
horizon: 0.14 0.10
borizon: 0.07 0.10
lake:
leaves: 0.15
watermari:: 0.16
midfltght: 0.24
0.07 0.09
suc 0.09 1 0.13
deciduous: 0.09 0.12
shote: 0.12 0,18
fling (1.14 1
coustal:
dock: 0.37 0.300
seabird: 0.08 1 0.12
river: 0.09 0.13
harizom: 0.07 =: 0.10
foam: 0.09 =: 0.13
Table 28: Ablation study on Waterbirds, where we analyze the impact of the number of selected examples on the found keywords (k) and their respective similarity values.


Reviews:
[{"review_id": "yeZmG1pcgE", "reviewer": "Reviewer_LSBx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The method is straightforward and intuitive. This is a reasonable use of a VLM to attach semantic meaning to image components for analysis.\n\nI see two weaknesses: \n\n1) Determination of representative samples uses k-medoids for correct and a hinge-inspired metric for incorrect classifications (Eq. 1). This is reasonable. Due to reliance on decision layer outputs (which can assign high confidence to incorrect predictions), it might be better to use metrics such as Mahalanobis or even Euclidean distance at the embedding layer. Euclidean distance from class centroids may be simpler and equally if not more effective for representative samples as well, given the tendency toward Neural Collapse (NC) (Papyan et al. 2020) in classification networks. There are nuances here, such as class imbalances, etc. and I don't think the paper needs to be revised to include these, but I think it is important to conside NC collapse geometry when exploring distance metrics in vision models for classification tasks.\n\n2) The VLM that describes images and embeds text is assumed to be objective, and I don't see any discussion on debiasing the observer.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper suggests using a Vision Language Model to textually describe representative and strongly misclassified image examples from a previously trained classifier in order to attach semantics to potential model biases. Representative keywords of class are created and then compared with embedded descriptions that have common features across all classes removed. The keyword embeddings and \"unique feature\" embeddings are then compared via cosine similarity. When similarity is high, it indicates that a keyword is strongly associated with that class. In cases where strongly associated keywords are not strictly relevant to the class, a bias exists.\n\nThe method does not require a validation set, is useable at training and inference time, is validated on several benchmarks and can be coupled with existing debiasing strategies.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As per point 2 above, a quick search turns up recent work in this field: \n\nhttps://arxiv.org/abs/2207.01056?utm_source=chatgpt.com\nhttps://arxiv.org/abs/2305.15407?utm_source=chatgpt.com\nhttps://arxiv.org/abs/2503.08368\n\nIt is important to point out that the observer could have its own biases, and make note of the literature on this, even if there is empirical evidence that the observer is objective enough to get results on benchmarks.\n\nThe authors should account for why--if they cannot include an analysis of the observer's biases--this paper provides enough evidence for their results before I can fully support the claims that are made.", "broader_impact_concerns": "It is important to acknowledge that VLM's are not an objective source of truth when determining biases in models or data sets.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2Vilc4A2QX", "reviewer": "Reviewer_aTpa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- The authors suggest a novel setup to address the problem of bias mining without a validation dataset. To my awareness this idea is novel, and has the potential to be quite useful to detect biases in many real world datasets and settings where a validation dataset may not be readily available. \n- I appreciate the authors' fairly thorough comparison of their work to B2T (Kim et al. 2024) in the main text and Appx. D, given the apparent similarities between the work on first reading. Ultimately, as both methods involve multi-step pipelines with differences at each step, it is difficult to do an exhaustive comparison, but I believe the discussion of the differences is useful.\n\n# Weaknesses\nI believe there are three main weaknesses of the paper: 1) ambiguity in nature of keywords generated by SaMyNa, 2) lack of detail for the training time bias naming paradigm, and 3) general clarity. \n\n## Keywords generated by SaMyNa\nThe greatest weakness of this paper is in the description of what the keywords generated by SaMyNa actually represent. The stated definition of biases in this paper are *spurious correlations*, which the model uses as \"shortcuts\" for the network to infer a prediction rule on training data which does not generalize well to test data. Throughout the paper, there are prominent examples of keywords extracted by SaMyNa which do not represent biases in this sense, or for which the concept of bias has considerable ambiguity. These ambiguities reduce the utility and explainability of biases labeled by SaMyNa for end users. \n- The most prominent examples are in Figure 4, Table 2, and Figure 8. Here, some of the most strongly scored keywords are the target classes themselves (e.g. `vaulting` for `vaulting`), categories which are semantic generalizations of the target classes (e.g. `insect` for `cockroach`), or apparently useful features of classification (`metal` for `nails`). This observation has two potential sources. In one case, **SaMyNa could be extracting keywords which represent only correctly classified examples**, leading to extraction of useful, non-spurious features as keywords. In another case, **components of the SaMyNa pipeline share the same biases as the model under consideration**, leading to incorrect assignment of \"bias\" keywords to the misclassified examples. There are two concrete issues that arise from this feature of SaMyNa. \n  - 1. **Model vs. Dataset bias.** If SaMyNa extracts keywords which only appear in correctly classified examples, it is very difficult to distinguish between \"benign bias\" in the dataset, and biases which the model is actively using to classify the data. In the text, the authors state: `Unlike approaches that discover biases in the dataset, our focus is on naming the biases captured by the model under examination.` Consider the Colored MNIST dataset as studied in Nam et al. 2020. When training the model on the `color` feature, the `digit` feature is a benign bias (e.g. not used for classification). If the goal of SaMyNa is to capture model biases, as opposed to dataset biases, the `digit` feature should not be extracted as a keyword. It is not clear to me that this should be the case.\n  - 2. **Ambiguity in bias classification.** In section 5.3, the authors assert that the extracted keywords do not resemble biases. Why is the `wall` keyword not considered a bias for the `Nails` class in Imagenet, while `rock` is considered a bias for the `Climbing` class in BAR? Both are contextual cues, and the choice to consider one as a bias but not the other appears arbitrary. If SaMyNa can extract truly useful features in addition to spurious correlations, there are many datasets where it is non-trivial for a human user to distinguish which is which, and it may be difficult to mine biases in the presence of many real features. In Table 2, is `glossy` a bias feature for `Cockroach`? Without the assurance that extracted keywords are at least as related to misclassified samples as correctly classified ones, it can be difficult to identify biases from keywords, even for expert users. \n\n3. **Relative scale of cosine similarity.** I am also concerned with the interpretation of the cosine similarity score used to rank keywords. Across the different models and datasets studied, it is not clear how to interpret different values of the statistic in terms of the relative \"strength\" of biases present. For example, many biases in the `Landbirds` target class have a cosine similarity score $\\sim 0.3$ in Figure 3. Simultaneously, in Figure 7, for the `Waterbirds` target class the keyword `sea` has a similarity score of 0.3, **even fit on the debiased dataset**, and is described as \"low correlation\" in the text. Without some kind of relative scale to calibrate the meaning of the cosine similarity score, it is not clear how to define what constitutes significant levels of bias for each application of SaMyNa. \n\nMy impression overall is that SaMyNa aggressively generates candidate bias keywords, but that it is not clear how to distinguish real biases from useful features of the data. While this does not appear to be an issue for worst group classification (e.g. Table 1), imprudent suggestion of real features as bias keywords has concerning implications for many of the real world applications that the authors mention in their work. \n\n## Training time bias naming paradigm\nThere is relatively little discussion around the training time bias naming paradigm; in my opinion, one of the most interesting parts of the paper. \n1. Despite the existence of prior work that discusses similar training-time bias mining (Nahon et al. 2023), it would be important to understand how bias mining from training in this manner is related to the more typical practice of bias identification from a validation set, and how sensitive this process is to the choice of iteration $t^*$. \n2. It would also be useful to discuss how this work is related to other works which use similar metrics to identify mislabeled data, as opposed to spurious correlations. \n3. It is unclear if the bias mitigation in section 4.3 is performed using keywords which are derived from the training set or the validation set for SaMyNa + GDRO. \n\n## General clarity\n- In the text, it is not clear if  $\\text{k}$ is the mediod number, and $k$ the number of incorrect samples, or if $k$ is shared for both.\n- the citation style of the paper is very distracting, without distinction between textual and paranthetical citations. \n- In my personal opinion, creating an acronym out of an evocative title is not very informative, and does little to tell the reader how this methodology differs from other work that exists in the field.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The authors propose a pipeline to name model biases. The pipeline has four major steps:\n\t1. **Pinpointing a particular model for analysis.** If one has access to a validation set, use the final trained model. If not, one can use the checkpoint of the model for which it has the greatest proportion of confidently misclassified samples. \n\t2. **Extracting samples.** For each target class, identify the samples which are correctly and incorrectly classified into that target class. Run k-medioids to subsample the correct samples, and select the top k most confidently misclassified samples.\n\t3. **Generating candidate bias keywords.** Caption each image with LLM. Extract keywords with custom pipeline. \n\t4. **Filtering bias keywords.** Generate per-class embeddings for each class by averaging per-image embeddings generated with a pre-trained text encoder. Get a set of keywords which occur often within each class, align with each class embedding up to a certain threshold. \n- The authors then evaluate the performance of convolutional image classification architectures (ResNet-18/ResNet50) on four datasets previously studied in the debiasing literature: CelebA, BAR, Waterbirds, and ImageNet-A. All models are pre-trained on ImageNet-1K, and either fine-tuned (CelebA,BAR,Waterbirds) or tested directly (ImageNet-A) on these datasets of interest. \n\t- For WaterBirds, CelebA, and BAR, the authors demonstrate that their pipeline rediscovers known biases within the dataset, when evaluated only on the training dataset.  \n\t- For ImageNet-A, the authors demonstrate that using their pipeline on a model at inference time extracts keywords which potentially represent biases. \n\t- Once again for WaterBirds, CelebA, and BAR, the authors combine their keyword extraction framework with GroupDRO. In particular, they obtain per-sample bias labels by using a CLIP zero-shot classifier to extract potential alignment between per-sample classification and per-class bias keywords.\n-  Finaly, the authors present a variety of other analyses, showing the performance of SaMyNa on vision transformer models, using an unbiased dataset, a specific case where SaMyNa extracts non-bias keywords, correlation between keywords with human feedback on CelebA, BAR, and Waterbirds, as well as a set of other ablation studies.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have organized my requested changes to match the weaknesses described above. \n## Keywords generated by SaMyNa\nThese changes are critical to securing my recommendation for acceptance. \n- Please demonstrate if keywords generated by SaMyNa are truly reflective of model bias, and not dataset bias. As an example, if you were to use SaMyNa on Colored MNIST as in Nam et al. 2020, how would it handle models trained on the malignant vs. benign bias settings? \n- It would be useful to understand how bias keywords are associated with correctly vs. incorrectly classified samples. For example, for how many correct vs. incorrect samples is \"vaulting\" is a keyword for $c=\\text{vaulting}$ in the BAR Dataset? \n- Is there some \"null distribution\" of cosine similarities that can be used to define keywords which are signficantly indicative of bias? \n- Beyond these changes, please include a thorough discussion of the keywords generated by SaMyNa, and how they coincide with actual biases of the model. Additional discussion of how to identify real biases from SaMyNa keywords would be important as well. \n## Training time bias naming paradigm\nThese changes are critical to securing my recommendation for acceptance. \n- How are the keywords and cosine similarity scores generated from training time bias naming related to the keywords and cosine similarity scores generated from inference time bias naming, for Waterbirds and CelebA? How does this change as you consider different training epochs $t^*$? \n- Please discuss how this bias naming paradigm relates to previous work around memorization and mislabeled examples (e.g. , such as Pleiss et al. 2020.)\n- Please specify if in Table 1, comparisons are performed using Training time bias naming, or Inference time bias naming for SaMyNa + GDRO. \n- The text states: `Notably, since both our method and B2T rely on GroupDRO for the final bias mitigation step, our advantage has to be imputed on a finer semantic bias discovery.` In Table 1, if I understand correctly the comparison between B2T+GDRO and SaMyNa + GDRO involves training on datasets of different sizes. Can you tell us how SaMyNa +GDRO performs on the same size training set as B2T+GDRO (i.e.g holding back the same held-out validation set from the training set)? \n## General clarity\nI have described my concerns with general clarity above in the weaknesses section. These changes would strengthen the work in my view.\n\n# References\nPleiss, Geoff, et al. \"Identifying mislabeled data using the area under the margin ranking.\" Advances in Neural Information Processing Systems 33 (2020): 17044-17056.", "broader_impact_concerns": "My concerns around the nature of SaMyNa's bias keywords relates to my concerns about the broader impact statement currently included. I will refrain from making additional recommendations regarding broader impact until the authors respond to my initial round of comments.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Dwmkps4whV", "reviewer": "Reviewer_QtvF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\na. This paper is well-written and easy to follow.\n\nb. I appreciate the extensive experiments. \n\nc. I think the topic of biases is very interesting.\n\nWeaknesses:\n\na.\tMy main concern is about the proposed method. I don’t capture the technique contribution of the proposed method. Meanwhile, the motivation of each step in the proposed method is confusing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on the issue of biases which are the spurious correlations that the models over-rely on specific patterns that are non-representative of the real-world data distribution. To address this issue, the contributions of this paper are summarized as:\n\na.\tThis paper proposes a tool for naming biases in DL models that is text-based and usable both at training and at inference time.\nb.\tThis paper validates the proposed approach on established benchmarks, finding biases well-known by the community.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "a.\tIn Section 1, this paper describes the issue of biases which are very interesting. I think that it is better to provide experiments to demonstrate that the models sometimes do over-rely on the biases, I, e., specific patterns that are non-representative of the real-world data distribution.\n\nb.\tIn Section 1, Contributions 1, 2 and 3 could be merged as one.\n\nc.\tIn Section 3.1, It is started that “Inspired by the Hinge loss function Rosasco et al. (2004), we can define a per-sample distance metric …”. However, I think it is quite different from the Hinge loss. Please describe it in details.\n\nd.\tI think Eq.1 is not rigorous. Considering $ \\hat{y} _{t,n} = [0.7,0.2,0.1]$ and $y _{n}=2$, then Eq.1 is equal to 0.5, which indicates that the sample is misclassified into only label 1 with high probability. Considering another case, $ \\hat{y} _{t,n} = [0.41,0.2,0.39]$ and $y _{n}=2$, then Eq.1 is equal to 0.21. But in this case, the sample is misclassified into labels 1 and 3 with high probability. Therefore, Eq.1 and the statement “the higher $d _{t,n}$ is, the most $M$ s confident in misclassifying …” are not rigorous.\n\ne.\tIs $D^{misclass} _{train}$ changing with the increase of iteration $t$? What is the range of $t$? \n\nf.\tIn the first sentence of Section 3.2.1, why is there a $c$ in front of $D^{misclass}(c)$? Why do you use a cluster algorithm k-medoid? What are the differences between $D^{misclass}(c)$ ($D^{correct}(c)$) and $S^{misclass}(c)$ ($S^{correct}(c)$)? It is better to rewrite Section 3.2.1 for readability.\n\ng.\tIn Section 3.2.4, how to achieve the embedding matrices?\n\nh.\tIn Eq.3, it doesn’t need to be divided by 2.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)