ConceptScope: Characterizing Dataset Bias via Disentangled Visual Concepts
Authors: Jinho Choi
Hyesu Lim
Steffen Schneider
Jaegul Choo
Venue: NeurIPS 2025 poster ()
Keywords: dataset bias analysis
vision datasets
sparse autoencoder
Primary area: applications

Abstract: Dataset bias, where data points are skewed to certain concepts, is ubiquitous in machine learning datasets. Yet, systematically identifying these biases is challenging without costly, fine-grained attribute annotations. We present ConceptScope, a scalable and automated framework for analyzing visual datasets by discovering and quantifying human-interpretable concepts using Sparse Autoencoders trained on representations from vision foundation models. ConceptScope categorizes concepts into target, context, and bias types based on their semantic relevance and statistical correlation to class labels, enabling class-level dataset characterization, bias identification, and robustness evaluation through concept-based subgrouping. We validate that ConceptScope captures a wide range of visual concepts, including objects, textures, backgrounds, facial attributes, emotions, and actions, through comparisons with annotated datasets. Furthermore, we show that concept activations produce spatial attributions that align with semantically meaningful image regions. ConceptScope reliably detects known biases (e.g., background bias in Waterbirds) and uncovers previously unannotated ones (e.g, co-occurring objects in ImageNet), offering a practical tool for dataset auditing and model diagnostics.

ConceptScope: Characterizing Dataset Bias via Disentangled Visual Concepts
Jinho Choi KAIST AI
Hyesu Lim KAIST AI
Steffen Schneider Helmholtz Munich
Jaegul Choo KAIST AI
Abstract
Dataset bias, where data points are skewed to certain concepts, is ubiquitous ma- chine learning datasets. Yet, systematically identifying these biases is challenging without costly, fine-grained attribute annotations. We present ConceptScope, a scalable and automated framework for analyzing visual datasets by discovering and quantifying numan-interpretaan representations from vision foundation models. ConceptScope categorizes concepts into target, context, and bias types based on their semantic relevance and statistical correlation class labels, enabling class-leve dataset characterization, bias identifi- cation, and robustness evaluation through concept-based subgrouping. We validate that ConceptScope captures wide range of visual concepts, including objects, textures, backgrounds, facial attributes, emotions, and actions, through compar- isons with annotated datasets. Furthermore, we show that concept activations produce spatial attributions that align with semantically meaningful image regions. ConceptScope reliably detects known biases (c.g.. background bias in Water- birds [59]) and uncovers previously unannotated ones (e.g, co- co-occurring objects in ImageNet [12],, offering a practical tool for dataset auditing and model diagnostics. The code is available at https: /tte://ggiiie //github.... com/ jihoochoio/eeeeco cttp://oo-ccoo//ooceeeeeeeeeeeee choo//ooncptSccpp.....
concepts using Sparse Autoencoders trained on
Introduction
(a) Discovering categorizing concepts
(b) Characterizing via concept distribution
turtle" class images
Target
Context
Lench
SAE
Bias
sea turtle
Strong
Weak
Figure I: Overview. Visual concepts are unevenly distributed image datasets, e.g., many images labeled turtle taken the beach. (a) Our framework ConceptScope discovers and categorizes presenting visual concepts into target, context, and bias concepts based on a Sparse Autoencoder (SAE)-based concept dietionary. (b) We characterize datasets by averaging activation of each concept across the training set, where brighter elements indicate stronger concept presence.
Large-scale datasets have significantly driven recent breakthroughs in artificial intelligence, partic- ularly in large language models and vision-language foundation models, as described by scaling laws [26]. Alongside increasing dataset sizes, recent studies have highlighted the importance of careful dataset curation. For instance, curated subsets of data have demonstrated favorable perfor- mance compared to larger, less carefully constructed datasets [20]. However, there currently no
"Correspondenc:: jehoo@kiastacc.k kiast.ckkrr: 'Institute Computational Biology, Computational Health Center. Helmholtz Munich; "Munich Center for Machine Learning (MCML)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

systematic framework for designing effective datasets. In practice, researchers must train models on multiple curated versions of dataset using a fixed training protocol and compare their performances. making the process both resource-intensive and indirect.
One critical aspect influencing dataset quality is the diversity and distribution of visual concepts, defined as recurring patterns such as colors, objects, and textures. Since datasets are collected and filtered according to specific human criteria, they inevitably encode inherent biases. For example, as shown in Fig. 1, approximately 75% of images labeled as leatherback sea turtle in Ima- geNet [12] are captured on beaches, while only 15% are taken underwater. Models trained on such a biased dataset often generalize poorly when test images deviate from these dominant patterns, such as failing to recognize sea turtles without beach backgrounds. Recent research [75, 41] further shows that these biases also exist in large-scale visual datasets, including YFCC [66], CC [7], and
DataComp [20], to the extent that a simple classifier can easily distinguish among datasets.
Curating datasets to mitigate such biases requires a comprehensive understanding and quantification of the distribution of visual concepts. Existing approaches typically rely on manual annotation to quantify visual concepts within large-scale datasets, but this process is costly and impractical [38, 30]. Vision-Language Models (VLMs) [39] have recently emerged as scalable tools for automatically generating descriptions of visual content. However, because these captions are expressed in natural language, they often vary in granularity and use different synonyms to describe the same content. This inconsistency makes it difficult to extract structured and meaningful visual concepts from VLM-generated captions, often requiring additional human intervention [6] or extensive prompt engineering [45], both of which present significant practical challenges.
In this paper, we introduce ConceptScope, framework for analyzing visual datasets through the distribution of visual concepts, designed for image datasets with class labels (Fig. 1). ConceptScope operates in two stages: First, it constructs an interpretable concept dictionary by training Sparse Autoencoders (SAEs) [10] on image representations from vision foundation models [37]. SAES disentangle dense representations into a sparse set of numaan-nterppeetable concepts, enabling scalable, unsupervised concept discovery without manual annotation. Second, ConceptScope characterizes the dataset by measuring each concept's semantic relevance and co-occurrence frequency with every class, and grouping them into three categories (Fig. 1 (a)): (1) target, essential for recognizing the class (e.g, the body or shell of sea turtle); (2) context, frequently co-occur with the class but are not necessary for recognition (e.g., beach, underwater, or forest backgrounds); and (3) bias, subset of context concepts that are class-agnostic but exhibit strong statistical correlation with specific classes (e.g., the prevalence of beach scenes for images labeled as turtle). By analyzing the distributions of these categorized concepts, ConceptScope provides class-level dataset characterizations (Fig. 1 (b)) that enable bias identification and robustness evaluation.
Through extensive experiments, we show that our method captures broad range of visual concepts, including objects, textures, backgrounds, facial attributes, emotions, and actions. We further demon- strate that the spatial attributions derived from SAEs aligned closely with ground-truth segmentation masks. Moreover, ConceptScope reliably identifies known dataset biases (e.g.. background bias in Waterbirds [59]) and uncovers previously unannotated ones (e.g., cultural biases or co-occurring objects in ImageNet [12]). Finally, we highlight its practical utility for evaluating model robustness by identifying out-of-distribution samples and quantifying the resulting degradation in model accuracy.
Our contributions are summarized as follows:
1. We introduce ConceptScope, framework that systematically extracts visual concepts, catego- rizes them based on their relevance to target classes, and quantifies their distributions.
2. We empirically validate SAE activations as effective, interpretable concept extractors.
3. We demonstrate the practical utility of ConceptScope for detecting dataset biases and evaluating model robustness by grouping test samples into subsets having distinct concept distributions.
2 Related Work
Analyze vision dataset. A key line of research aims to comprehensively characterize visual datasets. Early works summarized datasets by analyzing local image patches [13, 57] or by selecting representa- tive images [67]. With the advent of VLMs [39]. more recent methods have leveraged VLM--genrated captions to provide deeper insights. Some approaches summarize datasets by aggregating captions to highlight overall characteristics [15, 75], but they do not provide detailed, sample-level statistical
2

---

analyses of visual concepts. Other works treat captions as pseudo-labels for visual attributes, but typically rely on manually curated attribute lists [6] or extensive LLM-basee prompt chaining to extract and assign attributes [32, 45]. Even after these labor-intensive steps, attributes may still be missing or inconsistently captured unless prompts are carefully crafted [8]. Our work addresses these limitations by automatically discovering semantically meaningful concepts and quantifying their sample-level distributions using SAEs.
Several studies detect model bias by analyzing failures on test samples. These approaches segment test samples into subgroups with low performance and then assign semantic labels to the resulting clusters using cross moddal embeddings [17] (e.g., CLIP [54]) or generated captions [73, 28]. However, such methods identify dataset bias only indirectly, through downstream model behavior. If the training data is biased (c.g.. birds frequently co-occurring with ocean backgrounds) but the test set lacks blas-conflicting examples (e.g., birds without ocean backgrounds), these methods cannot detect the underlying bias in the training data or the model's s reliance on it. More importantly, they do not directly inspect the training data and thus cannot determine which training samples contain bias-inducing features or how prevalent those features are. Another line of work identifies biases by inspecting neuron activation maps that highlight image regions unrelated to the target objects [48, 63, 51]. However, these methods depend on manual annotation to distinguish between target-related and unrelated regions, limiting scalability. In contrast, method operates fully automatically without requiring human intervention.
Bias identification via model behavior.
Sparse Autoencoders (SAEs) for interpretabiliiyy SAEs have recently emerged as a powerful tool for interpreting the internal mechanisms of deep learning models [10]. Unlike earlier approaches [52, 19, 2] that attempt to interpret individual network neurons, which often encode multiple meanings, SAES extract latent concepts that are monosemantic and thus more interpretable [16]. Following their success in large language models [10], SAES have been applied to vision models for interpreting classification decisions [55], explaining model adaptation [37], and analyzing diffusion models [64, 27J. Our work differs from these approaches in its primary focus: rather than interpreting model behavior, we leverage SAES to characterize visual datasets. By disentangling visual representations from strong vision encoders such as CLIP [54] into clearly interpretable concepts, ConceptScope provides a robust foundation for understanding dataset composition and uncovering inherent biases.
3 ConceptScope: A Framework for Characterizing Datasets
In this section, we present ConceptScope, framework for analyzing image classification datasets through the distribution of interpretable visual concepts. ConceptScope operates in two stages: (1) constructing concept dictionary by extracting human-interpretable visual concepts by training Sparse Autoencoders (SAEs) on representation from a vision foundation model, and (2) categorizing these concepts into target, context, and bias types for every class, based on their semantic relevance to the class label and their frequency of occurrence in images of that class. We begin by defining the concept types and formalizing the problem setting in $3.1, then describe the construction of the concept dictionary in $3.2. and finally explain the class-wise concept categorization process in $3.3.
3.1 Problem formulation
Consider an image classification dataset D = {(;, )1)) where each input x; E X' is an image and yı E y = (1,..., K} is its class label among K possible classes. Let C = {c1, c denote a set of visual concepts representing recurring visual patterns such as colors, shapes, textures, backgrounds, and objects. For each class y E V, we categorize concepts into three types: target, context, and bias.
Target concepts. For a class y, the target concepts Ty C C contains features that are both essential and distinctive for identifying y (e.g., turtle shell and turtle head for sea turtle). Their absence substantially hinders the correct classification of an image as belonging the class.
Context concepts. For a class y, any non-target concept E C \ T,, is considered context concept. These concepts co-occur with y but are not required for correct classification as long as the target concepts are present. For example, in an image of a sea turtle on a beach, the sandy environments and the ocean background are context concepts for the sea turtle class.
3

---

(a) Training SAE
(c) ciasscconcop somantic alignment
(4) Target / context discrimination
MSE (reconstruction)
Target dlass lyl:
turtle
Vision
z.ng
SAE Enc.
SAE Dec.
concepts
image
(image,
target
Encoder
Necessity
N
terget
L1 (sparsity)
(e) Bias concept identificaain
(b) Interpreting concepts
Rash
Sufficiency
seg mask
seg mask
.) 9. 1.95/0.9
f: 1455)
+1.25
SAE
Allamment
sea
turtle
sand
texture
#2.025
bise
Figure 2: ConceptScope. (a) We train an SAE and (b) construct a concept dictionary specify the semantic meaning of each latent through reference images with segmentation masks and generated textual descriptions. (c) We compute class-concept semantic alignment score and (d) categorize concepts into targer and context concepts. (e) We further distinguish context concepts above threshold concept strength (activation value) as bias concepts.
Bias concepts. For a class y, a bias concept is a context concept that disproportionately co-occurs with y in the dataset. For example, if sandy environments appears far more frequently in images of sea turtle than other backgrounds such as ocean or tropical scenes, then it is considered the bias concept of that class.
Given these definitions, the goal of ConceptScope as follows:
1. Concept dictionary construction: Identify a set of interpretable visual concepts C.
2. Concept categorization: Categorize each concept as target, context, or bias for each class.
We describe the concept dictionary construction procedure in the next section, followed by the class-wise categorization process.
3.2 Constructing concept dictionary using Sparse Autoencoders
Constructing concept set with both broad coverage and appropriate granularity is challenging, as requires exhaustively identifying semantically meaningful visual patterns across large datasets- process that labor-intensive, subjective, and prone to omission. To address this, we adopt SAEs as scalable concept extractors that automatically discover interpretable visual concepts without manual supervision. We train an SAE on representations from a strong vision encoder, such as CLIP vision transformers (VIT) [54] (Fig. 2 (a)), and interpret the learned concepts by examining their maximally activating images and identifying the shared visual patterns they capture (Fig. 2 (b)).
Architecture of SAEs and training. SAEs consist of a linear encoder and a decoder with non- linear activation function [10]. Given an input image x, we first extract intermediate-layee token embeddings
{21, 22, ..... 2 from pretrained VIT, where 24 E Rd is patch-level token embedding, is the token length (the number of patches plus one class (CLS) token), and d is the embedding dimensionality of the VIT. We train SAEs reconstruct each of the input token embeddings z through a sparse representation:
f(2 = (WTTc=)) SAB(z) = WT. (z),
(I)
where W ene dxd and Wa E I are the encoder and decoder weight matrices, respectively and Φ is ReLU activation function [37]. The SAE latent dimension d' set significantly larger than d, using large expansion factor (e.g. 16 32). We optimize the SAE by minimizing combination of a reconstruction loss and L, sparsity penalty:
(2) where the À controls the degree of sparsity in the latent representation (Fig. (a)). We utilize the pretrained CLIP-ViTE ViT-L model with a patch size of 14 and train the SAE using ImageNet- [12]. Further implementation and training details are provided in the Appendix $B.1.
L
|| SAB (2) A||2||1
Quantifying concept activations. After training, the SAE provides fine-grained quantification of concept activations at both patch and image levels. Each row of the decoder weight matrix Wass

---

corresponds to linear representation of a learned concept, while the encoder output f(2) represents the activation strength of each concept for a patch embedding z. These patch-level activations can be visualized as coarse segmentation masks, localizing concepts within images ($4.2). For image-level
analysis, we aggregate patch-level activations by averaging the encoder outputs across all patch
tokens: i.e., fz) = + >x,ex (zi); f(z) E IRd". The scalar value f(z) then represents the overall activation strength of concept e for the image.
Interpreting learned concepts. To interpret the semantics of each latent, we collect the top-activating images per concept and extract their corresponding segmentation masks, following the procedure introduced in Lim et al. [37]. We then use multimodal large language models (GPT-4o) to generate short natural language descriptions from these visual inputs (Fig. 2 (b)). More examples are illustrated in Fig. 6. It is worth noting that this annotation step is fully automated, except for light prompt tuning, and intended purely for human interpretability. The generated descriptions are not used in the concept categorization procedure described in $3.3. In Appendix B.2, we provide further details, including prompts, implementation specifics, and an evaluation of annotation quality.
3.3 Categorizing concepts as target, context, and bias
After building the concept dictionary, we categorize concepts for each class. Given a labeled dataset of (image, class) pairs, we first compute a class-concept aligument score, which measures how representative concept is of class and how essential it is for recognizing that class (Fig. 2 (c)). Based on this score, we separate concepts into target and context categories (Fig. 2 (d)). We then further divide the context concepts into bias and non-bias categories using concept strength, which measures how frequently and confidently a concept appears within a class (Fig. 2 (e)).
Separating target and context concepts via alignment scores. To distinguish target from context concepts for a class y, we define a class-concepp alignment score. Let C {cI...., cM be the set of learned concepts. For each concept c, we compare the model's prediction confidence under three conditions: (I) using all concepts C. (2) excluding e (C \ {c)), and (3) using only ((c)). We obtain the set C {e} by masking out the spatial attribution of from the image z (z @ (1 mx)))), and {c} by keeping only concept region (a @ mz(x)).
We formalize this comparison with two metrics: necessity [(., y) and sufficiency S(c, y). Necessity measures the drop in prediction confidence when the e is removed (i.e., case (1) vs. (2)), while sufficiency measures the retained confidence when only e remains (i.e., case (1) vs. (3)) (Fig. 2 (c)):
N(c, v)
Ply x) ZEX, Ply s (1 me())) S(c,
Σ
Σ ZEX,
Plu x θ mc(x))
(3)
X
X
P x)
where Xy is the set of images belonging to class y. Ply x)) is the model's confidence for class y given v, computed as the cosine similarity between the CLIP text embedding of y and the image embedding of xr (x(x) the binary mask for concept c, and @ denotes element-wise multiplication.
We combine the necessity and sufficiency metrics into a single alignment score A(c, y) = -(cy)) A concept is labeled target if A(c, 2 My (aliin + a X align and as context otherwise (Fig. (d)). Here, jay and align are the mean and standard deviation of alignment scores for class y. and is a threshold factor. In Appendix B.3, we provide further details, empirical validation, and illustrative examples (Fig. 7).
Distinguishing bias and context concepts via concept strength. We measure concept prevalence using the SAE encoder output (z) ($3.2), which non-negative by design. For each concept c, we define its concept strength for class as the average activation across all samples z E Z, belonging to class y: i.e., J.., avgzez, (f))))) A high concept strength indicates that the is frequently and confidently associated with the class y. After excluding the target concepts Ty, we compute the mean م = avg. (fr and standard deviation o std. w)) of concept strengths for all remaining context concepts E EC ECA Ty, We classify e as a bias concept if its strength exceeds one standard deviation above the mean (Je.y 2 e + 2 otherwise we classify it as a non-bias context concept (Fig. 2 (c)).
We validate the accuracy of these spatial attributions in $4.2.
5

---

Table I: Performance comparison for concept prediction. We compare our SAE-based method in ConceptScope with generated paption-based (BLIP-2, LLaVA-NeXT) methods and show that our proposed approach outperforms the baselines. We use six labeled datasets and report class-wise binary classification accuracy in F1 and AUPRC, averaged across classes. Values represent the mean and standard deviation across classes.
Method
Metric
Caltech101 (18) (Objects)
DTD [9] (Textures)
Waterbird [59]
CelebA (43) (Facial Attr)
RAF-DB [36]
Stanford40 [72]
Average
(Backgrounds)
(Emotions)
(Actions)
1I-P-- [85]
Fi
0.382075
0.37_am
0.27anm
0.24
0.66
0.43
I.vVA LLaVA-NeXT(4)
0.40
0.57_ame
0.62ga3
0.45
0.80
ConceptScope
Fi
0.83e021
0,57=402
0.78:tum
0.55
0.78:0113
0.72
(SAE)
AUPRC
0.89
0.57amm
0.83.
0.85
0.59
0.82.ans
0.76
Analyzing class-wise concept distributions. By analyzing the distribution of target, context, and bias concepts, ConceptScope allows practitioners evaluate the diversity and dominance of visual concepts for each class. Specifically, the distribution of target concepts reflects the range of visual appearances and prototypical features that define the class, while variation among context concepts captures the diversity in backgrounds and co-occurring objects (Fig. 8, Fig. 9). The concept strength of bias concepts highlights the dominance of spurious patterns, revealing potential shortcuts a model may exploit during training ($5).
4 Sparse Autoencoders are Reliable Concept Extractors
In this section, we verify that SAEs capture a diverse set of visual concepts and that their activation values are reliable indicators of concept presence. We first evaluate the predictive capability of SAE latent activations through binary attribute classification in $4.1. Subsequently, we assess the accuracy of segmentation masks derived from SAE activations in $4.2.
4.1 Evaluating concept prediction performance
Tasks. We perform binary attribute classification to validate that SAE activations accurately reflect the presence of visual concepts. Evaluation is conducted across six annotated datasets covering diverse visual attributes: Caltech101 [18] (objects), DTD [9] (textures), Waterbird [59] (backgrounds), CelebA [43] (facial attributes). RAF-DB [36] (emotions), and Stanford40 Actions [72] (actions).
Additional dataset details are provided in Appendix $C.1.
For each test sample, we obtain the image-level activation values f(z) E Rd and treat them as unbounded confidence scores for concept presence. For each target attribute, we identify the most relevant latent dimension from the training split that maximizes the Area Under the RRecision-Recall Curve (AUPRC), and determine the prediction threshold that maximizes the F1 score. We reported AUPRC, the Fi score, precision, and recall as final metrics.
Methods.
Baselines.
Recent approaches [45, 32] use VLMs to generate captions and apply extensive prompt chaining with L.LMs to assign attribute labels. To minimize dependence on LLMs, we implement a simplified -based baseline: for each generated caption, we query LLM to decide whether it implies the presence of the target attribute. We evaluate two -fate-of-thh-aar spen-source VLMs, BLIP-2 [35] and -LT [34], and report their Fi scores, precisions, and recalls for direct comparison with our SAE-based results. The details of baselines are provided in Appendix $C.2.
SAE captures diverse visual concepts. As shown in Table 1, SAE outperforms baselines, achieving an Fi of 0.72 and AUPRC of 0.76 with low variation across four random seeds (std. 0.01). Caption- based methods (BLIP-2, LLaVA-NeXT) perform worse due to linguistic variability, often using generic terms (c.g., simplifying "dotted pattern" " with "pattern"). In contrast, SAE directly encodes visual concepts through latent features, consistently outperforming baseline methods across diverse visual attributes. Additional metrics are reported in Table 6, precision mreall curves are shown in Fig. 10, and representative failure cases of caption-basedd methods are illustrated in Fig.
SAE activation highly correlated with concept presence. We further evaluate whether SAE activation strength reflects the degree of the presence of visual concepts. To this end, we treat the similarity between CLIP image embeddings and textual embeddings of class labels as pseudo- ground truth and assess their correlation with the image-level SAE activation values. Table 7 summarizes

---

Input
mask
SAE (Ours)
LLaVA-NeX* -NeXT
BLIP2
Take
AURRI
2.4
0.
0.4
bottle
0.2
SAE (Ours)
LLaVA-NeXT
SUP2
Model
Figure 3: SAE provides reliable segmentation masks. Quantitative and qualitative comparisons of SAE spatial attribution (ours) with attention map-based approaches from BLIP-2 [35] and LLaVA- NeXT [34] on ADE20K [77]. We report the Area Under Deocision-Recar Curve (AUPRC) of segmentation masks. SAE results averaged over four models trained with different random seeds, yielding standard deviation of 0.002. Error bars indicate the standard deviation across 150 classes.
the Pearson (r) and Spearman (p) correlation coefficients across the six attribute-labele datasets. We observe consistently strong correlations (r 0.71, p 0.65), confirming that SAE activations reliably reflect visual concept presence. Further details are provided in Appendix $C.4.
4.2 Evaluating concept localization
Tasks. Since we utilize the spatial attribution of SAE activations to distinguish between target and context concepts, we evaluate the localization accuracy of these activations. Specifically, conduct binary segmentation experiments on the ADE20K [77] dataset, a widely-used benchmark with pixel-level annotations across 150 semantic categories (e.g., people, animals, sky, and buildings). Evaluation is conducted on 2,000 images from the ADE20K validation set.
Methods. We use patch-level SAE activation for concept f(z), ($3.2) coarse segmentation
masks, and normalize the value into a range [0, 1]. We discretize these continuous-valued SAE activations varying thresholds and measure AUPRC. Following the approach in $4.1, I. assign each latent to the class for which achieves the highest AUPRC on the training split.
Baselines. We compare against attention maps derived from VLMs, as they provide spatial lo- calization with respect to textual input. Specifically, we use BLIP-2 [35] and L.L.VA-NeTT [34], Hate-offthhearr VLMMs evaluated in $4.1. We obtain the attention maps by taking the max value across heads and averaging across layers of self-attention weights of image tokens.
SAE generates reliable segmentation masks. Fig. 3 compares the segmentation performance of SAE with attention-based masks from LLaVA-NeXT and BLIP-2. SAE achieves the highest AUPRC (BLIP-2: 0.098. LLa VA-NeTT 0.302, Ours: 0.399), performing particularly well on large regions (e.g.."lake")
"lake"), "), where activations closely match ground-truth masks. In contrast, LLa LLaVA-NeXT maps contain noise, and BLIP-2 maps include irrelevant activations. Moreover, SAE efficiently produces segmentation masks for concepts simultaneously methods require specifying each target class and multiple inference passes per image. Although performance drops for small objects due to coarse 16 resolution, SAE still localizes them reliably (e.g., the "bottle" example in Fig. 3), supporting its use for computing alignment scores in $3.3.
single forward pass, whereas the VL.M-based
5 ConceptScope for Dataset and Model Bias Analysis
In this section, we demonstrate the capability of ConceptScope to discover dataset biases. We first perform quantitative evaluation on datasets with annotated bias attributes and show that our approach accurately detects known biases (e.g.. background bias from Waterbirds [59]) in $5.1. We then extend our analysis to real-world datasets such as ImageNet-1K [12], Food101 [4], and SUN397 [69], where bias attributes are not explicitly annotated, revealing previously unrecognized biases in $5.2. Lastly, we show an application of ConceptScope for diagnosing model robustness under concept distribution
shifts in $5.3

---

(o) Known Dioses
Celesk "bioed cless binosl "hrwske
GAMO
(t) Novel
"ecilarr Sinucts
"ridge dlleed
SUNDGO
rial
Figure 4: ConceptScope discovers dataset biases. We discover (a) known and (b) novel biases from bias attribute annotated and unannotated datasets, respectively. The top row of each panel shows bias-aligned examples, and the bottom row shows examples without bias attributes.
Table 2: ConceptScope outperforms training-based baselines in bias discovery (Precision @ 10). For NICO+ values in parentheses represent bias severity levels, with higher values indicating stronger correlations between classes and spurious attributes in the training dataset.
Method
Waterhirds [59]
CelebA [43]
Nico++(75) [76]
Nico++ (90) [76]
Nico++ (95) [76]
DOMINO [17]
90.0%
87.0%
24.0%
24.0%
24.0%
FACTS [73]
100.0 %
100.0%
55.0%
60.8%
61.0%
ViG-Bias [46]
100.0%
100.0%
60.0%
66.7%
65.0%
ConceptScope (Ours)
100.0% %
100.0%
72.9%
73.1 %
74.0% %
5.1 Detecting known biases
Tasks. We evaluate our framework on the bias discovery task introduced by Yenamandra al. [73]. In this setup, each class in the training set is associated with distinct spurious attribute, such as background scene. At test time, images of target class y are divided into subsets, each characterized by a visual attribute spuriously correlated with a different class y'. The goal is to correctly identify the spurious attribute associated with each subset. Performance measured using Precision @ 10 [17], where we retrieve the top 10 most strongly associated with the candidate attribute and compute the proportion that indeed contain the ground-truth spurious attribute. We conduct experiments on three widely used benchmarks: CelebA [43], Waterbirds [59], and NICO++ [76]. Example images are shown in Fig. (a). Details regarding datasets and evaluation protocols are provided in Appendix $D.1.
Methods and baselines. Our approach first identifies bias concepts within each training class. For each class, we extract SAE latent activations corresponding to the bias concepts from other classes and retrieve the top 10 images with the highest activations from the test set. We compare ours against three baselines: DOMINO [17]. FACTS [73], and ViG-Bias [46]. Further details regarding baselines are provided in the Appendix $D.3.
ConceptScope accurately detects known dataset biases. Table 2 shows that our proposed frame- work achieves state-of-the-art performance on the bias discovery task, substantially outperforming existing baselines. Reported results of ConcepScope are averaged over four independently trained SAEs, with standard deviation below 0.02, confirming the robustness of our approach. Unlike baseline methods that rely on classifier training and maluree-mode analysis, ConceptScope directly uncovers bias concepts from the structure of the training dataset. Moreover, once the SAE is trained, it can be applied to analyze biases in other datasets without retraining, leveraging the strong trans- ferability of SAFs [37]. Finally, our method provides localization of spurious attributes through segmentation masks, clearly highlighting the input regions responsible for the bias activation (Fig. 12).
5.2 Discovering novel biases in the wild
Biases in real-world datasets. Using our framework, we discover previously unrecognized biases in real-world datasets. As shown in Fig. 4 (b), we identify various types of biases across ImageNet-11 and SUN397. We detect unexpected object associations (e.g.. "necklaces" frequently appearing with "mannequins"), cultural biases (e.g., the class "bridegroom" correlated with "East Asian cultural" con-

---

Concept distribution
(b) Subgroup accuracy
(c)
IN-S
train
IN
(normalizee)
t
IRX
IN-S
Lest data
dnoioons
Subgroup division
Average accuracy
(9)
Terget
High
Low
Number of images
Fngh
CDR
IN
C21
S43
IN-S
target concept: hippo head blas concept: swimminng 2oo
Figure 5: Diagnose model robustness. (a) We divide test data into four subgroups (1-4) by comparing its concept strength (high low) the training distribution: (h, h), (h, (1. h), and for target and bias concepts.
(b) We use ImageNet-IK (IN) and ImageNet-Sketch (IN-S) for the analysis and observe groups (1) & (2) are the majority in both datasets, and their group-wise accuracy of 34 pretrained vision models (c.g., ResNet, ViTs) shows consistent tendency: from group (1) to (4) in descending order. (c) Example images categorized each subgroup for class "hippo".
texts), and location-specifie biases ("ice skating rink" correlated with "New York City"). Additional examples, including those from the Food101, shown in Fig. 13, with further details provided in Appendix $D.5. Across these datasets, the average number of bias concepts per class is 2.45, indicating that ConceptScope effectively captures multiple biases within a single class (Fig. 7).
5.3 Diagnosing Model Robustness
In this section, we demonstrate the application of our proposed framework, ConceptScope, as a diagnostic tool for model robustness against concept distribution shifts. Model robustness is typically evaluated by measuring the performance drop distribution-shifted test sets, such as from ImageNet [12] to ImageNet-V2 [56] or ImageNet-Shee -ketch [68]. However, in many domains (c.g., food scene classification), such curated out-of-distribution (OOD) benchmarks unavailable, and constructing them manually can be both costly and consuminn, ConceptScope provides complementary approach that enables robustness assessment directly within the original test set, without requiring additional OOD datasets.
Approach. Using target and bias concepts identified by ConceptScope, we partition the test set of each class into four subgroups: (1) high target-high bias, (2) high target-low bias, (3) low target-high bias, and (4) low target-low bias (Fig. 5 (a)). A test sample is classified as having "high" concept strength if its activation exceeds the average activation of that concept across all training images in the corresponding class. These groups can be interpreted as forming spectrum of distribution shift. Group represents the most typical setting seen during training, where both the target and bias concepts co-occur. In contrast, Group 4 represents the target nor the bias concepts are present, posing a stronger generalization challenge for the model.
unusual outlier scenario, where neither
Experimental setup. We compute concept distributions from the training split of ImageNet- KK, then partition the ImageNet validation set and the ImageNet-SSkech test set into distinet groups accordingly. We subsequently evaluate and compare model accuracy across these groups across 34 model weights trained on ImageNet- IK, including diverse architectures such as VGG [62], ResNet [23], and Transformer-basee models [14]. The complete list of models is provided in Appendix $E.
Key findings. In Fig. (b), the y-axis represents subgroup accuracy (with the four groups shown in different colors), and the x-axis indicates the average accuracy across all four subgroups. Each vertical line corresponds
single model, with four colored dots representing that model's accuracy on each subgroup. Among the 34 models evaluated, ConvNeXt-Largg [42] achieves the highest average accuracy and is shown on the rightmost line (in both ImageNet (IN) and ImageNet- Sketth (IN-S)). Notably, it also performs best on Group 4, the most challenging subgroup. We observe that models with higher accuracy on Group (the most frequent training setting) also tend to perform better on Group 4 (the rare outlier setting). This trend is consistent with prior findings [47], which

---

show that out-of-distribue (OOD) performance often correlates with in-distribution performance. Crucially, our approach enables such robustness analysis without requiring an external OOD test set, using only concept-aware subgroups derived from the original data. Interestingly, we observe the same pattern on the ImageNer-Skent despite its large domain shift and generally lower performance. Representative examples from each group are visualized in Fig. 5 (c).
6
Discussion
Concepts beyond ImageNet. Although the SAE is trained on ImageNet, our SAE-derived concepts generalize other datasets used in CLIP zero-shot evaluation, such Food101 [4] and SUN397 [69]. This generalization is expected, as the SAE is not trained using ImageNet class labels. Instead, it learns to reconstruct image representations from general-purpose foundation model, which is assumed to have already captured diverse visual concepts, as evidenced by its strong zero-shot performance. For datasets that differ substantially from ImageNet, such as medical images, the SAE should be trained on the target domain and/or paired with a more suitable backbone model. Prior work, including CytoSAE [11] and Mammo-SAE [50], has shown that SAE can be successfully applied to medical images, and a similar approach can be used for domain -spmcff- analysis.
Extension to multi-label settings. To examine its applicability beyond single-label classification, we apply ConceptScope to a multi-label setting using the MSCOCO 2017 test set [38]. We extract object mentions from ground-truth captions and treat them multi-label targets and conduct concept categorization as in Section 3.2. The analysis reveals distinct bias patterns: cat is often associated with indoor environments, dog with couches and living rooms, and bird with bird feeders and trees. These findings indicate that ConceptScope can detect meaningful bias concepts even in complex, real-world multi-label datasets.
Comparison with existing approach for bias analysis in vision datasets. To further contextualize framework, we additionally review prior work on bias analysis in vision datasets. LaViSE [71] interprets the semantic meaning of individual neurons and uses this to examine gender bias in the MS-COCO dataset [38]. Similarly, SpLiCE [3] decomposes CLIP representations into sparse, human- interpretable concepts based on a predefined concept set and analyzes bias between the Woman and Man classes of CIFAR-100 [31]. While these approaches provide valuable insights into dataset bias, they are limited to predefined gender categories, implicitly treating any unrelated concept as bias. For instance, such methods require manual specification of what constitutes bias (e.g., considering sand environment as a bias for sea turtle). In contrast, ConceptScope automatically determines whether concept is bias or not based on the semanties relationship the class label, removing the need for manually defined bias categories. Further analyses, including detailed comparisons with existing approaches to segmentation- concept discovery, are provided in the Appendix A.
7 Conclusion
Limitations. Our SAE-discovered concepts are constrained by the knowledge encoded in CLIP's representations. For example, concept prediction scores (Table 1) on domain-specific datasets such as RAF-DB [36] (emotion) or DTD [9] (texture) are lower compared to more general datasets. Leveraging vision foundation models fine-tuned on these Nommin-specific datasets could yield more concrete and task-relevant latent concepts. Additionally, and coarse, leaving room for improvement in localization accuracy. Despite these limitations, our concept-level analysis still uncovers previously unrecognized, non-trivial biases.
segmentation masks are patch-level
Conclusion. We introduced ConceptScope, a framework that uses SAE to detect and characterize biases in image classification datasets. Our pipeline automatically builds concept dictionary, categorizes concepts into target, context, and bias, and enables detailed dataset analysis. Experiments show that SARs are reliable concept extractors, surpassing prior methods and achieving state-of-the art results in bias discovery benchmarks. ConceptScope also uncovers new biases in complex datasets such ImageNet and can diagnose model robustness without external out-of- datasets.
10

---

Acknowledgements
This work was supported by Institute for Information communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (RS-2019-11190075 Artificial Intelligence Graduate School Program(KAN) KAIST)), the National Research Foundation of Korea (NRF) grant funded by the Korea (MVS) (No. RS-2025-00555621 and Institute of Information communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.RS-2021-121) 2068, Artificial Intelligence Innovation Hub).
References
[1] Herve Abdi and NJ Salkind. Z-scores. encyclopedia of measurement and statistics, 2007.
[2] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantity- ing interpretabiliity of deep visual representations. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6541-6549, 2017.
[3] Usha Bhalla, Alex Oesterling, Suraj Srinivas. Flavio Calmon, and Himabindu Lakkaraju. Interpreting clip with sparse linear concept embeddings (splice). Advances in Neural Information Processing Systems, 37: 84298-84328, 2024.
[4] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. food-101-mining discriminative components with random forests. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13, pages 146-461 Springer, 2014.
[5] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E. Burke, Tristan Hume, Shan Carter, Tom Henighan, and Chris Olah. Towards monosemanticity: De- composing language models with dictionary learning. tttt::://trasfforee crrrranstormer--rruuttttt circuits. www.ciis...u//22222....2 pub/2023/ conosemantic--eaturree 2023, Anthoopic, Oct 4, 2023.
[6] Rwiddhi Chakraborty, Yinong O Wang, Jialu Gao, Runkai Zheng, Cheng Zhang, and Fernando D De la Torre Visual data diagnosis and debiasing with concept graphs Advances Neural Information Processing Systems, 37: 37:106383 106410, 2024.
[7] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training recognize long-tail visual concepts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3558 3568, 2021.
[8] Muxi Chen, Chenchen Zhao, and Qiang Xu. Hibug2: Efficient and interpretable error slice discovery for comprehensive model dehugging. In International Conference on Learning Representations, 2025.
[9] Mireca Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-360 3613, 2014.
[10] Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.0mm 080000 2023.
[11] Muhammed Furkan Dasdelen, Hyesu Lim, Michele Buck, Katharina Götze,, Carsten Marr, and Steffen Schneider. Cytosae: Interpretable cell embeddings for hematology. arXiv preprint arXiv:2507.12466 2025.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenct: large-scale hicrarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 248-255. leee, 2009.
[13] Carl Doersch, Saurabh Singh, Abhinav Gupta, Josef Sivic, and Alexci A Efros. What makes paris look like paris? Communications the ACM, 100 2015.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1199 2020.
11

---

[15] Lisa Dunlap, Yuhui Zhang, Xiaohan Wang, Ruiqi Zhong, Trevor Darrell, Jacob Steinhardt, Joseph E Gonzalez, and Serena Yeung-Levy. Describing differences in image sets with natural language. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 24199-24208, 2024.
[16] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravee, Zac hatdield-Dodds. Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint 2rXiv:2209. 10652, 2022.
[17] Sabri Eyuboglu, Maya Varma, Khaled Saab, Jcan-Benoit Delbrouck, Christopher Lee- Aee-Messer, Jared Dunnmon, James Zou, and Christopher Ré. Domino: Discovering systematic errors with cross-modal embeddings. arXiv preprint arXiv:2203. 14960, 2022.
[18] Li Fei-Fei, Rob Fergus, and Pictro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In IEEE Conference on Computer Vision and Pattern Recognition Workshop, pages 778. IEEE, 2004.
[19] Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène. and Thomas Serre. Craft: Concept recursive activation factorization for explainability. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pages 2711-27211 2023.
[20] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smymiss, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36:27022-27112, 2023.
[21] Ame Grobrügge, Niklas Kühl, Gerhard Satzger, and Philipp Spitzer. Towards human-undenn uumman-underttndaable multi-dimensional concept discovery. In Proceedings of the Computer Vision and Puttern Recognition Conference, pages 20018-20027, 2025.
[22] Jiawei Gu, Xuhui Jiang. Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-asa-jjudee arXiv preprint arXiv:241 1.15944, 2024.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Rea, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 770-778, 2016.
[24] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedingy of the 2021 Conference on Empirical Methods in Natural Language Prucessing. Association for Computational Linguistics, 2021.
[25] Forrest landola, Matt Moskewiez, Sergey Karayev, Ross Girshick, Trevor Darrell, and Kurt Keutzer. Densenet: Implementing efficient convnet descriptor pyramids. arXiv preprint 1404: 1869, 2014.
[26] Jared Kaplan, Sam McCandlish, Tom Henighan. Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0888 2020.
[27] Dahye Kim, Xavier Thomas, and Deepti Ghadiyaram. Revelio: Interpreting and leveraging semantic information in diffusion models. arXiv preprint arXiv:2411.16725 2024.
[28] Younghyun Kim, Sangwoo Mo, Minkyu Kim, Kyungmin Lee, Jacho Lec, and Jinwoo Shin. Discovering and mitigating visual biases through keyword explanation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11082-11092 2024.
[29] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead. Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the JEEE/CVF international conference on computer vision, pages 4015-4026, 2023.
[30] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73. 2017.
[31] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[32] Sehyun Kwon, Jaeseung Park, Minkyu Kim, Jaewoong Cho, Ernest K Ryu, and Kangwook Lee. Image clustering conditioned on text criteria. In International Conference Learning Representations, 2023.
12

---

[33] RRchaad Landis and Gary G Koch. The measurement of observer agreement for categorical data. biomefricx, pages 159-174, 1977.
[34] Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li. Zejun Ma, and Chunyuan Li. Llava-eext-interllave:: Tackling multi-image, video, and 3d in large multimodal models, arXiv preprint arXiv:2407.07895, 2024.
[35] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language image pre-training with frozen image encoders and large language models. In International conference machine learning, pages 19730-19742. PMLR, 2023.
[36] Shan Li, Weihong Deng, and JunPing Du. Reliable crowdsourcing and deep docality-preservve learning expression recognition the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2584-2593. IEEE, 2017.
[37] Hyesu Lim, Jinho Choi, Jaegul Choo, and Steffen Schneider. Sparse autoencoders reveal selective remapping of visual concepts during adaptation. In International Conference on Learning Representations, 2025.
[38] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740-755. Springer, 2014.
[39] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems, 36:34892-34916, 2023.
[40] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[41] Zhuang Liu and Kaiming He. A decade's battle on dataset bias: Are we there yet? arXiv preprint 2024.
[42] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11976-11998 2022.
[43] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision, pages 3730-3738, 2015.
[44] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, December 2015.
[45] Yulin Luo, Ruichuan An, Bocheng Zou, Yiming Tang, Jiaming Liu, and Shanghang Zhang. Lim as dataset analyst: Subpopulation structure discovery with large language model. In European Conference on Computer Vision, pages 235-252. Springer, 2024.
[46] Badr-Eddine Marani, Mohamed Hanini, Nihitha Malayarukil, Stergios Christodoulddss, Maria Vakalopoulou, and Enzo Ferrante. Vig-bias: Visually grounded bias discovery and mitigation. European Conference on Computer Vision, pages 414-429. Springer, 2024.
[47] John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa. Pang Wei Koh. Vaishaal Shankar, Percy Liang. Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distributio and in-distribution generalization. In International Conference on Machine Learning, pages 7721-7735. PMLR, 2021.
[48] Mazda Moayeri, Wenxiao Wang, Sahil Singla, and Soheil Feizi Spuriosity rankings: Sorting data to measure and mitigate biases. arXiv preprint arXi::2212.02646 2022.
[49] Ron Mokady, Amir Hertz, and Amit H Bermano. arXiv:2111.0 10.9334, 2021.
Clipcap: Clip prefix for image captioning, arXiv preprint
[50] Krishna Kanth Nakka. Mammo-sae: Interpreting breast cancer concept learning with sparse autoencoders. arXiv preprint ar arXiv:2507. 152277 2025.
[51] Yannic Neuhaus, Maximilian Augustin, Valentyn Boreiko, and Matthias Hein. Spurious features
everywhere-larg scale detection of harmful spurious features in imagenet. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 20235-20242 2023.
13

---

[52] Tuomas Oikarinen and Tsui-Wei Weng. Clip-dissect: Automatic description of neuron representations in deep vision networks. arXiv preprint arXiv:2204.1095 2022.
[53] Krishna Pillutla, Sham M. Kakade, and Zaid Harchaoui. Robust Aggregation for Federated Learning. IEEE Transactions on Signal Processing, 70: 142-1154, 2022. doi: 10.1 10.110999SP.222.11533.33
[54] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamola Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In International Conference Machine Learning, pages $748-8763. PMLR, 2021.
[55] Sukrut Rao, Sweta Mahajan, Moritz Böhle, and Bernt Schicle. Discover-thor name: Task-agnostic concept bottlenecks via automated concept discovery. In European Conference on Computer Vision, pages 444-461, Springer, 2024.
[56] Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenct classifiers generalize to imagenet? In International Conference on Machine Learning, pages 5389-5400 PMLR, 2019.
[57] Konstantinos Rematas, Basura Fernando, Frank Dellaert, and Tinne Tuytelaars. Dataset fingerprints: Exploring image collections through data mining. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4867-4875, 2015.
[58] Peter J Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathemarics, 20:53 65, 1987.
[59] Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks group shifts: On the importance of regularization for worst-case generalization. arXiv preprint (rXiv:191.00873/) 2019.
[60] Christoph Schuhmana, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk. Clayton Mullis, Aarush Katta. Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021.
[61]
Ramprasaath R Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? arXiv preprint 1..055061.007450. 2016.
[62] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 101.1.4495555 1556, 2014.
[63] Sahil Singla and Sohcil Feizi, Salient imagenet: How to discover spurious features in deep learning? arXiv preprint arXiv: 10.01301 2021.
[64] Viacheslav Surkov, Chris Wendler, Mikhail Terekhov. Justin Deschenaux. Robert West, and Caglar Gulcehre. Unpacking sdxl turbo: Interpreting text-to-image models with sparse autoencoders. arXiv preprint arXiv:2400. 12.2266, 2024.
[65] Mingxing Tan and Quoe Le. Efficientne:: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-61 6114. PMLR, 2019.
[66] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Flizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfec100m: The new data in multimedia research. Communications of the ACM, 59 (2):6-73,, 2016.
[67] Nanne Van Noord. Prototype-based dataset comparison. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pages 1944-1954, 2023.
[68] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advonces in Neural Information Processing Systems, 32, 2019.
[69] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recogmition from abbey to zoo. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3485-3440 3492, IEEE, 2010.
[70] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4492-1500 2017.
14

---

[71] Yu Yang, Seungbae Kim, and Jungseock Joo. Explaining deep convolutional neural networks via latent visual-semantic filter attention. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 8333-8343, 2022.
[72] Bangpeng Yao, Xiaoye Jiang, Aditya Khosla, Andy Lai Lin, Leonidas Guibas, and Li Fei-Fei. Human action recognition by learning bases of action attributes and parts. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1331-1338. IEEE, 2011.
[73] Sriram Yenamandra, Pratik Ramesh, Viraj Prabhu, and Judy Hoffman. Facts: First amplify correlations and then slice to discover bias. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4794 4804, 2023.
[74] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint wrX1v.600.007146 2016.
[75] Boya Zeng. Yida Yin, and Zhuang Liu. Understanding bias in large-scale visual datasets. In Advances in Neural Information Processing Systems, 2024.
[76] Xingxuan Zhang, Yue He, Renzhe Xu, Han Yu, Zheyan Shen, and Peng Cui. Nico+:: Towards better benchmarking for domain generalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 16036 16047, 2023.
[77] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through adc20k dataset, In Proceedings of the IEEE Conference Computer Vision and Pottern Recognition, 2017.
15

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: In the abstract and introduction, we explicitly state the contributions and scope this work. Scope: We introduce framework for visual dataset analysis. Contribution: 1. We introduce a dataset analysis framework that discovers visual concepts and catego- rizing into target, contextual, and bias concepts. We show novel application Sparse Autoencoder as concept extractor. 3. We show how this framework can be used for model robustness diagnosis.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We present the limitation of this work in Section 6.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
16

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe details for reproducibility each experiment section and also in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
17

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The code with sufficient instructions will be ready by the time of decision. We describe details for reproducibility the paper at this moment.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//ipp..cccc httpsscccpbbcc/gccceceeeeeeccpee gwwii//ggge///e/eeeeeeeee /Coo/eeooeeeeeeeeeeeeee issisniin ssssonolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Details are provided Section 4, and and in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We report error bars for experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information on the computer resources in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / /ttp///eeeppp/eeeeee/eeeeeeeeeee cc httppublic/thiccauuuutuuu
Answer: [Yes]
Justification: This paper conforms with the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impact of the paper in the end of the paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper of dataset and methods.
Guidelines:
existing assets. The answer NA means that the paper does
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
e do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
21

---

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: This paper uses LLMs for textual description generation of the concept dictionary, which a part of our proposed method. We cite references and clearly mentioned how we prompt the models.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https: /attps://neet /ttps///nuuip....eeee wwteeuriss.c//ootte hct.//ooneeerrccee222///11/ for what should or should not be described.
22

---

Appendix
Additional Discussion
24
B Method Details: ConceptScope
24
B.1 SAE training
24
B.2
Concept dictionary
24
B.3
Concept categorization: target, bias, and context
26
B.3.1
Robustness of alignment score potential bias in CLIP
26
B.3.2 Empirical validation thresholds
27
B.3.3 Additional results: examples of concept categorization
28
C Experiment Details: Concept Prediction
28
C.1 Datasets
28
C.2 Baselines
30
C.3 Additional results: concept prediction
31
C,4 Measuring correlation between SAE activation with CLIP similarity
32
D Experiment Details: Bias Discovery
32
D.I Setup
32
D.2 Datasets
32
D.3 Baselines
34
D.4 Qualitative results: examples of discovered known biases
.i..............................
34
D.5 Additional Results: examples of discovered novel biases
34
Experiment Details: Model Robustness Analysis
36
E.1 Full list of models evaluated ImageNet
36
23

---

A
Additional Discussion
Concept discovery based on segmentation. Recent advances segmentation models, such as SAM [29], have achieved impressive generalization performance, motivating several studies to use segmentation for concept discovery. HU-MCD [21], for example, combines SAM 3AM-generated region masks with feature clustering to extract object- background-leved concepts. Although HU-MCD demonstrates potential in identifying high-level visual entities (e.g.. object parts, background ele- ments), it has notable limitations. It struggles to capture fine-grained attributes such textures, colors, or object subtypes, and lacks a formal mechanism to quantify how each discovered concept relates to class identity or dataset bias. HU-MCD typically identifies coarse-level object back- ground concepts (c.g.. "airplane", "airplane tail", "sky"), whereas the Sparse Autoencoders used ConceptScope uncover more granular and semantically diverse concepts such as "commercial airplanes", "military aircraft", and "jets", as well as state-dependent variations like "landing" and "flying".
On the Role of Segmentation in ConceptScope. Unlike traditional segmentation tasks, producing high-quality masks is not the primary objective of our framework. The segmentation derived from SAE spatial attributions is used to compute the alignment score, enabling ablation or isolation of each concept within an image. Our objective is to analyze concept-class relationships and categorize concepts into target, context, and bias types, rather than optimizing segmentation quality. Despite imperfect segmentation performance, ConceptScope effectively reveals meaningful real-world biases, and its interpretability can be further enhanced through integration with external segmentation models.
B Method Details: ConceptScope
In this section, we provide additional technical details and illustrative examples for our proposed framework ConceptScope, supplementing $3 of the main paper.
B.1 SAE training
We include details regarding hyperparameters and computational resources for training the SAE, supplementing the explanations provided in $3.2 of the main paper.
Hyperparameter settings. We train the SAE using OpenAl CLIP VIT-I/14 the vision encoder and extract embeddings from its penultimate (23rd) layer with embedding dimension d = 1024. The vision encoder resizes the input images 224 224, producing 257 tokens consisting of one class token ([CLS]) and 16 x 16 spatial tokens. We apply an expansion factor of 32, resulting in SAE latent dimensionality d' 32, 768. We set the L1 sparsity loss weight À = 8 x 10-5, and the learning rate to 4 x 10-4 with 500 warmup steps using constant scheduling. We initialized decoder bias with the geometric median [53] of the first training batch and applied ghost gradients [5] to resample dead neurons caused by sparsity regularization. We use batch size of 64 and train on ImageNet- 1K training split, comprising approximately 1.28 million images, for epochs.
Computation resources. We ran all experiments on a single NVIDIA RTX A6000 GPU. With precomputed CLIP image embeddings, each iteration (batch size 64) took about 0.79 seconds, used 24 GB of GPU memory, and required 2,208 GFLOPs.
B.2 Concept dictionary
We provide implementation details for constructing the concept dictionary introduced $3.2 of the main paper, including the filtering of meaningful latent dimensions and the use of multimodal Large Language Models (L.MMs) to assign concept names.
Latent filtering. As noted by Lim al. [37], not all SAE latent dimensions are interpretable or monosemantic. In some cases, we observe that multiple unrelated concepts activate the same latent. To improve interpretability.
filter out uninformative entangled latents and retain only interpretable ones for analysis. We first compute image-level activation values fz) for all latent
http://wwwggiinoocooooooeeeeeeen 1k resized__25
24

---

dimensions across the ImageNet training set and apply two filtering criteria. First, we discard latents whose maximum activation never exceeds 0.5, since we empirically observed that activations above 0.5 consistently represent clear concepts, whereas lower values fail to do so. Second, we discard latents whose concept strength exceeds 0.1, because they activate too frequently across many images and therefore lack interpretability. After filtering, we retain 3, 3,303 latent dimensions, which is about 10% of the original set.
Latent interpretation. To assign semantic labels to latent dimensions, we collect the top-activating image patches for each concept and extract their corresponding segmentation masks, following the standard procedure introduced in Lim et al. [37]. Using the top five images and their masks, we query the GPT-4o API alongside the following prompt:
Prompt
Identify the shared concept among images, focusing solely on non-blocked areas excluding dark silhouette areas, using 2-3 clear and specific words. Anawer with concepts only.
Annotating all 3,103 latent dimensions costs approximately $7 USD in total.
knitted fabric
lace patterns
patchwork quilts
yellow objects
orange objects
colors
group of dogs
puppies
leashes, walking
seat belts
car gauges
car bumpers
car mirrors
grills
car accident
writing
hand in pocket
hanging
Figure 6: Examples of discovered concepts. The top row shows the five images from ImageNet with the highest activations for each latent dimension, while the bottom row shows the corresponding segmentation mask aaplied images. The concepts identified include various colors, textures, objects, and actions. The assigned semantic labels are generated by GPT-4o.
25

---

Table 3: Examples of concept descriptions generated by LLaVA-NeXT and GPT-4o, along with their similarity scores computed in CLIP and OpenAl text embedding spaces.
LexVA-NeXT
GPT-4o
CLIP Similarity
OpenAl Similarity
curtains
curtains
1.00
1.00
rooster
red rooster
0.81
0.93
vehicles
convertible cars
0.76
0.88
textured
knitted fringe
0.53
0.81
Examples of discovered concepts. In Fig. 6, we visualize examples of the discovered concepts along with segmentation masks for the five most highly activated images from ImageNet for each concept. These latents captures diverse range of visual concepts, encompassing colors, textures, objects, and actions. Additionally, the discovered concepts capture fine-grained distinctions within broader categories. For instance, car-related concepts include detailed components such as seat belts, gauges, bumpers, rear-view mirrors, grills, and even scenes depicting car accidents. Similarly, dog-relates concepts captures specific scenarios, including groups of dogs, puppies, and dog leashes. Human action concepts include writing, placing hands in pockets, and hanging objects.
Generated annotation quality. Qualitative inspection shows that the generated annotations are meaningful across similar visual patterns, producing concise and interpretable phrases such as "knitted fabric" or "yellow objects" (Fig. 6). To verify that the top five activating images for each concept depict coherent patterns and that the resulting descriptions are stable across multimodal LLMs, we compared captions generated by GPT-4o and LLaVA-NeXT using the same prompt. The average cosine similarity between the two sets of captions was 0.754 (19. CLIP's text embedding space' and 0.862 (=00050) in OpenAl's cext-embedding-adeee -00- space", indicating strong agreement and convergence toward simple, intuitive descriptions. Examples of caption pairs are provided in Table 3.
B.3 Concept categorization: target, bias, and context
In this section, we provide implementation details and empirical validation of the concept categoriza- tion methods, complementing the description in $3.3 of the main paper.
Implementation details for computing alignment scores. explained in $3.3, we categorize each class's concepts into target and context concepts by measuring the differences in cosine similarity between class labels and concept-ablated images. To compute text embeddings, we use only the class name without any additional textual prompts. For each class, randomly sample 128 training images and calculate sufficiency and necessity scores (Eq. 3 in the main paper). Empirically, we observe that the top 20 concepts per class exhibit significantly higher concept strength, while the remaining concepts contribute negligibly. We therefore compute alignment scores only for these top 20 latents.
B.3.1 Robustness of alignment score to potential bias in CLIP
CLIP similarity scores are widely used in practice, including for data filtering [60] and image caption quality assessment [24]. Although CLIP may carry biases from its pretraining data, our method is robust to such potential biases by designing the alignment score to rely on the relative change in similarity when specific concepts are ablated or isolated, rather than on absolute similarity values.
Toy experiment setup. To demonstrate this robustness, we designed simple toy experiment. The concept ocean a potential bias for the class albatross seabird), they often co-occur. Using the Waterbirds dataset [59], we sampled 100 albatross images and applied segmentation masks to isolate the bird from its background. We then composited the segmented birds onto either ocean or forest backgrounds, producing two groups of 100 images each.
Toy experiment results. The CLIP similarity scores between these two groups and the text prompt albatross were 0.281 and 0.252, respectively. As expected, the absolute scores suggest that the
"httss/ttugginngffec.olllaaa htwp://.ww.oolaaaaa/aa/aaaaaaaae
26

---

CLIP slightly favors ocean background than the forest background for the albatross class. If our aligmment scores were sensitive this bias, might incorrectly classify ocean as target rather than a context concept. However, as shown in the table below, the alignment scores for ocean and forest are consistently lower than those of the seabird concept in both background settings, which correctly categorizes them as context concepts. This confirms that our approach is robust to potential CLIP bias.
seabird
ocean
forest
ocean background
1.05
0.83
0.85
forest background
1.15
0.88
0.84
For reference, the average alignment score across datasets is 1.01 + 0.05 for target concepts and 0.87 + 0.04 for context concepts, supporting the correct classification in this example.
Consistency across CLIP variants. To further assess the robustness of our method to CLIP's internal biases, recompute aligmment scores using three different CLIP models trained on diverse datasets: (1) OpenAl CLIP (used in our main experiments) (2) CLIP trained on LAION-28 1n (3) CLIP trained on DataComp" We evaluate the consistency of concept categorization across these models on Food101 [4], SUN397 [69], and ImageNet [12] datasets. Using Cohen's kappa [33] measure agreement, obtain average pairwise score of 0.875, indicating almost perfect agreement. The standard deviation of alignment scores across models is only 0. 10, suggesting that our categorization remains stable despite potential differences in model bias.
B.3.2 Empirical validation of thresholds
Empirical validation of alignment score threshold. We observe that per-class alignment score distributions typically form two clusters: high and low, corresponding target and context concepts. To separate them, we apply threshold defined as the mean plus standard deviation. We select as the value that maximizes the silhouette score [58], a standard clustering metric that quantifies how well each point fits within its assigned cluster compared to other clusters. In Table 4, we report the silhouette score [58] for six datasets: Waterbirds [59], CelebA [43], Nico++ [76], ImageNet [12], Food101 [4], and SUN397 [69]. In most cases, the silhouette score peaks near 0, while some datasets, such as ImageNet, achieve slightly higher scores at a
Table 4: Average silhouette scores (# standard deviation) for different a values across datasets. Each cluster is formed by grouping target and context concepts.
3
-2
Waterbinds
0.00
0.00
0.00
0.74
0.79
0.56
0.00
CelebA
0.00
0.26
0.42
0.53
0.43
0.23
0.00
Nico++
0.00
ImageNet
0.00
0.01
0.21
0.59
0.61
0.48
0.00
Food101
0.00
0.00
0.29
0.60
0.55
0.30
0.05
SUN397
0.00
0.05
0.34
0.56
0.55
0.37
0.06
AVG
0.00 (=0.00) (=0.10)
0.27 (=0.15)
0.59 (=0.08)
0.541
0.32 (=0.20)
0.02 (=0.03)
Empirical validation of concept strength threshold.
Concept strength distributions typically include a small number of outliers representing concepts that occur unusually frequently or with high activation. We detect these outliers using the threshold of f + a=* X oex across all context concepts for each class, where we set a* based on the z-score method, standard technique for outlier detection [1]. Silhouette score analysis support this choice: Table 5 shows that a = 1 yields the highest average score (0.67) across datasets, indicating the clearest separation between bias and non-bias context concepts.
b90K
27

---

Table 5; Average silhouette scores (# standard deviation) for different a values across datasets. Each cluster is formed by grouping context and bias concepts.
Dataset
a
a
α 2
3
Waterbirds
0.55
0.78
0.66
0.00
CelebA
0.55
0.59
0.39
0.00
Nico++
ImageNet
0.62
0.71
0.61
0.27
Fond101
0.62
0.63
0.39
0.02
SUN397
0.61
0.65
0.42
0.05
AVG
0.60 - 0.04
0.67 0.07
0.49 -
0.16 0.25
B.3.3 Additional results: examples of concept categorization
Example of computing alignment scores. An illustrative example from the freight car class in ImageNet presented in Fig. 7. Masking out regions activated by the concept freight train reduces the cosine similarity relative to the original image (from 0.25 to 0.20). In contrast, retaining only these activated regions increases the similarity (from 0.25 0.27). On the other hand. masking regions activated by concepts such as railroad tracks or colorful graffiti art causes negligible changes cosine similarity, and retaining only these regions decreases the similarity. Because the alignment score for freight train exceeds the threshold, it is classified target concept, whereas railroad tracks and colorful graffiti fall below the threshold and are classified as context concepts.
Example of categorized concepts. Fig. shows the concept activation and their categorization using ConceptScope. The y-axis denotes the concept strengths (z) computed over all images belonging to class i.e., fa.y EZ, (f (Zy)c), where Zy represents the set of image embeddings labeled as class y. The x-axis lists concept sorted by type. The right side of the figure shows example images corresponding each concept type. For the freight car class, the primary target concept is zoomed-in in view of freight train, while secondary target concepts include side or front views of stream locomotivees as illustrated in the figure. Concepts such as gravel stones or rusted metal chains are infrequently activated and generally irrelevant. In contrast, colorful graffiti art and railroad tracks are more commonly activated but are not sufficiently discriminative to independently identify image a freight car.
Concepts discovered by ConceptScope provide valuable insights into target classes by capturing both their prototypical representations and diverse visual states. Fig. 9 illustrates additional ex- amples categorized concepts from various ImageNet classes, particularly focusing on classes where typical visual representations and their diversity may not be immediately intuitive. ConceptScope clearly identifies prototypical examples through target concepts, reveals variations using context concepts, and highlights dominant patterns as bias concepts.
C Experiment Details: Concept Prediction
In this section, we provide additional experimental details to supplement $4 of the main paper, including descriptions of datasets, baselines, and further results.
C.1 Datasets
To assign SAE latents to target attributes, we randomly sample 100 images per class from the training split and select the latent that achieves the highest performance (AUPRC). We then evaluate the selected latents using all test samples.
Caltech-101 [18]. Caltech-101 dataset is widely used benchmark for object classification masks,containing 9,146 images across 102 categories. In our experiments, we use the split pro- vided by HuggingFace 12, which includes 3,066 training images and 6,080 test images.
28

---

mask out
retain
Alignment score
Categorization
concept Trright train
N(cly) 0.25
S(=1y) 0.27 0.25
1.08
Atelyl 11.25 +1.08)/2
A (el,) threshold
+1:7
+ target
class Freigght
sim (mage-ct y) 0.20
(e1, y) 0.27
concept
rraircad tracks
N( 0.25 0.25
1.00
0.255
0.84
A(c2,)) (1.0010.84)2 +0.82
(c2,)) threshold
+ 02 la context
(image,
(mage- 0.25
sim 1o2, 0.21
concept e3; "colorful graffit art"
N(3,y) 0.25 0.25
1.00
S(33,)) 0.24 00.25
0.96
A(33,) (1.00+0.961/2
A (c3,y) < threhshold
sim (magee-3, 0.25
sim [c3, y) 0.24
Figure 7: Examples illustrating the computation of alignment scores. We present the detailed process for calculating alignment scores of concepts associated with the class freight car from the ImageNet dataset, including their actual cosine similarity values, as supplementary Fig. 2 (c) in the main paper.
target
freight train
steam locometives
target
Concept strength
calorful graffiti art
railraad tracks
bias
blas
TOUP
0.
metal
gravel stones
Inlpeter
looet wwwtslaaasaie
Cles
context
Figure 8: Examples of concept categorization. As a supplement to Fig. 2 (d) in the main paper, we show the average concept activation values for each identified concept within the class freight from the ImageNet dataset. Example images corresponding to each concept are provided on the right.
DTD [9]. The Describable Textures Dataset (DTD) is designed for texture recognition tasks and contains 5,650 images in total, with 3,770 images in the training split and 1,800 images in the test split 13
Waterbirds [59]. The Waterbirds dataset contains bird images across four different background types: bamboo forest, forest, ocean, and lake. There are 4,056 images in the training split and 5,794 images the test split.
CelebA [44] The CelebA dataset consists of cropped celebrity images annotated with 40 binary facial attributes. The training split includes 162,770 images, and the test split includes 19,962 images. Although this dataset is widely employed, many attributes are subjective (c.g., attractive.
29

---

clsss name
target
bias
context
caulifiower
cauiflower
green ceulillwer
green iceves
bowl
mir Hel
cooked
bubble
soap bubbics
glass art
child
urban
park
lendscapps
blue
summits
hair wig
wearing wig
mannequin
toy models
models
colorful hars
Figure Additional examples of categorized concepts from ImageNet classes. To further illustrate the outputs of concept categorization, we provide additional examples of target, context, and bias concepts. ConceptScope effectively identifies prototypical images for target classes, as well as the contexts in which these targets typically appear (context concepts), and highlights the most frequent contexts that may introduce bias (bias concepts).
big 1ips, or pointy nose) or overly fine-grained (e.g., facial hair divided into 5 o'clock shadow, mustache, sideburns, goatee). To maintain clarity, we select subset of attributes primarily related to hairstyles and accessories. The selected 15 attributes are: bald, bangs, black hair, blond hair, brown hair, eyeglasses, gray hair, male, smiling, wavy hair, straight hair, wearing earrings, wearing hat, wearing necklace, wearing necktie, and young.
RAF-DB [36]. The Real-world Affective Faces Database (RAF-DB) is widely used for emotion recognition research and includes seven emotional categories: happy, sad, surprise, fear, disgust, anger, and neutral. The dataset contains 12,271 training images and 3,068 test images.
Stanford40 Actions [72]. The Stanford40 Action dataset contains images of humans performing 40 distinct actions, including playing guitar, riding bike, cooking, and reading. The dataset contains 5,532 training images and 4,000 test images.
C.2 Baselines
The primary objective of this binary attribute classification experiment in $4 is to verify whether individual latent consistently and selectively correspond to distinct visual concepts. Specifically, latent dimension representing given concept (e.g., "chair") should reliably activate across diverse instances that concept while remaining inactive for unrelated concepts (e.g.. "table"), Given the absence of directly comparable basclines, instead compare approach against VLM-base methods as suitable alternatives.
Setup. For the VLMM-based baseline methods used in $4.1 (Table 1) in the main paper, we follow the SSD-LLM [45] framework with modifications. SSD-LLM identifies the distribution attributes such as backgrounds, actions, co- co-occurring objects related to a target class through four steps: (1) caption extraction using VLMs, (2) identification iterative refinement of the attribute list, and (4) final assignment of attributes captions. The original method heavily relies on L.LMs to earry out steps 2 to 4, our task focuses on evaluating the presence of known target attribute. Therefore, omit steps 2 and 3 (attribute proposal and refinement) and reformulate step 4 as a binary classification task: given a caption, does it imply the presence of the target attribute? This choice follows the growing practice of using LLMs automatic evaluators or "judges" [22]. Specifically, we use gemini -2. 0 fflss-lhitte 14 as the LLM, uinsg the following prompt:
major attributes from generated captions, (3)
30

---

Prompt
"Given the following image caption: {caption) Determine whether the attribute (attribute) is present in the caption, either explicitly or implicitly. Consider synonyms, paraphrases, and implied meanings.
the caption includes the attribute semantically meaningful way, anawer 'yes'. Otherwise, anawer 'no'. Reapond strictly with 'yes' or 'no'."
VLM details. We employ the BLIP-2 [35] model architecture with a 2.7B parameter model 15, Input images are resized to 256 x 256, and captions are generated using the prompt: "Describe this image detail." For LLaVA-NeXT [34], we utilize the 8B parameter model 16, The same prompt used for BLIP-2 is applied, with a maximum token limit set to 256.
C,3 Additional results: concept prediction
Table 6: Performance comparison for concept prediction. We compare SAE-base method (ConceptScope) against caption-based baselines (BLIP-2, LLaVA-NcX), reporting and recall scores across classes.
reporitng average precision
Method
[18]
Waterbird [39]
CelebA (Facial Attr)
RAF-DB [36]
Stanford40 [72]
Average
(Objects)
(Textures)
(Backgrouads)
(Emottion))
(Actions)
[35)
preciston recal!
3.62
0.74
n.50
0.39amcs
a.
0.25
0.82
0.19
0.19am14
0.37 1822
0,85 jass
0.59cazi
0.68
0.37
LLaVA-NXT [4]
precision recall
0.65ea37
a.35
0.63ann2
0.50
a.88
0.53
0.43am
0.82
0.59
ConceptScape (Ours)
rocal!
precision
0.86
D.62anm
4.57
0.77ucre
0.87.0m
0.77an1
0.55gmm
0.58
0.77 ss
0.73
0.74
Caltechr01 (Onjects)
TTextures]
Waterbirds (Beckgrounds)
1.4
12
044
Procoun
04
A1 2rd Daw
02
B1182
RxXT
62
0.0
Reaal
Secal
CalobA (Facial attributes]
RAF-DB (Emotrons)
Stanford 40 (Actions)
06
2.8
3.2
3.0
Rocal
Rocal
Rlocll
Figure 10: Average precision-recall curve for attribute prediction using SAE activations. The shaded blue region represents +1 standard deviation. Red and green markers indicate the mean precision and recall scores of BLIP-2 [35] and LLaVA-NeXT [34], respectively, with dashed lines showing +1 standard deviation.
VLM captions capture general traits but miss specific concepts. In Table 6, we report precision and recall scores for concept prediction tasks as complementary results to Table I of the main paper. Additionally, Fig. 10 shows precision-recall curves for SAE at varying activation thresholds (blue), with baseline performances indicated by red "x" marks for BLIP-2 captions and green "x" marks for LLaVA-NeXT captions. As shown in both the table and the figure, the FLMM-asen baselines exhibit relatively high precision but significantly lower recall. This reduction in recall primarily arises from
15 httpp://uuugnnnniiiccoossssseeep "htpss//tuuggingffce..o/// wwwp://iiigga.......oaaaaaaaaaaa wwt.waawaaalllllaaaaaaaaaa htw.:/allaaaalllaaeeetetttee
31

---

Table 7: Correlation of SAE activation (Ours) & CLIP similarity. High Pearson and Spearman correlations indicate that SAE activation strength consistently reflects the presence of visual concepts. Each value represents the mean and standard deviation across classes, averaged over four models trained with different random seeds.
Metric
Caltech10l [18]
DTD [9] (Textures)
Waterbird [59]
CelebA [43]
RAF-DB [36]
Stanford40 [72] (Actions)
Average
(Objects)
(Backgrounds)
(Facial Attr)
(Emotions)
Pearson
0.64
0.55
0.74 2
0.76
0.65
0.90
0.71 0.65
Spearman
0.45 022
0.53 0.4
0.91 ox7
0.74 042
0.60 0.4
0.71 6.14
the linguistic diversity inherent in VLM-generated captions. Specifically, as demonstrated in Fig. 11, VLM "LM-generated captions tend to describe general visual characteristics but often fail to capture the specificity needed for identifying target attributes. For example, they describe bubbly pattern merely as irregular shape. blonde hair 1 light-colore than specifically fishing. These observations suggest that while VLMs excel at capturing broad visual descriptions, they are less effective in consistently identifying and quantifying discriminative visual concepts within image datasets.
hair, or simply standing rather
C.4 Measuring correlation between SAE activation with CLIP similarity
In 14.1, we report the average correlation between SAE activations and CLIP similarity scores across six datasets. Here, we provide supplementary implementation details and comprehensive results for each individual dataset in Table 7.
Implementation details. For each target class, we first select images where the SAR activation for the target concept latent is greater than Next, we compute the cosine similarity between the image embeddings and the text embedding derived from the class label. We sort the selected images based on SAE activation values and partition them into 100 percentile-based groups. For each group, we calculate the average SAE activation and average CLIP similarity. Finally, we evaluate the correlation between SAE activation and CLIP similarity using Pearson and Spearman correlation coefficients.
D Experiment Details: Bias Discovery
This section provides additional details for the bias discovery experiments presented in $5.1 and $5.2 of the main paper.
D.1 Setup
This subsection provides detailed explanations of the experimental sctups, evaluation metrics, datasets, and baselines, well as additional qualitative results, for the bias discovery experiment described in $5.11 of the main paper.
Evaluation metric. We measure the performance using Precision @ 10 [17], which is calculated as follows: for each candidate spurious attribute, the ten test images predicted with the highest confidence for containing that attribute are selected. Precision @ 10 is then computed the fraction of these selected images that truly contain the attribute. The final score is the average Precision @ 10 across all subset-atrribute pairs.
D.2 Datasets
Waterbirds [59]. The Waterbirds dataset contains two classes: waterbirds and landbirds. The training set comprises 4,795 images. Of these, 95% of the waterbird images have ocean or lake backgrounds, while the remaining 5% feature forest or bamboo forest backgrounds. Conversely, 95% landbird images have forest or bamboo forest backgrounds, with the remaining 5% featuring ocean or lake backgrounds. The test set consists of 5,794 images, balanced equally across backgrounds (50% water backgrounds, 50% land backgrounds) for both classes.
32

---

Dataset Cless
input image
SAE activation
BLIP2
LLaVA-Nox NaXT
Caltech101
illustraion grasshopper. grarshopper depicted in side view. oody oriented honionnaally across image. body gracshogper elongsteed segmenieed, with each segment clearly delineed"
image
diagram am a fly and a bee"
"ant"
'The Image displays cicse-up of textured surface with a blue huc. The texture appears to be pettern of shapes and lnee, giving the impression of cracked or shattered surface.
"abstract blue gliter texture background"
rregular
"bubily"
Waterbrids
"In the image, grass. She dressed in blue shirt and white hat,
giri standing in fiald of tall, grean
girl holding bird the air"
"bambod"
providing a contrast to the greanery around her. In har hands, she holds a black bind, which is perched on her arm."
"The image a close-up portreil e person with neutr beokgr ound that appears be wall with patterned texture, incividual has short, light-oolored hait that is styled the
CeleaA
with beard and jackett
"blond hair"
"The Image shows a close-up of a persen's face pasking through hale in a cardboard box. The person appcars bo young child with fair sknn and dark har. The chiic's is eyes are wide open, and therr mouth slightly open, revealing teeth. The child's expression is one of surprise or shock.
RAF-DB
boy peeking out cardboard box"
"angry"
Stanford- action
"In image,
standing grassy bank, gaze dirested towards body water, dressed baseball cap, plaid green color, bordered by touch nature scona."
man standing the grass near river"
blue joans. The water, light
"ishing"
shrubs, adding
Figure 11: Examples of failure cases from VLM-based baselines. This figure provides illustrative examples explaining why the VL.M-based baselines yield lower performance compared to SAF, as reported in Table of the main paper. Although the generated captions accurately describe images at a general level, they frequently lack the specificity needed to effectively capture target attributes, leading to overly simplified or generalized descriptions.
mammal with
mammal "grass"
mammal 'dimi
mammal with "outdoor*
mammal with 'water"
mammal with "outumn"
The qualitative examples of discovered biases. We illustrate the top 5 test samples of the Nico++ dataset, labeled as "mammals" from each attribute-specific subset. The subset highlighted with red box corresponds to the attribute correlated with the "mammals" class during training (rock background), while the remaining subsets represent spurious attributes associated with other classes. As shown in the figure, each subset consistently shares the same background attributes, clearly visible in the corresponding segmentation masks shown in the bottom row.
Figure 12:
CelebA [44]. Following prior work, we use CelebA for a binary classification task focused on the attribute blond hair. In the training set, approximately 93% of individuals with blond hair are female. The training dataset consists of 162,770 images, and the test dataset consists of 19,962 images.
Nico++ [76]. Yenamandra et al. [73] modified the original dataset into six-way classification task, including the classes: mammals, birds, plants, waterways, landways, and airways. Each class is strongly correlated with particular background: rocks, grasses, dim lighting, water, outdoor, and autumn, respectively. The severity of the bias the training dataset varied across three levels, 75%, 90%, and 95%, indicating the proportion of images in each class exhibiting the correlated background
33

---

attribute within the training set. The training sets contain 9,359 samples for NICO++ 8, 1,124 for NICO+ and 7.844 for NICO+ contrast, the test set designed to be balanced: for each class, images are evenly distributed across the six background types. This results in 300 samples per class (50 per background), for a total of 1,800 test images.
D.3 Baselines
Typical baseline methods for bias discovery involve two stages: first, training models on biased datasets; and second, identifying biases based on model failures in test data through clustering approaches that leverage various features such as model predictions, ground-truth labels, and feature embeddings.
DOMINO [17]. DOMINO fits Gaussian mixture models (GMMs) using model predictions, ground- truth labels, and CLIP image embeddings for each input image. The GMM clusters data samples based on error types, such as false positives or false negatives. Subsequently, DOMINO generates natural language descriptions of biases by identifying the most similar keywords corresponding to each cluster within the CLIP embedding space.
FACTS [73]. FACTS extends DOMINO by adding specialized training objective that amplifies the model's reliance on biased attributes. Specifically, this objective encourages higher confidence scores for samples with biased attributes and lower scores for those without. FACTS then fits separate GMMs within each class to identify distinct subsets associated with bias attributes. Finally, FACTS generates image captions using pretrained captioning models [49] and identifies biases by extracting the most frequently occurring keywords across these captions.
ViG-Bias [46]. ViG-Bias further improves upon FACTS by incorporating visual explanations into the bias discovery process. It first generates masks using Grad-CAM [61] heatmaps derived from the trained model. In the second stage, ViG-Bias uses these masked image embeddings, rather than original CLIP embeddings, to perform clustering. We use the values reported in the original ViG-Bias [46] paper for comparison in Table 2, and apply the same data split when evaluating method
Comparison with baselines. All baseline methods described above require model training and identify biases based on model errors observed in the test dataset. contrast, ConceptScope does not require training any additional models and instead directly identifies biases from the dataset itself. Furthermore, baseline methods assume that each class contains only a single type of bias with high correlation (typically greater than 75% of samples per class). However, this assumption rarely holds true for real-world datasets such as ImageNet. Thus, these baseline methods have not been demonstrated to generalize effectively to real-world settings. In contrast, ConceptScope successfully identifies biases directly within real-world datasets, as demonstrated in $5.2.
D.4 Qualitative results: examples discovered known biases
As discussed in the last paragraph of $5.1, one key advantage of ConceptScope in identifying biases its ability to pinpoint specific regions of the input image responsible for introducing bias. Fig. 12 illustrates this capability by presenting examples of biases discovered in the Nico++ dataset. Specifically, the figure shows the top five test images labeled as mammals from each group, selected based on high activation of the latent dimension identified as a bias for each class. Each group consistently exhibits a particular background attribute, as visually highlighted by the segmentation masks provided in the bottom row. Since ConceptScope directly associates latents with clear semantic meanings, it eliminates the need to generate captions to determine the semantic labels of the discovered biases.
D.5 Additional Results: examples of discovered novel biases
In this subsection, we provide additional examples of discovered biases from real-world datasets, supplementing the findings discussed in $5.2 of the main paper.
ConceptScope identifies diverse biases, including backgrounds, co-occurring objects, and events, in real-world datasets. We illustrate examples of biases discovered by ConceptScope in real-world
34

---

ImageNet "balance beam" blesed "compatition" high activated rate: 36%
ImageNot- "afghan hound" blased "dog show" high activated rate: 43%
ImageNet- "White-headed capuchin" "rainfores"" high activated rate: 53%
ImageNet- "dog sledd blasad "snow" high activated rate: 79%
ImageNet "scuba blased to "underwater" high activated rate: 78%
ImageNot- "leathopper" blased "green leaves" high activated rate: 62%
ImageNet- "red-breasted merganser blased to 'water' high activated rate: 77%
ImageNet smooth green snake biesed to "eerrarumm" high activated rate: 58%
ImageNet "triiobite" biased to "stone" high 58%
ImageNet- - "indigo bunting" biesed to " 'bird feeder' nigh activated
ImageNet- "pier'
ImageNet- "fireboat' biased "water fountains' nigh 50%
biased to "suspension bridges' high rate:
ImageNet- "carbonara" blesed 'bacoo" high activated
ImageNet- "cheeseburger" blased french fires' high activated rate:
ImageNet "trifle" blased to strawberries" high activaled 32%
"hotdog" biased to "oood wrappers' high activated rate: 61%
Food101 "bibmbap" blased to "fried eggs" activatad rata:
Food101 "risotto" blased "white plates" high activated rate: 65%
Figure 13: ConceptScope discovers dataset biases in the wild. We illustrate examples of biases identified by ConceptScope in the ImageNet and Food101 datasets. For each panel, the top row displays samples exhibiting the identified bias attribute, while the bottom row presents samples without the bias attribute. Additionally, we report the proportion of dataset samples with high activation (above a threshold of 0.5) for the corresponding bias latent.
datasets such as ImageNet and Food101 [4]. Our analysis reveals multiple types of biases, including background biases, co-occurring object biases, and event-related biases. Animal-related classes, for example, often exhibit background biases: birds frequently appear near water, reptiles in terrariums, and insects on leaves. Food-related classes commonly display biases involving co-occurring objects, such as burgers paired with french fries. Additionally, we identify biases arising from specific events or contexts, such as balance beams appearing predominantly in gymnastics competitions and certain dog breeds frequently featured at dog shows.
35

---

E Experiment Details: Model Robustness Analysis
In this section, we provide additional experimental details to supplement $5.3 of the main paper.
E.1
Full list of models evaluated on ImageNet
The following list includes all models evaluated on the ImageNet and ImageNer-Sketch datasets. The pretrained weights used for evaluation available https: https://docs. pyttsccytoccccccco org/vision/ stable/models html. For models offering multiple pretrained weight options, we consistently select the ImageNet1K. MK weights.
1. VGG11 [62]
2. VGG16 [62]
3. VGGI9 [62]
4. ResNet18 [23]
5. ResNet34 [23]
6. ResNet50 [23]
7. ResNet101 [23]
8. ResNet152 [23]
9. resnext50 [70]
resnext101 [70]
11. ViT-B-16 [14]
12. ViT-B-32 [14]
13. ViT-L-16 [14]
14. ViT-L-32 [14]
15. ConvNeXI-Tiny [42]
16. ConvNeXt-Base [42]
17. ConvNeXt-Small [42]
18. ConvNeXt-LLarge [42]
19. EfficientNer:BB0665]]
20. EfficientNe:-BI [65]
21. EfficientNer-B22 [65]
22. EfficientNNet-333 [65]
23. EfficienttN:t-B44 [65]
24. EfficientNet-B55 [65]
25. EfficientNez-B6 [65]
26. EfficientNet-B7/66 [65]
27. DenseNet121 [25]
28. DenseNet169 [25]
29. DenseNer201 [25]
30. WideResNer50 [74]
31. WideResNer101 [74]
32. SwinTransformer-Be [40]
33. Trin [40]
34. Swin TwintTraasoomme-SSSaalll [40]
36


Reviews:
[{"review_id": "EuRfCgsTLH", "reviewer": "Reviewer_P7Ht", "summary": "The paper proposes ConceptScore, a framework for discovering bias concepts in class-labeled image datasets. It first uses SAE and VLM to build a concept dictionary from a pretrained model and an image dataset. Then it calculates several metrics based on SAE activations to determine the set of \"target\", \"bias\", \"non-bias contextual\" concepts for a given class y. Experiments demonstrate SAE is able to accurately capture and localize concepts, and on bias discovery task (both known and \"in the wild\") it outperforms existing methods.", "questions": "See weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The tackled problem is very important and underaddressed, especially evidenced by recent research.\n2. The framework of conception dictionaries together with target, bias, and non-bias contextual concepts is novel and intuitive.\n3. The usage of SAE is well motivated and demonstrated to be effective.\n4. The experiments conducted are diverse, covering both the SAE part, and the whole pipeline, with different kinds of datasets and tasks.\n\nWeaknesses:\n\n1. A core contribution is the definition to distinguish the concepts into 3 categories. However, I find this part not clear, and it is a very important part to improve.\n\nThe difference between the alignment scores (equation (4)) and the bias score (equation (5)) should be clarified. Are they highly correlated? What is the difference intuitively? To the readers they all seem to rank the relevance/co-occurance of the concepts in images of class y. In figure 2 (c) confirm this impression where the target concept also ranks higher in formula (5) but its definition is from eqn (4). The scores should be explained in an intuitive manner, not just mathematical languages, especially in clarifying the differences when it is quite subtle. \n\nAlso, the threshold in both (4) and (5) require some explanation/motivation.\n\n\n\n2. The necessity and sufficient score computation uses CLIP embedding of class label y as the \"ground truth\" - masked images with retained patches that generate closer image embeddings to this y embedding are considered closer to the \"true\" class definition. This could be problematic as CLIP itself could be biased in *its* training. The CLIP text encoder may already have learned to associate beaches with turtle, thus when the image contains beach but not turtle, CLIP can still give a high score. This would disturb the calculation of the scores and make it an inaccurate estimation.\n\n\n\n3. The final and very important step of identifying the concept dictionary is using a VLM (e.g., GPT-4o) to annotate, given a few concept-masked referenced images. This step is only described in one sentence. It would be great if more details are given, e.g., the prompt used, how confident the answers are, and how well it aligns with human judgement. Some examples can be presented.\n\n4. This work tackles image datasets with classes (e.g., imagenet), but image-text datasets are gaining dominance, especially in MLLM and generative model training. The work would be more significant if it tackles image-text datasets. This is also acknowledged by authors in the limitation section.\n\nMinor:\n\nLine 202-203: above the average of that of other context concept. Should it be \"above one standard deviation to the right of the average...\"?", "was_revised": true, "final_justification": "Concerns were mostly addressed in rebuttal, and the remaining ones were understandable.", "rebuttal": "***Thank you for the constructive feedback. We hope our responses address the reviewer's concerns.***\n\n### *Summary of responses*\n\n- (W1) We **clarify concept categorization** with both intuitive and mathematical explanations, and **validate threshold**s using silhouette scores.\n- (W2) We show that alignment scores are **robust across CLIP variants** through controlled experiments with high inter-model agreement.\n- (W3) We emphasize that VLM-based annotation is **fully automated** and does not affect the core algorithm.\n- (W4) We demonstrate **broader applicability** by evaluating on the MSCOCO image captioning dataset.\n\n---\n### *(W1) Clarification on concept categorization*\n**Alignment score**\n\nThe **alignment score** quantifies how representative and essential a concept is for predicting a specific class $y$, with higher values indicating greater importance. Let $\\mathcal{C} =$ { $c_1, \\cdots, c_n$ } be the set of concepts that fully reconstruct an image. For a given concept $c_i$, we compute the alignment score by comparing the model's prediction confidence for the target class when the image is reconstructed using $\\mathcal{C} \\setminus $ { $c_i$ } (the necessity score in Eq.(3)) or using only { $c_i$ } (the sufficiency score in Eq.(3)), relative to the original image (i.e., reconstructed using $\\mathcal C$).\n\nIf $\\mathcal{C} \\setminus$ { $c_i$ } explains the class $y$ significantly *less* than $\\mathcal{C}$, while { $c_i$ } alone explains $y$ nearly as well as $\\mathcal{C}$, the alignment score is high—indicating that $c_i$ is an important, class-defining **target concept**. \n\nConversely, if $\\mathcal{C} \\setminus$ { $c_i$ } explains $y$ almost as well as $\\mathcal{C}$, and { $c_i$ } alone fails to explain $y$, the alignment score is low—suggesting that $c_i$ is a non-essential **context concept**.\n\n**Alignment score -- Threshold**\n\nWe observe that per-class alignment score distributions typically form **two clusters**—high and low—corresponding to **target** and **context** concepts. To separate them, we apply a threshold of **mean + α × standard deviation**.\n\nWe evaluate this separation using the **silhouette score** [1], which measures clustering quality. The score peaks near α = 0, supporting the use of the mean (i.e., normalized alignment score > 0; line 194) as a robust, generalizable threshold. Results are averaged over six datasets: Waterbirds, CelebA, Nico++, ImageNet, Food101, and SUN397.\n\n| |  α = -2 | α = -1 | α = 0| α = 1 | α = 2 | \n| - | - | - | - | - | - | \n| **AVG** | 0.05 (±0.10) |0.27 (±0.15)|**0.59 (±0.08)**|0.54 (±0.10)|0.32 (±0.20)| \n\n---\n\n**Concept strength**\n\nWe assess non-target concepts using **concept strength** (line 200), defined as the average SAE activation across all images labeled with the target class. This continuous metric captures both the **frequency** and **magnitude** of concept activation, without relying on arbitrary thresholds.\n\nDue to the sparse nature of activations—typically near zero when absent and high when present—concept strength effectively reflects concept presence. Concepts with significantly high strength are labeled as **bias concepts**; others as **context concepts**.\n\nTo validate its reliability as a frequency proxy, we compared it to a binary frequency measure (activation > 0.5) and found strong agreement (Spearman ρ = **0.73**).\n\n**Concept strength -- Threshold**\n\nWe observe that the distribution of concept strength includes a few outliers, suggesting certain concepts are significantly more frequent or strongly activated than others.\n\nTo identify these outliers, we apply a threshold of **mean + α × standard deviation** across all context concepts for a given class, with α set to 1 (Eq. 5). This aligns with the **z-score** method—a standard technique for outlier detection [2].\n\nTo validate this choice, we compute silhouette scores [1] for varying α values. The score peaks at α = 1, averaging **0.67**, indicating that this setting provides the most coherent separation between **bias** and **context** concepts. Results are averaged over six datasets: Waterbirds, CelebA, Nico++, ImageNet, Food101, and SUN397.\n\n| | α = 0 | α = 1| α = 2 | α = 3 |\n| - | - | - | - | - |\n|**AVG**|0.60 (±0.04)|**0.67 (±0.07)**|0.49 (±0.12)|0.16 (±0.25)|\n\n\n[1] Rousseeuw, P.J. \"Silhouettes: a graphical aid to the interpretation and validation of cluster analysis.\" Journal of computational and applied mathematics (1987).\n\n[2] Abdi, H. \"Z-scores.\" Encyclopedia of measurement and statistics 3 (2007)\n\n---\n### *(W2) Robustness to potential bias in CLIP*\nCLIP similarity scores [1] are widely used in practice, including for data filtering of LAION-400M [2]. Although CLIP may carry biases from its pretraining, our method is **robust to such potential biases** by designing the alignment score to rely on the *relative change* in similarity when specific concepts are ablated or isolated in the image, rather than on absolute similarity values. \n\n**Toy experiment**\n\nTo test robustness against potential CLIP bias, we conduct a toy experiment using the Waterbirds dataset [44]. The concept \"ocean\" is a potential bias for the class \"albatross\" (a seabird), as they often co-occur. We create two groups of 100 \"albatross\" images by segmenting the birds and compositing them onto either \"ocean\" or \"forest\" backgrounds.\n\nCLIP similarity scores with the text prompt \"albatross\" are **0.281** (ocean) and **0.252** (forest), slightly favoring ocean. If alignment scores were sensitive to such bias, **ocean** might be misidentified as a target.\n\nHowever, the alignment scores for **seabird**, **ocean**, and **forest** show that **ocean** and **forest** are consistently categorized as **context concepts**:\n\n|                      | seabird | ocean | forest |\n|----------------------|---------|-------|--------|\n| **ocean background** | **1.05**| 0.83  | 0.85   |\n| **forest background**| **1.15**| 0.88  | 0.84   |\n\nFor reference, the average alignment score across datasets is **1.01 ± 0.05** for target concepts and **0.87 ± 0.04** for context, supporting the correct classification in this example.\n\n---\n\n**Consistency across CLIP variants** \n\nTo assess **robustness** to CLIP’s internal biases, we repeat our analysis using three CLIP models trained on different datasets:  \n(1) OpenAI CLIP (main), (2) LAION-2B, and (3) DataComp.\n\nWe evaluate consistency in concept categorization across Food101, SUN397, and ImageNet. Using Cohen’s kappa [3], we observe an average pairwise score of **0.875**—indicating _almost perfect agreement_—and a standard deviation of alignment scores of **0.110**, showing high consistency despite model differences.\n\n[1] Hessel, J. et al. \"Clipscore: A reference-free evaluation metric for image captioning.\" arXiv (2021).\n\n[2] Schuhmann et al. (2021). Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv (2021).\n\n[3] Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977.\n\n---\n### *(W3) Explanation of VLM-based concept annotation*\n**Purpose and role of VLM annotation**\n\nUsing VLM for concept annotation is a post-hoc step intended to make the extracted concepts easier for human interpretation. Importantly, this step does **not directly influence** the concept categorization, which includes alignment score or concept strength computation and thresholding.\n\n\n**Detailed annotation procedure** \n\nThe full procedure, including the prompt template, is described in **Supplementary Section A.2**, with examples in **Supplementary Fig. 1**. We follow PatchSAE [28] to collect top-activating image patches and their segmentation masks, then use GPT-4o to generate concise natural language descriptions. The process is fully automated, aside from light prompt tuning to select a format that consistently produces accurate and meaningful labels.\n\n\n**Annotation quality**\n\nAlthough we do not manually verify every annotation, qualitative inspection shows that the generated descriptions are consistent and meaningful across similar visual patterns (see **Supplementary Fig. 1** and **Fig. 4**). Examples include concise, interpretable phrases like \"knitted fabric\" or \"yellow objects.\"\n\nTo assess consistency, we compared generated captions by GPT-4o and LLaVA-NeXT (`llama3-llava-next-8b`). The average cosine similarity was **0.754** (±0.119) in CLIP’s text embedding space and **0.862** (±0.050) in OpenAI’s `text-embedding-ada-002` space, indicating strong agreement and convergence toward simple, intuitive descriptions. Examples of caption pairs:\n- LLaVA-NeXT vs. GPT-4o generated -- CLIP similarity, OpenAI similarity\n- curtains vs. curtains -- 1.00, 1.00\n- rooster vs. red rooster -- 0.81, 0.93\n- vehicles vs. convertible cars -- 0.76, 0.88\n- textured vs. knitted fringe -- 0.53, 0.81\n\nWe acknowledge that annotation quality may vary across different VLMs and consider alignment with human perception an important area for future work. \n\n--- \n### *(W4) Generalization to image-text dataset*\n\nTo demonstrate generality beyond single-label classification, we apply our method to a **multi-label** setting using the MSCOCO 2017 test set [1]. We extract object mentions from ground-truth captions as multi-label targets and conduct concept categorization as in **Section 5.2** (lines 305–311). Representative findings include:\n- **Cat**: Bias – “indoor environments”; Context – “office workstation”, “bed sheets”.\n- **Dog**: Bias – “couch”, “living room”; Context – “grass”.\n- **Bird**: Bias – “bird feeder”, “trees”; Context – “cage”, “blue sky”.\n\nThese results show that our method effectively captures concept-level biases and contexts in complex, real-world multi-label scenarios.\n\n[1] Lin, T. et al. \"Microsoft coco: Common objects in context.\" ECCV 2014.\n\n---\n### *Typo*\nThanks for pointing out, we will revise accordingly."}, {"review_id": "YOqBmsGlea", "reviewer": "Reviewer_QZaG", "summary": "This paper introduces ConceptScope, a novel framework for identifying and characterizing biases within image classification datasets. The approach leverages a Sparse Autoencoder (SAE) trained on visual encoder representations (e.g., from CLIP) to automatically discover a dictionary of disentangled visual concepts. The framework then categorizes these concepts for each class as \"target\" (essential to the class definition), \"contextual\" (correlated but not essential), or \"bias\" (contextual concepts that spuriously co-occur with high frequency). The authors empirically validated the approach by investigating its effectiveness in concept prediction and localization tasks in comparison to VLM-based baselines. ConceptScore was then compared to other bias detection approaches in tasks such as detecting known biases in benchmarks like CelebA and Waterbirds. Its capacity to discover novel, unannotated biases in datasets like ImageNet was also investigated. Finally, the paper demonstrates a practical application for diagnosing model robustness by partitioning test sets based on concept activation strength.", "questions": "- Clarification on Definitions: Could you provide a more precise definition of \"concept strength\"? What is the exact threshold used to determine if a strength is \"high\" when identifying a bias concept? The paper states it is $\\mu_{c', y}​+\\sigma_{c', y}$​​ , which seems to imply that there will always be concepts identified as biased. Is this intended?\n- Handling of Multiple Biases: Is the framework fundamentally limited to identifying only one bias concept per class? How would it perform in a scenario with several distinct, equally prominent spurious concepts?\n- Supervision and Manual Effort: Could you clarify which steps of the ConceptScope pipeline are fully automated and which require manual intervention? Specifically, what is the exact process and VLM-prompting strategy used to annotate the latent concepts in the dictionary, and how robust is this to the choice of VLM?\n- Experimental Details:\n  - What population are the error bars in Figure 3 calculated over?\n  - Could you provide a brief definition or a more direct reference for the Precision@10 metric used for the bias discovery task? The paper cites a source but does not define the metric itself.\n  - Computational Cost: How does the computational cost of ConceptScope (including SAE training) compare to the baseline methods in Table 2, which require training models on biased datasets to identify failure modes?\n  - Validity of Target/Context Separation: The stratification of concepts into \"target\" and \"contextual\" seems critical. How does the framework ensure this initial step is not itself biased? For example, using CLIP similarity to measure necessity and sufficiency might inherit any biases present in CLIP's text-image embedding space.", "limitations": "I think it is important to include in the limitations that the core assumption that concepts can be cleanly separated into \"target-essential\" and \"contextual\" categories is a strong one. As raised in the weaknesses, this process may introduce its own biases by enforcing a narrow, monolithic definition of a class (e.g., \"happy face must have a smile\"), potentially causing the framework to mischaracterize the dataset's diversity.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Discovery of Novel Biases: A significant strength is the framework's demonstrated ability to uncover previously unannotated biases in widely used datasets, such as cultural biases in the \"bridegroom\" class or object co-occurrence biases in the \"necklace\" class in ImageNet. This highlights the practical utility of the tool for dataset curators and researchers. \n\n\n- Strong Empirical Performance: In the task of detecting known biases, ConceptScope achieves state-of-the-art results, outperforming several baseline methods on benchmarks like NICO++ without requiring the training of additional classifiers specifically for bias detection.\n\n- Automation and Scalability: The framework seems to be designed to be fully automatic, identifying concepts and their distributions without needing costly, fine-grained human annotations for biases.\n\nWeaknesses\n\n- Clarity and Presentation: The overall presentation of the methodology is confusing. Key definitions are either vague or circular. For example, the paper defines \"visual concepts\" as \"groups of visual concepts\" and introduces \"concept strength\" as a critical measure without defining it upfront, making it difficult to understand the logic before reaching later sections. Figure 2, which is central to the method, lacks sufficient detail, particularly regarding the role of the VLM in building the concept dictionary and the degree of manual intervention required.\n\n\n- Oversimplified Bias Definition: The framework designates a single \"bias concept\" per class based on the contextual concept with the highest activation strength. This is a restrictive definition, as a class may have multiple spurious correlations (e.g., the \"dog\" class could be biased toward \"grass,\" \"parks,\" and \"tennis balls\" simultaneously).\n\n\n- Potential for Introduced Bias: The initial step of categorizing concepts into \"target\" and \"contextual\" may itself introduce bias. For instance, the paper suggests a \"smile\" is an essential target concept for a \"happy face\". This assumption could cause the model to incorrectly penalize or misinterpret valid images of happy faces without a visible smile, thereby introducing a new layer of characterization bias.\n\n\n- Evaluation Gaps: While the method outperforms baselines, the absolute performance on concept segmentation (AUPRC of 0.344) and concept prediction (average AUPRC of 0.66) suggests that the underlying concept extraction has significant room for improvement before it can be considered \"accurate\" or \"reliable\" in a standalone capacity. Furthermore, the submission lacks a comparison of the computational cost against baselines, which is an important practical consideration.", "was_revised": true, "final_justification": "The authors have addressed my concerns during the rebuttal. I would be more confident in recommending accepting the paper if I could read an updated version of the manuscript, but as the authors provided a clear plan for how they are going to incorporate the changes, my final score is \"Borderline Accept\".", "rebuttal": "***Thank you for the constructive feedback. We hope our responses address the reviewer's concerns.***\n\n### *Summary of response*\n\n- (W1, Q1, W2, Q2) We **clarify the definition of concept strength**, the **thresholding procedure** for separating context and bias concepts, and the rationale behind it. \n- (W1, Q3) We emphasize that VLM-based annotation is **fully automated** and does **not influence the core algorithm**. \n- (W3, Q4) We demonstrate that alignment scores are **robust across CLIP variants**, supported by controlled experiments with high inter-model agreement. \n- (W4, Q4.3) We address **computational cost**, highlighting a key strength: ConceptScope is **reusable across tasks without retraining**, unlike baselines. \n- (Q4.1, Q4.2) We provide the **missing experimental details**.\n\n---\n### *(W1, Q1, W2, Q2) Clarification on concept strength*\n**Concept strength**\n\nWe assess non-target concepts using **concept strength** (line 200), defined as the average SAE activation across images of the target class. This continuous metric captures both **frequency** and **magnitude** of concept activation, without relying on arbitrary thresholds.\n\nDue to the sparse nature of activations—typically near zero when absent and high when present—concept strength effectively reflects concept presence. Concepts with significantly high concept strength are labeled as **bias**; others as **context**. To validate its reliability as a frequency proxy, we compared it to a binary frequency measure (activation > 0.5) and found strong agreement (Spearman ρ = **0.73**).\n\n**Concept strength -- Threshold**\n\nWe observe that the distribution of concept strength includes a few outliers, suggesting certain concepts are significantly more frequent or strongly activated than others.\n\nTo identify these outliers, we apply a threshold of **mean + α × standard deviation** across all context concepts for a given class, with α set to 1 (Eq. 5). This aligns with the **z-score** method—a standard technique for outlier detection [2].\n\nTo validate this choice, we compute silhouette scores [1] for varying α values. The score peaks at α = 1, averaging **0.67**, indicating that this setting provides the most coherent separation between **bias** and **context** concepts. Results are averaged over six datasets: Waterbirds, CelebA, Nico++, ImageNet, Food101, and SUN397.\n\n||α = 0|α = 1|α = 2|α = 3 \n|-|-|-|-|-|\n|**AVG**|0.60 (±0.04)|**0.67 (±0.07)**|0.49 (±0.12)|0.16 (±0.25)|\n\n**ConceptScope identifies multiple biases**\n\nThis threshold yields an average of **2.4** bias concepts per class. For example, in **Supplementary Fig. 3**, the \"freight car\" class includes bias concepts like “graffiti” and “rail”. This is intentional—our goal is to identify dominant context concepts likely to reflect dataset artifacts that can mislead model behavior. Empirically, this approach performs well on bias discovery benchmarks (**Tab. 2**) and reveals previously undocumented biases (**Supplementary Fig. 8**), validating its effectiveness.\n\n\n[1] Rousseeuw, P.J. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. (1987).\n\n[2] Abdi, H. Z-scores. Encyclopedia of measurement and statistics 3 (2007)\n\n---\n### *(W1, Q3) ConceptScope is fully automated*\nThe entire ConceptScope pipeline is **fully automated**. Given a class-labeled dataset, it outputs visual concepts with class-dependent categorization with no manual labeling or human feedback.\n\n**Step 1 (Fig. 2(a–b))**: After SAE training, we extract top-activating image patches and corresponding segmentation masks for each latent. To aid interpretation, we use GPT-4o to generate brief descriptions based on visual evidence. The process requires **no manual intervention**, except for minimal prompt tuning—we tried a few prompts and kept the ones that gave qualitatively good results (detailed in **Supplementary Sec. A.2** (lines 52–56) with examples in **Supplementary Fig. 1**). Importantly, this annotation step does **not directly influence** the concept categorization.\n\n\n**Step 2 (Fig. 2(c–d))**: We compute **alignment scores** to separate target vs. context concepts, and use **concept strength** to identify **bias concepts** among the context ones. This step is fully automated and VLM-free.\n\n---\n### *(W3, Q4.4) Robustness and validity of target vs. context*\n**Clarification on target and context concept definition**\n\nLet $\\mathcal{C} =$ { $c_1, \\cdots, c_n$ } be the set of concepts that fully reconstruct an image. For a given concept $c_i$, we compute the alignment score (Eq.(4)) by comparing the model's prediction confidence for the target class when the image is reconstructed using $\\mathcal{C} \\setminus$ { $c_i$ } (the necessity score in Eq.(3)) or using only { $c_i$ } (the sufficiency score in Eq.(3)), relative to the original image (i.e., reconstructed using $\\mathcal C$).\n\nIf $\\mathcal{C} \\setminus$ { $c_i$ } explains the class $y$ significantly *less* than $\\mathcal{C}$, while { $c_i$ } alone explains $y$ nearly as well as $\\mathcal{C}$, the alignment score is high—indicating that $c_i$ is an important, class-defining **target concept**. \n\nConversely, if $\\mathcal{C} \\setminus$ { $c_i$ }explains $y$ almost as well as $\\mathcal{C}$, and { $c_i$ } alone fails to explain $y$, the alignment score is low—suggesting that $c_i$ is a non-essential **context concept**.\n\n**Subjectivity of target concept definition**\n\nGiven the ambiguity of what constitutes an \"essential\" concept, there is no gold-standard annotation for labeling concepts as “target” or “context.” This makes evaluation using standard metrics like precision or recall infeasible. Instead, **we conduct a bias discovery experiment** on *known* biases and show that the identified bias concepts are at least as accurate, as shown in **Tab. 2**.\n \n---\n\n**Robustness to potential bias in CLIP**\n\nCLIP similarity scores [1] are widely used in practice, including for data filtering in LAION-400M [2]. While CLIP may carry pretraining biases, our method is **robust to such biases** by designing the alignment score to depend on the *relative change* in similarity when concepts are ablated or isolated, rather than on absolute similarity values.\n\n**Toy experiment**\n\nTo test robustness against potential CLIP bias, we conduct a toy experiment using the Waterbirds dataset [44]. The concept \"ocean\" is a potential bias for the class \"albatross\" (a seabird), as they often co-occur. We create two groups of 100 \"albatross\" images by segmenting the birds and compositing them onto either \"ocean\" or \"forest\" backgrounds.\n\nCLIP similarity scores with the text prompt \"albatross\" are **0.281** (ocean) and **0.252** (forest), slightly favoring ocean. If alignment scores were sensitive to such bias, **ocean** might be misidentified as a target.\n\nHowever, the alignment scores for **seabird**, **ocean**, and **forest** show that **ocean** and **forest** are consistently categorized as **context concepts**:\n||seabird|ocean|forest|\n|-|-|-|-|\n|**ocean background**|**1.05**|0.83|0.85|\n|**forest background**|**1.15**|0.88|0.84|\n\nFor reference, the average alignment score across datasets is **1.01 ± 0.05** for target concepts and **0.87 ± 0.04** for context, supporting the correct classification in this example.\n\n---\n\n**Consistency across CLIP variants** \n\nTo assess **robustness** to CLIP’s internal biases, we repeat our analysis using three CLIP models trained on different datasets:  \n(1) OpenAI CLIP (main), (2) LAION-2B, and (3) DataComp.\n\nWe evaluate consistency in concept categorization across Food101, SUN397, and ImageNet. Using Cohen’s kappa [3], we observe an average pairwise score of **0.875**—indicating _almost perfect agreement_—and a standard deviation of alignment scores of **0.110**, showing high consistency despite model differences.\n\n[1] Hessel et al. Clipscore: A reference-free evaluation metric for image captioning. arXiv (2021).\n\n[2] Schuhmann et al. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv (2021).\n\n[3] Landis et al. The measurement of observer agreement for categorical data. Biometrics. 1977.\n\n---\n### *(W4) Performance on concept segmentation and prediction*\n\nWhile concept segmentation and prediction can be improved, ConceptScope already outperforms SOTA baselines. Our primary goal is not perfect attribute prediction, but extracting open-vocabulary concepts for **dataset bias analysis**—a capability missing in existing methods.\n\nAs shown in **Tab. 2**, **Fig. 4** (main), and **Fig. 8** (supplement), we uncovers *previously unrecognized, non-trivial biases* through concept-level analysis, even without perfect accuracy.\n\nTo our knowledge, no prior work systematically evaluates both concept prediction and spatial attribution of SAE as done in Sec. 4.1 and 4.2, marking this as a key contribution of our paper.\n\n---\n### *(W4, Q4.3) Computational cost*\nWe report the SAE training cost in **Supplementary Section A.1 (lines 37–39)**: ~2,208 GFLOPs per batch (batch size 64) with precomputed CLIP activations. Direct comparison with Tab. 2 baselines is difficult due to differing goals and setups.\n\nConceptScope performs **a one-time SAE training** on a general dataset (e.g., ImageNet) and can be reused across downstream tasks **without retraining**. In contrast, baselines detect bias via model errors and must **train a new classifier for each dataset**. Although individual classifiers (e.g., ResNet50 at 4 GFLOPs/image) are cheaper, their cost adds up, making baselines less scalable for multi-dataset analysis.\n\n---\n### *(Q4.1, Q4.2) Experiment details*\n**Error bars in segmentation task**: The error bars in Fig. 3 represent the standard deviation across 150 classes in the segmentation task.\n\n**Precision@10 metric of bias discovery task**: The details of the bias discovery experiment are described in **Section C** of the Supplementary (lines 208-212). \n\n### *Typo*\nThanks for pointing out, we will revise accordingly."}, {"review_id": "t5rEPz8FFA", "reviewer": "Reviewer_6bLm", "summary": "The paper leverages sparse auto-encoders (Cunningham et al [7]) to discover concepts related to the image target class, the context, and potential bias. Methodologically, it follows a similar line work from Lim et al [12], which are leveraging sparse autoencoders to reveal selective remapping of concepts. The paper relies on multimodal LLM to discover concepts that emerge from a vision encoder trained on ImageNet. Evaluation assesses whether such framework can predict concepts in various datasets and discover biases present in visual datasets.", "questions": "I am happy to increase my score with improvements regarding discussion and comparison with missing related work, details about how concepts are annotated and their limitations.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n*S1. The method enables to discover novel biases*\n* Figure 4 illustrates novel biases being discovered by the proposed method. It is valuable that the method goes beyond known biases in a dataset and enables to discover relations between concepts that can affect the object classification performance.\n\n*S2. Concepts are visually grounded*\n* Figure 3 illustrates how discovered concepts are grounded in images. This offers a better interpretability of the discovered concepts.\n\n**Weaknesses**\n\n*W1. Related work section is insufficient*\n* The paper ignores related works in bias discovery, which is a new but already rich field. It would be beneficial to discuss differences in terms of datasets, biases, and evaluation protocol. For example, here are a few:\n  * Beery et al. Recognition in Terra Incognita. ECCV 2018.\n  * Hirota et al. SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIP. ICLR 2025.\n  * Lang et al. Explaining in Style: Training a GAN to explain a classifier in StyleSpace. ICCV 21.\n  * Li et al. Discover the Unknown Biased Attribute of an Image Classifier. ICCV21.\n  * Zhang et al. Discover and Mitigate Multiple Biased Subgroups in Image Classifiers. CVPR 24.\n\n*W2. The annotation process for concepts is unclear*\n* The paper only mentions that it \"annotates latent dimensions using multimodal LLM based on reference with localized activations\". How many concepts were annotated, and how was this decided? Are there any manual checks to verify the validity of the annotations? Can any bias be introduced in the annotation process? Is there any manual input in the annotation process, or all is done by an LLM? In the current form, it is hard to understand how concepts are discovered and reproduce the annotation process.\n\n*W3. Out-of-distribution concepts are unclear*\n* The paper is based on discovering concepts present in ImageNet. What if we want to characterize a dataset for which concepts were not discovered?\n* Furthermore, what is a sufficient set of concepts to be able to perform the different tasks assessed in the paper?\n* On a related topic, in Table 1, the performance on the DTD dataset is quite poor. What makes this dataset challenging? Similar question for the RAF-DB dataset.", "was_revised": true, "final_justification": "The rebuttal has addressed my concerns and is clarifying aspects regarding the related work, the annotation process, and properties of the method.", "rebuttal": "***Thank you for the constructive feedback. We hope our responses address the reviewer's concerns.***\n\n### *Summary of responses*\n- (W1) We clarify that unlike prior work focused on model bias, **ConceptScope directly analyzes the training dataset**, offering a fundamentally different approach.\n- (W2) We explain that **concept extraction relies solely on SAE**, not on multimodal LLMs. LLMs are used only post hoc for interpretation and do **not affect the core algorithm**.\n- (W3) We show that **SAE-learned concepts generalize across datasets**, supported by empirical activation statistics.\n\n---\n### *Clarification on summary*\nOur proposed method does not rely on multimodal LLM to discover concepts. We use SAE for concept extraction. We clarified the purpose and role of generated annotation in (W2) Clarifying the concept annotation process.\n\n---\n### *(W1) ConceptScope analyzes dataset*\nWhile existing work addresses important aspects of model bias, **none of them directly analyze the structure of the training dataset**. In contrast, our method focuses on **automatically discovering and characterizing bias by analyzing visual concept distributions in the training data**, which is largely orthogonal to these prior efforts. Below, we outline the key distinctions:\n\n**Model robustness benchmarks**\n- **Beery et al. (ECCV 2018)** introduced a benchmark to evaluate model robustness under distribution shifts across geographical locations. While it highlights the importance of dataset-induced bias, it does not propose methods for identifying or interpreting biased attributes within the dataset itself.\n\n**Societal bias mitigation**\n- **Hirota et al. (ICLR 2025, SANER)** aim to mitigate societal biases (e.g., gender, race) in CLIP by neutralizing sensitive attributes. In contrast, our work does not assume any predefined sensitive attributes. Instead, we focus on **automatically discovering unknown or subtle biases** directly from the dataset.\n\n**Concept-based explanation methods**\n- **Lang et al. (ICCV 2021, StylEx)** and **Li et al. (ICCV 2021)** use generative models (e.g., StyleGAN2) to discover interpretable latent directions that explain classifier decisions. While ConceptScope also extracts interpretable visual concepts—using a sparse autoencoder over frozen CLIP features—our goal is fundamentally different: we use these concepts to **characterize the dataset itself**, not to explain predictions of a specific model.\n\n**Subgroup discovery via failure analysis**\n- **Zhang et al. (CVPR 2024)** identify model bias by segmenting test samples into subgroups with poor performance and finding shared visual features. This is related to baseline methods we compare against in Table 2. However, these methods detect dataset bias indirectly—through downstream model behavior. If the training dataset is biased (e.g., birds frequently co-occurring with ocean backgrounds), but the test set lacks bias-conflicting examples (e.g., birds without ocean), such methods cannot detect the presence of bias in the training data or the model’s reliance on it.\n- More importantly, as discussed in the main paper (lines 80–81) and Supplementary Section A.5 (lines 251–258), these methods do not directly inspect the training data and thus cannot reveal **which training samples contain bias-inducing features** or **how prevalent** those features are.\n\nIn summary, **ConceptScope directly analyzes the training dataset** to uncover the distribution of learned concepts, including stereotypical visual states, contextual co-occurrences, and potential sources of spurious correlations (see **Supplementary Figures 3 and 4**). This enables a more comprehensive and interpretable understanding of dataset-level bias, independent of any downstream model or task. We believe this ability to inspect and interpret the internal structure of the training data itself represents a significant and novel contribution to the field of dataset bias analysis.\n\n---\n### *(W2) Clarifying the concept annotation process*\n**Purpose and role of VLM annotation**\n\nThe use of a VLM for concept annotation is a post-hoc step intended to make the extracted concepts easier for human interpretation. Importantly, this step does **not directly influence** the concept categorization, which includes alignment score or concept strength computation and thresholding.\n\n**Detailed annotation procedure**\n\nWe detail the annotation procedure—including the prompt template—in **Supplementary Section A.2**, with examples of concepts and their annotated labels shown in **Supplementary Figure 1**. We first collect the top-activating image patches for each concept and extract their corresponding segmentation masks, following the standard procedure introduced in PatchSAE [28]. We then apply a GPT-4o to generate short natural language descriptions from these visual inputs. The annotation process is fully automated, except for light prompt tuning: we tested several prompt formats and selected one that consistently yielded concise, accurate, and semantically rich descriptions.\n\n**Annotation quality**\n\nAlthough we do not manually verify every annotation, qualitative inspection shows that the generated descriptions are consistent and meaningful across similar visual patterns (see **Supplementary Fig. 1** and **Fig. 4**). Examples include concise, interpretable phrases like \"knitted fabric\" or \"yellow objects.\"\n\nTo assess consistency, we compared generated captions by GPT-4o and LLaVA-NeXT (`llama3-llava-next-8b`). The average cosine similarity was **0.754** (±0.119) in CLIP’s text embedding space and **0.862** (±0.050) in OpenAI’s `text-embedding-ada-002` space, indicating strong agreement and convergence toward simple, intuitive descriptions. Examples of caption pairs:\n- LLaVA-NeXT vs. GPT-4o generated -- CLIP similarity, OpenAI similarity\n- curtains vs. curtains -- 1.00, 1.00\n- rooster vs. red rooster -- 0.81, 0.93\n- vehicles vs. convertible cars -- 0.76, 0.88\n- textured vs. knitted fringe -- 0.53, 0.81\n\nWe acknowledge that annotation quality may vary across different VLMs and consider alignment with human perception an important area for future work. \n\n---\n### *(W3) Generalizability of ConceptScope*\n**Concepts beyond ImageNet**\n\nAlthough the SAE is trained on ImageNet, the resulting visual concepts are not limited to the ImageNet dataset; rather, they **generalize well to other datasets**--commonly used classification dataset for CLIP zero shot evaluation--such as Food101 and SUN397 (scene) (Lim et al., 2025). This generalization is expected, as the SAE is not trained using ImageNet class labels. Instead, it learns to reconstruct image representations from a general-purpose foundation model—the CLIP vision encoder—which is assumed to have already captured diverse visual concepts, as evidenced by its strong zero-shot performance.\n\nIf we aim to characterize a dataset that is significantly different from ImageNet—such as medical images—we should train SAE on that specific domain and/or with a more suitable backbone model. As demonstrated in prior work (CytoSAE [1], Mammo-SAE [2]), applying SAE to medical images is feasible, and a similar pipeline can be adopted for domain-specific analysis.\n\n[1] Dasdelen et al. (2025). CytoSAE: Interpretable Cell Embeddings for Hematology. arXiv.\n\n[2] Nakka et al. (2025). Mammo-SAE: Interpreting Breast Cancer Concept Learning with Sparse Autoencoders. arXiv.\n\n**Sufficient set of concepts**\n\nWe have not yet systematically analyzed how many or how granular concepts are sufficient for different tasks, but we consider this an important direction for future work.\n\nAlthough we do not provide a standardized benchmark through extensive comparisons, we present **empirical evidence** suggesting that approximately 3,000 concepts are sufficient for meaningful concept extraction and categorization.\n\nIn this work, we found that the SAE learns around `3,000` meaningful concepts. Among these, `2,537` concepts are frequently activated in ImageNet, `259` in Food101, `1328` in SUN397, and so on. On average, approximately `20` concepts are activated per class.\n\n**Performance on fine-grained dataset**\n\nSince our concepts are derived from CLIP’s representations, they are inherently constrained by CLIP’s learned knowledge. However, using vision foundation models fine-tuned on domain-specific datasets like RAF-DB or DTD could improve performance by yielding concrete and task-relevant latent concepts.\n\nRAF-DB is a facial expression recognition dataset with 7 emotion categories. It is particularly challenging because some emotions—such as \"fear\" and \"surprise\"—are difficult to distinguish even for humans due to their subtle visual differences. Although CLIP’s large-scale training data may contain some facial images with emotional expressions, it was not explicitly trained on facial expression recognition tasks. As a result, it lacks the fine-grained supervision needed to capture subtle facial cues, leading to relatively low zero-shot performance on RAF-DB (39.8%) [1].\n\nDTD is a texture classification dataset with 47 fine-grained texture categories. Many of these textures are abstract and visually similar, lacking distinctive semantic features. This makes zero-shot classification difficult, and CLIP achieves an accuracy of 50.4% on DTD [2].\n\n[1] Zhao, Zengqun, et al. \"Enhancing zero-shot facial expression recognition by llm knowledge transfer.\" WACV 2025.\n\n[2] Wu et al. \"How well does CLIP understand texture?.\" arXiv(2022)."}, {"review_id": "JbdjX1mPpZ", "reviewer": "Reviewer_usgc", "summary": "This paper introduces ConceptScope to detect biases in datasets by disentangling visual concepts through a sparse autoencoder. This is achieved by training a linear sparse autoencoder to reconstruct visual features for a pretrained model. The authors found that the activation of sparse codes reveals one concept, e.g., object and texture. Based on this, the normalized alignment score is proposed to distinguish concepts for target, contextual, and bias. The approach is able to discover biases in any datasets and achieves good performance in 5 benchmarks for bias discovery. ConceptScope can be also used for robustness evaluation.", "questions": "What are the advantages of SAE compared to clustering in concepts discovering?\n\nHow Figure 5 diagnoses model robustness? It is reasonable to find the target with biases getting the best accuracy. The trend cannot tell me how robust the model is. What is the robustest model from 34 models?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The proposed method for concept discovering via SAE is simple yet effective, providing insights on diagnosing biases in datasets. \n2. The metrics of detecting bias is novel. The experimental results reveal good performance in detecting biases among datasets.\n3. The paper is easy to follow. \n\nWeakness:\n1. This work only validates the concept discovering for CLIP. As we know SAM is a powerful segmentation model, how well it will be to apply SAE into SAM?\n2. The comparison to feature clustering is missing. The features can generate good segmentation mask for concepts if proper threshold is used [1]. Therefore, performing clustering can also get visual concepts. It is vital to know the advantages of sparse autoencoder comparing to clustering.\n\nLanguage Issue:\n1. In line 217, \"For each target attribute y, we image-level activation value...\"\n\n[1] Wu J, Mo S, Awais M, et al. Masked momentum contrastive learning for zero-shot semantic understanding[J]. arXiv preprint arXiv:2308.11448, 2023.", "was_revised": true, "final_justification": "This work proposes ConceptScope to detect the biases on the data, which is novel to provide a model robustness evaluation without OOD benchmarks. The response from authors resolves most of my concerns. However, we can see from the response that there is a gap between the proposed concept discovery and the SOTA. Since it is not the major contribution and can be improved in the future work, this work has the potential to be applied for model robustness diagnoses.", "rebuttal": "***Thank you for the constructive feedback. We hope our responses address the reviewer's concerns.***\n\n### *Summary of response*\n- (W1) Regarding the concern that SAE is only applied to CLIP, we clarify that our goal is to **categorize concepts and analyze class-level biases and visual characteristics**. We also provide a quantitative comparison across different CLIP variants for the concept prediction task.\n- (W2, Q1) Regarding the lack of comparison with feature clustering methods, we note that this line of work is fundamentally different. Instead, we **qualitatively compare ConceptScope with SAM-based concept discovery** and show that our method captures more fine-grained object and texture concepts that baseline methods miss.\n- (Q2) Regarding questions about model robustness diagnosis, we explain that **ConceptScope enables practical and interpretable robustness assessment** through concept-based subgroup analysis, even in the absence of explicit distribution shift datasets.\n\n---\n### *(W1) Discussions regarding SAM-based concept analysis*\nWhile our work applies SAE to the CLIP encoder, a similar approach can be applied to other vision models, such as Segment Anything Model (SAM). As mentioned in our related work (line 92), SAE-based concept analyses have been applied to diffusion models, suggesting broader generalizability.\n\nApplying SAE to the encoder of SAM and comparing its concepts with those from CLIP is indeed a promising direction for future research. However, this is beyond the current scope of our work. Here, our primary goal is to **categorize decomposed concepts** of CLIP representation **into target, bias, and context types** to **analyze class-level biases and visual characteristics** in class-labeled datasets.\n\nAlthough comparing SAE across model architectures is outside the scope of this work, we evaluate its **robustness across different CLIP variants**, (all ViT-L/14) pretrained on different datasets:\n\n- OpenAI CLIP (used in our main experiments)\n- DataComp CLIP\n- LAION CLIP\n\nTo evaluate performance, we conduct a concept prediction task (Tab. 1, Sec. 4.1 of the paper) across six benchmark datasets (see Table 1 of the paper).\n\n| Model|Caltech101|DTD|Waterbird|CelebA|RAF-DB|Stanford40|AVG |\n|-|-|-|-|-|-|-|-|\n|BLIP2*|0.74|0.34|0.60|0.40|0.25|0.47|0.41|\n|LLaVA-NeXT*|0.64|0.42|0.62|0.59|0.47|0.43|0.47|\n|OpenAI (**ours**)|0.83|0.52 |0.78|0.67|0.55|0.75|0.68|\n|DataComp (**ours**)|0.85| 0.52|0.76|0.68|0.39 |0.68|0.65|\n| LAION (**ours**)| 0.80  |  0.63 | 0.78| 0.69| 0.44|0.64|0.66|\n\n*: baseline methods from Tab. 1\n\nDespite some variation across CLIP variants, our method consistently outperforms baseline models (0.41, 0.47) by a substantial margin, achieving average F1 scores between 0.65 and 0.68.\n\n\nIt is also worth mentioning that, to the best of our knowledge, **no prior work has quantitatively evaluated both the concept prediction task (Section 4.1) and their corresponding spatial attributions (Section 4.2)** of SAE concepts. Evaluating these aspects systematically is one of our key contributions.\n\n---\n### *(W2, Q1) Comparison to feature clustering*\nThe paper referenced by the reviewer primarily focuses on enhancing zero-shot segmentation using visual prompts (e.g., point-click) for self-supervised segmentation models. This direction is fundamentally different from ours. In contrast, our goal is to **automatically discover semantically meaningful and class-dependent visual concepts**, without any human prompts or pre-defined labels, and use them to **analyze dataset-level bias**.\n\nUnlike traditional segmentation tasks, producing a high-quality mask is not the main purpose of segmentation derived from SAE spatial attribution in our framework. Instead, these attributions are used when computing the alignment score (Eq.(3) and Eq.(4)), to ablate or isolate each concept from an image. Our focus is to analyze concept–class relationships and **categorizing concepts** (target, context, bias), not on optimizing the segmentation quality.\n\nA relevant comparison candidate is HU-MCD [1], which combines SAM-generated region masks with feature clustering to extract object- or background-level concepts. Although HU-MCD demonstrates potential in discovering high-level visual entities (e.g., object parts, background elements), it has notable limitations:\n- It struggles to identify fine-grained attributes such as textures, colors, or object subtypes.\n- It provides no formal quantification of how each discovered concept relates to class identity or dataset bias.\n\nA direct quantitative comparison would ideally require a benchmark dataset annotated with diverse concept types (e.g., objects, textures, colors, contexts). However, such a benchmark does not currently exist, to the best of our knowledge. Alternatively, we provide **qualitative comparisons** using the ImageNet classes—“Airplane,” “Beach Wagon,” and “Tailed Frog”—selected from Figure 3 of the HU-MCD paper.\n\nWe observed that \n- HU-MCD identifies broad object or background concepts like “airplane,” “airplane tail,” and “sky\". In contrast, SAE discovers **more fine-grained and semantically diverse concepts**, such as \"commercial airplanes\", \"military aircraft\", and \"jet\" as well as different states such as \"landing\" and \"flying\".\n- For the class “Tailed Frog,” SAE captures **distinct attributes** such as \"dotted skin texture\", \"frog eyes\", along with diverse backgrounds such as \"mud\", \"terrariums\", \"walls\", \"swamp\".\n- In the case of “Beach Wagon,” SAE reveals **detailed components**, including \"car grille\", \"car bumper\", \"backlights\". Additional car-related concepts are provided in Supplementary Figure 1 (“seat belts,” “side mirrors,” “car accidents”).\n\nThese findings highlight SAE’s strength in producing nuanced, human-interpretable, and semantically disentangled concepts that go beyond what region-based clustering typically provides. Moreover, ConceptScope goes beyond discovering concepts—it also reveals which of these concepts appear frequently and disproportionately in specific classes, allowing us to identify potential sources of dataset-induced bias. To the best of our knowledge, no prior work provides this level of insight into how visual concepts are distributed across training data.\n\n[1] Grobrügge, Arne, et al. \"Towards Human-Understandable Multi-Dimensional Concept Discovery.\" CVPR 25.\n\n---\n### *(Q2) Model robustness diagnoses*\n*We will add the clarification for Section 5.3 in the camera-ready version.*\nModel robustness is typically evaluated by measuring the performance drop on distribution-shifted test sets—such as from ImageNet to ImageNet-v2 or ImageNet-Sketch. However, in many domains (e.g., food or scene classification), such curated out-of-distribution (OOD) benchmarks are unavailable, and constructing them manually can be both costly and time-consuming.\n\n---\n**Concept-based distribution shifts**\n\n**ConceptScope** provides a complementary approach that enables robustness evaluation **within the original test set**, even when no distribution-shifted dataset is available. To do this, we create four subgroups of test samples with varying concept distributions.\n\nAs shown in Figure 5, for the class *“hippo”*, ConceptScope identifies **“hippo head”** as a target concept and **“water”** and **“zoo”** as bias concepts. We then define the following subgroups:\n1. **Group 1**: Images containing both the hippo head and water and/or zoo.\n2. **Group 2**: Images with the hippo head, but without water or zoo—i.e., a different background context.\n3. **Group 3**: Images featuring water and/or zoo, but where the hippo head is either absent, unclear, or too small.\n4. **Group 4**: Images labeled as “hippo” but lacking both the hippo head and the identified bias concepts (water/zoo).\n\nThese groups can be interpreted as forming a spectrum of distribution shift. **Group 1** represents the most frequent and typical setting seen during training, where both the target and bias concepts co-occur. In contrast, **Group 4** represents a more unusual or outlier scenario, where neither the target nor the bias concepts are present—posing a stronger generalization challenge for the model.\n\nThis setup allows us to simulate concept-based distribution shifts and assess model robustness without requiring a separate OOD dataset.\n\n---\n**Interpreting subgroup accuracy**\n\nIn Fig. 5(b), the y-axis represents subgroup accuracy (with the four groups shown in different colors), and the x-axis indicates the average accuracy across all four subgroups. Each vertical line corresponds to a single model, with four colored dots representing that model’s accuracy on each subgroup.\n\nAmong the 34 models evaluated, **ConvNeXt-Large** achieves the highest average accuracy and is shown on the rightmost line (in both ImageNet (IN) and ImageNet-Sketch (IN-S)). Notably, it also performs best on **Group 4**, the most challenging subgroup. We observe that models with higher accuracy on **Group 1** (the most frequent training setting) also tend to perform better on **Group 4** (the rare or outlier setting).\n\nThis trend is consistent with prior findings [1], which show that out-of-distribution (OOD) performance often correlates with in-distribution performance. Crucially, our approach enables such robustness analysis **without requiring an external OOD test set**, using only concept-aware subgroups derived from the original data.\n\nTo clarify,  our goal in this experiment is not to rank all 34 models by robustness, but rather to demonstrate that ConceptScope can analyze how and why models fail. This capability makes ConceptScope a useful tool for estimating or understanding model robustness, especially in cases where no special OOD test set exists. \n\n[1] Miller, John P., et al. \"Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization.\" ICLML, 21.\n\n---\n### *Typo*\nThanks for pointing out, we will revise accordingly."}]

Author rebuttal:

Meta-review: This paper proposes a method to characterize dataset biases by concepts it discovers. Instead of relying on predefined attributes or manual labels, the paper explains present biases in a dataset by concepts automatically detected. The reviewers found the paper was well written and the evaluation was thorough. 

The paper also shows that the method can detect "novel" biases (not from any existing labels or predefined categories) in a dataset and the reviewers thought this would be practically very useful. The exact same idea and general pipeline have already been published in a previous paper [1], however this paper was not cited. The AC conditionally recommends acceptance of the paper, provided that the authors include missing references in discussion and clarify the novelty of the current paper in comparison to these papers. It is also strongly recommended that the authors perform a comparative evaluation between them. 

[1] Yang, Y., Kim, S., & Joo, J. (2022). Explaining deep convolutional neural networks via latent visual-semantic filter attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8333-8343).
Decision: Accept (poster)