Learning Concept Bottleneck Models from Mechanistic Explanations
Authors: Antonio De Santis
Schrasing Tong
Marco Brambilla
Lalana Kagal
Venue: ICLR 2026 Poster (2026)
Keywords: interpretability
concept bottleneck models
computer vision
explainable ai
Primary area: interpretability and explainable AI
TL;DR: We propose a novel CBM pipeline, namely M-CBM, that uses mechanistic interpretability to learn concepts directly from its black-box counterpart.

Abstract: Concept Bottleneck Models (CBMs) aim for ante-hoc interpretability by learning a bottleneck layer that predicts interpretable concepts before the decision. State-of-the-art approaches typically select which concepts to learn via human specification, open knowledge graphs, prompting an LLM, or using general CLIP concepts. However, concepts defined a-priori may not have sufficient predictive power for the task or even be learnable from the available data. As a result, these CBMs often significantly trail their black-box counterpart when controlling for information leakage. To address this, we introduce a novel CBM pipeline named Mechanistic CBM (M-CBM), which builds the bottleneck directly from a black-box model’s own learned concepts. These concepts are extracted via Sparse Autoencoders (SAEs) and subsequently named and annotated on a selected subset of images using a Multimodal LLM. For fair comparison and leakage control, we also introduce the Number of Contributing Concepts (NCC), a decision-level sparsity metric that extends the recently proposed NEC metric. Across diverse datasets, we show that M-CBMs consistently surpass prior CBMs at matched sparsity, while improving concept predictions and providing concise explanations. Our code is available at https://github.com/Antonio-Dee/M-CBM.

Published as a conference paper at ICLR 2026
LEARNING CONCEPT BOTTLENECK MODELS FROM MECHANISTIC EXPLANATIONS
Antonio De Santis 1,2+
Schrasing Tong²
Marco Brambilla'
Lalana Kagal*
Politecnico di Milano
2 MIT CSAIL
ABSTRACT
Concept Bottleneck Models (CBMs) aim for ante-hoc interpretability by learning bottleneck layer that predicts interpretable concepts before the decision. State- of-the-art approaches typically select which concepts to learn via human specifica- tion, open knowledge graphs, prompting an LLM, or using general CLIP concepts. However, concepts defined a-priori may not have sufficient predictive power for the task or even be learnable from the available data. As a result, these CBMs of- ten significantly trail their black-box counterpart when controlling for information leakage. To address this, we introduce a novel CBM pipeline named Mechanistic CBM (M-CBM), which builds the bottleneck directly from a black-box model's s own learned concepts. These concepts are extracted via Sparse Autoencoders (SAEs) and subsequently named and annotated using Multimodal LLM. For fair comparison and leakage control, we also intro- duce the Number of Contributing Concepts (NCC), decision-level sparsity met- ric that extends the recently proposed NEC metric. Across diverse datasets, we show that M-CBMs consistently surpass prior CBMs at matched sparsity, while improving concept predictions and providing concise explanations. Our code is available
selected subset of images
com/ Anton o- Dee/M-CRM
INTRODUCTION
As AI systems become increasingly complex and embedded in high-stakes applications such as healthcare, autonomous driving, and defense, there a growing demand for models that not only perform well but are also transparent and interpretable. To obtain explanations for Al decisions, we can generally take two approaches: (i) utilize post-hoe methods that try to gain insights into how black-box models produce their outputs, (ii) develop inherently transparent models that can explain their decisions by design (i.c., ante-hoc explainability) (Xu et al. 2019). A promising ante- hoc approach explainability is Concept Bottleneck Models (CBMs), which
trained to first predict an intermediate set of interpretable concepts and then use these concepts to predict the final output. Recent practice typically instantiates this concept set a-priori, either specified by human experts (Koh et al.) 2020), based on knowledge graphs (Yuksekgonul et al., 2023). 3, by prompting an LLM (Yang et al.] 2023 Oikarinen al., 2023, Srivastava et al.) 2024), using general concepts extracted from pre- preitr vision- languaee models (Rao et al. 2024). However, concepts defined a-priori may not have sufficient predictive power for the target task or even be learnable from the available data. As result, state-of-the-arr CBMs substantially underperform their black-box coun- terpart when controlling for information leakage. Beyond performance, further reason not to fix concepts a-priori is that modern ML systems often equal or exceed human expertise, creating op- portunity to use interpretability to learn from machines. For example, Schut et al. (2025) extracted concepts learned by the chess engine AlphaZero (Silver et al., (017) and were able to teach them to grandmasters. Furthermore, mechanistic interpretability has recently made significant progress in extracting concepts learned by black-box models, in particular via Sparse Autoencoders (SAEs) (Bricken et al. 2023). Motivated by this, we ask whether CBMs built directly from a model's own learned concepts can serve interpretable approximations of their black-box counterparts. Because these concepts originate in the backbone, we expect them to be easier to learn and to have better
Work done while visiting MIT CSAIL. Correspondence antoni o. desant ss0pol mi it

---

Published as a conference paper at ICLR 2026
predictive power. To test this, develop a novel CBM pipeline, which we refer to Mechanistic CBM (M-CBM), and compare it to state-of-the-ae art CBMs in both task accuracy and its ability to learn concepts, showing significant improvements.
RELATED WORK
Concept-based Explanations. Early approaches for explainable AI typically rely on saliency (Selvaraju et al. 2017) or attribution maps (Sundararajan et al. 2017 that show which part of the imput (e.g., regions or pixels of an image) contribute the most to decision. By contrast, concept-based methods aim to provide explanations in terms of higher-level, human- cumman-nndrrstnddblle concepts (e.g., stripes for a zebra). seminal contribution to the field was TCAV (Kim et al., 2018), method that investigates a model's sensitivity to a user-defined concept by collecting set of exam- ple images representing that concept. Later, De Santis et al. (2025) extended TCAV with per-instance concept attributions and saliency maps indicating where the concept is recognized. However, both methods have practical limitations as they require users to manually collect concept examples. To address this, unsupervised approaches have also been proposed (Ghorbani al. 2019 Zhang al.) 2021 Fel et al.] 2023; Bianchi et al., 2024) to automatically discover influential concepts. These methods typically perform some form of clustering of a model's activations extract groups of semantically similar inputs or cropped patches that correspond to a concept. However, with this approach, achieving completeness (i.e., extracting a concept set sufficient to recover the model's prediction) remains a nontrivial task (Yeh et al. (2020).
Mechanistic Interpretability. Mechanistic interpretability (Mf) aims to comprehensively reverse- engineer deep networks by converting their neurons and weights into interpretable features and algorithms. A central challenge to this is polysemanticity, i.e., neurons often respond to unrelated features, so they cannot be mapped one-to-one with concepts (Olah et al.. 2020 This could allow networks to learn far more features than there are neurons, which is known as the superposition hypothesis (Elhage et al., 2022). Recently, Bricken et al. (2023) showed this can be addressed post- hoc by disentangling features via Sparse Autoencoders (SAEs) that learn a sparse, overcomplete dictionary of monosemantic features. Given their effectiveness in both language (Gao et al. 2025) and vision [Gorton] [oootton[002]]]]]] Tha4aaaaaaaa Thasarathan et al.) (025), we also adopt SAES for concept extraction in our pipeline. Another emerging trend in MI is automated interpretability, i.e., using LLMs to automate the generation of natural language descriptions for the functional role of network components. This was first applied to explain language model neurons (Bills et al. 2023). but then also proved effective to explain vision models (Rott Shaham et al., 2024). We also use similar approach assign names to concepts extracted via SAEs. MI has also made progress in dissecting models into interpretable circuits (e.g., identifying algorithmic sub-structures within deep networks) via masking or patching procedures (Conmy et al. 2023). Those circuit-level analyses are currently not being used in our pipeline, but integrating them could be promising future work.
Concept Bottleneck Models. Concept Bottleneck Models (CBMs) are self-explaining neural net- works that learn a set of intermediate uumaan-understandable concepts to solve a task. The term was first introduced by Koh et al. (2020). who trained CBMs using datasets featuring concept annota- tions. Later, Yuksekgonul et al. (2023) relaxed this requirement with post-hoc CBMs that learn a Concept Bottleneck Layer (CBL) using Concept Activation Vectors (CAVs) (Kim et al.) 2018). only requiring manual selection of representative examples for each concept. Furthermore, when using a CLIP (Radford al. (202T) backbone, they could learn concepts directly from text sourced from the ConceptNet (Speer et al. 2017) knowledge graph. Yang et al. (2023) later showed benefits in gener- ating the concept set with LLMs. Oikarinen al. (2023) extended this paradigm also non-Cli con-CLIP backbones using CLIP-Dissed any backbone. known problem, however, that exists across all CBMs is information leakage, i.e., the fact that the CBL inadvertently encodes hidden class-relevant patterns beyond the concept semantics, which can be quickly learned by the final predictor to improve its accuracy (Havasi et al.) 2022). This issue is quite serious, as Yan et al. (2023) even showed that replacing concepts with random words can achieve similar accuracy. Information leakage also results in unsatisfying expla- nations, in which the most important concepts contribute significantly less than the sum of all other concepts, making the model basically a black-box. box. Furthermore, while for classical CBMs, leakage can at least be quantified using metrics based ground-truth concept labels (Havasi et al. 2022
(Oikarinen & Weng 2023) to map concept embeddings in CLIP to
2

---

Published as a conference paper at ICLR 2026
Concept Extraction
Sparso Nutoencodor (SAE)
Concept Namiing
Highly active ovarpiee
BAE Nourons
Fostures
Festures
Notvactive unarrçles
Concept Nome ballme
31
Backboos
Block Box
Enioddrr
Dosoder
Midimootl LUM
Picnrot neuron
looking
3 Dataset Annotation
c Concept Bottleneck Model
Chasses
Concest Labels
Crazu
Ni
Vier
No
Helint
Conoost
Spare Layyer
Phorppt enages Aehnor??
Snew
Figure I: Overview of the M-CBM pipeline. (1) Given trained black-box backhone, we extract its features and learn sparse, disentangled concept directions using Sparse Autoencoder (SAE). (2) A Multimodal LLM is prompted with examples of highly activating and non-activating images to assign concept names to each SAE neuron.
(3) The MILM then annotates subset of the dataset containing equal split of active and non-active examples, indicating the presence or absence of each concept in selected images. (4) Using these concept annotations, we train Concept Bottleneck Layer (CBL) and sparse linear classifier to predict target classes from the learned concepts.
Zarlenga et al.] 2023), this is not trivial for CBMs that learn concepts automatically. To address this, SSrivastava
(2024) introduced the Number of Effective Concepts (NEC) as metric to measure and control how many concepts CBMs use to make a prediction, effectively reducing infor- mation leakage. In this work, we also follow this idea and use the Number of Contributing Concepts (NCC), generalization of NEC, to control for leakage and explanation conciseness. More details on NEC and NCC are provided in Section Srivastava et al. (2024) also introduced VLG-CBM,
CBM pipeline that uses GroundingDINO (Liu al.) 2025). open-vocabulary object detector, to automatically annotate a dataset with LLM-generated concepts. The CBL is then trained on these annotations in multilabel setting. However, leakage still arises from the annotation being class-conditioned, as we show in Section J Another limitation of these CBM paradigms is that may have sufficient predictive power for the target task not be learnable from the available data, requiring the inclusion uninterpretable components in the bottleneck to recover accuracy (Yuksekgonul al. 2023 Zhang al., 20255] Shang et al. 2024). Sometimes concepts can also be non-visual (Roth et al. 2023) (e.g., "spicy", "loud"), making ex- planations less transparent. Instead, we propose extracting and using the black-bor box model's own leamed concepts, rather than guessing with an LLM. A first step in this direction is DN-CBM (Rao al., 2024), which learns concepts from CLIP with an SAE and uses its hidden layer as CBL, naming the concepts by selecting the nearest text embeddings to the decoder vector. However, this paradigm can only be applied with a CLIP backbone, limiting its accuracy across datasets, as we show in Section 6, and CLIP dependence can still introduce ninuvisual concepts.
L.M--generated concept
METHODOLOGY
In this section, we introduce methodology for transforming any black-box model into an interpretable-by-de by-eesnn CBM. Our approach, which we refer to as Mechanistic CBM (M-CBM), extracts human-inters iimanninterrreetbble concepts from trained black-box model, assigns names and annota- tions using a Multimodal Large Language Model (MLLM), and then trains sequential CBM (Koh al. 2020) using these concepts. An overview of the whole pipeline is provided in Figure
3

---

Published as a conference paper at ICLR 2026
Concept Extraction. Given black-box backbone & trained on arbitrary dataset D, the first step of our methodology is to decompose the features learned by the model during training into a set of interpretable concepts. To achieve this, we use the Sparse Autoencoder (SAE) approach, which was recently popularized in the mechanistic interpretability literature (Bereska & Gavves) 2024) and has proven effective disentangle model features into interpretable concepts for both vision (Gorton 2024
Thasarathan et al. 2025) and language models (Bricken et al., 2023; Huben et al. (0224).
An SAE is a neural network trained to reconstruct its input features while enforcing sparsity in the hidden representation (see step ( in Figure D In our case, the input features are the activations E IR. of the backbone & for each sample xx(*) in the training set ID. Following Bricken al. (202)), the SAE subtracts input bias bn and then passes the resulting vector to an encoder with weights Wg. bias bg. and ReLU activation, obtaining the hidden layer h E R.":
(1)) <(x ((((()
h = R (W (a - bD) + bbg
In the sparse hidden layer h. ideally, each neuron learns to recognize distinct concept. A decoder with weights WD and bias bn then maps h back to the reconstructed features à:
à W bb
where WE RT WD R and typically for large datasets > n to account for the superposition hypothesis (Elhage et al. 2022). i.c., the fact that neural networks tend to learn more concepts than the neurons they have. The input and output biases by are opposite in sign and equal in magnitude. While Bricken et al. (2023) train SAES with expansion factors (defined as m/n) ranging from 1x 2562, we avoid going above 4x to keep the annotation step computationally feasible. To train the SAE, we minimize the following objective:
D
1 DD]
CSAE
Σ
la 9
aa + ASAE (12
where ASAB > 0 is a hyperparameter that controls the strength of the sparsity penalty on the hidden representation. We also monitor the average to norm (i.e., the number of non-zero activations) to ensure to < n, as recommended by Bricken et al. (2023).
SAE training often leaves many neurons in the hidden layer h dead (never activated for any training sample) or nearly dead (activate only very rarely). To ensure that our set of candidate concepts is both meaningful and computationally efficient for subsequent annotation, we perform a filtering step to remove such neurons. To define the threshold for identifying nearly dead neurons, we measure, for each unit h, the number of training samples for which is active. We then select a cutoff value such that removing all units below this threshold does not reduce the recovered cross-entropy loss of the black-box model, defined This procedure ensures that only neurons with negligible contribution to predictive performance are pruned. This metric was also used to evaluate SAE quality in prior work (Bricken et al., 2023 Rajamanoharan et al. 2024 Gao et al. 2025). More details on SAES in Appendix
(e (ann(a))CCndd by more than a tolerance of 1%.
Concept Naming. After pruning, each remaining neuron in the SAE hidden layer h is treated as a candidate concept, where we denote by hj the j-th hidden SAE neuron. To assign human- interpretable names, adopt an automated procedure inspired by recent work on mechanistic in- terpretability of language model neurons (Bills et al., 2023). For each candidate concept, we first select set of inputs x D that maximally activate the corresponding neuron hy. For these in- puts, we also highlight the spatial regions that contribute the most to the activation, similarly to Rott Shaham et al. (2024). To compute these concept saliency maps, we use the method introduced De Santis et al. 2025) (i.c., weighted average of feature maps using WD as concept weights and followed by ReL.U). To provide a contrastive signal, we additionally sample set of aottactivan examples, of which half are drawn at random from D. and half are selected as the most cosine simi- lar to the activating examples to enhance discrimination of fine-grained visual features. The paired examples are provided to MLLM, GPT-4.1 in our experiments, which is prompted to produce concise natural-language description of the concept that the neuron represented by hy is responding to. At this stage, also explicitly instruct the model not to use class names concepts and re-try it does not adhere. Step ② of Figure [ shows an example of what the MLLM receives as input. In experiments, we used 10 activating examples and 10 non-activating ones.
4

---

Published as a conference paper at ICLR 2026
Finally, since do not want duplicate or semantically equivalent concepts, perform merging step similar to Oikaarinen et al. (2023), in which we embed all proposed textual names using a pre- trained embedding model and merge those with very high cosine similarity (i.c., above 0.98). We use OpenAl's Sext-embedding---laaaee
in our experiments. To make the embeddings context-aware, also wrap each concept name in the following template before inserting it into the embedding model: "This is visual concept in the context of {domain]: {concepr}". The variable {concept} contains the concept name, while {domain) } specifies the dataset domain (e.g., bird species, skin lesions). For simplicity, Figure I omits the merging step.
Dataset Annotation. With concept names assigned, we proceed to build a partially annotated dataset to train the Concept Bottleneck Layer (CBL). This step is necessary because a concept name only hypothesis, rather than faithful description of the corresponding SAE neuron's 's functional role. Since such hypotheses are often difficult to validate (Sharkey et al. 2025), we do not consider it ideal to use the SAE hidden layer directly as a bottleneck when interpretability is the primary goal.
Let C = fci.....cr denote the final set of concepts. For each concept cg. the goal is to obtain binary presence/absence labels on subset of images x E D. Since exhaustive annotation of the full dataset is not computationally feasible at the time of writing this paper, we annotate up to 1000 samples per concept. The annotation procedure is performed by prompting the MLLM with batches of 25 images arranged in a 5 5 grid, which is much cheaper than image at time, but also slightly less accurate (see Appendix G The model is asked to indicate, for each of the 25 grid images, whether the concept is present absent. See step 3 of Figure [ for high-level overview of the annotation procedure. Each call also includes a grid of the top-25 most activating images for the corresponding SAE neuron, which serve as a reference together with the textual concept name. To select the subset of images for annotation, we first select up to 500 active samples per concept. The active set is defined as all inputs for which hy > O. From this set, we select samples whose activation lies above the 95th percentile of the set. If fewer than 500 samples exceed this percentile, we take the top-500 activations overall within the active set. If the neuron has fewer than 500 active samples in total, we take all available examples, rounding the number down to the nearest multiple of 25 to match the batch annotation protocol. For merged neurons, activations are normalized across the group and treated as a single unit when computing percentiles. We then select equal number of non-active samples, of which half are drawn uniformly at random and half are chosen as the most cosine similar to the active samples, similarly to the naming procedure. Furthermore, to avoid biasing concepts toward particular classes, both active and non-active sets are stratified across class labels. Each batch of 25 images also contains balanced mix of active and non-active examples. At the end of this annotation step, we obtain a set of around 1000 annotated samples for each concept, containing both presence and absence cases across both training and test data. An image may be annotated for more than one concept for none. Formally, for each image x (9() E D, the annotation procedure creates ternrry vector of concept labels (+)) {-1,0, 13 with the following entries:
if cj is annotated as present in (+)
( 2/
0
if cı is annotated absent in x
if C₁ is not annotated for (+))
Concept Bottleneck Model. After generating the concept labels, we proceed with training a se- quential CBM (Koh et al. 2020). As shown in step 4 of Figure the CBM has three components: (i) a frozen backbone ò that maps input image to feature vector, (ii) Concept Bottleneck Layer (CBL) g that predicts the presence of K named concepts from those features in a multi-label setting, and (iii) sparse linear classifier f that predicts the class from the concept outputs.
For each input (+(() the frozen backbone produces n-dimensionnll features a = ((x(()) E RR. The CBL g: RR= + K takes these features as input and outputs concept logits, then a sigmoid produces probabilities $ = a(g(a ())) E [0, _]K From the annotation pipeline, each image carries a ternrry concept vector 2 E f 0, 1]K indicating present (1), absent (0), or not annotated (-1). Since not every image-concept pair is labeled, we train only on the entries we know. Let Ω =
(4) (0, 1}} be the of annotated pairs. The CBL is optimized to minimize a masked Binary Cross-Entropy (BCE) loss that averages over Q:
k(i,k) 2
LCBL = Ω
1
Σ BCE (2 2k
(3)
(LA))R
5

---

Published as a conference paper at ICLR 2026
(6) 1 effectively ignored in the loss computation. Therefore, images without Entries with F any concept annotation (all entries -1) are not used to train the CBL. Furthermore, since positives are often rarer than negatives, we weight each concept the BCE by the ratio of its class imbalance.
To map concepts to classes, we follow prior work (Srivastava et al. 2024 Yuksekgonul et al. 2023 Oikarinen al. 2023) and train sparse linear classifier on concept logits (i.e., CBL's pre-sigmoid outputs), optimized using the GLM SAGA solver (Wong et al. (021). Since GLM-SAGA assumes standardized input features, we z-normalize (zero mean and unit variance) the concept logits and use these to predict the classes. With frozen, we define fully connected layer f : R x + R with weights WF E Rx and bias by E IRC, where C' is the number of output classes, and minimize the following Cross-Entrope (CE) loss with an elastic-net (Zou & Hastie, 2005) penalty:
[D]
CCLF Σ CE (Joggoobaa )() )) + XCLF Ra
(4)
where e) represents the one-hot ground-truth class label for sample (+()) and R. (1 a) Wr + a [Wella denotes the elastic-net penalty. Following Wong et al. (2021), we use 0.99, while ÄCLF is tuned to obtain target sparsity.
4 NUMBER OF CONTRIBUTING CONCEPTS (NCC)
Prior work has shown that sparse layers are more interpretable (Wong et al. 2021; Yuksekgonul et al., 2023] Oikarinen et al. 2023), and Srriastavaa et al. (2024) also showed that sparsity in- versely correlated with information leakage. They demonstrated that a dense linear classifier built on top of random (i.e., untrained) CBL can recover Mox accuracy if the number of concepts K" approaches or exceeds the backbone feature dimension n, but this effect decreases with higher sparsity. Related studies (Shang et al., 2024 Yan et al. 2023) similarly report that when the concept set is large enough (e.g., K 2 1), dense linear classifiers can preserve black-box accuracy by re-estimating the backbone activations, therefore even using random words concepts can match the accuracy obtained with concepts defined by LLMs or humans.
While high sparsity improves interpretability and limits leakage, naturally tends to correlate with lower accuracy (Wong et al. 2021; Oikarinen et al. 2023); Srivastava et al.) [2024), making CBM comparison incomplete if only accuracy reported. To address this, (Srivastava et al. (2024) intro- duced an evaluation metric named NEC, which is defined as the average number (per-class) of non- zeros in the weights WF of the final layer f. They train ff at different regularization strengths ACLF and accuracies are compared at equal NEC. This is convenient for enabling a fair comparison be- tween CBMs, but it also has limitations. Controlling NEC forces concise decision explanations, but it does so by linearly restricting the effective concept vocabulary as the number of classes decreases. For instance, with three classes, NEC=5 forces < 15 (or even K = 5 for binary classification with single output head) after training so that average predictions are explained by 5 concepts. However, in datasets with substantial intra-class diversity (e.g., peeled or in-field pineapples are the same class in ImageNet). a class may require rich concept vocabulary (i.e.. larger K) to cover its different contexts, even though only a subset of them is needed to predict an individual image.
With this in mind, we introduce a generalization of NEC, named Number of Contributing Concepts (NCC), which does not impose hard cap on K but still enforces concise explanations by measuring sparsity at decision-level using concept contributions rather than weights count. To measure the con- tribution of concept k. for class and image i, we must consider the magnitude of both the concept logit g(a()) ]} and its weight [WF]a,r towards class r. We then define the absolute contribution of (g(a(()) [Wrle.r] |. Ideally, we want the model recognize a
a concept to a class as u class with only a small subset of concepts that cover the vast majority of the total absolute contribu-
tion, or, in other words, explain the vast majority of the decision. Let "(s),r denote the s-th largest absolute contributing concept, and fix coverage level E
[0, 1]. We define NCC as:
[D|
K
NCC,
JD ΣΣ min {N (0...., (O....,K:: E T
Σ
While here we average NCC over all C classes, it can also be alternatively computed considering only the model's predicted class. Intuitively, NCC, is the average number of concepts required to

---

Published as a conference paper at ICLR 2026
explain at least a fraction of the prediction of class. For example, NCC=5 with 7 0.95, means that, on average. just 5 concepts explain 2 95% of the decision. For controlling NCC, we fix and follow the approach Srivasstt et al. 2024), training at different ACLF and compare CBM accuracies at equal NCC levels. In practice, targeting a lower NCC generally means trading accuracy for explanation conciseness and vice versa.
EXPERIMENTAL SETUP
Baselines. We compare M-CBM with three state-of-the-art CBMs: LF-CBM (Oikarinen et al., 2023). VLG-CBM (Srivastava et al., 2024), and DN-CBM (Rao et al. 2024). For VLG-CBM, we compare with class-agnostic annotation variant, which we refer to as VLG-CBMcA, rather than the original pipeline. In the original VLG-CBM, concepts are assigned to classes before annotation and annotated only on images of their assigned classes. While this design reduces annotation cost, coupling concepts to classes can introduce substantial information leakage. We verify this on CUB in Figure 2 Using random words concepts, VLG-CBM reaches black-box level accuracy already at around NCC=1.5, showing that in this setup, performance is insensitive to both sparsity and con- cept semantics. Intuitively, this happens because learning concept that is labeled positive only on images of a class is nearly equivalent to learning that class directly. When we remove class condi- tioning by annotating each concept across all images (VLG-CBMca). accuracy drops substantially for both random and real concepts, and the expected interpretability-acoume accurrac trade-off reappears. We performed the same experiment with our M-CBM, and the performance of substituting concepts with random words is similar to using random words in VI.G-BMMcA- However, when real con- cepts are used, our M-CBM outperforms VLGG-BBMA at high sparsity (NCC=3 to 5), while for low sparsity (NCC=10+), accuracy becomes similar to the random words variant due to information leakage. Further implementation details for this experiment are provided in Appendix
Accuncy MCC
Accumny MCC
3.8
Accurscy MCC
3.7
5.7
277
2.55
3.4
03
02
V15 C3R tranccrr words)
VIG-CMM
VLC Dy
Me CWtte panden woods
12
MCIN CBN mondom worts)
31
to
20
100
61
10
75
100
51
1o
20
to
103
(a) VLG-CBM
(b) VLG-CBMcA
(c) M-CBM (Ours)
Figure 2: Accuracy vs NCC (T = 0.95) on CUB. (a) With class-conditioned annotation, VLG-CBM reaches near black-box accuracy with NCC=1.5 (i.e., using only 1 to 2 concepts per prediction). The same happens using random concept names, showing evidence of leakage. (b) Making annotation class-agnostic (VL.G-BMcc)) restores the accuracy-interpreaablit b trade-off, with real concepts both VI.G-BBMcA and the
slightly beating random words at low NCC. (c) M-CBM outperforms random baselines at low NCC, while leakage is significant for both methods NCC increases.
Setup. We evaluate three standard image classification datasets that vary in domain and class
count: CUB (Wah et al.) 2011), D. ISIC2018 (Codella et al.] 2019 Tschandl et al.] 2018), and Im- ageNet (Deng et al. 2009). CUB contains 6k training images and 5.8k test images of 200 fine-grained bird species. As backbone for this dataset, we use the pre-trained ResNet18 from py- torchey| ISIC2018 contains dermatoscopic images of pigmented lesions categorized in 7 classes, split into ~ 100 train, 193 validation, and ~ 1.5k test. Given high class imbalance, we report both accuracy and balanced accuracy for this dataset. Given the lack of public pre-trained models, we train a ResNet50 (weighting each class by its imbalance ratio) and use it as backbone. ImageNet includes 1A classes with ~ 1.3M training and 50A test images for general image classification. As backbone, we use the pre-trained ResNet50 from torchvision) Furthermore, for ImageNet and CUB, we extract 10% from the train and use it as a validation set. Regarding DN-CBM, since it only supports CLIP backbone, we evaluate it using both the ResNet50 and ViT-B/16 backbones. As diseussed in Section g we compare under the same NCC. We use 7 = 0.95 and measure accuracies at NCC=5 and NCC=avg, with the latter being the average of the levels: 5, 10, 15, 20, 25, 30.
7

---

Published as a conference paper at ICLR 2026
Compute Resources. We trained all neural components (SAE, CBL, and GLM-SAGA) HPC cluster using an NVIDIA H200 on a multi-core node (32 cores and 512GB of RAM). On CUB and ISIC2018, each stage takes 5-20 minutes, while 5-10 hours for ImageNet. The dominant step in terms of cost and runtime the annotation with GPT-4.1 API, which takes around 2 minutes and costs USD 0.14 per concept. Concept naming was lighter, taking around 10-20 seconds and USD 0.02 per concept, while concept merging costs were negligible. These costs scale linearly with the concept number, which was 278, 73, and 2648, respectively for CUB, ISIC2018, and ImageNet.
RESULTS AND DISCUSSION
Accuracy Comparison. We report results in T Our M-CBM consistently achieves the high- est accuracy across datasets and NCC values. An expected caapretability-acccaaa trade-off is also visible across all methods, as accuracy always increases when NCC is higher (i.e., explanations are less concise). DN-CBM consistently performs poorly, especially at NCC=5, indicating that small subset of generic CL.IP concepts may be insufficient to predict a class across datasets. VIGG-BMcCA shows better accuracy than LF-CBM and DN-CBM, but annotating per-concept the entire dataset with GroundingDINO makes it computationally prohibitive at ImageNet scale (~~ 300 GPU-days). In contrast, M-CBM uses SAE activations to pre-select candidate images per concept, so that only need to annotate ~ 1 images per concept. We exclude class- dlass-conditioned VLG-CBM from the comparison because, due to leakage, it is effectively a black-box (see Section
Table 1: Accuracy comparison NCC=5 and NCC=avg with best model in bold. The results for M-CBM are averaged over 3 seeds with same annotations. N/A denotes computationally unfeasible.
Dataset
CUB
ISIC2018
ImageNet
Metrics
Accuracy
Accuracy
Balanced Accuracy
Accuracy
Black-box
76.67%
79.37%
75.37%
76.15%
Sparsity
NCC=5
NCC=avg
NCC=5
NCC=avg
NCC=5
NCC=avg
NCC=5
NCC=avg
LF-CBM
58.08%
71.09%
61.44%
67.55%
64.29%
67.30%
62.20%
69.08%
DN-CBMRN
38.21% %
48.98%
35.38%
54.61%
39.85%
52.85%
46.71%
57.24%
DN-CBMyn
48.12%
66.19%
43.92%
56.08%
42.47%
53.56%
60.23%
69. 69.98%
VLG-CBMcs
69.12%
72.25%
64.55%
72.61%
64.63%
70.80%
N/A
N/A
M-CBM
73.70%
74.18% %
72.75%
75.51 %
70.14%
71.54%
72.18%
73.64 %
(Ours)
+ 0.133%
+ 0.06%
+ 0.100
0.08%
+ 0.09%
+ 0.05%
+ 0.21%
+ 0.15%
Evaluating Concept Prediction. We assess how well each method can learn its concepts by also annotating the test set. Because these labels are not ground truth, high scores do not guarantee that the model is learning the concepts intended, but only that they tent and learnable. Especially for ISIC2018, we found that LLM-generated concept sets are often non-visual (e.g., "warm the touch") or not in the data (e.g., "medical report"). Since M-CBM uses concepts extracted from the backbone, we expect some benefits in the concept predictions, which is what we see Table Another factor that could contribute to the lower performance the capa- bility of GroundingDING to annotate correctly, which may be inferior to asking GPT-4. 1. especially for medical images. However, due to lack of ground truth, this remains challenging to quantify.
least internally consis-
Table 2: ROC-AUC evaluation of concept predictions on test set. Each method is evaluated on its own concepts. We report the macro-average across concepts and the average of the worst 10%.
Dataset
CUB
ISIC2018
ImageNet
ROC-AUC
ROC-AUC
ROC-AUC
Metrics
Macro
Worst-10%
Macro
Worst-10%
Macro
Worst-10%
VLG-CBMcA M-CBM (Ours)
62.03% 90.04% %
45.60% 79.05% %
73.37% 80.57
52.92% 66.98 %
N/A 88.90
N/A 78.36 %
8

---

Published as conference paper at ICLR 2026
verical row Bumirated status Indicator iitt on electooiic devize
aroa
pirted sircsl board with wadrroiic components
Moders
doeed metalla to plactic rectangutar device with rounded edges
dark brown di fur lesin at the torder of lighter skin legion
while rouncied plastc surfice
pigmerter lesion
Melancopio
homagersuu
dual seriical artannss on reclang ular wladrone devioe
brown patch at border betwoon two dkl tones
rectangular electronts device with vibble ports or switches
spon ligter baceglound
owarlynng
analog contaal and frequency daas
Rado
perpnersi piprant network
handhelld electomiz device
Interface
Dammalstrrom
gorowe wirk with stteched cylinti wamment
pinceetie trudurebes ansa
stackos audio uquipment hont panvls
dioullrr spenker grlle
ImageNet
(b) ISIC 2018
Figure 3: Sankey plots of concept-class weights of our M-CBM NCC=5. Concepts the left and classes on the right. Concepts with negative weights are labeled as "NOT concept".
Explanations. We illustrate the behavior of M-CBMs through global (class-level) and local (instance-level) explanations, using models at NCC=5. Using the final layer weights WF, we can visualize how concepts globally contribute to classes. In Figure 3 show these weights using Sankey diagrams, with "NOT concept" indicating a negative weight. For clarity, we include only concepts with [Wrl 0.1. On ImageNet, the model's behavior aligns with intuition. The classes "Modem" and "Radio" share concepts related to ports/switches and antennas, while they are mainly differentiated by the presence of indicator lights for class "Modem" versus control knobs for class "Radio"" On ISIC2018, the model learns a richer concept set for "Melanocytic nevus" than for "Der- matofibroma" ", which could be explained by the large class imbalance. Still, the few concepts learned for "Dermatofibroma"" seem reasonable considering dermatological literature (Zaballos et al.) 20066) Some minor concepts for "Melanocytic nevus", such as skin-tone-relle related terms, are less clear. This likely arises from the concept naming (step 2), where visually highlighting the concept (in this case, the skin around the nevus) in the image can introduce mild artifacts that GPT-4. over-interpreted. CBMs can also explain individual predictions by showing. for an input (8), the contribution of con- cepts to class r. This contribution is computed directly by multiplying the logit of the k-th concept g(a I with its corresponding weight [Welk.r towards class r. Concepts with a negative logit are indicated "NOT concept". We show two examples in Figure including correct CUB predic- tion and a misclassification on ISIC, where the model incorrectly sees "clustered blue-gray greeggray ovoid nests", leading to a "Basal Cell Careinoma" prediction. Zeroing this concept flips the decision to the correct class. In both cases, we see that the decision is largely explained by the top 4-5 concepts. More examples explanations are provided in Appendix [
4. yellow upperpart with black body and beak (+0.38)
3. bright yellow plumage with black cap (+1.28)
with black
around yellow (+4.34)
around eyes
pink light papute/noctuee
or ulcerstion (40.15)
clustared blue gray nests (+3.32)
(-0.17)
deprassion
5. olive-green upperparts (+0.12)
NOT leeron with muliiple red and brown ulcersted or
concepts (+0.05)
Sum
concapts (+0.03)
(a) Correctly predicted Hooded Warbler
(b) Melanocytic Nevus wrongly predicted as BCC
Figure 4: Per-image explanations of M-CBM at NCC=5 for a correct prediction in CUB (a) and a misclassification in ISIC 2018 (b). Concepts with negative logit are labeled "NOT concept".
CONCLUSION AND LIMITATIONS
We presented Mechanistic Concept Bottleneck Models (M-CBMs), novel paradigm for training CBMs using concepts learned directly from a black-box backbone and automatically annotated by MLLM. With this approach, substantially improve the state-of-the-art, both in terms of

---

Published as a conference paper at ICLR 2026
task accuracy and concept predictions. We are also able to keep explanations concise by control- ling final layer sparsity to achieve a target Number of Contributing Concepts (NCC). One limitation general to all CBMs is that we still lack systematic way assess whether concepts are learned as intended and not via spurious correlations. This is because the final layer is interpretable, but the concept prediction remains black-box. Another limitation is that, while NCC allows trol the accuracy-leakage trade-off, it is still not enough to eliminate leakage, as CBMs trained on random words still achieve much higher accuracy than we would expect from random chance. In future work, it may be interesting to investigate whether adding more bottleneck layers can mitigate this by making it harder for information leak through. Furthermore, compared other base- lines, M-CBM is less plug-and-play. requiring some supervision to ensure that concepts extracted via SAE are interpretable (see Appendix B). and that the MILM is providing high-quality annota- tions. Finally, the high computational cost of using MILMMs for annotations can also be considered a limitation, especially for large datasets. Despite these limitations, given that, due to computational constraints, we annotate only a small subset of images, there might be great potential for improve- ment with the advancements of MLLMs in both performance and efficiency.
ACKNOWLEDGMENTS
Antonio De Santis was supported by the Progetto Rocca Doctoral Fellowship for his visit to MIT CSAIL. His doctoral scholarship is funded by the Italian Ministry of University and Research (MUR) under the National Recovery and Resilience Plan (NRRP), by Thales Alenia Space, and by the European Union (EU) under the NextGGeneratioNUU project.
REFERENCES
Leonard Bereska and Stratis Gavves. Mechanistic interpretability for Al safety review. Transac- lions Machine Learning Research, 2024. ISSN 2835-8856. pest/forumiiccm PetemnidddPUUeetPPP K Survey Certification, Expert Certification.
Matteo Bianchi, Antonio De Santis, Andrea Tocchetti, and Marco Brambilla. Interpretable net- work visualizations: human-in-the whe-ih-thhooopppeeee approach for post-hoc explainability of enn-based im- age classification. In Proceedings of the Thirty-Thiin International Joint Conference on Artificial Intelligence, IJCAI '24, 2024. ISBN 978- 178-1-956792-04-1.
doi: 10.24963/jjcai20244422 2014/4111 URL
https://doi. org. 249.0..4933/2.. 1tA.9//3///1a..22///5...
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. /tttps://ppeaaiuuse neuzonn-explaine//pppae /apeeer/inndexxxx index. html 2023.
blob. core. windows.nett
Trenton Bricken, Adly Templeton, Ben Chen, Jack Lindsey, Adam Jermyn, Shan Carter, Tom Henighan, Adam Pearce, and Chris Olah. Towards monosemanticity: Decomposing lan- guage models with dictionary learning.
htt///aaaffoomerr.oe.ee.eoeoeeee Cormer-circuute pub/ monoosmanticc featueesse 2023. Transformer Circuits Thread, Anthropic.
Noel Codella, Veronica Rotemberg, Philipp Tschandl, M. Emre Celebi, Stephen Dusza, David Gut- man, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, Harald Kittler, and Allan Halpern. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic), 2019. URL https::/pp/........... / h/trxxiivvrrrg///. abb/
1902. 03368
Arthur Conmy, Augustine N. Mavor-Parker Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonoo. Towards automated circuit discovery for mechanistic interpretability. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:
Antonio De Santis, Riccardo Campi, Matteo Bianchi, and Marco Brambilla. Visual-tcav: Concept- based attribution and saliency maps for hoe explainability in image classification, 2025. URL https:///arxt hww.voorggaaaa. 24 2411.05698
10

---

Published as a conference paper at ICLR 2026
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, doi: 10. 101.11///..00000 L099//CVR.22095526644..
pp. 248-255, 2009.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatficld-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superpo- sition, 2022. URL httptttt////ax...rrgg////
10652
Thomas Fel, Agustin Picard, Louis Béthune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. Craft: Concept recursive activation factorization for explainability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR),
pp. 2711-2721, June 2023.
Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, llya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In The Thirteenth Inter- national Conference on Learning Representations, 2025. URI.https:/ htttpss://ppeereeeee...rt
Corum?id-tcs20 9ZNKD
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buee E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Liv Gorton. The missing curve detectors of inceptionvl: Applying sparse autoencoders to in- ceptionv1 early vision. In ICML 2024 Workshop on Mechanistic Interpretabilltyy 2024. URL http::///ppeeeviewww net wwweee/oooommmddnnooo...
Marton Havasi, Sonali Parbhoo, and Finale Doshi- Addressing leakage in concept bottleneck models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh Advances in Neural Information Processing Systems, volume 35, pp. 23386-23397, Curran Associates, Inc.,
2022. URL
neurips cc http:/cccppppppeeecceeee wwwerpppppppeeeee File F119944ee55aaae7700002a006066000
pdff
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024, URL httpst /ttp::///ppereeeeeee net met/forru forum?
Cd=F76bwRSLe.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory sayres. Interpretabilitty beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Jennifer Dy and Andreas Krause (eds.), Proceedings the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp.
0668-2677. PMLR, 10-15 Jul 2018. URL https: httpss////cccceeingg..e. ml wwt.mm......e/....
kiml 1d.1dd.r html.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5338-5348. PMLR, 13-18 Jul 2020. URL https: htttp:::/prrceedinggs
mlr. www.presssssoe. v vlvvvll999 www.//ooaaa.... html
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Giil Varol (eds.), Computer Vision ECCV 2024, pp. 38-55, Cham, 2025. Springer Nature Switzerland. ISBN 978 978-3-031-72970-6.
Tuomas Oikarinen and Tsui- Wei Weng. Clip-dissect: Automatic description of neuron representa- tions in deep vision networks. International Conference Learning Representations, 2023.
11

---

httpss//diiilllppo////Dderrrrirr Published as a conference paper at ICLR 2026
Tuomas Oikarinen, Subhro Das, Lam M. Nguyen, and Tsui- Wei Weng. Label-free concept bottle- neck models. In The Eleventh International Conference on Learning Representations, 2023, URL https:://ppereett
net/ wwttteeuettteeee www.mmiddroioooo 7MNVBA
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 10.239 //diisill.000024.000
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 18-24 Jul 2021. URL https: petps://ppron httts::///proceeedgggs... mir wtt.://mm...p.......p..e..eeee.s sttess//v33/radfoorooe
PMLR,
ht.m
Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving sparse decomposition of lan- guage model activations with gated sparse autoencoders. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neu- ral Information Processing Systems, volume 37, pp. 775-818. Curran Associates, Inc., 2024. URL httts::/ www.:/iiinnnnnnnuiiieeeee nwwrips.s............. cw/ppppp.pppeeeeeeeeeeeeeeeee.ee http://eee.ellee//eeeeeeeeeeeeee fiaep//p/ssse//p/2/2////// fi Elle/A17/abb1200aec000aaseee2222
Paper Conference. P
Sukrut Rao, Sweta Mahajan, Moritz Böhle, and Bernt Schiele. Discover-then-namee Task-agnostic concept bottlenecks via automated concept discovery. In European Conference on Computer Vision, 2024.
Karsten Roth, Jae Myung Kim, A. Sophia Koepke, Oriol Vinyals. Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15700- 15711, 2023. doi:
Tamar Rott Shaham, Sarah Schwettmann, Franklin Wang, Achyuta Rajaram, Evan Hernandez, Jacob Andreas, and Antonio Torralba. A multimodal automated interpretability agent. In Forty-first International Conference Machine Learning, 2024.
Lisa Schut, Nenad Tomašev, Thomas McGrath, Demis Hassabis, Ulrich Paquet, and Been Kim. Bridging the human-ai knowledge gap through concept discovery and transfer in alp- hazero. Proceedings of the National Academy of Sciences, 122(11)) 2025. doi: 10. 2406675522.. BRLnttts::/// http::///wwwwppe pnas www.pnan.....ooooooooeooooeeeeee /ttp://////////////1////////// pnas. 24C6675122
Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. In 2017 IEEE International Conference on Computer Vision (ICCV), IEEE, October 2017. doi: 10. 10.1109/icx222177.4
BtLhttp://dee doi. org /10 1109/ ICCV. 2017.
Chenming Shang, Shiji Zhou, Hengyuan Zhang, Xinzhe Ni, Yujiu Yang, and Yuwang Wang. In- cremental Residual Concept Bottleneck Models In 2024 IEEF/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 11030-1104p Los Alamitos, CA, USA, June 2024. IEEE Computer Society. doi: 10.1 10.1199CCVPR3233.22244.0000
URL https:
7/doi. htt.loeeeeecppptrrroooeeeeroooor
2024. C1049
Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeffrey Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Isaac Bloom, Stella Biderman, Adrià Garriga-Aoons, Arthur Conmy, Neel Nanda, Jessica Mary Rumbelow, Martin Watten- berg, Nandi Schoots, Joseph Miller, William Saunders, Eric J Michaud, Stephen Casper, Max Tegmark, David Bau, Eric Todd, Atticus Geiger, Mor Geva, Jesse Hoogland, Daniel Murfet, and Thomas McGrath. Open problems in mechanistic interpretability. Transactions Machine Learning Research, 2025. ISSN 2835-8856. URL https: http:///ppereviiiee net/ forum? Survey Certification.
id=91 6m 9794
12

---

Published as a conference paper at ICLR 2026
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel. Timothy Lillicrap, Karen Si- monyan, and Demis Hassabis. Mastering chess and shogi by self-play with general reinforce- ment learning algorithm, 2017. URL https: htw.https:/araae htttt:::/aaxxxv.oorgg aatp/////////////// wwwpaa/s..1///../ 01815
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. Proceedings of the AAAI Conference Artificial Intelligence, 31(1), Feb. 2017. doi: 10.1609/aai..3311..111 AAAI BAAI/article/vee../
URL httthtss://osss aaai. org/ index. inde
Divyansh Srivastava, Ge Yan, and Tsui-Wei Weng. Vig-cbm: Training concept bottle- neck models with vision-language guidance.
A. Globerson, L. Mackey, D. Bel- grave. A. Fan. U. Paquet. J. Tomezak, and C. Zhang (eds.), Advances in Neural In- formation Processing Systems, volume 37. pp. 79057-79094. Curran Associates, Inc., 2024. URL httpss://prceeddngs www.inuuri.... cc/pppr. http://aaprrreeppeeeeeeepeeepeee files/paper/202e/ f1 File/ 00000eeedd80000ffffffffffeeeoeee
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference Machine Learning -Volume 70, ICML' pp. 3319-3328, JMLR.org, 2017.
Harrish Thasarathan, Julian Forsyth. Thomas Fel, Matthew Kowal, and Konstantinos G. Derpanis. Universal sparse autoencoders: Interpretable cross-model concept alignment. In Forty-second International Conference on Machine Learning, 2025. URL https: pestps:///penrevieeee BeruumiddDoaxxN6B..
Philipp Tschandl, Cliff Rosendahl. and Harald Kittler. The ham10000 dataset, a large col- lection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data, 5:180161, 2018. doi: 10. 10.1038/sdtt.2011 2018.......1....... https:/::: //wp: nature. com/
articles/sdata200
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-uesd birds-200-2011 dataset. Technical Report CNS-TR-200 1-01- California Institute of Technology. 2011.
Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging sparse linear layers for debug- gable deep networks. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th Inter- national Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11205-11216. PMLR, 18-24 Jul 2021. URL nttps://proceedie www.nnssmsr.......m... .mlr. press/vi39/ http://ogg2lb......mm
Feiyu Xu, Hans Uszkoreit, Yangzhou Du, Wei Fan, Dongyan Zhao, and Jun Zhu. Explainable Al: Brief Survey on History, Research Areas, Approaches and Challenges, pp. 563 573-574. 09 2019.
ISBN 978-3-030-32235-9, doi: (10007978-33000-32226----
An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong, Zexue He, Yujie Lu, William Yang Wang. Jingbo Shang, and Julian McAuley. Learning Concise and Descriptive Attributes for Visual Recognition In 2023 IEFE/CVF International Conference on Computer Vision (ICCV), pp. 3067-3077, Los Alamitos, CA, USA, October 2023. IEEE Computer Society. doi: 10.1 10.1109//CCCV5117022222 00287. URL https://dom 2023. 00287
Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison- Burch,, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable im- age classification. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19187-19197, 2023.
Chih-Kuan Yeh, Been Kim, Sercan Ö. Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-awee aware concept-based explanations in deep neural networks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20 Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 97817138295A4
13

---

Published as a conference paper at ICLR 2026
Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. In
The Eleventh International Conference on Learning Representations, 2023. URL https:
Zaballos, Á. Llambrich, M. Ara, Z. Olazarán, Malvehy, and S. Puig. Dermoscopic findings of haemosideroticc and aneurysmal dermatofibroma: report of six patients. British Journal of Der- doi: 10.111 1355-2133..055.... 1111/1 1365 2133. 2005. 06844.
matology, 154(2)224--550, 02 2006. ISSN 0007-0963.
URL httpttttt//oooo...
Mateo Espinosa Zarlenga, Pietro Barbiero, Zohreh Shams, Dmitry Kazhdan, Umang Bhatt, Adrian Weller, and Mateja Jamnik. Towards robust metries for concept representation evaluation. In Proceedings of the Thirty- AAAI Conference Artificial Intelligence and Thirty- Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Arlificial Intelligence, AAAAI223AAAAT22MAAAAA AAAI Press, 2023.
ISBN 978-1-57735-880-4 doi: 1t.1609/aii..33711.2222. x37110026392. URL https: //dois::/ddi...o org/ 3009/aaai.3371100
Rui Zhang, Xingbo Du, Junchi Yan, and Shihua Zhang. The decoupling concept bottleneck model. IEEE Transactions Pattern Analysis and Machine Intelligence, 47(2):1250-122) 2025. doi: 10.1 10.1099/PAMM.2224444999999
Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A. Ehinger, and Benjamin I. P. Rubinstein. Invertible concept-based explanations for enn models with non-negative concept activation vec- Proceedings of the AAAI Conference Artificial Intelligence,
May URL petps::/ooj. aaai oag.aai.rrrr///e www.indew..innooeeeooooooo.ee..e php/
2021. doi: 10..1699/aaai.3553......... AAAI AAAIaarticleeviee/111555
Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B: Statistical Methodology, 67(2):301-32) 03 2005. ISSN
1369-7412. doi: 10.111 160.:11....77-9800000000000.. URL https: //tccps://di..ooo 0tt1://00101...01100111111
5.1467 9868 2005 00503.
A APPENDIX OVERVIEW
In the appendix, we provide:
B. Details on training SAEs
C. Details on random words CBMs
D. Summary of CBMs parameters and backbones
E. Visualizations CBL neurons
F. More examples of explanations
G. Additional experiments on dataset annotation
H. Accuracies under NEC
Visualizing SAE features
J. Explanations compared with baselines
K. Performance with an open-source MILM
B DETAILS ON TRAINING SAES
In this section, we detail how trained and evaluated SAEs for concept extraction. We did not find a single hyperparameter configuration that worked uniformly across datasets or backbones. Some dataset-specific adjustments were typically required. Following [rickenn et al. (2023). we relied on a mix quantitative and qualitative proxies to judge whether an SAE was "good enough" for downstream concept use. Specifically, tracked the following metries:
14

---

Published as a conference paper at ICLR 2026
1. L2 reconstruction loss. We want the reconstruction loss to be low to ensure extract comprehensive set of concepts.
2. Average lo. We aim for number significantly lower than the backbone dimensionality to ensure concepts are disentangled.
3. Feature density histogram. It shows how many hidden neurons fire at different activation frequencies across the training set. In the ideal scenario, neurons are either dead or rep- resent interpretable concepts, so we look for a histogram with two clusters, one with very low density representing dead or noisy features and one with higher density, which should represent the actual concepts.
4. Recovered cross-entropy loss. We ideally want the extracted concepts to recover model performance, so know they have predictive power.
5. Recovered accuracy. Same 4. For ISIC2018, we also consider balanced accuracy.
6. Manual inspection. Inspecting top-activating images for random neurons in the high- density cluster to assess whether the learned concepts interpretable. Empirically, found that when all other metrics are healthy, most concepts are interpretable, although this cannot be guaranteed.
In Table we provide the training hyperparameters for the SAEs used in the paper, while in Table we show the results in terms of the evaluation metrics we monitored. In Figure S we provide the Feature Density Histograms for each SAE. We can see that low-density neurons are generally well separated from high-density neurons. Furthermore, most of the low-density neurons are dead, i.e., never activating. Some neurons are neither dead nor high-density, and these are typically noisy and not very important for the task. As explained in Section 3 we perform a filtering step to remove these neurons before naming and annotation. In Figure 6 we show how choosing a different feature density cut-off impacts recovered loss, accuracy, and the number of neurons kept. We highlight the cut-off we used with a red star symbol. As we see, removing neurons with very low density has little impact cross-entropy loss and accuracy. After pruning, recovered loss and accuracy for CUB become 89.40% and 98.41 41% %. For ISIC2018, recovered loss and balanced accuracy become 99.41 % and 96.86.88 96.84% For ImageNet, recovered loss and accuracy become 97.63% and 96. 96.60%.
Table 3: Training hyperparameters for the SAES used the paper.
Hyperparameter
CUB
ISIC2018
ImageNet
Backbone layer dimension
512 1x
2048
2048
Expansion factor Optimizer
0,25x
4x
Adam
Adam
Adam
Learning rate L1 coefficient (ASAE)
1 10-
1 x 10-4
1 x 10-3
2 x 10-3
5 x 10-4
1 x 10-3
Epochs
1000
1000
1000
Patience for early stopping
50
50
50
Table Evaluation metries for the SAEs used in the concept extraction phase (pre-pruning). These are computed on the validation set, except for fo. which is computed on the training set.
Metric
CUB
ISIC2018
ImageNet
1.2 reconstruction loss
0.0231
0.0066
0.0462
Average tp
7.66
17.14
39.23
Recovered loss (CE)
89.49%
99.58%
97.74%
Recovered accuracy
98.39%
acc: 96.08%, bal. acc: 96.84%
96.68%
15

---

Published as a conference paper at ICLR 2026
Deedyy Dep
0
(a) CUB
(b) ISIC 2018
(c) ImageNet
Figure 5: Feature density histogram for CUB, ISIC2018 and ImageNet. Purple indicates dead neu- rons (never active in the training set). Yellow indicates neurons that were pruned due to low density and little to no impact on recovered loss. Green indicates neurons that are kept as concepts for the subsequent steps.
Rocovered CE Loss vs Praning Threchold
Recovered Accuracy is Pruring Threshold
hearcna Kapt tS Threchold
0.41
0013
300
307
0.02
201
300
0.500
6425
200
0.00
200
20
90
133
2
20
40
90
100
20
40
10
90
100
Adivvion Cuar Tlasoluud
Adiidoon Doat Tlesslind
Addidian Daak:
(a) CUB
Recovared CE Loss Pruning Threchold
Recovered Balonoed Accuracy vs Pruning Threshols
Neurons Kapt vs Threshold
12
ou to
22
22-
144
05
250
300
02
1000
3257
1000
1/50
2003
150
500
152
M00
1250 1500
3/50 2000
250
300
2000 150
3500
120
1500
(b) ISIC 2018
Recovered CI Loos Prunin Theechold
Recowered Accuracy as Pruning Threshold
Neurons Kept vs Threshold
0.55
2302
0.00
Numarr of 6wp
2300
1330
0.66
0.00
2.00
0.00
0.75
1300
3300
690
2000 4000 5000 8003 20000 :2000 34003
2000
4000 1900 2000 20030 22000 24002
2002
4003
100G
5000
22008 12000 30000
Activiisn Coant Thncaladd
Acthulon Doar
(c) ImageNet
Figure 6: Effect of pruning by activation-comm threshold for CUB, ISIC2018, and ImageNet. Higher thresholds typically reduce recovered performance, but when discarding low-density neurons, the reduction tends to be negligible. The point highlighted by red star indicates the cutoff used in our experiments.
C DETAILS ON RANDOM WORDS CBMS
In this section, we provide additional details on how we implemented the experiments with random words for VLG-CBM, VI.G-CBMcA. and M-CBM. For each method, replace every concept
16

---

Published as a conference paper at ICLR 2026
Pommon Yellowthroat Conf: 0.974 Logit:10.12 Blas:-0.02
bird balsamy
+1.35
bird acne
+0.23
Sum of 493 other features
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Concept contributions
Pred:Blue Jay
0.999
Logit:11.28
Blas:0.04
bird punditic
+2.05
bird
+0.22
Sum of 493 other features
+0
0.5
1.0
Concept contributions
Pred:Downy Woodpecker Conf: 0.999 Logit:13.03
Blas:0.04
bird unsown
features
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Concept contributions
Figure 7: Examples of VLG-CBM explanations where replacing concept names with random ones still yields correct predictions using just 1-2 concepts, illustrating how class-conditioned an- notation can leak speciffe information unrelated to concept semantics.
name with random, semantically meaningless text while preserving the cardinality the original concept sets and their class conditiond assignment for vanilla VLG-CBM. We draw words from the NNTK''s 's words corpus) filtered lowerease alphabetic strings of length 3-8, and added the prefix "bird so that the result is a short phrase like "bi rd pi zza". The prefix helps maintain min- imum image relevance so that models like GroundingDINO or GPT4. are more likely to annotate the random concepts positive in some of the images. For VLG-CBM, the annotation is done using their official codecia without modifications, while for VLG-CBMcA. we remove class conditioning and annotate each (random) concept across all images. Because GroundingDINO accepts at most 256 input tokens, we batch concept lists and run multiple passes until all concepts are processed. For M-CBM, we follow standard pipeline but substitute random names before annotation. Further- more, when annotating random concepts, we omit reference grids of top-activating images to avoid leaking information about the original concepts. Figure 7 illustrates how, under class-conditioned annotation, VLG-CBM can effortlessly predict correctly using only 1-2 random concepts.
D SUMMARY OF CBMS PARAMETERS AND BACKBONES
In this section, we report the number of parameters used at inference time for all methods and datasets. For each configuration, we decompose the total parameter count (i) the pre-trained backbone, and (ii) the CBL plus the final classifier, which we denote as CBM. The CBL size K corresponds to the dimensionality of the concept bottleneck (i.c., the total number of concepts). It differs across methods, because each CBM constructs and filters its concept set using its own procedure. Taalle5Jsummarize the parameter counts (in millions) for all methods considered in our main experiments. Differences are largely dominated by the choice of backbone (e.g., CLIP ViT- B/16 vs ResNet-18/50), while the additional parameters introduced by the CBM head are relatively small in all cases.
17

---

Published as a conference paper at ICLR 2026
Table Inference-time parameter counts (in millions). Backbone counts include only the pretrained feature extractor. CBM counts include the CBL and the final classifier.
Method
Dataset
Backbone
Backbone (M)
CBL K
CBM (M)
Total (M)
LF-CBM
CUB
ResNet-18
11.69
208
0.15
11.84
LF-CBM
ISIC2018
ResNet-50
25.56
35
0.07
25.63
LF-CBM
ImageNet
ResNet-50
25.56
4523
13.79
39.35
DN-CBMRN
CUB
CLIP RNSO
38.30
8192
10.04
48.34
DN-CBMRN
ISIC2018
CLIP RN50
38.30
8192
8.45
46.75
DN-CBMRN
ImageNet
CLIP RN50
38.30
8192
16.59
54,89
DN-CBMwI
CUB
CLIP ViT-B/I6
86.20
4096
2.92
89.12
DN-CBMvI
ISIC2018
CLIP ViT-B/I6
86.20
4096
2.13
88.33
DN-CBMAT
ImageNet
CLIP ViT-B/16
86.20
4096
6.20
92,40
VLG-CBMCA
CUB
ResNet-18
11.69
535
0.38
12.07
VLG-CBMcA
ISIC2018
ResNct-50
25.56
80
0.16
25.72
M-CBM (Ours)
CUB
ResNet-18
11.69
278
0.20
11.89
M-CBM (Ours)
ISIC2018
ResNet-50
25.56
73
0.15
25.71
M-CBM (Ours)
ImageNet
ResNet-50
25.56
2648
8.07
33.63
E VISUALIZATIONS OF CBL NEURONS
In Figures and 10 we show the top-5 activating test images for representative concepts on CUB, ISIC20T 8, and ImageNet, respectively. These visualizations qualitatively assess whether CBI. concepts align with their intended semantics and, when paired with model explanations, help convey what the model is actually sceing in the image that influences prediction.
F MORE EXAMPLES OF EXPLANATIONS
In this section, provide additional examples of local explanations of our M-CBMs NCC=5. The explanations are shown in Figures I1 ImageNet test sets.
12 and 13 respectively from the CUB, ISIC2018, and
18

---

Published as a conference paper at ICLR 2026
(a) Bird held in human hand
(b) White eyebrow stripe
(c) Birds water
(d) Black cap with white body
(c) Bird in leafy background
(f) Very long forked tail
(g) Red berries branches
Black head with bright orange breast
(i) White patch black back
Black mask around yellow face
(k) Reddish-brown throat patch
(I) White fan-shaped head patch
(m) Iridescent green-blue plumage
(n) White wing bars on dark wings
(o) Yellow chest with black v-shaped mark
(p) Dark, slender waterbird with long neck
(q) All-white plumage
(r) Black and white longitudinal stripes
(s) Birds perched on rocks
(t) Large crango-yellow beak
Figure 8: Top-5 activating images for CUB concepts
19

---

Published as a conference paper at ICLR 2026
(a) Blue-white veil
(b) Peripheral pigment network
Dark brown homogencous patch
(d) Pink-white structureless
(e) Red to purple homogeneous area
(I) Clustered blue-gre gray ovoid nests
(g) Small, dark brown or globule
(h) Scalp lesion surrounding hair
(i) Clustered pink-to-red vascular network
(j) Pink scaly plaque with white and brown dots
(k) Elongated pink structure with homogeneous tex-
ture
(I) Parallel lines or streaks within pigmented lesion
(m) Pink- to-lght red, structurcless, scaly erythema- pression or ulceration
Pink light red papule/nodule with central
tous plaque
Pinkish-white structureless with multiple
(0) Atypical irregular dark brown black blotches small eircular target-like brownish spots (milia-like with asymmetric borders and variegated pigmentation cysts and comedo- openings)
(q) Clustered white-to-pink. round structures (milky (r) Dark, structureless central with irregular, dif- keratotic dots) on a pink background
fuse pigmentation
(s) Irregular brown blotches
(t) Asymmetric patch with variegated pigmentation
Figure 9: Top-5 activating images for ISIC2018 concepts
20

---

Published as a conference paper at ICLR 2026
(a) Tiger stripes
(h) Hairless dog skin
(c) Sunset over water
(d) Exposed human skin
(e) Colorful umbrella canopy
Hexagonal grid structure
(g) Hands using laptop keyboard
(h) Wooden stick held by hand
Intricate lace crochet pattern
ii Yellow banana peel with brown spots
(k) Animal swimming at the water surface
Pairs footwear placed side by side
(m) Erupting lava and glowing volcanic ejecta
Person holding object near their mouth
Werde
CaKe
Canon logo text circular black surface
(p) Printed text and logos white plastic bags
(q) Set dining tables with tablecloths and chairs
(r) Dimpled white spherical surface with printed
(s) Creamy pasta with visible bacon pancetta pieces (t) Person bending forward wearing light-colored and black pepper
shirt
Figure 10: Top-5 activating images for ImageNet concepts
21

---

Published as a conference paper at ICLR 2026
Preciter Smallou Cent Lio.--ooo--00
Chai wingt orange thisst
-0.32
wosder pert
non cas soger jaasege
10.00
272
(a) Correctly predicted Barn Swallow
while
NOT sal binkk systimee stripes
white streaked back
NOT pulliu vood with black by and beak
Sum 273 other features
(b) Correctly predicted Blue headed Vireo with significant negative reasoning.
Bright orange curned bear
HOT
eye
peictidd on braachh +0.00
HOT witoe beily derc chart
271 other bunare +0.00
Contect cottitutios
(c) Correctly predicted Crested Auklet
Preeskeii femm
Logiti 3.0
erange
black head
Not with winggannn.o bcrwnn sings
+0.27
wing sunch
keep
festures
1..00
(d) Correctly predicted Arctic Tern
PrediRed headed Woodpecker Conf: Logit6.42 Bias: 0.10
red head with black soackk-an--hiie white body
+1.34
bright
+1.81
upperparts underperts
+0.32
iridescent magente throat patch
Sum 274 other features +0
(e) Correctly predicted Red headed Woodpecker
Loggeriedd
perthed
light grey bedy. btack wings
bleck acttun ayea
red shouder parch on bluck plumage -6.0
ferturm
Concept cotritttins
(i) Correctly predicted Loggerhead Shrike
Figure 11: Examples of local explanations from CUB test set.
22

---

Published as a conference paper at ICLR 2026
choteroii rosls
+2.04
FRA light papur etreeuee with central Iepression or atoration
pue weon with mumplle noc and brewn commadd or cruated prase
-0000
bose homogee pacch
of other festures
(a) Melanocytic nevus wrongly predicted as Basal Cell Carcinoma.
Prefiterrcma Cast 256. Logid Blaid
while wed
-0.00
NOT stepical brown blick tuAches with asprenecrr, tonder repettd oegracnaaion +5.25
reatrconpeeee dmamnee with muntt custrnn sosen tlack auts guotures
+0.00
NOT unihere hight medeom browe siynentation smnooh,, regular boeters
NOT scathered trown pigrentaatio
-0.03
teston surroudding hair
Sum other features
6.20-.15 0---55 8
0.13 015
Concept CORmiRAtiO
(b) Melanocytic nevus wrongly predicted as Melanoma.
PreduVascular lesion Conf: Logit:5.55 Blas:-1.03
red purple homogeneous
NOT pink scely plaque with white and brown dots
NOT light brown, structuroless, homogencous areas
+0.36
Sum of 70 other features
+0
Concept contributions
(c) Correctly predicted Vascular lesion
PrediActinc keratosis / Bowen's disease (intr intraspitheliaa carcinoma) Conf: 0.935 Logib4.65 -.75
pink-to-red structureless areas with white scales
+3.06
NOT small, dark brown dot or glooule
pink eraly pleque white and brown dots
Sum other
contributions
(d) Correctly predicted Actinic keratosis / Bowen's 's disease (intraepithelial carcinoma)
welonergthi Cwrl a.ds teget.77 Bu
MOT dark browe homagemans parcc
SUT brow parch the sander bntweee tao
33
sine ausion pla brow' sicaraeed cructed
HOT dant border Semv 66 other festunes
(e) Correctly predicted Melanocytic nevus
Mericaana Coal Lagit
ahysical Irepuiar brown treccha in myprmmetic benders and ragutted plomentattes inmttaataionn
unmoodt bondery
Mi white well
sinoitandoo, intipgmmel
Indian
-000
Saav othirr bostaney
Dororet conyituties
( Correctly predicted Melanoma
Figure 12: Examples of local explanations from ISIC2018 test set.
23

---

Published as a conference paper at ICLR 2026
Pretdook Cent Lagiz
MIT
wandaa atrrttai will radar rails
obeaited modilr solliviy over wallr
Sum 2541 lortures
-05
+03
00
02
(a) Correctly predicted Dock
0333 LopESA2 Stex
cinate no
celling deconatie
HOT stane and may with petiost
NOT rvod
Pepture
bustona
(b) Projectile wrongly predicted as Vault
Paces wearing berte sooples and smarkers
peodle standing at edge of bes
Sum of 2511 other festures
Conkept wwweeccccnnntee
(c) Correctly predicted Snorkel
Prediprogy_ Conti 0,231 Logg::045 65 Bla5:-0.77
large quadrupedal
decorated pig shaped ceramic container
NOT cartoon pig face with round snout and large eyes
00.0
NOT pumplcn squach with
0.00
-066
0.44
-0.2
0.0
0.2
0.4
Concest contributions
(d) Pitcher wrongly predicted as Piggy bank.
Frod watthe yonn
OUBSS Lagit 14.00 Baae 6:28
eotte parsereed caardigg planns
with tendo busleen rectudiss planlss Bitcren
+0.17
mebbrsed cocuier wriiber reated text and syettoos -0.9
Sum 2042 other
-as
23
(c) Correctly predicted Woffle iron
WIACOON Srick Cont 6,300 Logic:2 54 Blc
trock catre with light ber are bowiing boom
lngge suppedt
VOT exposed metal whenh and betts fand machinery +0.14
suppertion cables on indgee
suspenstoo
0.25
3.0
0.2 033
(1) Racer wrongly predicted as Tow truck.
Figure 13: Examples of local explanations from ImageNet test set.
24

---

Published as a conference paper at ICLR 2026
G ADDITIONAL EXPERIMENTS ON DATASET ANNOTATION
On the effect of the reference set for annotation quality. Our annotation pipeline provides the MLLM with a reference set of active examples for the target concept before asking it to annotate new samples. The goal is to visually ground the concept, which can be especially important when the textual label alone may not be precise enough (e.g., long tail, rufous crown, or small nevus). To quantitatively assess how much this reference set contributes to annotation quality, we use the human-annotated concept labels available for the CUB dataset and compare the MLLM's perfor- mance with and without providing the reference images. For each concept, we two parallel annotations over the same set of randomly selected 25 images: (i) one where the model is first shown 25 positive examples of the concept, and (ii) one where the model sees only the concept name and the images to annotate. Ground-truth concept labels from CUB allow us to compute accu- racy in each condition. Of the 312 CUB attributes, we exclude 19 concepts that contain fewer than 50 positive examples, leaving 293 concepts for evaluation for a total of 7325 images to be annotated. Across these concepts, providing the reference grid achieves a measurable improvement as accuracy increases from 65, 95.91% (without reference) to 69.46% (with reference images). This shows that the MILMM effectively leverages the visual examples for better annotation. However, it is important to note that these numbers may not be representative of the absolute annotation quality, given that CUB concept annotations are known to be very noisy. As noted by Koh et al. (2020), each attribute was annotated by a single non-expert crowdworker, and many concepts are fine-grained or subjective (e.g., red belly vs rufous belly). While such inconsistencies limit the maximum attainable accuracy, the relative improvement provides evidence in favor of the reference set being helpful in improving the quality of the annotations.
Grid vs single-image for dataset annotation. In the main annotation pipeline, we present the MLLM with a 5 x 5 grid of 25 images once, asking it to output list of binary predictions corresponding to each image. This batch style strategy is critical for keeping annotation costs man- ageable, especially for large datasets such ImageNet. However, it is natural ask whether the MILM would produce higher-quality annotations if instead queried with one image at a time. To investigate this, we repeat the experiment described in the previous section, but modify the prompt- ing so that each image is shown individually. For every concept, the model receives the same visual reference examples, but is presented with each of the 25 test images in separate calls. We compare the resulting accuracy with the grid-based setup.
The single image approach has an accuracy of 74.00%, compared to 69.46% for the grid. This indicates that the MLLM indeed benefits from focusing on one image at time, likely because it reduces cross-image interference and allows the model to allocate all attention to the visual details of a single example. In principle, this also suggests that prompting granularity is an important design dimension in MLLM-based annotation systems. However, the practical implications of this improvement are highly limited. The single-image method increases the number of model calls by a factor of 25, as each of the 25 images in the original batch now requires its own MILM request. As example, for ImageNet this would increase annotation time from days months and the API cost from USD 370 to USD 9250. As a result, while the single-image approach achieves better annotations, it is currently impractical for most applications. Nevertheless, these findings are encouraging, as they indicate that as MLLMs become more cost-efficient, single-image prompting would be a straightforward way to improve results further.
Robustness of MILM annotation under reference set poisoning. Providing set of reference images for a target concept generally improves annotation quality, but a potential risk is that if this reference set is noisy (i.c., containing some images that represent different features than the concept), the MLLM might inadvertently internalize these spurious visual features and associate them with the concept. To test whether our annotation pipeline is susceptible to this effect, we design a controlled poisoning experiment in which the reference images for concept are deliberately contaminated with images from a different concept. We then also contaminate the batch of images to be annotated with images from this secondary concept and measure whether these poisoned images are predicted as positive more often than typical negative examples. This setup provides a direct quantitative assessment of whether the MLLM forms spurious visual associations from the reference examples or remains faithful to the intended concept semantics. The experiment is carried out on the CUB dataset, where ground-truth concept annotations allow measurement of false positive behavior.
25

---

Published as a conference paper at ICLR 2026
For each target concept A, we construct two types of negative examples: (i) standard negatives (im- ages that do not contain A) and (ii) poisoned negatives (images that do not contain A but do contain different concept B). For every concept A, we randomly select a concept B and contaminate its reference set by replacing 5 out of the 25 reference images with poisoned negatives. Then, during annotation, we also replace 1 standard negative image per annotation batch with a poisoned nega- tive. The remaining pipeline is kept unchanged. We repeat this procedure for all 293 concepts with sufficiently many positive samples and annotate 25 images per concept, resulting in 293 MLLM annotation calls and total of 7325 images.
After annotation, we compute the False Positive Rate (FPR) separately for poisoned negatives and for standard negatives, quantify the difference using Cohen's 's h effect size, and test whether the two proportions differ significantly using a two-proportion 2-test. Results are reported in Table 6 The results reveal only small difference between the FPR of poisoned and normal negatives: 0.232 vs 0.198. The corresponding effect size, Cohen's h = 0.084, well below the conventional threshold for "small" effect (/ 0.2), indicating that the magnitude of this difference is tiny and negligible in practical terms. Consistently, the two-proportion z-test does not find a statistically significant difference between the two FPRs (p-value 0.157), meaning that the observed effect is statistically indistinguishable from random variation. Taken together, these findings suggest that the MLLM does not meaningfully internalize the spurious visual features introduced through reference-set poisoning. Even when 20% of the reference examples are deliberately contaminated with a conflicting concept, the model's s annotation behavior remains largely stable, indicating that the annotation is primarily guided by the semantic description of the target concept rather than incidental correlations present the reference examples.
Table 6: Results of MLLM-basee concept annotation under poisoning of the reference set on the CUB dataset using human-annotate concepts as ground truth. We report the false positive rates (FPR) for poisoned vs normal negative examples, the two-proportion z-test, and the effect size (Cohen's 's h).
Metric
Value
FPR (poisoned)
0.232
FPR (normal)
0.198
Two-proportion z-test (p-value)
0.157
Cohen's 's effect size
0.084
H ACCURACIES UNDER NEC
In this section, report M-CBM accuracies when controlling for NEC instead of NCC. For context, NEC measures the average number of non-zero classifier weights per class (e.g.. NEC=5 means that, on average, 5 concepts per class have non-zero weights), while NCC is defined in detail in Section 4 and intuitively measures how many concepts, on average, are needed to explain predictions. NCC can be viewed as a generalization of NEC, as when the coverage level is set to T 1, NCC reduces to NEC (see Section H.T for proof). Because we use a relatively high coverage level T = 0.95, the resulting accuracies are not dramatically different (see Table Intuitively, NCC differs from NEC in that it measures sparsity at the decision- using concept contributions (weights multiplied by activations), rather than just weights. This relaxes the hard cap that NEC implicitly imposes on the effective concept vocabulary. Indeed, with NCC, 5, the model may assign non-zero weights to
Table 7: Accuracy results at NEC=5 and NEC=avg.
Dataset
CUB
ISIC2018
ImageNet
Metrics
Accuracy
Accuracy
Balanced Accuracy
Accuracy
Sparsity
NEC-S
NEC=avg
NEC=5
NEC=avg
NEC-5
NEC=avg
NEC=5
NEC=avg
M-CBM
72.54%
73.70%
69.38%
73.47%
68.97%
70.45%
71.23%
73.00%
26

---

Published as a conference paper at ICLR 2026
many concepts, long only about concepts required, average. explain each prediction at coverage level т. For instance, M-CBM at NCCu.95 = 5 has 271 concepts with non-zen zero weights CUB, 47 ISIC2018, and 2572 ImageNet, while NEC=5 it has 253 CUB, 29 ISIC2018, and 2343 on ImageNet. In Figure 14 we provide some examples of concepts that are used by the NCC=5 model but not by the NEC=5 model.
(a) Irregular brown blotches
(b) Clustered pink-to-red vascular network
(c) Scalp lesion with surrounding hair
(d) Scattered light brown granular pigmentation
(e) Diffuse reddish pigmentation with structureless (1) Pink skin with fine hair and erythematous (reddish) background
pattem
(g) Birds on water
(h) Black plumage
(i) Perched on branch
(j) Yellow belly or breast
(k) Long pointed wings in flight
(I) Green leafy background
(m) Hands manipulating objects
(n) Wooden cylindrical stick
(o) Gothic ribbed vaulted ceiling
(p) Car dashboard audio control panel
(q) Motion of fast-moving vehicles
(r) Formation multiple flying objects against the sky
Figure 14: Top-5 activating images for CBL neurons that have all zero weights NEC=5, but not at NCC=5 for datasets.
27

---

Published as a conference paper at ICLR 2026
H.1 RELATIONSHIP BETWEEN NEC AND NCC
We now formalize the connection between NEC and NCC at 7 Proposition H.l. Let
ΣΣ NFC(WP)) C =lk=
1
([Wlle #03,
and let NCC, be defined as in Section Assume that whenever [Wr)kr F O, the corresponding concept logit [g( is non-zero for all images i. This assumption is reasonable, since concept logits are continuous, z-normalized values, so the probability of encountering an exact zero is neg- ligible at machine precision. Then
NCC NEC(Wr).
Proof. By definition in Section E
[[((a()) [Wrlk.d 2 0.
[Wel,, = O, then "krr = 0 for all i. Under our assumption, if [Weler * 0, then [g(a 11 + for all i, hence "krr > for all i. Thus, for fixed r,
ker >
[Wrk.r # 0 for all
For each class define
[[W + #0},
my
the number concepts with non-zero weight for class r. For any image i, exactly m, of the "kr are strictly positive and the remaining K - m, contributions are zero.
(8) Let (1).r 2 (K)r be the sorted contributions. Sorting moves the m, positive terms to the front,
my
(4)
u (s).r
(1) and (s),r O for all 8 > my.
By the definition of k, (1) in Section 4 is the smallest number of top-ranked contributions whose (6) sum matches the total contribution Σk=1 "k.r- Since this total is exactly the sum the m, strictly positive terms and all remaining terms are zero, it follows that
( mr for all i,r.
Substituting into the definition of NCC1,
C Σ
NCC,
C
Σ
(1)
D
Σ m,
0 m,
4,"
1.5
r=1
Using the definition of m,,
C K Σ
NCC1 C
Σ
((Weer # 0) = -NFC(WF).
r=1k=
which completes the proof.
28

---

Published as a conference paper at ICLR 2026
I VISUALIZING SAE FEATURES
In Figures 15 and 16 we show the top-5 activating images for SAE neurons on CUB, ISIC2018, and ImageNet, respectively. We also provide their respective description given the MLLM. These visualizations qualitatively assess whether the MLLM descriptions align with the neuron behavior. We also overlay the activating images with their respective saliency map show where that neuron was looking.
(a) Bird on tree bark
(b) White wing patch on dark bird
(c) Bird held in human hand
(d) All-white plumage gull
Bird ground rocky surface
(f) Small bird perched bare branch
(g) Orange bill with white facial plumes
(h) White eyebrow stripe on a brown bird
(i) Outstretched like wings in flight
(j) Bright yellow head and chest with black body
(k) Green back with pale yellow spotted underparts
(I) White patch the back of black bird's neck
(m) Bright green body with black face and blue crown (n) Bright yellow throat and upper chest on a songbird
(o) Black bird with bright red and yellow shoulder (p) Iridescent or green plumage on medium- sized hird
patch
(q) Bold streaks on pale underparts
(r) Glossy black plumage
Figure 15: Top-5 activating images for SAE features in CUB
29

---

Published as a conference paper at ICLR 2026
(a) Large crowd at an event
(b) Deep snow
(c) Tiger nose muzzle
(d) Handwritten or printed text on paper
Sttting
MARAN'S
ARBERS
UEEE SHOE
SAUER
(c) Tiger stripes
(f) Barbershop window signage
(g) Cluster of unripe green bananas
(h) Circular speaker grille
White rounded plastic surface
(j) Stacked audio equipment front panels
(k) Analog control knobs and frequency dials
(I) Gooseneck with attached cylindrical element
(m) Dual vertical antennas rectangular electronic (n) Vertical row of illuminated status indicator lights electronic device
device
(o) Green printed circuit board with electronic compo- (p) Handheld electronic device screen displaying text ments
or interface
(q) Rectangular electronic device with visible ports or (r) Closed metallic or plastic rectangular device with rounded edges
switches
Figure Top-5 activating images for SAE features in ImageNet
30

---

Published as a conference paper at ICLR 2026
(a) Brown globules
(b) Blue white veil
(c) Peripheral pigment network
(d) Dark brown homogeneous patch
Asymmetric patch with variegated pigmentation
(f) Pink-white structureless
(g) Red to purple homogencous area
(h) Clustered blue-gray ovoid nests
Brown macule with overlying hair
(j) Scalp lesion with surrounding hair
(k) Diffuse homogencous brown pigmentation
(I) Clustered pink-to-red vascular network
(m) Brown patch at border between two skin tones (n) Parallel lines or streaks within pigmented lesion
(o) Central dark spot or blotch on a lighter pink back- (p) Dark brown circular lesion at the border of lighter ground
skin region
(q) Small, well-cireumseribed, dark brown to black (r) Clustered white-to-pink round structures (milky or keratotic dots) on pink background
central pigmented spot on light skin
(s) Pink or light red papule/nodule with a central de- (0) Pink structureless area scattered brown dots or pression or ulceration
clods
Figure 17: Top-5 activating images for SAE features in ISIC2018
EXPLANATIONS COMPARED WITH BASELINES
31

---

Published as a conference paper at ICLR 2026
Ground Truth: Bridegroom
M-CBM (Ours)
DN-CBMVIT
Predicted: Bridegroom (Conf. 0.753)
Predicted: Bridegroom (Conf. 0.948)
1. bride white wedding dress holding bouquet (+4.07)
1. bridal (+6.42)
2. group_ of_ peopleppppppp in celebratory _o ro_formal___gg _attire (+2.06)
2. couple (+2.05)
3. men's suit jacket with dress shirt and necktie (+1.62)
bejing (+1.12)
wedding dress bodice upper skirt (+1.27)
suits (+0.3)
5. standing person in formal distinctive clothing (+0.46)
5. parties (+0.17)
6. Sum of all 2643 other concepts (+0.26)
6. Sum of all 4091 other concepts (+0.1)
LF-CBM
Predicted: Bridegroom (Conf. 0.135)
wedding (+4.34)
2. waiting at the altar (+0.55)
groom (+0.35)
Sum 4520 concepts (0.0)
Figure 18: ImageNet Dataset
Great Grey
M-CBM (Ours)
VLG-CBMcA
Predicted: Great Grey Shrike (Conf. 0.70)
Predicted: Great Grey Shrike (Conf. 0.41)
black mask eyes (+4.64)
NOT red shoulders (+1.8)
2. gray body with white underparts (+1.75)
2. NOT stocky body (+1.35)
wingbars on gray wings (+1.6)
large black spider (+0.88)
4. dark with cheek (+0.3)
NOT small blue green body (+0.77)
light gray body, black wings (+0.23)
long black and white tail (+0.6)
Sum of 273 other concepts (0.0)
Sum 530 other concepts (+0.85)
LF-CBM
DN-CBMyIT
Predicted: Great Grey Shrike (Conf.. 0.08)
Predicted: Sciesor tailed Flycatcher (Conf. 0.09)
colors (+0.62)
1. black cap and white "eyeline" (+1.51)
2. kramer (+0.48)
white
(+1.48)
field (+0.46)
3. white and black coloration (+0.9)
beige (+0.45)
4. a white face and underparts (+0.11)
5. flea (+0.42)
5. Sum all 204 other concepts (0.0)
Sum of 4091 other concepts (+0.05)
Figure 19: CUB Dataset
32

---

Published as a conference paper at ICLR 2026
Ground Truth: Melanoma
M-CBM (Ours)
VLG-CBMCA
Predicted:
Melanoma (Conf. 0.70)
Predicted: Melanoma (Gonf. 0.45)
1. atypical irregular dark brown black blotches with
1. cancer (+1.17)
asymmetric borders and variegated pigmentation (+5.0)
redness erythema (+0.75)
2. NOT uniform light-lo-medium brown pigmentation with smooth, regular borders (+1.03)
3. NOT sun damaged skin (-0.19)
scaly or rough patches (+0.14)
3. blue-white veil (+0.76)
NOT scaly crusted surface (-0.12)
4. multicomponent structure with multiple clustered brown
6. pathological entity (+0.08)
black dots globules (+0.75)
ulceration some cases (+0.08)
5. NOT scattered light brown pigmentation (-0.03)
Sum all 73 other concepts (+0.05)
Sum other concepts (0.0)
LF-CBM
DN-CBMVIT
Predicted: Vascular lesion (Conf. 0.54)
Predicted: Melanoma (Conf. 0.17)
1. NOT dryness scaling (+1.16)
2. NOT hair follicies (+1.09)
1. magnesium (+0.05)
3. NOT Bleeding (-0.38)
(+0.03)
4. NOT Scaly crusted surface (+0.07)
3. Sum of all 4094 other concepts (0.0)
5. NOT Epidermis and dermis layers (+0.01)
30 other concepts (0.0)
Figure 20: ISIC Dataset
33

---

Published as a conference paper at ICLR 2026
Table 8: Accuracy results at NCC=5 and NCC=avg of M-CBM using an open-source MLLM (1ttemnL3.5-2411--42283 for naming and annotation and e5-large-v2 for concept merging.
Dataset
CUB
ISIC2018
ImageNet
Metries
Accuracy
Accuracy
Balanced Accuracy
Accuracy
Spansity
NCC=5
NCC=avg
NCC=5
NCC=avg
NCC=5
NCC=avg
NCC=5
NCC=avg
M-CBM
71.40%
74.02%
67.33%
74.25%
69.26%
72.20%
56.76%
68.76%
Table 9: Evaluation of concept predictions on the test set for M-CBM using an open-source source MLLM (ntern/L335-2411---A00 for naming and annotation and e5-large-v2 for concept merging.
Dataset
CUB
ISIC2018
ImageNet
ROC-AUC
ROC-AUC
ROC-AUC
Metrics
Macro
Worst- 10%
Macro
Worst-10%
Macro
Worst-1092
M-CBM
74.49%
58.14%
74.36%
56.53%
70.97%
59.28%
K PERFORMANCE WITH AN OPEN-SOURCE MLLM
In this appendix, we evaluate how M-CBM behaves when replacing the proprictary components used for concept naming, dataset annotation, and concept merging with open-source alternatives while keeping the rest of the pipeline unchanged. For this experiment, we use |ntern A28B as the MILM for both naming and annotation (it supports multi-image inputs, so we can preserve our prompting protocol), and we use e5-large-v2 to embed and merge concept names with the same similarity-
merging strategy as in the main paper and the same threshold (i.e., 0.98). Table 8 reports the downstream performance obtained with this fully open-source variant. On CUB, obtain 71.40% accuracy NCC=5 and 74.02% at NCC=avg: on ISIC2018, and 74.25% accuracy, with balanced accuracy of 69.26% and 72.20%; and on ImageNet, we obtain 56.76% accuracy at NCC=5 and 68.76% NCC=avg- Since do not have ground-truth concept annotations, we follow the same proxy evaluation as in the main paper by annotating the test set with the same pipeline and then measuring ROC-AUC of concept predictions against these labels. Table reports the macro-average ROC-AUC across concepts and the average ROC-AUC over the worst 10% concepts. On CUB, the ROC-AUC is 74.49% (macro) and 58. 14% (worst 10%); on ISIC2018, it is 74.36% and 56.53%; and on ImageNet, it is 70.97% and 59.28%. Overall, using the open- source MILM leads to lower concept-prediction quality and lower downstream performance on all three datasets, but the effect is not uniform across them. On CUB and ISIC2018, the degradation noticeable but relatively limited, whereas on ImageNet it substantially larger. Qualitatively, we observe that Intern VL3.5 tends to produce more generic and highly similar concept names, which increases the amount of merging and reduces the final concept vocabulary. For example, on CUB obtain 195 final concepts instead of the 278 concepts produced with GPT-4.11 on ISIC2018 we obtain 38 concepts instead of 73, and on ImageNet we obtain 1768 concepts instead of 2648, again substantially fewer than with GPT-4.1. This effect appears to be particularly problematic on ImageNet, where the much larger and finer-grained set of classes seems to require more concepts and higher-quality annotations. This is also consistent with the lower proxy ROC-AUC values for concept prediction. At the same time, the same qualitative trend of lower annotation quality and fewer final concepts is also present on CUB and ISIC2018, suggesting that concept naming and annotation quality are important across all datasets, even when the downstream impact is smaller. One possible explanation for why the downstream drop is smaller on CUB and ISIC2018 is that these datasets may be more prone to information leakage, since the amount of information needed to separate classes may be substantially smaller than in ImageNet. Overall, these results make more evident that the quality of concept naming and annotation is critical for downstream classification in M-CBM. On a positive note, since GPT-4.1 annotations are themselves far from perfect and the MLLM component is modular, stronger models may further improve concept quality and, in turn, downstream performance.
obtain 67.33%
34


Reviews:
[{"review_id": "WyF38qFQv8", "reviewer": "Reviewer_gj53", "summary": "The paper proposes M-CBM, a pipeline that (i) trains an SAE on a frozen backbone’s features, (ii) uses an MLLM to name SAE neurons, (iii) asks the MLLM to annotate concept presence/absence on a subset of images (partly from test), and (iv) trains a CBL and a sparse linear classifier. The authors also introduce NCC (Number of Contributing Concepts) as a decision-level sparsity metric.", "questions": "1. How do you prevent test-time contamination when concept labels on test are produced via the same activation-conditioned pipeline (SAE saliency + top-activating grids)? Please clarify precisely which model components see any information derived from test images and at what stage. \n2. Can you re-run the full pipeline without proprietary models (e.g., open-source MLLMs/embeddings) and report gaps in naming/annotation quality, costs, and final NCC-matched accuracy?\n3. Provide sensitivity of your results to (a) SAE expansion factor, (b) pruning tolerance, (c) activating vs non-activating examples per neuron during naming/annotation.  \n4. For fairness, could you supply ImageNet-scale VLG-CBM_{CA} with cost-matched budgets (e.g., same wall-clock as your GPT calls), or conversely report M-CBM results with the same annotation budget as VLG-CBM_{CA}?\n5. On the absence of an explicit concept-leakage metric (e.g., Impurity Score).\nThe manuscript highlights concept leakage as a major challenge motivating M-CBM, yet the evaluation currently focuses on sparsity (NCC) rather than a dedicated leakage measure. It might strengthen the empirical section to include or discuss a quantitative leakage metric, for example, the Impurity Score or a mutual-information-based variant. Such an analysis would help clarify whether the observed improvements stem from genuine leakage mitigation or primarily from sparsity regularization.\n6. On comparison with CBMs that explicitly address concept leakage.\nSeveral works have proposed concrete strategies to quantify or reduce concept leakage. Positioning M-CBM relative to these methods, either through direct comparison or a discussion of conceptual differences, could provide valuable context and highlight the distinct contributions of your work beyond sparsity control.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Clear, modular pipeline with concrete engineering choices. Training of the CBMs is standard and well-described.  \n2. NCC formalizes decision-level sparsity (concept contribution × weight) and can be computed per image/class.  \n3. Results tables/curves are easy to read.", "weaknesses": "1. Methodological leakage / evaluation contamination.\nThe paper explicitly annotates concepts on 20–30% of the test set (per concept) “solely for a final CBL evaluation.” But concept labels on test are produced using the same pipeline (SAE activation pre-selection + MLLM conditioned on highly activating images). This risks circularity and optimistic estimates of “concept learnability/consistency,” since test annotations are themselves guided by backbone activations and neuron saliency that were derived from training. At minimum, this violates a clean separation of test supervision and model analysis and undermines the claimed generalization of the concept predictor. \n2. Reproducibility concerns due to proprietary dependencies.\nCore steps depend on GPT-4.1 for naming/annotation and OpenAI embeddings for merging, with specific costs/timings reported; there is no open substitute evaluated. Reproducibility and fairness of comparison suffer because model behavior can change with closed APIs and prompts, and the paper does not provide robustness to alternative open MLLMs/embeddings.\n3. Novelty over prior work is incremental.\nDN-CBM already uses SAE-based concept discovery and naming; M-CBM mainly claims backbone-agnosticism and a different annotation path. NCC, while useful, is a straightforward generalization of NEC from weight-count to decision-level contribution. \n4. Baselines and fairness.\nThe paper drops the original (class-conditioned) VLG-CBM from main comparisons on grounds of leakage, and reports VLG-CBM{CA} as the “fair” variant—yet **ImageNet-scale VLG-CBM{CA}** is marked N/A, while M-CBM still uses thousands of GPT annotations, a heavy but different cost. The choice of backbones is also uneven (DN-CBM forced onto CLIP-RN50 across all datasets), which can disadvantage it (reported accuracy much lower than original paper) and muddle conclusions about accuracy–sparsity trade-offs. \notential confirmation bias in concept discovery.\n5. Concept naming/annotation is conditioned on highly activating examples of the SAE neuron and saliency computed from SAE decoder weights. This can bias the MLLM toward naming whatever the neuron already encodes, rather than establishing human-grounded semantics independent of the backbone. \n6. Inconsistent focus on “concept leakage” without any quantitative metric.\nThe authors repeatedly stress concept leakage but never evaluate it quantitatively. The proposed NCC only measures sparsity, not leakage strength or label information flow, so their claims of “leakage control” remain unsubstantiated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LlpfYroiqw", "reviewer": "Reviewer_kxfy", "summary": "This paper presents a mechanistic Concept Bottleneck Mode --- M-CBM --- that brings a new opportunity to extract concepts on the fly using Sparse Autoencoders. For annotating and giving the concepts a human-readable “entity,\" the authors prompt multimodal LLMs, with a special accounting for compute and costs. Notably, the authors introduce a new metric for measuring the sparsiness and consequently, expressiveness of the activated concepts. This metric is based on the prior work and evaluated throughout the paper. Finally, by rigorous comparisons with popular baselines in the field, M-CBM emerges as a robust and strong baseline, facilitating future research.", "questions": "See the **weaknesses** part", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**The baselines are recognized in the field and strong-performing.** The authors conduct their comparison in a simple and rigorous manner evaluating the same backbone encoder for each model on the same dataset. Considering that some datasets require a broader concept set to perform well on, the authors employ larger backbone models for all baselines, e.g., on ImageNet.\n\n**The idea is sound and well-presented.** I like the way how authors described their framework in Sec. 3. Effort in proposing a new metric for measuring sparsity leveraging concept contribution.\nAlso interesting how the authors ablate their meric vs. accuracy in Fig. 2 using the baseline VLG-CBM model\n\n**Accounting for compute and costs.** When using (M)LLMs to assign each SAE neuron to concept name or to annotate some subset of data to create a matrix of \"Present\" / \"Missed\" concept per image, the authors mention a precise cost of the OpenAI API per one annotation.\nAs well as stating that money spent scale linearly with the number of concepts. Thus, one can do the math and calculate the total spendings.\n\n**Demonstrating the concept \"yes / no\" selection and concepts pertaining to image.** In Fig. 3, the authors show a precise example of which concepts are being selected to a particular class and which receive the negative weight. The provided concepts look very meaningful and not monosyllabic, unlike concepts in many prior works that were obtained with less sophisticated methodology.", "weaknesses": "**Your NCC metric.** I do appreciate the introduced metric as the extension of the NEC metric from the prior work on VLG-CBM. As far as I understand, the parameter $\\tau$ controls, let us say, the \"sparsity\" of the impactful concepts. Meaning that the model with smaller NCC (and also the smaller NEC metric) provides more explainable decision since only few the main important concepts are activated. So it is naturally good to keep both metrics relatively small, or contrary, a large NEC (NCC) can result in information leakage. Given that both metrics are pretty similar in their essence, I would like to see more comparisons and discussion between two of them, in flavour of your Fig. 2 (or the Fig. 3 from the VLG-CBM work). Importantly, results of VLG-CBM demonstrate only minor, even negligible, drop in accuracy when their NEC is small. Which leads me to questions:\n1) Does the same holds with VLG-CBM under your NCC metric?\n2) Does the accuracy drop of your M-CBM is also negligible under small NCC and NEC metrics?\nIn my opinion, offering discussion and, which is better, experiments on this merit in your manuscript will significantly increase the reader's confidence in your NCC metric.\nAlso, it would be nice to include results on NCC while sweeping $\\tau$. Could it be that with $\\tau = 0.5$ or less many concepts will satisfy this fraction of information and, thus, the decision-making process would be rubbish? How does this connect to the fact that even with randomly initialised concept bottleneck layer the model can give near-optimal accuracy when the VLG-CBM NEC metric is large?\n\n**Regarding your comparison in Fig. 2 (a)**, I am also curious in the following: the VLG-CBM baseline uses up to 128 concepts for CUB --- see their Fig. 3 (c). Additionally, prior work Res-CBM [1] claims that the number of concepts should not be the larger the better, meaning there is some critical k, beyond which, the accuracy will drop. Do you think this will be observed when NCC (or similarly, NEC) are very large? Offer some discussion. Ideally, it would be nice to provide figures based on your Fig. 2 with very large NCC --- at least a couple of hundreds.\n\n**Backbone model choice.** From \"Sec. 5 Setup\", I see that you use ResNet-18 for CUB, ResNet-50 for ISIC1028, and ResNet-50 for ImageNet. I think that using the same pretrained backbones architectures for all baselines on the same dataset is a consistent choice and it deserves a credit for being applied --- as long as the number of total parameters (both pretrained and trainable), including your SAE is mentioned --- I would like to see a simple table summarizing this. More importantly, in Lines 369-370, you mention that DN-CBM supports only CLIP backbone and, thus, you use their CLIP ResNet-50. However, CLIP ViT-B/16 is also supported by their framework and offer better results.\nMoreover, such a backbone is the widely supported option, used almost in every work on CBM. To make the results more appealing and aligned with prior literature it is better to present a version of your Table 1 with this backbone.\n\n**Missed concepts for ISIC2018.** I am not sure wether it is my issues or yours, and apologize for potential mistake, but I cannot verify the number of concepts (and those concepts itself) for the ISIC2018 dataset, because the file `isic_concepts.txt`, unfortunately, is not found in the repo.\nHowever, I can confirm that, mentioned in Line 377, 278 concepts for CUB, and 2648 for ImageNet are matched.\n\n**Table 1 with the main results.** I do acknowledge that you now have different backbones, but the different in accuracy of the baselines evaluated by you in your framework with the backbones you are using differs quite dramatically from the results reported in the original papers of your baselines and their concurrent works. To give a better understanding of such a difference, I will write a precise metrics form your baselines, and their competitors that you cite but not evaluate.\nFor example, LF-CBM, in their original work, results in 74.31% on CUB and 71.95% on ImageNet, while the competing work --- DN-CBM --- reports for them 67.5% on ImageNet with CLIP ResNet-50 backbone and 75.4% on ImageNet with CLIP ViT-B/16 backbone. At the same time, in their paper, DN-CBM shows 72.9% on ImageNet with CLIP ResNet-50, and 79.5% with CLIP ViT-B/16 --- both results are superior to those you have mentioned for DN-CBM on the ImageNet dataset (46.71% when NCC=5, and 57.24% with NCC=avg).\nFurthermore, the work you cite, but do not compare with --- Post-hoc CBM and also their model PCBM-h --- report an accuracy of 73.6% and 80.01% on ISIC, resp., which makes them better than M-CBM --- so offer a comparison or discussion regarding this.\n\n**Comparison in concept extraction with other baselines.** It would straighten your work if you presented an analogue of Fig. 3 for several other baselines, e.g., like Fig. 3 in the very recent preprint [2].\n\n**Missing baselines and discussions.**  Despite evaluation very popular baselines, I think you are missing comparisons and (or at least) discussions regarding the following works [1,3,4,5,6]. All of them claim to outperform the long-standing LF-CBM to some extend, and are a decent option for comparison in the field. Summarizing a ranking those in your Table with results will further guide the community towards using better and more promising baselines.\n\n**A minor comment.** As I see, in your codebase you implement the basic Adam optimizer with the old \"coupled\" weight decay. I would suggest to stick to the decoupled weight decay of ~0.1 and AdamW instead of Adam.\n\n[1] \"Incremental Residual Concept Bottleneck Models\", 2024\n\n[2] \"Graph Integrated Multimodal Concept Bottleneck Model\", 2025\n\n[3] \"Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification\", 2023\n\n[4] \"Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning\", 2024\n\n[5] \"The Decoupling Concept Bottleneck Model\", 2024\n\n[6] \"Improving Concept Alignment in Vision-Language Concept Bottleneck Models\", 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1G7DpYVsWe", "reviewer": "Reviewer_pxTa", "summary": "The authors propose a new method for extracting concepts in concept bottleneck models by leveraging sparse autoencoders (SAEs) on top of a black-box architecture. They first train an SAE and identify concepts by examining the non-dead neurons. For each neuron, they select the most and least activating images from the black-box model’s dataset and use a multimodal large language model (MLLM) to assign concept labels. To train the concept bottleneck layer, they construct a semi-automated dataset containing 1,000 images per concept to determine concept presence. In addition, they introduce a softer measure of concept sparsity (NCC), with the aim of enabling more controlled and flexible comparisons across concept bottlenecks.", "questions": "- While this is a clearly well-written and valuable paper, I remain uncertain about the role of NCC. One way to address this would be to also report performance under NEC; another would be to demonstrate that the extracted concepts have an intuitive and interpretable basis.\n- Clarification: Beyond instructing the MLLM not to use class names as concepts, did you perform any filtering to ensure that class names were indeed excluded? (line 210)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The methodology is intuitive, and its benefits are evident: leveraging concepts already present in the black-box model enhances interpretability.\n- The approach achieves performance improvements over baselines when controlling with NCC.\n- The paper is clearly written, and most aspects of the methodology are well justified (with the exception of the weaknesses noted below).", "weaknesses": "While the motivation for NCC is intuitively explained (lines 303–305), it would be valuable to provide evidence for why the additional concepts are necessary. When concepts contribute substantially less than the dominant ones that NEC would capture, are they still useful and interpretable, or do they risk overfitting? NCC appears to strike a balance between interpretability and overfitting, but it is unclear how much the method benefits from this added flexibility. Would the same performance patterns hold under the stricter NEC criterion? Alternatively, is the improvement primarily due to the inclusion of negative concepts? Reporting NEC scores alongside NCC results could help clarify the extent to which this flexibility contributes to the method’s effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fp2uhCeVTM", "reviewer": "Reviewer_GTFX", "summary": "This paper introduces Mechanistic CBM (M-CBM), which introduces advancements in the field of mechanistic interpretability to the concept bottleneck framework. M-CBM takes a pretrained blackbox and trains a sparse autoencoder on its internal activations to extract human-interpretable concepts that are relevant for prediction. After having named and filtered the discovered concepts, a MLLM is used to label a non-random subset of images with these concepts. Subsequently, a standard CBM is trained on the discovered and labelled concepts for human-interpretable predictions. Furthermore, the authors introduce the Number of Contributing Concepts (NCC) metric, which measures the number of concept contributions required to explain $\\tau$ of all absolute contributions. Results show improved performance on NCC and concept prediction, as well as qualitative prediction examples.", "questions": "I invite the authors to address any of my statements from the Weaknesses section, as they are my main source of concern.\n\nFor W1: \n* Could the authors provide the highly activated images used when querying the MLLM for obtaining the concept names from Figure 3, jointly with the heatmap overlayed, such that it is clear whether concept names relate to the highlighted image parts?\n* What is the prompt used for dataset annotation?\n* How do the top-25 most activating images affect the MLLM during dataset annotation? Can they be safely removed, or do they affect predictive performance downstream?\n\nOther questions:\n* In similar spirit to above, overlaying the heatmaps in Fig. 8-10 would provide additional informativeness.\n* Was the consistency of dataset annotation for the 5x5 grid analyzed? I.e. when processing each image separately, would the annotations change? Because as far as I know, MLLMs are not the best at localizing inputs, therefore I am unsure whether a 5x5 grid might pose difficulties.\n* Is there a specific reason why elasticnet with $\\alpha=0.99$ was chosen instead of just using the simpler LASSO (i.e. $\\alpha=1$).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This work proposes to combine the two thriving fields of concept bottleneck models and mechanistic interpretability. As such, it is relevant in the current state of interpretability. It makes use of tested mechanisms from both fields, thereby fostering trust in the proposed method. The method is introduced clearly and the figure aids in understanding. M-CBM also shows NCC improvements compared to relevant baselines. Also, the extensive visualization of discovered concepts and examples in the Appendix is appreciated. Reproducible code is provided too.", "weaknesses": "1. While MechInterp is an exciting upcoming field, I am not convinced by the reliability of the concept annotation framework such that there is not a substantial amount of leakage in M-CBM. (i) To name the SAE activations a heatmap is used to localize the concepts. Heatmaps are famously unreliable [1] such that they might not capture what information the blackbox model is using. Then, the highlighted images are passed to the MLLM (Fig. 1), but it is highly unclear to me whether the MLLM truthfully adheres to the highlighted region, and whether the concept interpretation/naming is unique. (ii) It is a good leakage-reducing choice that concepts are subsequently labelled using a MLLM, rather than directly using the SAE activations. Still, there might be a considerable influence from the leaky concept naming critiqued in (i) because \"Each call also includes a grid of the top-25 most activating images for the corresponding SAE neuron\", which could lead the MLLM to mimic the concept annotation strategy from the leaky SAE, rather than focusing on actual concept presence. Additionally, the MLLM might also use class-leakage where it recognizes an object and determines a concept that relates to the object to be present even if it is not visible in the image. \n\n   To give a concrete example that might ease comprehension, lets say we are interested in the class \"Radio\". A SAE on a blackbox classifier might find a neuron that always activates when a radio is present. Then a good heatmap highlights most of the radio in each highly activated image. The MLLM that outputs concept names might consequently call the concept \"analog control knobs and frequency dials\", as it appears in many of the images and is often even highlighted. Note though that the concept name has nothing to do with the actual concept that the model uses, which is \"presence of radio\". Then, an MLLM is queried to concept annotate a selection of images. Two things that could lead the MLLM to annotate concept presence even if the concept is not there are (a) the 25 highly activated SAE images do not all contain the named concept, thereby the MLLM learns to still assign concept presence for radios even if it is not visible on the image. (b) The MLLM even without context might understand the presence of the object radio and that this object has the aforementioned concept, even if it is not visible, thereby labeling it positive. In the end, the concept labels essentially behave like a class label, but hidden behind an interpretable concept name, which does not accurately capture how it behaves and/or when it activates.\n\n   I am not saying this is necessarily always the case, however, with the current evaluation setup I do not think the amount of this leakage is quantified. Therefore, it is very hard to say whether the extracted concepts (and therefore the method) are meaningful and interpretable.\n\n2. I am not convinced that NEC / NCC are sufficient metrics for capturing information leakage (as stated e.g. in the Abstract). Similar to the previous example, lets say I repurpose a frozen black-box's classifier to be \"concept predictor\" and rename the classes to some good concept names. Then, initialize the Identity Mapping from my new concept predictor to the classes. As I understand it, this method would have the black-box's accuracy at NCC=1 while being completely uninterpretable and leaking (similar to VLG-CBM). Thus, I do not think the current evaluation setup is able to differentiate meaningful, interpretable posthoc CBMs versus uninterpretable class-leakage.\n\n   To resolve this, off the top of my head, I could think of either performing human user studies or measuring alignment of discovered concepts with known ground truth concepts on a controlled dataset where actual concepts are known.\n\nOther weaknesses:\n* In my opinion, the proposed combination of CBM and SAE does not provide a lot of novel ideas but rather combines two existing approaches, which is meaningful, but not original.\n* Some concepts in Figure 10, e.g. Tiger stripes or Yellow banana peel with brown spots essentially describe the class, thereby not providing any additional interpretability beyond a blackbox classifier.\n\n[1] Adebayo, Julius, et al. \"Sanity checks for saliency maps.\" Advances in neural information processing systems 31 (2018).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)