FaCT: Faithful Concept Traces for Explaining Neural Network Decisions
Authors: Amin Parchami-Araghi
Sukrut Rao
Jonas Fischer
Bernt Schiele
Venue: NeurIPS 2025 poster ()
Keywords: Concept Explanations
Inherent interpretability
Faithfulness
Primary area: deep_learning
TL;DR: A new model with inherent mechanistic and concept-based explanations and a new concept consistency evaluation framework

Abstract: Deep networks have shown remarkable performance across a wide range of tasks, yet getting a global concept-level understanding of how they function remains a key challenge. Many post-hoc concept-based approaches have been introduced to understand their workings, yet they are not always faithful to the model. Further, they make restrictive assumptions on the concepts a model learns, such as class-specificity, small spatial extent, or alignment to human expectations. In this work, we put emphasis on the faithfulness of such concept-based explanations and propose a new model with model-inherent mechanistic concept-explanations. Our concepts are shared across classes and, from any layer, their contribution to the logit and their input-visualization can be faithfully traced. We also leverage foundation models to propose a new concept-consistency metric, C$^2$-Score, that can be used to evaluate concept-based methods. We show that, compared to prior work, our concepts are quantitatively more consistent and users find our concepts to be more interpretable, all while retaining competitive ImageNet performance.

FaCT: Faithful Concept Traces for Explaining Neural Network Decisions
Amin Parchami-Araghi
Sukrut Rao
Jonas Fischer!
Bernt Schiele
{amin. parchami, sukrut. rao, jonas. fischer, accheeee)@mpii iompi-inf.... mpg de (amin.parchami
Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbrücken, Germany
Abstract
Deep networks have shown remarkable performance across wide range of tasks, yet getting global concept-level understanding of how they function remains a key challenge. Many post-hoc concept-based approaches have been introduced understand their workings, yet they are not always faithful to the model. Further, they make restrictive assumptions on the concepts model learns, such class- specificity, small spatial extent, or alignment to human expectations. In this work, we put emphasis on the faithfulness of such concept- -ased explanations and propose Our concepts
new model with model-inheren. mechanistic concept-explanation are shared across classes and, from any layer, their contribution to the logit and their input- visulliialiiinionn can be faithfully traced. We also leverage foundation models propose new concept-consistency metric, C²-score, that can used to evaluate concept-based methods. Compared to prior work, we show that our concepts are quantitatively more consistent and that users find them to be more interpretable, while retaining competitive ImageNet performance,
1 Introduction
Deep learning has proven effective across a wide range of tasks, yet understanding the inner workings of such models remains challenge. which is critical for their in sensitive applications such as healthcare. Attribution methods [5] [56] have been typically used to understand the decisions of deep models, but they only show where in the input features of importance located, and not which high-level concepts a model uses.
To address this, concept-based based explanation methods have become popular tool to decompose model
decisions [ (35] and arbitrary activations [88] 22 into high-level interrrattereee concepts.
These include part-prototype networks 9 [5 and concept bottleneck models [35] 42 49 that aim to
create inherently interpretable models, yet often provide unfaithful explanations [27
39 61]. Other
approaches like CRAFT [18] decompose model activations post hoc, but the extracted concepts are not directly used for prediction, leading to reliance on approximate methods to estimate the importance of concepts to the output [17] and to visualize which region of the input the concept is activated
for [52 61 62]. which may also not be faithful to the original decision-main
2 Furthermore,
concepts are often subject to restrictive assumptions, e.g. being class-specific 18 22
36 being
limited to small patches or object parts [8 22 57), or belonging to a pre-defined set 35
hindering such methods from faithfully explaining the true concepts used by the model.
In this work we propose FaCT, an inherently interpretable model that provides concept decompositions that are bithful-by-deiigh (Fig. D.. Our model uses B-cos transforms [8] across its layers to facilitate
obtaining faithful attributions for any activation, and sparse autoencoders (SAEs) [6] intermediate
Code available at gigihab...oommmppcchaa/iaaaa Denotes equal contribution advisors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

9.4%
output contribution
4.3%
imsggh
School Bus
Input activation
1.19
Logits
Cross-layer contribution
Our proposed model FaCT offers faithful decoppp-dooomooiitionn across layers with shared basis across classes, c.g., the late-layer 'wheel' concept or early-layer 'yellow' concept are shared across classes and used by the model. Further, every concept is faithfflll visualized input-level (Concept Activation Σ Pixel Contributions) and every logit is faithfully explained at concept-level (Logit= = Σ Concept Contributions), e.g. yellow-color concept contributes to 4.3% of School Bus logit. Also contributions between different concept layers can be faithfully computed.
Figure I: It All Adds Up:
layers decompose the activations into interpretable concepts. The use of B-cos architecture together with concepts being part of the model's forward pass ensures that (1) model decisions can be faithfully attributed to concepts, and (2) concept activations can be faithfully visualized input. This is characterized by the concept attributions adding up to the logit and pixel attributions adding up to the concept activations. FaCT can also decompose across layers to build a concept hierarchy, see e.g. Fig. [ where the school bus logit can be fully decomposed to high-level late-layer concepts well as simpler early-layer concepts, with each being faithfully visualized. Unlike CRAFT [8] or VCC (36), our concepts are shared across classes (see late-layer 'wheel' concept in Fig. D which provides
shared basis that aids misclassification analysis (Fig. 8 and do not include any assumptions on size or spatial extent, leading to a diverse concept decomposition (Fig. r-right).
Since the concepts used by the model may not align with any predefined numan-annotin object parts (see Fig. B] where annotations fail to capture our concepts), we also find that existing metrics for evaluating concept consistency [30] that make such assumptions are suboptimal. To address this, inspired by its recent success for semantic correspondence 3 65 [6], we utilize DINOv2 [44] features to evaluate concept consistency without human annotations. Our proposed C2 -core takes into account the input features that activate the concept and evaluates consistency independent of pre-defined annotations, while correlating well with human notions of consistency.
Our contributions are thus as follows:
We propose model with inherent concept-basis that is used as part of the forward pro- cess. The concepts are shared across classes and exist across depth and generalize across architectures (CNNs and ViTs).
We demonstrate how to f'iitffully measure contribution of every concept to the output, and quantitatively show it outperforms existing approximate concept-importance measures.
We demonstrate how to faithfully visualize every concept at input-level and through user-study with control groups show how such visualizations impact the interpretability.
We propose novel concept-consistency evaluation metric for concepts across images for both shared (ours) and class-specific (prior-work) concept sets.
while providing faithful concept-based explana- tions with diverse (shared) concept basis. We quantitatively demonstrate that our concepts are more consistent, and through a user-study, that they are more interpretable than basclines.
Our models remain competitive on ImageNet 12

---

2 Related Work
Concept Extraction Methods
18 36 43 help understand a model's s activations in post-hoc
manner by decomposing them into set of high-level human-interpretable concepts. This decomposi- tion is typically unsupervised, using techniques such as non-negative matrix factorization [10] [8). hierarchical clustering 43, pattern mining [19], or by directly using channels of the layer being examined [5]. However, grounding such concepts in the input requires using post-hoc attributions which may not be faithful 2 34] to the model. The use of channels as the concept basis [ also assumes that the model learns separate human interpretable concepts per channel, which need not be true [66 26 Further, some approaches [18] 36
use a class specifcc concept basis which provides a limited view on how the model shares concepts across classes. In our work, we propose a model with inherent concept-band sancept-based explanations using B-cos layers [8] that ensures that concepts can be faithfully grounded by design. We use sparse autoencoders (SAEs) to extract concepts at each layer, which provides shared class-agnostic basis of decomposed concepts.
B-cos Networks 7 g are a class of inherently interpretable models that use architectural modifica- tions to emphasize weight- input alignment, leading to faithful and interpretable attributions of the model's decisions (cf. Eq. (3)). Since their introduction, B-cos models have proven to extend to large-scale training schemes
shown to be a strong proxy for guiding these models [46] 48). However, B-cos models only provide a local attribution that highlights pixels of importance and do not explain what concepts the model uses (Fig. B In our work, design a model with B-cos layers and sparse autoencoder bottlenecks to extract concepts, leveraging the "ynamic-llnearing of B-cos to obtain faithful and interpretable attributions for grounding concepts, as well as obtaining faithful attributions on how the concepts contribute to the output.
20 and their faithful attributions
have recently become popular as a tool to decompose model activations into a sparse set of human interpretable concepts, and have been used for understanding large language models 66
Sparse Autoencoders (SAEs) (6 32
49
64
49 64] with downstream uses such as model stecring [1] and constructing concept bottlenecks [49 In our work, we propose to use bias-free SAFs to extract concepts across different layers in an image classification model with B-cos layers, and build an inherently interpretable model with a concept hierarchy.
32 and vision models (37
Inherently Interpretable Concept-based Models such as Part Prototype Networks 9 and Concept Bottleneck Models (CBMs) [35
13 40
57
first predict set of concepts using the feature extractor, based on image patch similarity or annotated labels, and then use these concepts for classification. However, the feature extractor is still uninterpretable, and attempts to ground concepts from such models have suggested that they may not be faithful [27 31 39 61]. Recent work [55] 60] explored using B-cos models for better prototypes. however, in contrast to them, we directly use B-cos attributions to obtain fine-grained explanations of concepts.
45
49 63
Evaluating Explanation Methods is essential ensure that explanations can trusted and are interpretable. For attribution methods, many works proposed sanity checks for measuring their faithfulness
metrics for evaluating their localization ability [53] [88, as well as synthetic datasets 25 50 for a controlled evaluation. For concept-based explanations, prior work (30) proposes evaluating concepts in terms of binarized attributions having high IoU with part annotations. This however is limited to class-specific concepts. requires annotations for every part, and assumes every concept should correspond to an annotated part. In this work, we propose a novel consistency metric C--score, which leverages foundation models to evaluate consistency of concepts in a class-agnostic manner. Close to our metric are works that measure 'monosemanticity" of individual neurons [16) as the similarity of set of images or crops. In contrast, our work considers full-image attributions and, through appropriate baselines, can evaluate various concept extraction methods.
3 Faithfully Explaining with Concepts
In the following, we describe how our proposed model FaCT provides the user with inherent concept- based explanations (Section (.11. Since model leverages B-cos transforms (7] and SAEs [32]. we first provide a brief introduction to each respectively. In Section 4 we propose our novel concept consistency evaluation metric C²-score.
Notation. Given matrix M E DD and vector e E D2: ReI.U(.) clamps negative values in tensor to zero; c( (M;2) v) E IR outputs a vector of cosine similarity between each row of matrix
3

---

M with the vector v; (M @ v) E D applies element wise row scaling M by 2i |o] denotes element-wise absolute value; and the operator applies row-wise l normalization.
B-cos Networks [7]. For simplicity, let consider an imput vector z R and learnable weight matrix W I2 x and bias vector b E R= A conventional ReLU block can be defined as
S (x) ReLU(W x+ b).
B-cos transform removes bias terms, uses row-normalized weights W, and applies cosine non-linearity instead of ReLU, This pushes the rows W to be aligned with input a for higher activations. The transformation can thus summarized as dynamic-linear transform W(x)
B-1
B-cos (a; B) = (W z) o d
(W
(W
((z) T.
W(x)
A series of B-cos transformations thus be summarized dynamic-linenn linear transform of the input
f1 (x) = (J BB-os
f2
BB-cos o f/l-con (x) = II W. (.(aa-)) x W Wi-n(r)a
(3)
where a; is the output of layer Therefore, for every input æ. B-cos networks can produce model- inherent B-cos explanations WI-nn (3)) which faithfully reproduce the output logits. For every category e, the respective row e of W (1) would serve the B-cos explanation, i.e. [W1+t dynamic-linen
As shown in Eq. (3), series of B-cos transforms can be faithfully summarized combination of imput features. In our work, we use this property B-cos transform to, irrespective of the layer, faithfully compute contribution of each concept to every output logit, as well as, to faithfully visualize every concept at input-level.
Sparse Autoencoders [32]. In order discover interpretable concepts from neural network's representations in an unsupervised manner, Sparse Autoencoders (SAEs) 32 have been proposed to obtain sparse dictionary representation of features. For given collection of N training features X E IR Nxd, an SAE is trained to learn dictionary V E IR igx'dd and per-feature sparse codes R R as linear combination features such that:
E x x s.t. =u
Encoder(x) ReLU(Wz + b).
(4)
Sparsity be induced by e.g. l regularization the sparse codes, by explicitly enforeing k-sparse codes, i.e. TopK-SAE 21.38.. In our work, we use TopK-SAES, with the encoder having no biases, the sparse codes can be faithfully represented as strictly linear combination of features.
3.1 FaCT: Faithful Concept Traces
Having B-cos transforms and SAEs introduced, in this section discuss proposed model FaCT, and how it explicitly uses a concept-based representa- tion for its decision, while faithfully providing output contributions of every concept, together with faithful input-level attributions from any layer.
Concept Activations
B-cos Layers
Encoder
Decoder
B-cos Layers
Output Logit
Let fi-m be an n-layer deep network of B-cos trans- forms mapping an input image E Ha x3 to output logits L E RM for M categories. We denote intermediate representation F as
Summarize with Eg. (1.)
Summarize
with Eq.
Figure 2: Overview of FaCT.
F' HxWXC = fi-((() s.t. =1=() (fim (I).
(5)
We define our SAE module similar to Eq. (4) but without encoder biases. Both encoding and decoding stages are performed independently all embedding patches, i.e. using 1 x convolution operation. We thus transform F concept-activation tensor U E H x for K concepts
FRF s.t. F = conv (V.U). U Encoder F) = ReLU(conv
(6)

---

Our model then uses the reconstructed features F for the final output logits L FacT
F = Ji-a(F) - Ji- conv (V,
(7)
Eq. thus demonstrates that the output logits LFFCTT are only based concept activations U. Since Ji i8 composed of B-cos layers, we can summarize it as a dynamic-lineal transform, thereby explaining every logit FaCT for category as dynamic-linear combination of concepts W(U):
H.W.C
H.W.C Σ s.h.c
Le
FaCT_
= [fimm(F)
Σ
[Wimn(F)lFF
[Wi- (onD),, (V.U) U)
(8)
ijich
H.W.K
K
H.W
Σ
W(UJU = Σ Σ
U
Σ
Contribution
SJ
where Contribution denotes the contribution of concept category c. Therefore, the logit is summation concept contributions, regardless of the layer, which means FaCT provides model- inherent concept contributions. This is in contrast to prior work [17] 18 36 which do not directly use concepts for computing the logits and rely on post-hoc measures for concept importance.
Further, since in Eq. (6) we use bias-free SAE, the entire concept activation tensor U can be faithfully explained as dynamic-linear transformation intermediate features (see Appendix [ on why ReLU is dynamic-lineary As the previous layers fi-1 leading F have B-cos transforms, we can faithfully attribute our concept activations to imput pixels:
H.W
H.W
Concept Activation
Σ
U
Σ
[eeLU (W, (W,F))] ig
(10)
15
H.W.C.
H.W.C
Σ
[W(F)))
Σ (W(F)
(11)
iie
whe
H.W.C.
Ho Wo.3
= Σ (W(P((
Σ t.eee
[W(M ise
(12)
We can therefore represent each concept activation as a dynamicc-inenn combination of input pixels at input-level, which can be visualized similar to the original B-cos [7] explanations. This is in contrast to using approximate visualizations, c.g. showing image crops or up-sampled heatmaps 9 18 36 (7).
We have now demonstrated how FaCT faithfully explains every logit as summation of concept contributions (Eq. (9) and how faithfully attributes concept activations to the input (Eq. (T2). In Appendix H we further derive how one faithfully visualize Contribution i.e. the output- contribution concept, as well as how one can measure cross-layer concept contribution, e.g. see Fig. I The next section focuses on evaluation of concepts, where propose our novel C that can robustly evaluate consistency concepts across images class-agnostic manner.
4 Concept Consistency Evaluation
systematic evaluation of concept consistency, i.e. whether a concept activates for similar patterns across images, has been great interest the community. Prior work (30] proposes to use human- annotated part masks [23). This however is limited to a small set of classes with class-specific parts and assumes that every concept should correspond an annotated part. Further, the set of concepts may be shared across multiple (non-object) classes; see how in Fig. the annotations do not match the concepts. To tackle this, we propose C² -coree which uses foundation models' features to measure the concept consistency across images in class-agnostic manner. We consider model H(T) H x3 R xW x transforming image to a feature-tensor of same resolution.
We use DINOv2 44 for its success tasks such as semantic correspondence 3 65 66) We also
apply LoftUp 29). an off-the--hhelf feature upsampler, to obtain high-resolution feature maps. The LoftUP features are additionally centered using the mean feature computed over the dataset (sec
5

---

Human Annotations
DINOv2 Features
Our Concepts
Appeneskor
Calile
Candggnn
Boogle
Figure 3: Beyond Annotations: We observe that annotations 23 fail capture concepts, either by not having them annotated ('ship mast' top-right) or not matching the granularity ('dog blaze' bottom-right). Our proposed C2-score tackles this by considering concept attributions together with DINOv2 features, leading to class-agnostic evaluation framework. See also Appendix C.1 C.T
Appendix C.3 for further details). For every concept & from the concept set and image 7 from dataset D, we define the concept activation value I and input attribution Att as the following:
H.W
Sk. R2o
HxW Attr E =Roo
s.t.
Sth =1
Σ
Attr
k.
(13)
IED
For each concept k activating on image I, we can now define the embedding & what the activation of concept & corresponds to the output representation of H.
representing
Ex(I) E ERRE = Σ (ur)
(14)
i,f
We then measure the consistency of such embeddings each other over the set of images D.
Consistency R ==
E
ski11 gk. cos(E* (1),
(15)
(I,J)ED
Notice that Eq. (15) is defined over the set DD This would be biased towards methods with class- specific concepts, where each concept is only evaluated on the set images of the same category. To ensure comparability across methods, define C²-score the difference between the consistency of the concepts and that of a 'random' concept with a random attribution map on every image.
C -c-oe Σ K
Consistency Consiitencyyrad
(16)
K
Therefore, C2-score lends itself simple yet robust evaluation framework that takes the concept attribution into account and can evaluate both class-specific and shared concept sets. Having such evaluation framework, in Section 5.2 we compare we further validate the proposed C=-score and how it is close to human definition of consistency, by
concepts with prior work. In Fig. 4 right,
showing randomly sampled concepts with high and low C²-score. In Appendix C we elaborate -c-score, showing how it outperforms annotations and how it correlates with user interpretability.
Inspecting the Concepts
In this section we inspect the model-inheren concept- concept-based representation of FaCT. We begin by evaluating FaCT across architectures and layers in Section 5.1 measuring ImageNet performance and concept consistency (Ca-score), together with concept diversity in terms spatial extent. Afterwards, Section 5.2 quantitatively compares consistency of our concepts to prior work. Lastly, Section 5.3 compares the interpretability of our concepts and visualizations across layers compared to baselines in user study. With our concepts evaluated in this section, in Section O we focus the decision decision-making

---

Mean
Random Baseline
CRAFT (per-class)
CRP
Percentage of Concepts
Copen mare
Come
Arand 0.02
Bradd 0.26
B-cos Channels
FaCT (ours)
45
30
Arand 0.09
Deand 0.37
15
0.5
100
0.5
1.
Consistency Score
Figure 4: Evaluating Concept Consistency: We evaluate the CP-score (cf. Section 4 for both FaCT's concepts and prior work's. (left) plot the percentage of concepts for different consistency ranges, finding our concepts to be more consistent than those of prior work. (right) We randomly sampled concepts from different ranges of C2-score, Notice that the C² CC-score correctly assigns high consistency to the 'helmet' or 'muzzle' concepts, despite them being shared across classes. See also Appendix
demonstrate the effectiveness of the C2-score.
of the model, by quantitatively evaluating concept contributions (Eq. (9) to prior works' approximate measures, well as leveraging the shared concept basis of FaCT study misclassification.
Before we begin, let us briefly discuss our implementation details. We used ImageNet-pretrained
B-cos 8 networks with fixed parameters and used bias-free TopK-SAE 21 38 sweeping T Topk e (8, 16, 32), with K (8192, 16384) total concepts. We constructed the training dataset by sampling feature patches from ImageNet's training set. Throughout the paper, each concept is visualized by top-activating test images, with attribution (Eq. (12) and the image category. See also Appendix
5.1 FaCT is Competitive and Diverse
We begin by evaluating FaCT across layers (early, mid, late) and architectures (CNNs 28 and ViTs (14) on ImageNet [51]. In Fig. Heft, we observe that our models are able to maintain competitive accuracy compared original B-cos, despite being trained on having sparse concept- representation, even at intermediate layers. We observe that for a small drop in performance (< 3% across), we are able to obtain significantly higher C2-score for our concepts, e.g. 0.11 0..99 for DenseNet at Block 3/4. In fact, in Section 5.3 we will also show users found our concepts more interpretable than the bascline. We also see that FaCT readily generalizes to ViT architectures. Further, in Fig. right, we plot the diversity of our concepts in terms of spatial extent, i.c. number of highest- attribution pixels needed to cover 80% of the total attribution. We observe concepts variety of sizes across layers and architectures, e.g. see the small late-layer 'Bike Helmet' concept for DenseNet
the large early-layer 'Wooden Texture' for VIT. This is in contrast to prior work [18 36 40 that assume fixed sizes for every concept. Further analysis and visualizations can be found in Appendix D
5.2 FaCT is more Consistent
As discussed Section 3 we leverage SAES for extracting concepts intermediate layers, while faithfully visualizing every concept at input-level. In Section 4] we also discussed how our proposed Ca-score is stronger benchmark for evaluating consistency concepts. In Fig. 4 we evaluate consistency of our concepts using C²-score, well concepts from prior work. In Fig. left, we observe that our concepts are more consistent than prior work, with significant increase compared to channels (co-r 0.09 0.337). We also randomly sampled concepts from different consistency
7

---

B-cos
FaCT (aurs)
Concepts (9%)
DenseNet-121
ViT.-
ResNet-50
DenseNet-121
V
0.4
B04
'64
120'
16e
126"
64
120'
6e²
120
parca
Spatia Size (pixe sl
BROD
C1-Score
0.3
0.2
0.3
77
75 ImageNet Accuracy
72
Figure 5: FaCT for Diverse Concepts. (left): We observe significant gains in terms of concept- consistency for FaCT compared to B-cos channels. This holds across architectures (columns) and layers (points) with competitive performance on ImageNet (largest drop < 3%), see Appendix B for further analysis and comparison to standard models. (right): We observe high diversity for our concepts in terms of spatial extent (top) and show samples the bottom. See also Appendix [
ranges and visualized the concepts in Fig. right. We observe consistent concepts such as 'Helmet' and 'Muzzle' obtain higher scores than inconsistent ones. Notice that here the consistent concepts are shared across classes, yet C3-score reliably assigns them high consistency scores.
5.3 FaCT is more Interpretable
As explanations should aid humans to gain insights, we evaluate the interpretability of concepts based on whether users can retrieve a meaning from our visualizations. In particular, we randomly sampled 100 early and 100 late concepts from FaCT, together with 30 late and 30 early randomly chosen B-cos channels as a baseline. For each concept, we visualized top- 10 activating images along with their input attribution for each image. We randomly assigned participants to one of 10 groups to rate the shown figures in terms of interpretability on 5-point scale (higher is better). Each group was shown samples from all cases. We further conducted a counterbalanced AB/BA study to evaluate whether our faithful input attribution increases interpretability.. For each of our concepts, we generated one version with and one without the input attribution as control. Each group experienced concepts from the experimental and from control condition. The study was conducted fully anonymized with 38 volunteer participants. For further details and sample questions, refer to Appendix [E]
The study results in Fig. 6 show that the participants found our concepts far more interpretable than B-cos channels, for both early- and late-layer concepts. We also observe that providing our input attributions can change the given scores, observing that on average they lead to an increase in interpretability, particularly for earlier layer concepts with about 0.5/5 average increase. In Fig. G right, we visualize four early-layer concepts with the largest increase in score between their two groups. These are low-level visual patterns such as background (+3.0/5) and up-facing curves (+2.5/5) which users were able to interpret only when aided with our faithful input-level visualizations.
Our results thus find FaCT's concepts to be more consistent (Fig. and interpretable (Fig. (6) than the baselines. This was achieved by solely relying on faithful visualization of concepts without putting any assumptions such as having a fixed spatial-size being only object parts or coming from a pre-defined concept set, as often seen in prior work [3 18, 35| 40 42 [7]. That said, aligning the concepts with textual labels may be of interest some users. For this, one can accompany FaCT with existing neuron-labeling methods. In particular, we applied CLIP-Dissed [4] to our SAE latents, using common English words to match each concept to the text label with the most similar activation statistics in CLIP [47]. This results in concepts (A-F) in Fig. to be named as follows (best of top-three per concept): A+ B- "jenseys'; C- 'rurry D 'lexible": E- "==>vooleyball'; and F-+ "basketballl We thus observe that FaCT's concepts also lend themselves well to be named using existing neuron labeling methods. In the next section, we turn our attention towards contribution of concepts to the predictions.
or

---

à Score
User Interpretability Score
(w/ w/o Explanation)
5.0
3.5
3.0
3.2
3.0
2.5
1.0
Ours (Late]
Channels (Late)
Durs
Channels (Barly)
Ours
Ours
LSS PuTW
[Earry]
(Late)
(Early)
Figure 6: User Study on Concept Interpretability: (left) We evaluate the interpretability and consistency of concepts and visualizations through user-study. We observe significant gains in terms of interpretability compared to B-cos channels. (center) We show the results of our control study for concepts with average score Σ 2.5 when viewed only with images, and observe that the explanations increase their interpretability score, in particular for earlier layers. (right) we show four early-layer concepts, which had highest score increase when users saw the explanations, e.g. for the 'curve' concept (top-right), the explanations increased the score by A 2.5(/5). See also Appendix
Inspecting the Decision-Making
As in Section 3.1
proposed model uses shared concept basis and is able to faithfully decompose the output logits into concept contributions Eq. (9) We next empirically evaluate our concept- contributions compared to approximate importance measures from prior work [22 18 36
and then study a misclassification case to understand the model's confusion on a concept level.
further empirically validate the faithfulness our concept- contributions from Eq. (9) using the concept-deletion metric [17]. In short, this metric removes concepts from most to least important under different importance measures, and compares the drop in logit. As concept-basic basis is shared across classes and the concepts in fact used produce the output logits, we additionally evaluate the overall accuracy drop when removing the concepts. This gives broader understanding of how all of the ouppus change the concepts removed. In Fig. 7 we observe that our concept contributions, as defined in Eq. (9) provide significantly sharper drops, both for top-1 logit as well as overall accuracy, indicating that the reported contributions are better signal for measuring importance of concepts, We see significantly higher gains compared to other attribution methods such as Saliency or Sobol indices used in prior work 33 [ Further details on this evaluation can be found in Appendix F
Validating concept contributions.
Interestingly, for Block (2/4) in Fig. [7] concepts, which indeed highlights the faithfulness of our concept contributions. In Appendix we further study the concepts which cause such sharp drop upon deletion and find them to be small set concepts that always contribute to every decision. In Appendix [F] we further run a similar experiment in Fig. 7 but preventing such concepts from being removed, and find that concept contributions still significantly outperform other baselines.
sharp drop of accuracy with deletion of only few
Ours
Saliency
Random
Sobol (n=4)
DenseNet Block (2/4)
DenseNet Block (4/4)
DenseNet Block (2/4)
DenseNet Block (4/4)
Accurary
50
0
50
100
50
100
50
100
50
100
Concepts dropped
Concepts dropped
Concepts dropped
Concepts dropped
Figure Concept Deletion: We iteratively delete concepts in order of importance based on different attribution methods (steeper drop is better). We observe significant improvements using our Eq. 99) compared existing concept- amnceptatttributonn especially at earlier layers. See also Appendix F

---

Mutually Contributing Concapts
3.0 %
1.5 %
Volleyball Logit: 3.7
Exclusively Contributing Concepts
14.7%
4.9 %
0.0 %
Basketball Logit: 3.0
11.6%
14.2%
0.7
6.3
6.1
Having shared concept basis, inspect a misclassifi- cation and attribute each of the logits (i.e. Basketball and Volleyball) to inspect what are the concepts that contribute to the confusion. (center): the two logits with their original B-cos explanation [8] are reported. (left): we see four concepts (A-D) that mutually contribute to both logits. These include common features between the two, such as 'ball' (A) and 'jerseys' (B). (right): we see concepts (E) and (F) that exclusively contribute to volleyball and basketball logits. See also Appendix G
Figure 8: Understanding What is Shared:
Leveraging shared concept basis. Through FaCT, we can understand wrong decision-making inside the neural network, such as a Basketball image that is misclassified as Volleyball (Fig. 8). Just by looking the attribution (in the middle), it is not clear why the model is confused. Through our concept-level explanation, we get which concepts are active and what they perceive (Fig. A-F) as well as how much the concept contributes to each logit. This allows understand the model's confusion on concept-level. For the mutually contributing concepts, we observe that these are confounding factors, such as 'ball' (A), "jersey' (B), 'person with shirt' (C), or 'limbs' (D), that appear for both classes. In fact, for the 'jersey' concept (B), we already see samples of both volleyball and basketball classes within the top-activating images. Further details together with comparison to class-specific methods can be found in Appendix G In summary, FaCT provides us with a deeper understanding where and why model reasoning goes wrong.
Discussion
In this work presented FaCT, model with faithful and inherent concept-based explanations, which uses exactly these concepts for its decision-makinn Our model fully decomposes every logit into concept contributions and explains each concept faithfully at input-level, providing interpretable and consistent concepts across layers and architectures while remaining competitive on ImageNet. This was achieved by having the concepts shared across classes and not putting any restrictive assumptions on the concepts. In Appendix D we further explore the diversity of our concepts, with additional samples across layers and architectures. We also demonstrate the generalization of FaCT to other datasets in Appendix [
Our work also introduced consept-consistency metric (C*-score) evaluate both shared and class- specific concepts at scale, avoiding limited comparison against annotations (Fig. while agreeing with human notion of consistency (Fig. 4). In Appendix C superior to using human annotations for evaluating concepts.
we we further demonstrate how C² -corc is
In summary, we proposed an architecture that inherently encodes interpretable concepts across its layers, provides foriiesffithhy faithful attribution maps for each concept exactly reflecting the underlying model perception, and provides faithful contribution scores for each concept revealing its exact impact on downstream classification.
10

---

Acknowledgments
We would like to thank our colleagues Thomas Wimmer, Olaf Dünkel, Moritz Böhle, Nhi Pham, and Siddhartha Gairola for the helpful discussions. We are especially thankful to Thomas for helping with Fig.
References
[1] R. Achtibat, M. Dreyer, I. Eisenbraun, S. Bosse, T. Wiegand, W. Samek, and S. Lapuschkin. From Attribution Maps to Hluman-understnndde Explanations Through Concept Relevance Propagation. Nature Machine Intelligence, 2023.
[2] J. Adebayo, J. Gilmer, M. Muelly, I. Goodfellow, M. Hardt, and B. Kim. Sanity Checks for Saliency Maps. In NeurlPS, 2018.
[3] S. Amir, Y. Gandelsman, S. Bagon, and T. Dekel. Deep ViT Features as Dense Visual Descriptors. ECCVW What is Motion For?, 2022.
[4] S. Arya, S. Rao, M. Böhle, and B. Schiele. B-cosification: Transforming Deep Neural Networks be Inherently Interpretable. In NeurIPS, 2024.
[5] S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Müller, and W. Samek. On Pixel-wise Explanations for Non-Linear Classifier Decisions by Layer-wise Relevance Propagation. PloS one, 2015.
[6] T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Caafield-Doddss A. Tamkin, K. Nguyen, B. McLean, J. E. Burke, T. Hume, S. Carter, T. Henigham, and C. Olah. Towards Monosemanticityy Decomposing Language Models with Dictionary Learning. Transformer Circuits Thread, 2023.
[7] M. Böhle, M. Fritz, and B. Schiele. B-cos Networks: Alignment All We Need for Inter- pretability, In CVPR, 2022.
[8] M. Böhle, N. Singh, M. Fritz, and B. Schiele. B-cos Alignment for Inherently Interpretable CNNs and Vision Transformers. JEEE TPAMI, 2024.
[9] C. Chen, O. Li, D. Tao, A. Barnett, C. Rudin, and J. K. Su. This Looks Like That: Deep Learning for Interpretable Image Recognition. In NeurIPS, 2019.
[10] E. Collins, R. Achanta, and S. Susstrunk. Deep Feature Factorization For Concept Discovery. In ECCV, 2018
[11] B. Cywiński and K. Deja. SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoders. In ICML, 2025.
[12] J. Deng, W. Dong, R. Socher, L..J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, 2009.
[13] J. Donnelly, A. J. Barnett, and C. Chen. Deformable ProtoPNet: An Interpretable Image Classifier using Deformable Prototypes. In CVPR, 2022.
[14] A. Dosovitskiy, I.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR, 2021.
[15] M. Dreyer, R. Achtibat, W. Samek, and S. Lapuschkin. Understanding the (Extra-) Ordinary: Validating Deep Model Decisions with Prototypical Concept-based Explanations. In CVPRW, 2024.
[16] M. Dreyer, F. Purelku, Vielhaben, W. Samek, and Lapuschkin. Pure: Turning Polysemantic Neurons into Pure Features by Identifying Relevant Circuits. In CVPRW, 2024.

---

[17] T. Fel, V. Boutin, L. Béthune. R. Cadène, M. Moayeri, L. Andéol, M. Chalvidal, and T. Serre. Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation. In NeurlPS, 2023.
[18] T. Fel, A. Picard, L. Bethune, T. Boissin, D. Vigouroux, J. Colin, R. Cadène, and T. Serre. CRAFT: Concept Recursive Activation FacToriiztiin for Explainabilityy In CVPR, 2023.
[19] J. Fischer, A. Olah, and J. Vreeken. What's in the Box? Exploring the Inner Life of Neural Networks with Robust Rules. In JCML, 2021.
[20] S. Gairola, M. Böhle, F. Locatello, and B. Schiele. How to Probe: Simple Yet Effective Techniques for Improving Post-hoc Explanations. In ICLR, 2025.
[21] L. Gao, T. D. la Tour, H. Tillman, G. Goh, R. Troll, A. Radford, I. Sutskever, J. Leike, and
J. Wu. Scaling and Evaluating Sparse Autoencoders. In ICLR, 2025.
[22] A. Ghorbani, J. Wexler, J. Y. Zou, and B. Kim. Towards Automatic Concept-based Explanations.
In NeurlPS, 2019.
[23] J. He, S. Yang. S. Yang, A. Kortylewski, X. Yuan, J.-N. Chen, S. Liu, C. Yang. Q. Yu, and Dataset of Parts. In ECCV, 2022.
A. Yuille. PartImageNet: A Large,
[24] K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016.
[25] R. Hesse, S. Schaub- -Myerr and S. Roth. Funnybirds: A Synthetic Vision Dataset for a Analysis of Explainable AI Methods. In ICCV, 2023.
[26] R. Hesse, Fischer, S. Schaub-Meyen and S. Roth. Disentangling Polysemantic Channels in Convolutional Neural Networks. In CVPRW, 2025.
[27] A. Hoffmann, C. Fanconi, R. Rade, and J. Kohler. This Looks Like That... Does it? Shortcomings of Latent Space Prototype Interpretability in Deep Networks, 2021.
[28] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely Connected Convolutional Networks. In CVPR, 2017.
[29] H. Huang, A. Chen, V. Havrylov, A. Geiger, and D. Zhang. LoftUp: Learning a Coordinate- Based Feature Upsampler for Vision Foundation Models. In ICCV, 2025.
[30] Q. Huang, M. Xue, Huang, H. Zhang, J. Song, Y. Jing, and M. Song. Evaluation and Improvement of Interpretability for Self-explainable Part-prototype Networks. In ICCV, 2023.
[31] Q. Huang, J. Song, J. Hu, H. Zhang. Y. Wang, and M. Song. On the Concept Trustworthiness in Concept Bottleneck Models. AAAI, 2024.
[32] R. Huben, H. Cunningham, L. R. Smith, A. Ewart, and L. Sharkey. Sparse Autoencoders Find Highly Interpretable Features in Language Models. In ICLR, 2024.
[33] B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, et al. Interpretability beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). In ICML, 2018.
[34] J. S. Kim, G. Plumb, and A. Talwalkar. Sanity Simulations for Saliency Methods. In ICML, 2022.
[35] P. W. Koh, T. Nguyen, Y. S. Tang, S. Mussmann, E. Pierson, B. Kim, and P. Liang. Concept Bottleneck Models. In ICML, 2020.
[36] M. Kowal, R. P. Wildes, and K. G. Derpanis. Visual Concept Connectome (VCC): Open World Concept Discovery and their Interlayer Connections in Deep Models. In CVPR, 2024.
[37] H. Lim, J. Choi, J. Choo, and S. Schneider. Sparse Autoencoders Reveal Selective Remapping of Visual Concepts during Adaptation. In ICLR, 2025.
12

---

[38] A. Makhzani and B. Frey. K-sparse autoencoders. In ICLR. 2014.
[39] A. Margeloiu, M. Ashman, U. Bhatt, Y. Chen, M. Jamnik, and A. Weller. Do Concept Bottleneck Models Learn as Intended? In ICLRW, 2021.
[40] M. Nauta, J. Schlötterer, M. van Keulen, and C. Seifert. PIP-Net: Patch-Based Intuitive Prototypes for Interpretable Image Classification. CVPR, 2023.
[41] T. Oikarinen and T.-W. Weng. CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks. In ICLR, 2023.
[42] T. Oikarinen, S. Das, L. M. Nguyen, and T.-W. Weng. Label-Free Concept Bottleneck Models. In ICLR, 2023.
[43] L. O'Mahony, V. Andrearczyk, H. Müller, and M. Graziani. Disentangling Neuron Representa- tions with Concept Vectors. In CVPRW, 2023.
[44] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo. M. Szafraniec, V. Khalidov, P. Fernandez, D. HAZIZA, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y. Huang, S.-W. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jegou, J. Mairal. P. Labatut, A. Joulin, and P. Bojanowski. DINOv2: Learning Robust Visual Features without Supervision. Transactions on Machine Learning Research, 2024.
[45] K. P. Panousis, D. Ienco, and D. Marcos. Hierarchical Concept Discovery Models: A Concept Pyramid Scheme. arXiv preprint arXiv:2310.02116, 2023.
[46] A. Parchami-Araghi, M. Böhle, S. Rao, and B. Schiele. Good Teachers Explain: Explanation- Enhanced Knowledge Distillation. In ECCV, 2024.
[47] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning Transferable Visual Models from Natural Language Supervision. In ICML, 2021.
[48] S. Rao, M. Böhle, A. Parchami-Araghi, and B. Schiele. Studying How to Efficiently and Effectively Guide Models with Explanations. In ICCV, 2023.
[49] S. Rao, S. Mahajan, M. Böhle, and B. Schiele. Discover-then-Namee Task-Agnostic Concept Bottlenecks via Automated Concept Discovery. In ECCV, 2024.
[50] A. Ross, M. C. Hughes, and Doshi Velez. Right for the Right Reasons: Training Differen- tiable Models by Constraining their Explanations. In IJCAI, 2017.
[51] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015.
[52] M. Sacha, B. Jura, D. Rymarczyk, L. Struski, J. Tabor, and B. Zielinski. Interpretability Benchmark for Evaluating Spatial Misalignment of Prototypical Parts Explanations. In AAAI, 2024.
[53] W. Samek, A. Binder, G. Montavon, S. Lapuschkin, and K.-R. Müller. Evaluating the Visualiza- tion of What a Deep Neural Network has Learned. IEEE Transactions on Neural Networks and Learning Systems, 2016.
[54] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Baser Basee Localization. In ICCV, 2017.
[55] S. Sivaprasad, D. Kangin, P. Angelov, and M. Fritz, COMIX: Compositional Explanations using Prototypes. arXiv preprint arXiv:2501.06653 2025.
[56] M. Sundararajan, A. Taly, and Q Yan. Axiomatic Attribution for Deep Networks. In ICML,
2017.
[57] A. Tan, F. Zhou, and H. Chen. Post-hoc Part-prototype Networks. In ICML, 2024.
13

---

[58] H. Wang. Z. Wang, M. Du, F. Yang, Z. Zhang, S. Ding. P. Mardziel, and X. Hu. Score-CAM: Score- Weighted Visual Explanations for Convolutional Neural Networks. CVPRW, 2020.
[59] Welinder, S. Branson, T. Mita, C. Wah, Schroff, Belongie, and P. Perona. Caltech-UCSD Birds 200. 2010.
[60] N. Wolf and C. Wachinger. WASUP: Interpretable Classification with Weight-Input Alignment and Class- Disssiiscrimiiative SUPports Vectors. arXiv preprint arXiv:2501.17325 2025.
[61] T. N. Wolf, F. Bongratz, A.-M. Rickmann, S. Pölsterl, and C. Wachinger. Keep the Faith: Faithful Explanations in Convolutional Neural Networks for Case-based Reasoning. In AAAI, 2024.
[62] R. Xu-Darme, G. Quénot, Z. Chihani, and M.-C. Rousset. Sanity Checks for Patch Visualisation in Prototype-Based Image Classification. CVPR, 2023.
[63] M. Yuksekgonul, M. Wang, and J. Zou. Post-hoc Concept Bottleneck Models. ICLR, 2023.
[64] V. Zaigrajew, H. Baniecki, and P. Biecek. Interpreting CLIP with Hierarchical Sparse Autoen- coders. In ICML, 2025.
[65] Zhang, C. Herrmann, Hur, L.. P. Cabrera, V. Jampani, D. Sun, and M.-H. Yang. A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence. In NeurIPS, 2023.
[66] Zhang, C. Herrmann, Hur, Chen, V. Jampani, Sun, and M.-H. Yang. Telling Left from Right: Identifying Geometry-Aware Semantic Correspondence. In CVPR, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main contribution is model with faithful concept-based explanation, i.c. faithful output contribution and input visualization of concepts. In Section 3.1 we theoretically justify the claimed faithfulness, with no approximation or simplification. Our claims over out-performing prior work are also quantitative.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the drop in performance in Section 5.1 and the presence of less-
consisten/interpretable concepts reported in Fig. and Fig. 6
We have also put further
discussion on the limitations and future directions in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: In Section 3.1 we begin by defining our notation and our assumptions about the underlying networks are also explicitly stated in Section 3.1
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes we discuss the exact models types and architectures that we use, as well the layers that we inspect. For our proposed evaluation metric, we also state which models are being used, We provide a brief training details right before Section 5.1 which is extended in Appendix A The code will also be published upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
16

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code for training the models can be found together with the supplement. We will also release it publicly upon acceptance. Our work does not introduce any new data.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://niis..c.
Dublic/guides//Cn
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https www.eooaawieeiinnoooooooooooaaae
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: A brief overview of this is given right before Section 5.1 with the extended version in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our results are reported as plots over the distribution, Fig. or,, the distribution itself, e.g. Fig. and Fig. 5
Guidelines:
17

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, these are part of the extended training details in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:///wwrppp.cccppppbe//ebeeaa
Answer: [Yes]
Justification: We have read the NeurIPS Code of Ethics and acknowledge that our work conforms with
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
18

---

Justification:
These are discussed in Appendix M
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: We propose classification model with inherent concept-based explanations. There is no data release generation involved.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: As discussed in Appendix we used available APIs such PyTorch and
official B-cos code-base [8] and evaluated on ImageNet [51] (and CUB (59 in Appendix K) datasets. for all of which have ensured to comply with their license. The complete discussion on librarics, APIs, and datasets, together with appropriate citation, can found under Appendix A
Guidelines:
19

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssithcosee thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Besides the code release, will release trained models together with sufficient documentation on how to use them at github.com//m hwwpaa/owoooomaaoaa/aaaaaa wwwchami/aaaaCC
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Our work has user-study of 38 participants, the details of which is briefly described in Section 5.3) with the detailed version in Appendix together with screenshots the interface, instructions, and sample questions.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
20

---

Answer: [Yes]
Justification:
Yes we received all the necessary approvals for our study.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
We did not use LLM for any part of evaluation, method development, or writing.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not non involve LLMs as any important, original, non- standardd components.
Please refer to our LLM policy for what should or should not be described.
21

---

FaCT: Faithful Concept Traces for Explaining Neural Network Decisions
Appendix
In this supplement to work faithful concept traces (FaCT), we provide further results, deriva- tions, and implementation details, as indexed below. We particularly encourage the reader to see we demonstrate the suitability of our proposed C3-score, well as, Sec. D where the diversity of our concepts and their generalization across layers and architectures are demonstrated.
A
Training Details
23
Additional Analysis Sparsity-Accuracy Tradeoff
24
C
-ccor:: Superior to Annotations
25
FaCT Provides Diverse Concepts
28
Details on the User Study
32
Details on Concept Deletion
Details on Inspecting Misclassification
Further Derivations
36
Computation Overhead
38
Stability of SAE training
Results on CUB
Limitations and Future Work
42
M
Societal Impact
22

---

A Training Details
In this section provide details on implementation, constructed dataset, training sessions, as well as checkpoint selection.
Implementation. As discussed in Section 3.1 our model consists of models with B-cos trans- forms [S2] [S3] with Sparse Autoencoders at layers where we want concept-based representation. We therefore use pre-trained B-cos checkpoints (i.e. B-cos ResNet-50 [S17], B-cos DenseNet- 121 [SI8], and B-cos ViT,-S (S8]) from the official release Within the paper, we refer to the following layer names as such:
ResNet-50: layer2 2/4; layer3 >Blockk 3/4; layer4++BBloo 4/4
DenseNet--211
-tt.aasiiiiiiitoooooieoe -Block4/4
ViT c-S: encoder8 +Block 8/10; encoder9 9/10; encoder 10oidderr1++Bocc 10/10
For the SAEs, we base our implementation on the Dictionary-Leariing codebase [S27] We use the
TopK-SAE [S14] (26), and modify it be bias-free (i.e. data whitening and no biases for encoder
or decoder). This is needed so that faithful output contribution in Eq. (9) and faithful input attribution in Eq. 12 can be obtained.
Dataset. Our SAEs are trained for reconstruction ImageNet [S30]'s training set. We first take 50 samples per class as a held-out validation set and leave the rest for training. We accumulate the features from the training set D, but as this would be very large dataset, especially at earlier layer of CNNs with high spatial dimension, For every image I D we perform importance sampling over the spatial dimension of F₃ R Xxxxxx based the contribution to the output.
DSAE = U ( [in Jal Contribution ((J))) n = 1..... M}. IED
(A.1)
Thus from every image's intermediate features F, we randomly sample M feature vectors weighted towards features of higher importance. For convenience, we set the M per model and layer, such that each constructed dataset DSAE uses at most 170 GB. The Contribution j.j] is measured within the original B-cos model with B B-cos attributions (S3] of the predicted logit to every spatial position in Fy. i.e. similar to Eq. (3) but for intermediate layers.
Training. For training the SAE, we use a batch size of 32,786 (individual feature vectors), with sweep of learning rates X E (0.001, 0.0001), total number of latents K (8192, 16384), and sparsity factor of TopK-SAE topk E [8, 16, 32] (except for ViT @ Block 4/10, where we also tested topk 64, as the lower values led to low accuracy). We use Adam Optimizer [[223], together with cosine learning-rate scheduler, with initial warm up of 2 epochs. We trained each model for 16 epochs, but in general observed that many runs plateaued at earlier epochs.
Checkpoint Selection. When training the SAEs, we observed that some runs may end up with many 'dead' latents, i.e. latents that never get activated. We also observed that runs may sometimes have 'always-active" latents, i.e. latents that are active on more than 60% of the data points (i.e. features). This can be attributed to (a) the use of TopK-SAE. which may repeatedly take the same bias-free SAE, as the model may learn
subset of latents the TopK, and (b) the fact that set of 'mean features' that are always active. We observed this also for vanilla SAEs (i.c. with LI regularization) on our initial experiments and also found traces of this in related work, e.g. the dots with frequency of 1 in Figure: of [S25. For every sparsity factor, we thus selected top two best performing runs (using held-out set from training data), and for similar performances, chose the one with fewer 'dead' or "always-active" latents. Finally, we evaluated our set of candidate runs per model and layer on the ImageNet's official test set, which are reported in Sec. B
The final checkpoints, which were also used throughout the paper for visualizations and evaluations, were obtained with the above procedure. For convenience and reproducibility, we report the configu- ration of these checkpoints in T
23

---

Architecture
Position
Learning Rate
Number of Concepts (K)
Sparsity Factor
(Layer Name)
(TopK)
Block 4/4 (layer4)
0.001
16384
16
ResNet-50
Block 3/4 (layer3)
0.001
16384
32
Block 2/4 (layer2)
0.001
8192
32
Block 4/4 (norm5)
0,001
16384
16
DenseNet-121
Block 3/4 (transition3)
0.001
16384
32
Block 2/4 (transition2)
0.0001
8192
32
Block 10/10 (encoder10)
0.001
8192
32
VIT C-S
Block 9/10 (encoder9)
0.0001
16384
32
Block 8/10 (encoder8)
0.001
8192
32
Block 4/10 (encoder4)
0.0001
16384
64
Table Al: Configuration of chosen checkpoints (based on the procedure described in Sec.
B Additional Analyses on Sparsity-Accuracy Tradeoff
In Fig. B1 we report detailed results on ImageNet. For each architecture, we report standard variant, B-cos, and FaCT for different layers. For standard models, we rely numbers reported in [S3] which evaluates under comparable settings, except for Standard DenseNet-121, for which there is checkpoint with modern torchvision recipe available. In Fig. (top-hall) we show ImageNet accuracy against per-image fg-norm, which shows the total number of concepts that activate per- image for different models (columns) and different layers (colors within each subplot), with the pareto front being displayed as line, and the selected checkpoint highlighted with star (*)).
As one would expect, on the top we see that with higher per-image tu fo-norm can achieve higher accuracy for our models, and for the same accuracy, one would require higher per-image fo-norm for earlier layers. Note however that the total number of concepts per-image are used by all of the logits Eq. (7), and to explain single logit, one would only require to look at fewer concepts. The bottom half of the plot reports fo-norm for covering 80% of the positive contribution per-image.
ResNet-50
DenseNet-121
ViT~
75
Imagetet
02
Standurd Bron
(39)
Scanardd
Get
Standord
(ox)
Back 0e1rt
3200
2003
3000
200
400
600
800
200
400
600
800
3200
Ee per image
per image
per image
75
Imagette test Acouracy
72
1891
(44)
Boten
Beck (101))
Block
35-
Block
105
contribution)
prediction contribution)
prediction 180% contribution)
Figure B1: Complete ImageNet Results with Sparsity-Accuracy Tradeoff. (Top): observe the sparsity-accuracy tradeoff and that with higher per-image to one can achieve higher accuracy. (Bottom): We observe that for explaining 80% of the positive contribution to the logit, one would need much fewer concepts compared to the above half.

---

C -'socre:: Superior to Annotations
In this section elaborate
proposed C2-score Eq. (16) for concept consistency. In Sec. C.1
we show limitations existing human-annotation by showing concepts that do not match annotations.
In Sec. C.2 we discuss how C--score correlates with our user-study results, and lastly in Sec. C.3
we
provide further details on the implementation.
C.1 Comparison human annotations
In Fig. we show three examples (major rows) from PartImageNet [S16] against the PCA decom- position of DINOv2 [S29] features (upsampled with LoftUP (S19). We find DINOv2 features be significantly more inclusive than annotations. Note that the DINOv2 features have in fact even richer semantics, as the displayed colors here are simply PCA decomposition of features.
Daod
Craale
Partebend
Human Annotations
DINOv2 Features
Seowmolile
Fooibal trcknot
Mubill home
Snowmale
Stuvel
Unkysic
Scoreword
Aprrry
Annolations
Features
Back dorieed In
Limennine
Poenhn
Fallenn
Annotations
Pinva
Denortic
Miriaa
Boach
Figure Cl: Comparing DINOY2 features vs. Human Annotations: We find DINOv2 features, as used in C2 score, to be much superior for evaluating our diverse concepts on the right. For every row, show two sample concepts from FaCT that consistently being activated, yet not considered within the human annotations. On the first and second examples we see ( ('Grass' and 'Helmet') and ('Person' and 'Floral dress") not being considered within the annotations. Lastly, we see the minivan has concepts such as 'Windows' or 'Chasis' that do not match the granularity of human annotations.
25

---

As per Fig. C1 we find using DINOv2 S29 features under -core to be superior evaluation framework compared to human annotations, which do cover all concepts and do not support shared concept bases. F.g., concepts such as 'Grass" or 'Helmet' on the top row are not even considered as part the annotation, which would falsely indicated in-consisent IoU under prior metrics [S20]. We however see that, even in the simplified PCA visualization of DINOv2 features, we observe different color for different parts in the image. Even if significant part of the image was annotated, e.g. in the last row of Fig. CI for Mini-van, we again see the granularity of annotations may not match the granularity of concepts. With concepts such as 'Windows' not being considered in the annotation.
C.2 Comparing C2 -ccoe with user study results
To further validate our proposed Cc-scor,, we compared the user interpretability scores from our user study (done for Section 5.3 in the main paper) against our proposed CC-score Section 4 In particular, we compared how the study participants sorted the late-layer concepts that were shown to them (by rating each from to 5), and compare the Spearman correlation of these sorting with the consistency scores that we get per concept, according to Eq. (T6) in the paper. Note that, per Section [53] our user study was conducted for evaluation of our concepts compared the baselines terms of interpretability. One could further tailor a study solely dedicated to evaluating the CC-score metric itself, e.g. asking users to rate ranking of concepts.
Nevertheless, in Fig. C2 we observe that for all of the 38 participants the C² -c-sce'''s ranking of concept has positive spearman correlation with how they rated the concepts throughout the study. In particular, 33/38 and 20/38 participants have 'moderate' (> 0.4) and 'strong' (> 0.7) correlation [S6] respectively. We thus find C--score lends itself well for evaluating agnostic manner and it correlates well with what users found interpretable, while also acknowledging that user study dedicated C²- could provide further insights.
shared concept-base in class-
12
Number Users
2
-1.0
-0.5
0.5 1.0
Spearman Correlation (C2-Scores, User Scores)
Figure C2: Comparing User Scores with CS-score for DenseNet-121. We observe that the rankings achieved by -core are highly correlated with how users ranked the late-layer concepts that were shown to them. In particular, all users had positive correlation, with 33/38 having "moderate" and 20/38 having 'strong' correlation.
C.3 Implementation Details
As discussed Section our C²-score leverages DINOv2 S29 features, in particular DINOv2- Small, together with state-of-the-ar feature up-sampler LoftUP [S19) We then evaluate the consistency concepts according to Eq. (16) on ImageNet's official test set.
Evaluating the Concepts. For Fig. the main paper, we evaluated our concepts against baselines on Block 4/4 of ResNet-50 (S17]. For CRAFT [S12 and CRP [S9], we used standard torchvi- sion (S34) checkpoints. We trained CRAFT S12 with concepts per class for every ImageNet the concept- attribution, as done in the original paper (ST2). For CRP [S9], we used the official zennit-based [S implementation together with EpsilonFlat rule, used in the original paper [SI].
class with 600 training samples and considered the up-sampled NMF-coefficients
26

---

Input Image
Nearest (raw)
Nearest (centered)
Loftup (raw)
LoRUP (centered)
AnyUp (raw)
AnyUp (centered)
Cosine similarity
Figure C3: Comparing Different Up-samplers. We measure the cosine similarity the marked green pixel in the first image (top-left) to all of the other features across different images (rows), under different up-sampling methods (columns). We observe that feature up-samplers (cols. 4-7) provide much sharper feature maps with similarity across all feature points, even across unrelated images (bottom-3 rows), which mitigated when features are centered around dataset-mean. We that for more-recent up-samplers [S37] the centering is less essential.
accurate details. We observe that LoftUP exhibits baseline
For all models, layers, and concept methods in Figs. 4 and 5 in the main paper, considered top 5% of activating test images for every concept, ensuring to select at least 10 images per concept. For concepts with fewer than 10 activating images, considered all of their images. We discarded concepts that activate on fewer than 5 images.
Up-samplers and Centering. When using LoftUP (19 (state-of
at the time submission), we noticed a baseline similarity of up-sampled features, for which we applied dataset-centering i.e. subtracting the mean up-sampled feature over the dataset. This is demonstrated in Fig. where we take the features on the broom point (green point the first row) and measure its cosine similarity to all feature points of other images. We observe that LoftUP [S19] features provide very sharp motivation of using up-sampler. However,
and detailed maps, which also matches that the point on the broom exhibits a baseline similarity with other non-relevant pixels (see bottom three non-relevant images). This is well addressed when performing centering the features (subtracting the global dataset mean). We also see that more recent up-samplers (AnyUp [S37] on the right) may not exhibit such baseline similarity. Note that C2-score would directly benefit from any advancements on foundation models S31 as well as feature up-sampling methods [S37]
Random Baseline. key part of C2-score is the inclusion of random baseline, which allows to evaluate concepts while taking into account the baseline similarity of the probe dataset. For every image in the evaluating set, the random baseline samples a spatial attribution map with a random threshold (both from uniform distribution). The consistency score Eq. (15) of random concept was at 0.0005 (a=8e-6), averaged across three seeds, while on a per-class setup it was at 0.4008 (a 0. 166), averaged across the 1000 classes. This agrees with the original assumption that when the evaluation set is restricted a single category, all the output features of the foundation model at use (here DINOv2) become more similar. Hence the Cc-se tackles this by considering the difference with respect to the random baseline.
27

---

D FaCT Provides Diverse Concepts
In the paper demonstrated how method provides shared concepts with high variety in spatial size Fig. In Sec. [D.]] we extend the Fig. from the paper and additionally provide insights on the class-specificity of our concepts. Afterwards, Sec. D.2 demonstrates more qualitative samples across layers and architectures.
D.1 Diversity in class-specificity and spatial size.
In Fig. DI we report diversity of our concepts in terms of class-specificity (through concept-labed entropy) and also in terms of spatial size. We also provide sample concepts further demonstrate the diversity qualitatively.
Concept-label Entropy [S25 is defined for a concept k, activating on image / with value S as follows:
1 _ Pk(1) (l)
H4 = 1=1
(D.1)
with (t) corresponding to aggregate activation mass of concept l on samples with label yı = over total unique labels.
N
Σ
SkI
(1)
(D.2)
N
Σ
SK.
Therefore, a class-specific concept that only activates for a single class would be assigned entropy of 0, while concept that uniformly activates across images of different labels, would be assigned an entropy of log(1000) z 6.9.
In Fig. DI (upper left), we observe that both DenseNet and ViT models have concepts of different sass-specificciyy, even at penultimate blocks (red). We see that for DenseNet, as we go to an earlier block, the portion of shared, higher-entropy, concepts increases (yellow). We further demonstrate samples (a-d) different class-specificity. We observe that, e.g., concept (a) corresponds to 'Fox head', which is exclusive two classes, while concept (b), 'Animal eyes' are shared across many species, resulting in higher entropy. In concept (c) 'Fur' and (d) 'White shirt', we similarly observe high degree of sharing across classes. This is also evident within the top activating images coming from a variety of classes. The existence of such diverse set of concepts is in clear contrast to prior assumption concepts being class-specific [112 S24 S33
Spatial Size is defined as the number pixels needs to cover the top 80% of the input-resolution positive attribution of a concept (i.e. 80th percentile), averaged over the dataset. In Fig. DI (upper right) we observe that our concepts come from a wide range spatial extents. This is in clear contrast to prior work [S12. $28) which assumes concepts to be fixed-sizes patches. Our visualized samples in Fig. furthher validate this by having smaller concepts such (a) 'Fox head" and (b) Animal eyes', as well as larger spatial concepts such as (c) 'Fur'. The diversity in spatial size is further demonstrated in Figs. D2 and D3 for both architectures and across the layers.
28

---

DenseNet-122 121
VIT.
DenseNet-121
VIT -S Block (10/10)
Percentage of Concepts
Block 1424)
Bokk (20/10)
ock (4/4)
(a)
(c)
(d)
(a) (b)
(d) (c)
10
10
5
5
13/4)
9/10)
( 9/10)
10
10
5
0
70
7
64
128"
64
128
Concept-Label Entropy
Spatial Size (pixels)
(a)
(b)
Cougar
Cheetah
Cheetan
Lynoe
(c)
(d)
Newfloundland
Bison
Bison
Liama
Pur coat
Pansipe
Lab coat
Racket
Basoball
Bal player
Figure D1: FaCT offers diverse concepts. We observe that concepts, both for intermediate and late layers, come from a diverse distribution of class-specificty (upper left) and spatial size (upper right). We validate the plots by showing samples from different bins (a-d). We see class-specific concepts such as (a) head' and more class-agnostic concepts such as (c) and (d) Whitt shirt', For more qualitative results from both architectures, see also Figs. D2 and D3
D.2 Generalizing across layers and architectures
In Figs. D2 and D3 we show how FaCT generalizes across early and late layers, as well as across architectures (CNNs and ViTs). The displayed concepts additionally extend the observations made in Fig. D1 demonstrating the diversity of concepts in terms of spatial extent and class-specificity.
In Fig. D2 generally find simpler concepts for earlier layers, such as 'Red dots', 'Green back- ground field', 'Curves', 'Tiles', and 'Small animal legs', from top to bottom. We also observe a mix of high-level and simpler concepts for later layer (right col.), such 'Baby faces', 'Shoreline', 'Red surface', 'Archs', and Ping pong ball'. These concepts also serve as counter examples for what prior work considers concept to be. Unlike [S1 S12 S24 many of concepts are shared across class, while class-specific ones such as 'Ping-pong ball' on the bottom right also exist. Our concepts also not confined to be object- part-centric S44 S28 e.g. 'Shoreline' and 'Archs'.
In Fig. D3 we further demonstrate how FaCT generalizes across ViT architecture with interpretable concepts. We again see a mix of class-specific concepts, such as 'Junco beak" (first col. 4th row) and 'Saint Bernard brown fur' (second col. 5th row), together with shared concepts such as 'Bright yellow' (first col. 5th row) and 'Branches' (second col. first row). We observe that FaCT is able to encode small part-based (e.g. 'Sharp ears' first col. 3rd row) and large scene-based (e.g. 'Tiles' second col. 4th row) concepts, all being faithfully visualized at input. This is in contrast to prior work [S4 S28 S12 which assumes parts to be small patches and visualizes concepts by upsampling low-resolution intermediate maps, that does not generalize for ViT architectures.
29

---

DenseNet- 121 (Block 2/4)
DenseNet- 121 (Block 4/4)
Fox squirell
Water jug
Pojama
Hotdog
Soccer ba
Crib
Vacuum
Horizontal bar
Whippet
springer
Sperdboat
The
Big
Barrel
Mixing
Dough
Petri dish
Fire engine
Book ipaint
pump
DUOE
0000
T roof
Tab lamp
Waffle iron
Scabbard
Viaduct
Viaduct
Viaduct
Triumphal arch
Dogsled
Ibex
Dogsled
Impala
Ping-pong bell
Ping-pong ball
Ping-pong ball
Ping-pong ball
Figure D2: Sample concepts from early (left column) and late (right column) concepts of our DenseNet-121 FaCT models. We observe simple concepts such as 'Red dots", "Green Background Field', 'Curves', 'Tiles', and 'Small animal legs for early layer and concepts of higher semantics such as BBby Face", 'Shoreline', 'Red Surface'. 'Archs', 'Ping-pong ball' for late-layer concepts. See also Fig. D3 for ViT concepts.
30

---

ViT S (Block 9/10)
ViT S (Block 10/10)
Pembrake
Goldion retrieve
Red fox
Dingo
Obclisk
Ptarmigan
Bam
Projector
camera
camera
Projector
compu Fountain pen
teleph
Harmanica
Inols
Schipperke
squirrel
Schipperke
Cab
Limousine
Sliding
Junco
Junco
Junco
Junco
Soap dispenser
Shower curtain
Binder
Shower curtair
Rapeseed
Giant schnauzer European fire Long-horned be
Saint bernard
Saint beraard
Saint bernard
Seint bernard
Figure D3: Sample concepts from Block 9 (left column) and Block 10 (right column) of our VIT FaCT model. Similar to Fig. D2 for DenseNet, here we also observe mix concepts of different spatial size and class-specifiiiity. On the left column, we see 'Dog faces', 'Lens', 'Sharp dark ears', 'Junco beak", and 'Bright yellow', and on the right column we ee 'Branches', 'Human hands", 'Car wheels', "Bathroom Tiles', and 'Saint Bernard's ' fur'.
31

---

E Details on the User Study
In this section provide further details paper, we randomly sampled 100 late and 100 early concepts of our proposed FaCT model for DenseNet-121. We also randomly sampled 30 late and 30 early B-cos channel concepts baseline. For each concept, we visualized top-10 images with highest activation together with their input attribution, as derived in Eq. (12). We additionally performed counterbalanced AB/BA test, were for each of our 200 (early and late) concepts, we additionally plotted the top- 10 images without any input attribution. The resulting 460 questions were then distributed
study. As discussed in Section 5.3 in the main
randomized groups, such that each group has samples from both early- and late- FaCT and B-cos concepts. For the controlled study we made sure that user sees both experimental (with input-attribution) and control (without) version of the same concept.
Our amonymous survey received in total 38 complete responses. At the beginning instructed the participants with set of sample concepts that one would consider interpretable and uninterpretable, accompanied with explanation on why, see Fig. ET
After reading the introduction, the participants would then be randomly assigned to of the 10 groups with 46 questions. See Fig. B2 for screenshots of two sample questions from two groups.
Introduction
* G1000001
In following survey, we kindly ask you to look 46 collections of images decide whether they correspond to sin- gle, aaman-understandable concept. concept can be colors (e.g. yellow-color concept), patterns (e.g. strlpe-pattern concept), or more complex such as eyes, windows, or sand on the beach.
Each question displays collection of Images that may correspond to single concept. Additionally, some questions show additional visualization below every image, highlighting which part of the image belongs to the concept. For every tion, we ask you to rate how interpretable and consistent find the concept, rating from (least) to 5 (most) inter- pretable. Only a few seconds of your attention are needed for each concept.
provide example figure below, with three interpretable concepts (with visualization) on pretable concepts the right. Here, example (c) shows a simple concept may difficult to come up with word for, concept visually clearly interpretable. the right for (d), black-color seems reasonable concept, but it been highlighted for entire region.
set uninter-
Examples with single interpretable concept
Uninterpretable examples
(a)
[Tres)
(d)
(b)
[ Green Leaves
(c)
Figure E1: Screenshot initial introduction provided to the user. We instructed the users with general definition of concept, followed by sample interpretable and uninterpretable examples with explanation. For sample questions, see Fig.
32

---

300 Q4721
Please rate if the displayed figure points to a single human-understanddabe concept.
1: Strongly disagree: I cannot conclude an interpretable concept from this.
5: Strongly agree: can clearly see single interpretable concept.
No
185 q4401
Please rate if the displayed figure points to single human-undderstandable concept.
1: Strongly disagree: cannot conclude an interpretable concept from this.
5: Strongly agree: can clearly see a single interpretable concept.
No answer
Figure E2: Screenshot of two sample questions from randomized group 8 (top) and group 4 (bottom).
The sample at the top (without input attributions) serves as a control sample for the one at the bottom.
33

---

F Details on Concept Deletion
We evaluated concept contributions (Eq. 99 in the main paper) using concept-deletion met- ric [SI1] Given a concept importance measure, this evaluation framework deletes concepts in the order of most to least important and measures the changes of the target logit. If the importance measure is accurate, one would expect a sharper drop in the logit as the concepts are removed. In our case, since our concept-basis is shared across classes and are used for the final prediction, additionally plot the drop in overall accuracy. In order to compare to prior work [S13] S12 S24], we evaluate their used concept importance measures, namely Saliency [S32] and Sobol indices [S1O]. Note however that the comparison has to be done on the same concept set of the same layer, so that one only compares the difference in importance measures, without adding confounders such as different number of concepts or different distribution of significance over concept bases.
We evaluated the importance measures over 50 randomly selected classes of ImageNet, with 8 samples per class chosen from the test set. We based our implementation of Saliency and Sobol indices on definitions in [SII]. For Sobol indices [S10]. we used Janson Estimator [S22], similar to CRAFT [S12]. While CRAFT configures Janson Estimator to 32 designs for the few per-class concepts, we were only able use 4 designs, given the high number of concepts K [8192, 16384), which still created more than 32,000 concept perturbation masks and forward passes per image. This further highlights how some of the importance measures tailored for setups with few class-specific concepts may be difficult to scale to large and shared concept bases.
Additional Results for ResNet-50. Fig. FT we provide additional results for ResNet-50 (similar to Fig. 7 in the main paper for DenseNet-122 121). We observe consistent trend as in Fig. 7 with our concept contributions (Eq. ( outperforming existing importance measures, for both logit and accuracy drop.
We particularly see larger difference for earlier layers, with a larger gap between the curves.
Ours
Saliency
Random
Sobol (n=4)
ResNet Block (2/4)
ResNet Block (4/4)
ResNet Block (2/4)
ResNet Block (4/4)
Accuracy
Lage
50
100
100
50
100
50
100
Concepts dropped
Concepts dropped
Concepts dropped
Concepts dropped
Figure Fl: Additional Concept Deletion Results for ResNet-50. We observe that our proposed concept contributions (Eq. (9)) outperform existing concept importance measures with a sharper drop both in terms of top-1 logit and overall accuracy. This is similar to Fig, but for ResNet-50.
'Always-on' Concepts. As discussed in Section oi the main paper, we observed a very sharp drop for B-cos contributions in Fig. of the main paper (and in Fig. Fl for ResNet). In both cases, we found these concepts to be a small set that occur on 100% of the samples. This also matches the highly-frequent concepts observed in training the SAES (see Sec. A.. We hypothesize that these mean feature vectors that are used for reconstruction in every sample. The sharp drop in Figs. 7 and Filis is caused when these concepts get deleted. Nevertheless, as we use the same concept-set for methods, this still shows that B-cos contributions (Eq. (9) are
evaluating the concept-importanee best in identifying the most impactful concepts. To investigate the identification of most relevant concepts beyond these "always-on" concepts, in Fig. F2 we re-evaluated the concept deletion at early layers without removal of these few latents, only evaluating on the rest of the concept set. We see that the sharp drop disappears, yet Eq. still outperforms other concept-importance measures.
Ours
Saliency
Random
Sobol (n=4)
ResNet Block (2/4)
DenseNet Block (2/4)
ResNet Block (2/4)
DenseNet Block (2/4)
Accuracy
Logit
50
50
100
50
100
Concepts dropped
Concepts dropped
Concepts dropped
Concepts dropped
Figure F2: Similar to Figs. 7 and FT] we performed concept-deletion at early layers of ResNet and DenseNet, but without allowing the few 'always-on' concepts to be deleted. We observe that proposed concept contributions (Eq. (9)) nevertheless outperform existing concept importance measures, with a sharper drop both in terms of top-1 1 logit and overall accuracy.
34

---

G Details on Inspecting Misclassification
In this section, begin by providing further details the Fig. 8 in the main paper and afterwards additionally compare the same confusion case for CRAFT [S12]. a prior work with class-specific concepts, show how our shared basis in Fig. g can provide additional insights.
In Fig. [ the main paper, we demonstrated the confusion between Basketball and Volleyball image with a set of concepts that mutually and exclusively contribute to both. To create this figure, we used DenseNet- 121 FaCT model with concept-decomposiie Volleyball and Basketball logits individually in terms of concept contributions, using Eq. ( leading to two contribution values per concept, for basketball and volleyball logits. For each concept-logit pair, we computed the contribution as the percentage of overall positive contribution of concepts. Finally, the 6 concepts shown in Eq. were selected from the top- 12 concepts that appeared for each logit.
at Block 4/4. We then explained the
Comparison to class-specific methods. Through Fig. 8jin the main paper, we were able to understand the confusion between the two logits Volleyball and Basketball on concept-level, with concepts such as 'Ball' (A), 'Jerseys' (B), 'Man in sports-shirt's (C) and 'Limbs' (D) contributing to both logits. Below, we try to understand the same confusion case for ResNet--50 with class-specific method CRAFT [S12]. In Fig. G1] we report the two Basketball and Volleyball logit for the same image in the middle, and show class- blass-specific concepts that CRAFT offers for each on the side. While the concepts of CRAFT [S12 are indeed interpretable, we see that similar concepts repeat for each of the classes without any connection between the two, as opposed to our FaCT model that uses shared basis with concepts contributing to both, e.g. the third and fourth row seem to both point to a similar 'floor' concept. yet this is not captured by the explanation method.
CRAFT for Volleyball
Sobol Indices
CRAFT for Basketball
0.18
0.54
YYAAL
0.14
0.10
18)
0.01
0.08
Volleyball:
3.4
Basketball:
0.00
0.00
0.00
0.00
Figure G1: Comparing class-confusion analysis with class-specific methods. Here we plot CRAFT [S10 explanations for Volleyball (left) and Basketball (right) categories. Each row shows concept with the up-samnpled concept attribution (left sub-cols) and Sobol indices (numbers in the middle) for each concept. While the concepts are quite interpretable, we see that they are repeated between the two classes (see the 'floor" concept each side). This is in contrast to our setup, where shared concept basis is used.
35

---

H Further Derivations
In this section we extend the derivations provided in Section 3.1 in the main paper. We begin by discussing why ReLU is dynamic-liness transform, which was used in Eq. (6. We then demonstrate in Sec. H.2 how to derive input attribution of concept contribution opposed concept activation derived in Eq. (12) in the paper. Lastly, in Sec. H.3 we derive how the cross-layer contribtion of concepts each other can be faithfully measured.
H.1 Dynamic Linearity of ReLU
In the main paper, we discussed how in Eq. (6) concept activations U E Bllxxx is dynamic- linear transformation of F E Hx xc While our SAE definition in Eq. 7 is bias-free, it still uses explain how it can be considered
dynamic-linear transform. For any tensor X E R2 DD a ReLU operation can be formulated as an element-wise multiplication with tensor ((X) E R DixxxDD of the same shape, such that
ReLU non-linearity, which in this section
[1 ReLU(X) = W(X s.t. W(X) (0 if
if
(H.1)
In fact, any piece-wise linear function can similarly considered as dynamic-linear, but not vice versa (e.g. B-cos transforms [S2 in Eq. 33) in the main paper are dynamic-linear but not piece-wise linear). Therefore Eq. (H.T) shows that our concept activations defined in Eq. (6) are a dynamic-linear transformation of features, which allows our derivations for concept contributions in Eq. (9) and
concept-attributions in Eq. 12 to hold true.
H.2 Faithfully attributing concept contribution at input
In the main paper, we demonstrated how the activation concept for an image can be faithfully attributed to the input Eq. (12) in the paper). Note that in Eq. (1O), the activation of concept is defined by a summation over spatial dimensions.
output logit, however, may not rely on every spatial position equally. This is evidenced by Eq. (9), where the logit is dynamic-linear combination of concept activations (both over spatial dimensions H, W and concepts K). For convenience, we repeat the Eq. (9) below.
K
H.W Σ 1
K
L FCC
Σ
W(U)
U.j.k
Σ
Contribution
(H.2)
Therefore, for a particular concept contributing to logit é, the Contributionf, can be formulated as follows:
H.W
Contribution
Σ
[WU U.A
(H.3)
Analogous to Eq. 12 in the main paper, we can now further decompose the U tensor dynamic- linear transform of input pixels to obtain a faithful input attribution of concept contribution. The only difference here is that we have weighted sum spatial positions (i.e. (W))) instead of uniform summation at the beginning of Eq. (2) in the main paper.
H.W
H.W
Contribution
Σ
W(U),,
U.jA
Σ
W(U), [ReLU(conv(W) (W, ij
(H.4)
22j
H.W.C
H.W.C.
= Σ
Σ [W(F)
(H.S)
$J.Cc
i.ee
H.W.C.
Ho.Wo.3
Σ
(W
Σ
[W(I)
(H.6)
iJe
We therefore see that similar to Fq. where the activation of a concept could be faithfully attributed to the input, the contribution of concept to particular logit can also be faithfully attributed to input.
36

---

In our experiments, we did not find distinguishable difference in visualization of concept activation and concept contribution. Throughout the paper we therefore consistently visualized the activation of concepts, i.e. Eq. [2 in the main paper.
H.3 Faithfully measuring cross-layer concept-contribution
In Eq. (9) in the main paper, we demonstrated how every individual logit can be faithfully decomposed as a summation concept contributions. If one considers logit as a 'neuron', , then essentially, Eq. (9) explains how a late-layer neuron (concept activation or logit) can be explained as contribution of earlier concepts. Thus one can use Eq. ( for cross-layer contribution, by simply replacing the initial logit with concept activation. Below will nevertheless further derive this in detail.
Suppose have model ((x) Jg-m Jt-1 [1-1(1)), where have early and late concept decompositions the outputs at layers and 9, named U E HR x respectively.
x and U9 RR' x xw xx'
F no F9 S.A. = conv( (vonv(Vo, U*) conv (V,, ReLU(coov (W", (?))
(H.7)
LeaTT fg-n(F9) fa-n conv ReLU(conv (W,, F2))),
(H.8)
with intermediate feature tensor F itself being function of earlier features F*:
F S.L. Fl = conv( V U conv (V', ReI.U(conv W F))))
(H.99
F = fi-g(F fi conv (v', ReLU((cov(WW) F))))
(H.10)
Therefore, the activation of concept c from the late concepts U can be explained as a summation of contributions from early-layer concept activations U
H'WW"
H'WW" ReLU(conv(WW (W),F))).s
Activation =
U Ci.c
(H.11)
H'WW"
Σ
(H.12)
H'.w"
Σ
ReLU(conv(W*) fi-g conv(V' U'I))
(H.13)
H.W.C.
WWU'
Σ
Contribution
(H.14)
where Contribution denotes the contribution of early-layer concept è at layer 1, to the activation of late-layer concept at layer g. This is the exact derivation that was used for Fig. i the main paper, with the early-layer 'Curve' concept contributing 2.9% to the activation of late-layer 'Wheel' concept.
37

---

I Computation Overhead
As discussed in Section 3.1 FaCT leverages Sparse Auto-Encoder bottleneck during the forward process, ensuring that the model only relies on the concept activations. To ensure that this does not result in significant computational overhead, in Table [[ we measured the time it takes for FaCT to process the entire ImageNet's validation set (i.e. 50,000 images) for DenseNet-122 121 at different layers. We compare the inference time with the corresponding B-cos model of the same depth. The results were averaged across three runs. We find FaCT to be quite comparable to the original B-cos architecture. Specifically, we observe less than 0.2 milliseconds anference-overheaa per-image, which is likely to reduce further with inference optimization (e.g. changing the SAE-hook to fixed layers and using 'orch.compile".
Method
B-cos
FaCT @ Block 2/4
FaCT Block 3/4
FaCT Block 4/4
Time (scconds)
104
112
108
112
Table I1: Time required process the entire ImageNet's test set for DenseNet-121 models (averaged over three runs). FaCT has very comparable inference-time compared to the B-cos model.
J Stability of SAE Training
As discussed in Section 3.1 FaCT trains Sparse Auto-Encoder to form a concept-basis. While different SAE training sessions may result in different final models, in general for the same layer observed many concepts being repeated across configurations as well as across architectures at similar depth. To verify this quantitatively, we evaluated the recently proposed Stability Score (S13) for the same layer and dictionary size, but with different (Top-K) sparsity factor. We did this both for early and late-layer decompositions of DenseNet-121 and computed the pair-wise stability-score (ST3) for (Top-K E {8, 16, 32} experiments. We observed score of 0.76 and 0.70 for Block 2/4 and Block 4/4 of DenseNet-122 121 models. While being trained on different models, datasets, and layers, our stability scores are quite on par with the ones in Table 1 of [S13], where the authors report 0.5 for the TopK-SAr under different seeds.
Additionally, we would like to point out that FaCT would directly benefit from new variants of SAEs, c.g. Archetypal SAEs in [S13], that may be introduced by the community. We would also highlight that with faithful
of concepts, as discussed Eq. (22) of the main paper, FaCT allows the community to evaluate different concept-discovery methods (e.g. different SAE variants) under the lens of faithful attribution.
38

---

K Results for CUB Dataset
Throughout the paper, put partieular emphasis having scalable setup which remains compet- itive on ImageNet [S30] (see Section [5. in the main paper). In this section, we demonstrate how FaCT generalizes
other datasets, namely CUB-200 dataset [S30] for fine-grained classification. While there are many works that propose tailored models for this dataset [S4] [7] S35 S28), which have shown to often not scale more challenging datasets [S33], our focus obsever whether
FaCT remains competitive while providing interpretable shared concepts.
Following our ImageNet setup from the main paper, we thus trained FaCT on early, middle, and late blocks of a B-cos ResNet-34 (pre-trained on uncropped CUB). We collected the uncropped training set's features similar to Sec. A and trained our bias-free TOP-K SAFs with learning rate of 0.001, total concepts E [1024, 2048]. and sparsity factor TOP-K = 16.
We observe that across the layers, FaCT is able to maintain high accuracy while providing consistency gains. In particular, with less than 1% accuracy drop, we observe significant consistency gains for Block 3/4 (0.24->0.37) and Block 4/4 (0.32-+0.58).
B-cos
FaCT (ours)
ResNet-34 -4
For reference, standard ResNer-34 [S34], ProtoPNet [S4]. and Deformable ProtoPNet [S7] all below 77% accuracy, accord- ing to [S7]. though the pre-training recipes may not exactly match. Nevertheless, we observe FaCT to remain competitive and provide more consistent concepts.
0.6
0.5
C2-Score
When inspecting the concepts, we found the results agree with what we observed for ImageNet Section 5.1 with diverse set
0.4
of shared concepts across the layers. In Figs. K2 and K3
we
show concepts for Block 3/4 and Block 2/4, We observe many part-based concepts, in particular in Fig. K2 for Block 3/4, which are shared across classes. For earlier Block 2/4, i.e. Fig. K3 observe lower-level concepts such 'curves' or 'yellow-fur' while some also correspond to exact parts, e.g. 'wing edge' on the top-left. Interestingly, saw increased number of concepts for the background, which can be seen on the bottom of Fig. K3
B2/9
0.2
we
0.1
79
CUB Accuracy
Figure KI: CUB results for ResNet-34 across layers.
Our results thus show that FaCT, without having any assump- small
tion on the concepts being object parts, class- class-specifie, patches, generalizes other datasets. Having no restriction on concepts becomes more crucial when one moves to larger-scale datasets such as ImageNet, where concepts required for task may not necessarily correspond to parts, e.g. see scene-centric 'Shoreline' concept in Fig. D2 (second-row) or 'Bathroom Tiles' in Fig. D3 (fourthrrow.)
39

---

Figure K2: Sample FaCT Concepts from Block 3/4 on ResNet-34. We observe many corresponding to object parts, such heads and beaks (top-two rows), legs and tails (rows three and four). Notice that many these parts are shared among classes, e.g. 'legs on the branch' (fourth-row left).
40

---

Figure K3: Sample FaCT Concepts from Block 2/4 on ResNet-34. We observed many concepts corresponding simpler features, such as 'yellow-fur' (second-row left) or 'curves' (third-row right). We also observed an increased number of background concepts such as branches (fourth row) or water/land backgrounds (bottom row).
41

---

L Limitations and Future Work
In this work, discussed a model FaCT with inherent concept-based explanations with concept basis shared across classes. Our proposed model can further explain every logit in terms of concept contributions Eq. (9). while faithfully attributing every concept to input Eq. (2). Such faithful concept-based explanation however does not guarantee the interpretability of our concepts. Indeed, in Fig. in the main paper, we demonstrated that our concepts are more interpretable than baselines, yet, we also see that there exist uninterpretable concepts with low scores from the users. We believe the next step would be further inspections on less interpretable concepts, their contributions to different predictions (through Eq. (9). While uninterpretable concepts are less desirable for the end-user, whether model without such concepts can be as performant FaCT, an open question. Perhaps relevant direction could be single-stage training paradigm, where FaCT is trained from scratch and is regularized towards interinterpeeetblle concepts, as opposed to the two-stage training.
Further, our work leverages SAEs for arriving at the concept basis in an unsupervised manner. While SAEs offer the advantage of having concept activations a linear (and in case bias-free) transform of features, which we used for faithful input attributions in Eq. 12, we acknowledge the on on-going discussion SAEs, and note that approach would also benefit from further research in this direction. For example, recently [S3] proposed a new paradigm of training SAEs by constraining the dictionary vectors lie on the manifold of training features. Of course, this would directly help with training our FaCT models as well. In fact, we argue that one could now experiment different directions of training SAEs our proposed model FaCT, so that the resulting concepts hetter be studied, through faithfully visualizating them at input-level and faithfully measuring their importance the final logits.
Additionally, we proposed a new concept-consistenge metric C?-score which leverages DINOv2 features for a class-agnostic concept consistency evaluation. While in Sec. C (e demonstrated how this is superior to existing annotations and how it aligns well with our user study, a further study on what are the limitations of DINOv2 for such tasks across concepts of different semanties, and whether there exist better alternatives indeed still an open question, which could significantly influence how concepts evaluated in the future.
Lastly, the main ingredients of FaCT are the of B-cos layers SAE for concept extraction. This allows FaCT to readily extend to other modalities such as language, as both SAEs and B-cos layers have been shown extend to other modalities [S21 (36). Such applications would also allow exploring whether the faithful concept-based explanations of FaCT allow for better concept-editing for steering models (S5).
S2 S3
for faithful attributions and
M Societal Impact
Our work puts great emphasis on the faithfulness of concept-based explanations and proposes a new model FaCT which can faithfully report existence of concepts and their significance to the prediction. It is therefore a step towards models with explanations that can be trusted for safety- critical applications, such as health domain, where the explanations should not mislead the user. We also proposed a novel metric C2 -core for assessing quality of concepts, which can benefit future concept-based methods automated evaluation tool.
42

---

References
[SI] J. Anders, D. Neumann, W. Samek, K.-R. Müller, and s. Lapuschkin. Software for Dataset- wide XAI: from Local Explanations to Global Insights with Zennit, CoRelAy, and ViRelAy. arXiv preprint arXiv:2 102000000 2021.
[S2] M. Böhle, M. Fritz, and B. Schiele. B-cos Networks: Alignment is All We Need for Inter- pretability. In CVPR, 2022.
[S3] M. Böhle, N. Singh, M. Fritz, and B. Schiele. B-cos Alignment for Inherently Interpretable CNNs and Vision Transformers. JEEE TPAMI, 2024.
[S4] C. Chen, O. Li, D. Tao, A. Barnett, C. Rudin, and J. K. Su. This Looks Like That: Deep Learning for Interpretable Image Recognition. In NeurIPS, 2019.
[S5] B. Cywiński and K. Deja. SAcUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoders. In ICML,002.
[S6] C. P. Dancey and J. Reidy. Statistics without Maths for Psychology. Pearson Education, 2007.
[S7] J. Donnelly, A. Barnett, and Chen. Deformable ProtoPNet: An Interpretable Image Classifier using Deformable Prototypes. In CVPR, 2022.
[S8] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An Image Worth 16x 16 Words:
Transformers for Image Recognition at Scale. In ICLR, 2021.
[S9] M. Dreyer, R. Achtibat, W. Samek, and S. Lapuschkin. Understanding the (Extra-) Ordinary: Validating Deep Model Decisions with Prototypical Concept- based Explanations. In CVPRW, 2024.
[S10] T. Fel, R. Cadene, M. Chalvidal, M. Cord, D. Vigouroux, and T. Serre. Look the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysis. In NeurlPS, 2021.
[S11] T. Fel, V. Boutin, L. Béthune, R. Cadène, M. Moayeri, L. Andéol, M. Chalvidal, and T. Serre. A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation. In NeurlPS, 2023.
[S12] T. Fel, A. Picard, L. Bethune, T. Boissin, D. Vigouroux, J. Colin, R. Cadène, and T. Serre. CRAFT: Concept Recursive Activation FacTorization for Explainability. In CVPR, 2023.
[S13] T. Fel, E. S. Lubana, J. S. Prince, M. Kowal, V. Boutin, I. Papadimitriou, B. Wang. M. Watten- berg, D. E. Ba, and T. Konkle. Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models. In JCML, 2025.
[S14] L. Gao, T. D. la Tour, H. Tillman, G. Goh, R. Troll, A. Radford, I. Sutskever, J. Leike, and J. Wu. Scaling and Evaluating Sparse Autoencoders. In ICLR, 2025.
[S15] A. Ghorbani, J. Wexler, Y. Zou, and B. Kim. Towards Automatic Concept-based Explana- tions. In NeurlPS, 2019.
[S16] J. He, S. Yang. S. Yang, A. Kortylewski, X. Yuan, J.-N. Chen, S. Liu, C. Yang, Q. Yu, and A. Yuille, PartImageNet: Large, High-Quality Dataset of Parts. In ECCV, 2022.
[S17] K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016.
[S18] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely Connected Convolutional Networks. In CVPR, 2017.
[S19] H. Huang, A. Chen, V. Havrylov, A. Geiger, and D. Zhang. LoftUp: Learning a Coordinate- Based Feature Upsampler for Vision Foundation Models. In ICCV, 2025.
[S20] Q Huang, M. Xue, W. Huang, H. Zhang, J. Song, Y. Jing, and M. Song. Evaluation and Improvement of Interpretability for Self-explainable Part-prototype Networks. In ICCV, 2023.
43

---

[S21] R. Huben, H. Cunningham, L. R. Smith, A. Ewart. and L. Sharkey. Sparse Autoencoders Find Highly Interpretable Features in Language Models. In ICLR, 2024.
[S22] L. Janson, F. Barber, and E. Candès, EigenPrism: Inference for High-Dimensional Signal- to-Noise Ratios, 2016.
[S23] D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. In ICLR, 2015.
[S24] M. Kowal, R. P. Wildes, and K. G. Derpanis. Visual Concept Connectome (VCC): Open World Concept Discovery and their Interlayer Connections in Deep Models. In CVPR, 2024.
[S25] H. Lim, J. Choi, J. Choo, and S. Schneider. Sparse Autoencoders Reveal Selective Remapping of Visual Concepts during Adaptation. In ICLR, 2025.
[S26] A. Makhzani and B. Frey. K-sparse autoencoders. In ICLR, 2014.
[S27] Marks, A. Karvonen, and A. Mueller. dictionar learning. hetrmarrk///icciiarry...oo Learning 2024.
https://gittub. com/
[S28) M. Nauta, J. Schlötterer, M. Keulen, and C. Seifert. PIP-Net: Patch-Based Intuitive Prototypes for Interpretable Image Classification. In CVPR, 2023.
[S29] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, HAZIZA, Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, Howes, P.-Y. Huang. S.-W. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jegou, J. Mairal, P. Labatut, Joulin, and Bojanowski. DINOv2: Learning Robust Visual Features without Supervision. Transactions on Machine Learning Research, 2024.
[S30] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015.
[S31] O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508. 10104, 2025.
[S32] K. Simonyan, A. Vedaldi, and A. Zisserman. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. In ICLRW, 2014.
[S33] A. Tan, F. Zhou, and H. Chen. Post-hoc Part-prototype Networks. In ICML, 2024.
[S34] VischVision maintainers and contributors. Torch Vision: PyTorch's Computer Vision Library.
https://github con//pyoocch/vision. 2016.
[S35] J. Wang, H. Liu, X. Wang, and L. Jing. Interpretable Image Recognition by Constructing Transparent Embedding Space. In ICCV, 2021.
[S36] Wang, S. Rao, J.-U. Lee, M. Jobanputra, and Demberg. B-cos LM: Efficiently Transforming Pre-trained Language Models for Improved Explainability, arXiv preprint 17X1v:2502.12992, 2025.
[S37] Wimmer, Truong, M.-J Rakotosaona, M. Oechsle, Tombari, B. Schiele, and J. Lenssen. Anyup: Universal feature upsampling. arXiv preprint arXiv:2510.12777 2025.
44


Reviews:
[{"review_id": "JdqfVIArbN", "reviewer": "Reviewer_V8ET", "summary": "The paper proposes FaCT, an inherently interpretable concept-based method that is capable of tracing the contributions that concepts, encoded throughout the network, have on the output produced by the model. The main motivation for this design lies on its goal of emphasizing the faithfulness that explanations produced from these concepts have on the underlying decision-making process followed by the model.\n\nIn practice, layer-level concept extraction is achieved through the use of sparse autoencoders (SAEs) and the links between the extracted concepts and the model outputs is enforced through a B-cos type architecture that is the basis of the proposed method. This is paired with a metric (C2-Score) to measure the consistency of extracted concepts across different examples.\n\nQuantitative experiments as well as a user study conducted on the ImageNet-1k dataset show the capabilities of the proposed method in the downstream classification task, the faithfulness of the produced explanations, the intelligibility of the extracted concepts and their consistency.", "questions": "- When doing the, there is a very significant drop in performance in the performance went a relatively short amount of concepts is suppressed. While a drop is expected (assuming an explanation method is effective at assigning importance of concepts), the sharpness of the drop is rather abnormal and raises doubts. Is there any evidence or hypothesis of why this very sharp drop in performance is observed? How are the concepts been suppressed in that experiment? Is there any relation between the size (spatial extent) of concepts that are suppressed at earlier stages and the observed drop in performance?\n    \n- The third contribution and early parts of manuscript hint at the faithful visualization of concepts at the input-level, I would appreciate some clarification on what is meant by “faithful visualization”? Moreover, some clarify on how it differs from the aspects covered by the first two contributions could be helpful\n    \n- Does the user study  reported in Sec. 5.3 follow an existing protocol? are there steps taken to ensure not bias nor subjectivity affects the observed results?\n    \n    The second term of Eq. (14) covers a sum over H and W. This hints at a summation over a 2D matrix space, f(i) in this case.  Similarly in the first term the . operation takes f(I) as matrix. but according to l.141, isn’t f(I) an 1D vector of logits?\n    \n- When defining the random concept in Sec. 4 (l.180-183), what are the characteristics of its random attribution map in terms of its distribution, both across the spatial extent and intensity, that are considered for its generation? what is the effect that these two variables have in the effectiveness of the proposed C2-Score metric?\n    \n- What procedure is followed for computing the spatial extent/size of the concepts (l.210-215)? Is there a thresholding of an activation and/or attribution map taking place? if yes, how is that threshold defined?\n    \n- The ViT-S column in Fig.4 seems to suggest that the spatial extent of concepts is very similar at different depth, why is this the case? how is their size distributed at earlier layers?", "limitations": "In addition to the points listed as weaknesses I would stress the following aspects:\n\n- The third contribution and early parts of manuscript hint at the faithful visualization of concepts at the input-level, this is an aspect that has not been properly defined nor evaluated in the rest of the manuscript\n    \n\n- In Sec. 5.1 (Fig.4, left), the performance of the proposed method in the downstream classification task is only reported with respect to the considered B-cos-variant of the considered architecture (i.e. ResNet-50, DenseNet-121, ViT-S) method and not the original vanilla architectures. Without that reference, it is hard to assess whether the proposed method has an effect on the performance achieved by the original architecture.\n    \n- The user study (Sec. 5.3) aims at evaluating the interpretabilty of a given concept by assessing whether users can retrieve meaning from visualizations. Here it is not clear how “meaning” is defined nor what protocol is followed to measure it.\n    \n\n- Due to the way the content is presented, the amount of conducted experiments and results reported; the level of detail provided in the content of the paper and depth of analysis of the results obtained in the conducted experiments is rather shallow. As indicated earlier, some parts of the manuscript are missing important details, other relevant parts of the manuscript have been delegated to the appendix. Perhaps a better structure must be adopted in order to ensure a proper depth is achieved in relevant areas.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The proposed method has been validated on top of a good variety of network architectures (CNNs and Tranformers) and, in some parts, compared w.r.t. various existing explanation methods.\n    \n- The proposed method seems to effectively ensure that the units used for producing the explanations, i.e. the concepts, do have a significant effect determining the outputs produced by the model.\n    \n- The explanations produced by the proposed method are extracted from different layers of the model, as such it enables nearly complete coverage of the architecture. This is in contrast to the common practice of existing efforts which produced explanations from features/concepts extracted from a very specific part/layer of the architecture.\n    \n- The validation of the proposed method includes both a quantitative, more machine-centered, evaluation as well as a user-study.\n    \n\n**Weaknesses**\n\n- In the formal presentation of the proposed method, see Notation paragraph (l.112-115), the meaning of several symbols (with a wide established meaning) have been modified. This gets in the way when following the equations and other formal details that follow.\n    \n- Even when the evaluation of the proposed method included several existing methods from their literature, their inclusion in the different conducted experiments is not consistent.\n    \n- Some parts are missing important details, other relevant parts of the manuscript have been delegated to the appendix.", "was_revised": true, "final_justification": "Thanks for the extra efforts invested on the clarification of my question regarding the concept-deletion curves. I think the obtained insight of the 4 shared concepts is interesting and intriguing. If space allows, I would encourage the authors to include part of that finding in the main body of the paper.\n\nThanks for the clarification on faithful visualization, please add some related text in body of th paper in order to ensure this \n\nRegarding the applicability of certain methods on specific metrics, I would argue that if the explanation provided by a method, e.g. the mentioned CRAFT, does not provide the means to assess its effect in the downstream prediction task, it might not be a valid competitor/baseline to compare against (as it is clearly impaired next to the proposed method). On the contrary, I would rather limit the comparison w.r.t. methods that allow for a complete comparison among different angles as defined by the considered metrics. In this regard\n\nRegarding the evaluation of explanation methods via user studies. I agree with its usage as a means to complement a proper objective quantitative evaluation, and not as a the sole means for validation, especially if no proper protocol is followed. As that seems to be the same principle under which the user study presented in paper is conducted, the procedure described in the rebuttal seems to be more than sufficient. Having said for future (user-study) experiments/submissions, please have consider the line of work under the umbrella of Human-centered Explainable Artificial Intelligence, as that is research building on work from the Human-Computer Interaction community and have significant expertise conducting user studies.\n\nRegarding concerns raised by other reviewers, considering the proposed method is bottom-up in nature, I do not find the lack of explicit human alignment as a strong weakness. I agree on that forcing alignment, as in classical CBMs, would steer the model and perhaps reduce its capabilities as a representation learning machine.\nI agree with Reviewer qa8G on that one-stage training from scratch would be a very desirable capability to have. This would not only remove the dependency on pretrained B-cos models, but would in general remove the issues introduced by cascaded training schemes.\n\nI thank the authors for their efforts in this rebuttal, in light of the above, I have opted to increase my initial rating.", "rebuttal": "We thank the reviewer V8ET for their constructive feedback. We are glad that they found the variety of our experiments and evaluation setup as a strength. In the following, we will further address the Weaknesses **(W1-3)**, Questions **(Q1-6)**, and Limitations **(L1-4)** raised. \n\n**Faithfulness of input-attributions (L1 & Q2):** One main limitation of existing concept-based methods (or similarly part-prototype networks) is that they struggle with correctly attributing the activation of a concept (or prototype) at late-layer to the image. These attributions are usually approximated with up-sampling similarity maps from the late-layer [10, 17, 46] or post-hoc attribution methods [1, 48] and have shown to be unfaithful visualizations [23, 32, 48]. In addition, such approximations are even worse in  ViT architectures, since the inductive (locality) bias of CNNs no longer holds. B-cos networks, however, provide model-inherent attributions that are faithful to the model in that **they are the actual model’s computation**. Precisely, for every input sample, they express the activation of the logit (or in our case concepts) as a linear mapping of the input, which summarizes the computation (See Eq. 3 for B-cos networks’s logits and Eq. 12 for FaCT’s concepts). This is independent of the layer of choice or the architecture. On top of the theoretical grounds, B-cos attributions have been shown to outperform other (approximate) attributions in terms of localization (see e.g. Fig. 6 in [9]). \n\nBy using B-cos transforms for layers in FaCT, we are able to retain all of these benefits when it comes to visualizing concept attributions allowing us to precisely demonstrate where in the image the concept is being activated for. We would like to highlight this further by referring to Fig. E2 and E3 in Appendix E. Notice that it is only with such high-resolution faithful attributions at input-level that one can retrieve the correct meaning of each of the shown concepts, e.g. ‘Red Dots’.\n\n**Different baselines across experiments (W2):** Not all baselines make sense to compare in the context of every metric, as the metrics are task-specific. We elaborate below:\n\n***a)** Accuracy (Fig 4):*  Post-hoc concept-based methods such as CRAFT do not directly use concepts for prediction, and hence one cannot measure “accuracy” for such methods. As a result, we compare against the B-cos model as a baseline, which is close to the accuracy of conventional models (see Table 1 below). For CRP the accuracy would correspond to the accuracy of the standard model, which as per your suggestion in (L2), we will add for easier reference.\n\n***b)** Concept consistency (C2-score) (Fig 3):* We evaluate this against all baselines, i.e. against channels of the B-cos model, as well as post-hoc methods such as CRAFT and CRP.\n\n***c)** Concept Deletion (Fig. 2):* This metric measures the accuracy of the importance value assigned to each concept, *given a fixed set of concepts*. Since each concept extraction method (FaCT, CRAFT, CRP, etc.) yields a distinct concept set, a direct comparison between them is not possible on this metric. As a result, we focus on comparing our approach of using B-cos attributions against alternative *concept attribution methods* used with these baselines, e.g. Sobol indices used in CRAFT and Saliency used in TCAV and VCC, and find that B-cos attributions achieve the sharpest drop, showing their effectiveness.\n\n***d)** User Study (Fig. 5):* The goal of this experiment was to complement the C2-Score gains observed for SAE latents compared to B-cos channels, under the same visualizations, with the same architecture, weights, layer, with both having a shared concept basis. . In addition, since we were working with sampled concepts, we opted for having more samples rather than many methods with very few samples from each.\n\n**User Study (Q3 & L3):** To the best of our knowledge, there is no pre-defined protocol for evaluating fine-grained visual concepts. We have put our emphasis on developing a fair and unbiased study by introducing a double-blind control scenario, where we randomly distributed the users to 10 groups, each with randomly ordered samples from both methods. Similarly, when studying the effectiveness of visualizations, we performed counter-balanced AB/BA testing, where top-activating images of each concept were shown with and without the input-attribution in two different groups (see Fig. D2 in the Appendix). Regarding the definition of an interpretable concept with `meaining’, as this is abstractly defined in the literature, we followed the common approach of providing the user with examples. In particular, we provided examples of low-level, high-level, object-centric, and non-object-centric concepts to the users (see Fig. D1). \n\n**Vanilla Baseline (L2):** We will add the standard baselines for the final version for easier comparison. For convenience, we have put a comparison table below, where we compare FaCT, B-cos, and the vanilla architecture. \n### Table 1: Comparison to Vanilla Baseline\n| Architecture | Vanilla Accuracy | B-cos Accuracy |  (FaCT@LastBlock) Accuracy  | Consistency Gain |\n|-|-|-|-|-|\n| ResNet50 | 80.9 | 79.5 | 77.3 | 0.38 → 0.49 |\n| DenseNet121 | 74.4* | 77.3 | 75.2 | 0.31 → 0.50 |\n| ViT_c-S | 75.8 | 74.5 | 73.2 | 0.35 → 0.51|\n\n\\* Trained with torchvision’s v1 recipe, as v2 is not available for DenseNets. (Other rows are trained with the v2 recipe.)\n\nWe observe a significant increase in the consistency (i.e C2-Score) of concepts, while remaining on-par on ImageNet accuracy across architectures.\n\n**Concept-Deletion Curves (Q1):**  We were indeed also curious about such a sharp drop for early layers (Block 2/4) seen in Fig. 2, as well as Fig. F1 in Appendix. For these early-layer experiments, we investigated the first few concepts that are removed, and found a very small subset of latents that occur on every image. This was primarily the case for early-layer experiments. In the case of Fig. 2, there are exactly 4 concepts that occur on 100% of the samples. We had also identified these while training the SAEs, discussed it in Appendix A (lines 675-682), and found similar phenomena in the existing literature (see points with frequency of 1 in Figure 3 of [31]). We hypothesize that these are mean feature vectors that are used for reconstruction in every sample. It is the removal of these latents that results in the sharp drops for early-blocks in Fig. 2. Nevertheless, as we use the same concept-set for evaluating the concept-importance methods, this still shows that B-cos is best in identifying the most impactful concepts.\n\nTo investigate the identification of most relevant concepts beyond these “always-on” shortcut concepts, we now investigated concept deletion when not allowing removal of these four latents and evaluated on the rest of the concept. We plotted the results similar to Fig.2 and found B-cos contributions to again consistently out-perform other attribution methods on concepts. Since we cannot upload images for the rebuttal, we report the Area Under the Deletion-Curve here in Table 2: (Lower means faster drop, hence better).\n\n### Table 2: Area under Deletion Curve for Block 2/4 on DenseNet-121\n| Method | Removing the 4 concepts is allowed (Fig. 2) | Removing the 4 concepts is not allowed |\n|-|-|-|\n| B-cos  | 0.9 | 24.0 |\n| Saliency | 56.7 | 73.7 |\n| Sobol (n=4) | 108.2 | 111.0 |\n| Random | 98.9 | 116.2 |\n\nWe observe that the sharp drop disappears once we do not allow the removal of these four concepts for any method, yet we observe that B-cos contributions still significantly outperform other concept-importance measures. We thank the reviewer for raising this point. We will make the discussion around always-occurring concepts at early layers (lines 675-682 in Appendix A) more prominent for the final version, while also adding this new experiment which excludes removal of these concepts.\n\n**Notation in Eq. 14 (Q3.5):** Thanks for pointing this out. Indeed, in Eq (14), $f(I)$ should be replaced by $H(I)$, which refers to the foundation model’s features. We will correct this for the revision.\n\n**Random Concept in C2-Score (Q4):** The random concept activates on every image and assigns random importance to each pixel (sampled uniformly from  within [0, 1]). In a multi-class setup, the consistency (Eq. 16) of random concept was at 0.004 ($\\sigma$ 0.0005), averaged across three seeds, while on a per-class setup it was 0.13 ($\\sigma$ 0.021), averaged across the 1000 classes. This agrees with the original assumption that when the evaluation set $\\mathcal{D}$ is restricted to a single category, all the output features of DiNOv2 become more similar. This motivated us to define C2-score as the difference w.r.t the random baseline.\n\n**Spatial Extent (Q5):** The spatial extent is the average number of pixels required to cover 80% of the total positive attribution of a concept. This was defined so that we can demonstrate the variety of concepts, as opposed to prior work that often assumes concepts to fit in small patches [17, 33]. We have defined the spatial extent in Appendix E (Lines 786-792) and will make sure to make it more prominent in the revision.\n\n**Spatial Extent of ViTs (Q6):** We believe the similarity in spatial extent across two layers is due to the full-image receptive field at all the layers of ViT. Following your suggestion, we will run an experiment at a very early-layer and extend the plot for the revision.\n\n**Clarity and Notation (W1, W3, L4):** We will take all of the questions raised by the reviewer into account and revise for clarity. Regarding the notation, while we were trying to closely follow the notation in B-cos paper [9], we will make sure to adjust it to be more intuitive. We will also highlight references for the Appendix sections more clearly. We would be grateful if the reviewer could specifically mention what they find missing so that we revise for the revision."}, {"review_id": "p3fCWy9CHx", "reviewer": "Reviewer_Lkv1", "summary": "The authors introduces FaCT, a model that makes neural network decisions easier to understand by breaking them down into concepts, like \"yellow\" or \"wheel\", that can be traced across all layers of the network. Different from prior art which don't truly reflect the model's internal reasoning, FaCT ensures that these explanations are true to how the model works and can be visualized clearly. The core idea is to use B-cos transformations which preserves linear contribution additivity and sparse autoencoders (SAEs) to present human-interpretable concept vectors. The paper also introduces a new metric called C2-Score (concept consistency score), which leverages DINOv2 features to quantify how consistent a concept is across different images. Experiments show the proposed method learned concepts more semantically consistent and human interpretable than the prior art.", "questions": "The paper would be stronger if the authors can address the following:\n- While the paper shows how to visualize concepts, it doesn't resolve how to name or interpret each concept as some concepts are not easily aligned with human concept.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n- The approach is novel in achieving complete, faithful decomposition of predictions into concepts across layers. Prior concept models\noften provided only approximate importance of concepts to outputs, or needed additional tools to visualize concepts. FaCT’s design ensures by construction that all attributions are faithful and additive.\n- FaCT’s concepts are shared among all classes and across layers, which results in a more compact and general explanation basis. A concept like “wheel” can contribute to recognizing buses, cars, etc. \n- High accuracy: FaCT achieves competitive performance on ImageNet, which shows it doesn't compromise much predictive power to obtain interpretability.\n- Comprehensive Evaluation, including a C2-score quantitative metric, a user study, and qualitative examples.\n\nWeakness\n- FaCT’s concepts are more consistent and interpretable than baselines but they are not necessarily align with human-defined categories. This at the end still limit how easily users can label the concepts. \n- The metric C2-score relies on DINOV2, which could introduce errors if not reliable in all cases.\n- It may need to test on more datasets to ensure it works broadly\n- Lack of discussion on computational overhead", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer Lkv1 for their detailed feedback. We are happy to see that the reviewer finds our approach and our evaluation novel and comprehensive. In the following, we will further address the Weaknesses **(W1-4)**, Questions **(Q1)** raised by the reviewer. \n\n**Human-alignment of Concepts (W1 & Q1):** There is an inherent difference between human-aligned models, such as CBMs, that are forced to learn a mapping to pre-defined (human) concepts and conventional models, that learn concepts that they deem useful, which do not need to correspond to what we as humans would like to see or even comprehend. The advantage of the latter is obvious in terms of performance, achieving much higher accuracy than e.g. CBMs at the cost of being less transparent. With FaCT, we aim at making the **concepts that the network learned** for decision-making transparent, which **do not need to align with pre-defined human concepts**. This allows us to maintain the performance of powerful models while allowing to interpret their reasoning. Here, a visualization of concepts will give a much better understanding of what is happening than text, given that we work on a vision task, which was also a key motivation for ProtoPNet, PipNet and others.\nRegardless, we can use off-the-shelf methods to label our concepts such as CLIP-Dissect [R1] which name individual neurons using a language-aligned model (i.e CLIP). Following your suggestion, we thus applied CLIP-Dissect [R1] to our SAE latents, and got the following names for the concepts (A-F) shown in Fig. 6 of the paper: (for each concept we show the top-3 matches):\n\n\n[A]: volleyball, rugby, balls\n\n[B]: jerseys, uniforms, players\n\n[C]: rugby, sergio, concentration\n\n[D]: gymnastics, acrobat, flexible\n\n[E]: volleyball, tournaments, serve\n\n[F]: basketball, layup, contested\n\nWe find these names to be very consistent with how we would name the concepts in Fig. 6. We thank the reviewer for their suggestion and will happily add this discussion for the camera-ready.\n\n\n\n**Use of DiNOv2 for C2-Score (W2):** Indeed, output features of DiNOv2, or any other foundation model used within our C2-Score metric, should not be considered as the oracle or ground-truth semantic space. The underlying motivation of using DiNOv2 features is two fold: First, DiNOv2 has recently shown remarkable performance for similar tasks such as semantic correspondence [3, 52, 53], which makes it suitable for the context of evaluating consistency of concepts across images. Second, we intentionally steered away from evaluating the concepts with respect to human annotations, as the annotations are class-specific, subject to arbitrarily chosen granularity level, and are independent of what a model might learn to be useful for a decision.\nWe have demonstrated this extensively in Fig C1 in  Appendix, where we show sample concepts that are not considered in Part-ImageNet annotations, while even the PCA visualization of DiNOv2 features seem to be better suited. We will make this comparison more prominent for the final version. Appreciating that DiNO is not the ground truth, we had complemented our analysis with a human user study to evaluate found concepts. We will further highlight this complementarity of evaluations and their individual limitations in the discussion.\n\n**Extending to More Datasets (W3):** We mainly experimented with ImageNet as it is a realistic dataset covering a wide range of different classes and diverse images, which many existing interpretability works such as ProtoPNet or PipNet fail to scale to. Following your suggestion, we evaluated FaCT also on the (uncropped) CUB dataset for ResNet-34 architecture. Similar to ImageNet experiments, we started with a pretrained B-cos checkpoint (ImageNet B-cos finetuned on CUB) and trained our SAEs at different layers, having the rest of the model frozen. The results can be seen in the table below.\n\n### Table 3: Performance for ResNet34 on uncropped CUB dataset\n### (Standard: 76.0 from [13])\n| Method | Accuracy | Consistency Gain (C2-Score) |\n| -  | -  | -  |\n| B-cos   | 78.8     | -   |\n| FaCT @Block 2/4     | 78.6     | 0.27 → 0.38   |\n| FaCT @Block 3/4     | 78.0     | 0.33 → 0.51   |\n| FaCT @Block 4/4     | 78.5     | 0.49 → 0.63   |\n\nWe observe that FaCT works out of the box on smaller-scale datasets such as CUB and is able to maintain the accuracy (<0.8 % drop across), while training much faster than e.g. ProtoPNets. We will add the CUB results together with qualitative visualizations for the revision.\n\n\n**Discussing Computational Overhead (W4):** Following your suggestion, we measured the inference time over the entire ImageNet validation (50,000 Images). We did this for DenseNet-121 architecture on an A40 GPUs and averaged the results across three runs. \n### Table 4: Time to process ImageNet’s Test set on DenseNet-121 (average across three runs)\n| Method            | Time (seconds) |\n| - | - |\n| B-cos    | 104 |\n| FaCT @Block 2/4  | 112  |\n| FaCT @Block 3/4  | 108  |\n| FaCT @Block 4/4  | 112  |\n\nWe observe only a negligible increase in inference time, which is likely to reduce further with inference optimization (e.g. changing the SAE-hooks to fixed layers and using torch.compile). We will add these results for the final version.\n\n---\n**[R1]:** CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks, ICLR 2023"}, {"review_id": "rPwAHVsSdM", "reviewer": "Reviewer_RaLA", "summary": "This paper introduces a concept-based model, FaCT, that aims to ensure faithful concept attribution both to the output logits and the input image space. The model integrates B-cos networks and sparse autoencoders (SAEs) across intermediate layers to learn concept representations that are actively used in decision-making. Furthermore, the authors propose a new evaluation metric, C²-score, for concept consistency using DINOv2 features. The proposed FaCT model is validated through quantitative metrics, including ImageNet accuracy, concept deletion, C²-score, qualitative visualizations, and a user study.", "questions": "1. How sensitive is the concept decomposition to SAE hyperparameters?\n2. Could FaCT be applied to other modalities (e.g., text)?\n3. Can your concept contributions be used for counterfactual generation or concept editing?", "limitations": "Yes, the authors discussed the limitations in the Appendix. However, it would be better to move some of the discussions to the main text.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. The proposed idea is technically sound and novel.\n2. In contrast to post hoc methods, the proposed model inherently ensures the learned concepts are faithful to the decision-making process.\n3. The proposed method is generalizable to different architectures, including CNN and ViT.\n4. User study is provided to evaluate the actual quality of interpretations.\n5. The paper is well-organized and easy to follow.\n\n**Weakness:**\n1. The learned concepts still lack human explicit human-grounded semantics. The labeling of the concepts rely on input feature attribution or image retrieval, which is subjective to the human annotator and not scalable.\n2. The evaluations are primarily focused on ImageNet, which is mismatched with the concept-level interpretations offered by the proposed method. Evaluations on concept-level annotations, such CUB, Part-ImageNet, and PASCAL-part, will benefit this paper.\n3. Missing a comprehensive ablation study on the hyperparameters. For example, the effect of SAE sparsity (top-k) and expansion factor (dictionary size) need to be studied systematically.", "was_revised": true, "final_justification": "Thank you to the authors for their thorough and detailed responses to my questions. After reviewing the authors' responses and considering the feedback from other reviewers, I am inclined to accept this paper.", "rebuttal": "We would like to begin by thanking reviewer RaLA for their detailed feedback and questions. We are happy to see the reviewer finds our idea novel, our method generalizable, and our writing easy to follow. In the following, we will address the Weaknesses **(W1-3)**, Questions **(Q1-3)** raised by the reviewer. \n\n\n**Human-alignment and Scalable Labeling (W1):** Please note that there is an inherent difference between explicit human-aligned models, such as CBMs, that are forced to learn a mapping to pre-defined concepts and conventional models, that learn concepts that they deem useful, which need not correspond to what we as humans would like to see or even comprehend. The advantage of the latter is obvious in terms of performance, achieving much higher accuracy than e.g. CBMs at the cost of being less transparent. With FaCT, we aim at making the **concepts that the network learned** for decision-making transparent, which **do not need to align with pre-defined human concepts**. This allows us to maintain the performance of powerful models while allowing to interpret their reasoning. Here, we believe a visualization of concepts will provide a better understanding of what is happening than text, given that we work on a vision task, which was also a key motivation for ProtoPNet, PipNet and others.\nNonetheless, if textual interpretations are preferred, we can use off-the-shelf methods to label our concepts such as CLIP-Dissect [R1] which name individual neurons using a language-aligned model (i.e CLIP). To satisfy our curiosity, we applied CLIP-Dissect [R1] to our SAE latents, and got the following names for the concepts (A-F) shown in Fig. 6 of the paper: (for each concept we show the top-3 matches):\n\n\n[A]: volleyball, rugby, balls\n\n\n[B]: jerseys, uniforms, players\n\n\n[C]: rugby, sergio, concentration\n\n\n[D]: gymnastics, acrobat, flexible\n\n\n[E]: volleyball, tournaments, serve\n\n\n[F]: basketball, layup, contested\n\n\nThese human-understandable textual names seem meaningful with respect to the concepts in Fig. 6. While not our primary focus, we will add this downstream application for textual explanations to our paper, thank you for raising this point.\n\n\n**Evaluation on other Datasets (W2):** We primarily avoided using fixed human annotations to evaluate our shared concept basis as the annotations are class-specific, subject to arbitrarily chosen granularity level, and are independent of what a model might learn to be useful for a decision. We would like to refer to Fig. C1 in  Appendix, where we show sample concepts that are not considered in Part-ImageNet annotations, while even the PCA visualization of DiNOv2 features seem to be better suited. We will make this comparison and discussion more prominent for the final version. We have also extended our evaluation to the CUB dataset, to ensure that our findings also hold on smaller-scale, but fine-grained classification (kindly refer to Table 3 in response to reviewer Lkv1).\n\n**Ablation on Hyper-parameters of SAE (W3):** We have described our hyper-parameter sweep and model selection criteria in (Appendix A, Lines 669-684). We have also plotted the models in terms of sparsity vs. accuracy in Fig. B1, where each point corresponds to a particular (top-k, expansion factor, and learning-rate) configuration. We had deferred these results to the Appendix in lack of space. We will make the references more prominent for the revision. \n\n\n**Sensitivity to Hyper-parameters of SAE (Q1):** In general for the same layer we observed many concepts being repeated across configurations as well as across architectures at similar depth. Following your question, we evaluated the recently proposed Stability Score [R4] for the same layer and dictionary size, but with different (top-k) sparsity factor. We did this both for early and late-layer decompositions of Densenet-121. While being trained on different models, datasets, and layers, our stability scores are quite on par with the ones in Table 1 of [R4], where the authors report 0.5 for the TopK-SAE under different seeds. We would also like to point out that FaCT would directly benefit from new variants of SAEs, e.g. [R4], that may be introduced by the community. We will extend the quantitative stability evaluation for the final revision.\n### Pair-wise Stability Score \n### Averaged over three pairs of experiments (TopK ∈ 8, 16, 32)\n| Method  | Stability Score |\n| -  | -  |\n| FaCT @Block 2/4  | 0.76  |\n| FaCT @Block 4/4  | 0.70  |\n\n\n\n**Applying FaCT to other Modalities (Q2):** The main components of FaCT are the use of B-cos layers (for faithful input-attribution and output-contributions) together with SAE for interpretable concept-based representation. Both of these have been recently explored for other domains such as natural-language processing. In particular, B-cos LM [R2], introduces a B-cos language encoder and evaluates the interpretability and faithfulness of attributions on the input sequences. Additionally SAEs [6, 11] were originally introduced for language models. We thus believe FaCT would lend itself well for other modalities and will add this to the future work discussion for the revision.\n\n\n\n**Concept-Editing (Q3):** Thank you for this interesting suggestion. Indeed one of the advantages of FaCT is that it uses concepts *as part of the decision-making process*, and hence modification to the concept representation can be explored and removing concepts does indeed affect the output behaviour of the model (e.g. Fig. 2). While B-cos models have shown to lend themselves well for being guided through their attributions, e.g. to avoid relying of spurious features (see Fig. 1 in [R3]), FaCT directly allows for steering the model *on a concept-level* (e.g. by explicitly masking certain SAE latents). We will add this as a future work direction in our revision.\n\n\nWe will also move the limitations section to the main text in our revision.\n\n---\n**[R1]:** CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks, ICLR 2023\n\n**[R2]:** B-cos LM: Efficiently Transforming Pre-trained Language Models for Improved Explainability, preprint Feb 2025\n\n**[R3]:**  Studying How to Efficiently and Effectively Guide Models with Explanations, ICCV 2023\n\n**[R4]:** Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models, ICML 2025"}, {"review_id": "8XxtCsWUIC", "reviewer": "Reviewer_qa8G", "summary": "This study aims to understand the behavior of deep learning models at a global concept level, and unlike previous post-hoc concept-based approaches, it proposes an inherently interpretable model architecture that enables faithful explanations of model decisions. In addition, it introduces a novel evaluation metric called C²-score, designed to fairly compare different concept-based methods in a class-agnostic and annotation-free manner.", "questions": "- The proposed method builds on a pretrained and frozen B-cos network, with sparse autoencoders trained separately. Would it be possible to train the full FaCT architecture from scratch instead of relying on a pretrained B-cos network? Specifically, to obtain meaningful concept representations, could the model first be trained without the SAE modules during the initial epochs, and then continue training with the SAEs integrated, potentially enabling a single-stage training process instead of a strictly two-stage setup? Have the authors explored joint or progressive training strategies to this end?\n- While the authors report that FaCT maintains performance competitive with B-cos, it is worth noting that B-cos networks themselves incur a 2–3% drop in accuracy compared to original models. Given that FaCT introduces an additional drop of up to 3%, the overall performance gap from the original architecture may approach 6%. This is a non-trivial trade-off in accuracy. Could the authors comment on how they justify this gap in practical scenarios, particularly when applied to real-world tasks?\n- Finally, the proposed C²-score quantifies concept consistency as the difference between the consistency of a given concept and the consistency of a random concept. However, since the consistency of a random concept may vary across runs, would it not be more robust to compute the C²-score over multiple random seeds and report the mean along with its standard deviation? Have the authors examined the sensitivity of the C²-score to the randomness?", "limitations": "- The semantic interpretation of each concept ultimately relies on human judgment, as the model does not assign explicit labels to the learned concepts.\n- The method is only evaluated on a single dataset (ImageNet), limiting the assessment of its generalizability.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n- This paper proposes a concept-based explanation method that is inherently interpretable model. Since concepts are explicitly used in the forward pass, model predictions are directly computed based on concept representations.\n- The proposed approach is generalizable to both CNN and ViT architectures.\n- The method achieves competitive performance on the ImageNet classification task while maintaining strong explainability compared to baseline models.\n- The paper introduces a novel evaluation metric, C²-score, which can evaluate various concept-based methods without requiring additional annotations.\n\nWeaknesses\n- Although the model provides faithful visualizations of concepts, their semantic interpretation still requires human inspection, as the model does not assign explicit concept labels.\n- The proposed method builds on a pretrained B-cos network with frozen weights, and introduces an additional training stage by separately learning sparse autoencoders, thereby breaking the end-to-end training pipeline.\n- All experiments are conducted solely on ImageNet. Evaluations on other datasets would strengthen the claim of generalizability.\n- As an inherently interpretable approach, the method should be compared with other inherently interpretable baselines such as Concept Bottleneck Models or Part-Prototype Networks.", "was_revised": true, "final_justification": "The authors have addressed most of the concerns. I think the paper has merits for the community.", "rebuttal": "We thank the reviewer qa8G for their in-depth feedback and questions. We are very happy to see that the reviewer finds our approach generalizable, our model competitive and our metric novel. In the following, we will discuss the Weaknesses **(W1-4)**, Questions **(Q1-3)**, and Limitations **(L1-2)** raised by the reviewer. \n\n**Interpreting the Concepts (W1, L1):** We intentionally avoided using pre-defined concept sets that would bias the interpretation towards what we as humans expect the model should learn, and instead used an unsupervised concept-discovery method (SAE) for identifying the concepts the model finds useful. With FaCT, we aim at making the *concepts that the network learned* for decision-making transparent, which **do not need to align with pre-defined human concepts**. This allows us to maintain the performance of powerful models while allowing to interpret their reasoning. That said, we agree that having explicit names assigned to the concepts can benefit the interpretability for the end user, and have therefore followed this suggestion and used CLIP-Dissect [R1], which labels neurons (SAE latents in our case) using a CLIP model. Applying CLIP-Dissect on our concepts to generate the names would name the the concepts (A-F) shown in Fig. 6 to the following words (for each concept we show the top-3 matches):\n\n\n[A]: volleyball, rugby, balls\n\n\n[B]: jerseys, uniforms, players\n\n\n[C]: rugby, sergio, concentration\n\n\n[D]: gymnastics, acrobat, flexible\n\n\n[E]: volleyball, tournaments, serve\n\n\n[F]: basketball, layup, contested\n\nWe find these names to be quite consistent with how we would interpret these concepts. We will add this discussion to the final revision.\n\n**Using a pre-trained network (W2, Q1):** We thank the reviewer for raising this point. There are two main reasons why we focused on training FaCT with pre-trained B-cos checkpoints. \n\nFirst, our current two-step procedure for training FaCT allows it to be readily applied to existing trained B-cos models, which is particularly important given the recent focus on extending B-cos networks to different domains such as Natural Language Processing [R2] and Self-supervised models [R5].\n\nSecond, SAEs have been mainly proposed as an unsupervised concept-discovery method to be applied on top of a structured (trained) representation, with the sparsity constraints being the only prior guiding the model towards identifying concepts (i.e clusters). We thus suspect that training them on top of an unstructured representation may add to their instability or lead to less-interpretable latents.  \n\nWe do however agree that having the choice of one-stage training from scratch can be desirable in certain cases, and will add this in the future work discussion.\n\n\n**Other Datasets (W3, L2):** We thank the reviewer for their suggestion. We mainly experimented with ImageNet to ensure the competence of FaCT. Following your suggestion, we evaluated FaCT also on the (not-cropped) CUB dataset for ResNet-34 architecture. Similar to ImageNet experiments, we started with a pretrained B-cos checkpoint (ImageNet B-cos finetuned on CUB) and trained our SAEs at different layers, having the rest of the model frozen. The results can be seen in the table below.\n\n\n### Table 5: Performance for ResNet34 on uncropped CUB dataset\n### (Standard: 76.0 from [13])\n| Method | Accuracy | Consistency Gain (C2-Score) |\n| -  | -  | -  |\n| B-cos   | 78.8     | -   |\n| FaCT @Block 2/4     | 78.6     | 0.27 → 0.38   |\n| FaCT @Block 3/4     | 78.0     | 0.33 → 0.51   |\n| FaCT @Block 4/4     | 78.5     | 0.49 → 0.63   |\n\n\nWe observe that FaCT works out of the box on smaller-scale datasets such as CUB and is able to maintain the accuracy (<0.8 % drop across). We thank the reviewer for their suggestion and will add the CUB experiments, together with qualitative results of the shared concepts for the final version.\n\n**Comparison to ProtoPNets and CBMs (W4):** As our main focus was large-scale ImageNet evaluations, we mainly compared with models that extend to ImageNet, such as B-cos channels and post-hoc methods such as CRAFT and CRP. Existing implementations of Part-Prototype often do not extend to ImageNet. In the two tables below, we compare a set of ProtoPNets and CBMs that had a comparable setup to ours.\n\n### Table 6: ResNet34 comparison on CUB and ImageNet\n| Method | CUB Accuracy | ImageNet Accuracy |\n| - | - | - |\n| vanilla  | 76.0*  | 73.3   |\n| ProtoPNet   | 72.4* | 65.5**  |\n| Deformable ProtoPNet [13]  |  76.8*  | 63.4** |\n| B-cos | 78.8 | 72.3 |\n| FaCT @Block 2/4 | 78.6 | 71.3 |\n| FaCT @Block 3/4 | 78.0 | 70.4 |\n| FaCT @Block 4/4 | 78.5 | 71.0 |\n\n *Numbers from [13], trained on uncropped CUB images ; **Numbers from [46]\n\n### Table 7: ResNet50 comparison on ImageNet\n\n| Method             | ImageNet Accuracy |\n| -                  | -                 |\n| LF-CBM[34]             | 72.0***           |\n| Torchvision V1     | 76.1              |\n| Torchvision V2     | 80.2              |\n| B-cos              | 79.5              |\n| FaCT @Block 2/4    | 77.6              |\n| FaCT @Block 3/4    | 77.1              |\n| FaCT @Block 4/4    | 77.3              |\n\n***Numbers from [34], using Torchvision V1 recipe\n\nBesides the significant performance advantage on CUB and ImageNet, FaCT offers a shared concept-basis, extends to different layers, and architectures (CNNs and ViTs), all while providing a faithful input-level attribution of concepts.\n\n\n**Performance Margin to Vanilla Models (Q2):** We would like to highlight that the recent V2 version of B-cos models [9] that we use obtain much closer performance to vanilla models. (e.g. only 0.8% drop on DenseNet-121). We would like to point to Table 5, 6, and 7 above. In all the three tables we observe that FaCT models are performing on par to standard baseline (Maximum 3.3% drop across all tables). We also observe that on a smaller-scale (CUB) dataset presented in Table 6, the performance gap compared to B-cos is even smaller (<0.8%). In the revision, we will add the vanilla baseline numbers to the comparisons for clarity. \n\n**Random Baseline in C2-Score (Q3):** We observed consistent results for the random baseline across runs. In particular, for multi-class evaluations, the consistency score of the random baseline was at 0.004 ($\\sigma$=0.0005), averaged across three seeds. For the per-class setup, the random baseline score was 0.13 ($\\sigma$=0.021), averaged across the random baselines of all 1000 classes. We will add these for the final revision.\n\n---\n**[R1]:** CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks, ICLR 2023\n\n**[R2]:** B-cos LM: Efficiently Transforming Pre-trained Language Models for Improved Explainability, preprint Feb 2025\n\n**[R5]:** How to Probe: Simple Yet Effective Techniques for Improving Post-hoc Explanations, ICLR 2024"}]

Author rebuttal:

Meta-review: This paper introduces FaCT, an inherently interpretable concept-based method for explaining neural network decisions. Unlike previous methods, FaCT breaks down predictions into clear and faithful concepts across all layers of the network. The proposed model integrates sparse autoencoders and B-cos networks to ensure these concepts truly reflect the model’s reasoning. The authors also propose a new metric, C^2-score, to measure how consistent concepts are across images. They validated their approach using quantitative experiments on ImageNet, qualitative visualizations, and a user study.

Reviewers praised the clear ideas, strong experiments, and good writing of the paper. The method is applicable to different types of networks and maintains competitive accuracy while improving interpretability. Some concerns were raised by reviewers in the initial reviews, like how easy it is to label concepts, limited dataset evaluation, and comparisons with other baselines, but the authors addressed most concerns well. All reviewers have positive ratings for this paper. Overall, this paper is an important step forward in making deep learning more explainable, and is recommended for acceptance.
Decision: Accept (poster)