FACE: Faithful Automatic Concept Extraction
Authors: Dipkamal Bhusal
Michael Clifford
Sara Rampazzi
Nidhi Rastogi
Venue: NeurIPS 2025 poster ()
Keywords: computer vision
faithful interpretability
concept based explanations
interpretability
explainable AI
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We propose FACE, a concept-based explanation framework that ensures faithful alignment between model predictions and concept-based representations by combining Non-negative Matrix Factorization with KL divergence regularization.

Abstract: Interpreting deep neural networks through concept-based explanations offers a bridge between low-level features and high-level human-understandable semantics. However, existing automatic concept discovery methods often fail to align these extracted concepts with the model’s true decision-making process, thereby compromising explanation faithfulness. In this work, we propose FACE (Faithful Automatic Concept Extraction), a novel framework that combines Non-negative Matrix Factorization (NMF) with a Kullback-Leibler (KL) divergence regularization term to ensure alignment between the model’s original and concept-based predictions. Unlike prior methods that operate solely on encoder activations, FACE incorporates classifier supervision during concept learning, enforcing predictive consistency and enabling faithful explanations. We provide theoretical guarantees showing that minimizing the KL divergence bounds the deviation in predictive distributions, thereby promoting faithful local linearity in the learned concept space. Systematic evaluations on ImageNet, COCO, and CelebA datasets demonstrate that FACE outperforms existing methods across faithfulness and sparsity metrics.

FACE: Faithful Automatic Concept Extraction
Dipkamal Bhusal Rochester Institute of Technology Rochester, NY db17028rit.edi
Michael Clifford Toyota InfoTech Labs Mountain View, CA
mi cicchel..l.iffoodttoooooooo
Sara Rampazzi University of Florida Gainesville, FL srampazzi@ufl. sufzzeiffff....
Nidhi Rastogi
Rochester Institute of Technology Rochester, NY nxrvsedrit.cmm edu
Abstract
Interpreting deep neural networks through concept-based explanations offers bridge between low-level features and high-level suman-understandabl seman- ties. However, existing automatic concept discovery methods often fail to align these extracted concepts with the model's true becision-making process, thereby compromising explanation faithfulness. In this work, we propose FACE (Faithful Automatic Concept Extraction), a novel framework that augments Non-negative Matrix Factorization (NMF) with Kullback-Leibler (KL) divergence regulariza- tion term to ensure alignment between the model's original and concept-based predictions. Unlike prior methods that operate solely on encoder activations, FACE incorporates classifier supervision during concept learning, enforcing predictive consistency and enabling faithful explanations. We provide theoretical guarantees showing that minimizing the KL divergence bounds the deviation in predictive distributions, thereby promoting faithful local linearity the learned concept space. Systematic evaluations on ImageNet, COCO, and CelebA datasets demonstrate that FACE outperforms existing methods across faithfulness and sparsity metrics.
Introduction
Interpreting the decisions made by deep learning models is essential for understanding their behavior, diagnosing biases, correcting failed models, and fostering user trust. Among a wide array of explainable AI (XAI) methods proposed in recent years, feature attribution methods dominate the
30 35 These methods assign importance scores to individual input features (e.g., pixels in images), based their influence on the model's output. However, such fine-grained attributions fail to deliver semantic interpretability as they do not clarify how specific high-level concepts drive model's decision [16]. For instance, in animal image classifiers, assigning importance to isolated pixels reveals little about the body parts or visual cues the model actually uses, making it difficult for users to interpret decisions assess failure modes mcaningfully 6 23 29)
current practice 3 10
31
Concept-based explanation methods have recently emerged to address the interpretability shortcoming of feature attribution methods [8] 12 16
34 36] These methods explain model predictions using cannrninterprettbble concepts (e.g., texture, part, color), enabling a more intuitive understanding of the model's is reasoning process. For instance, rather than attributing importance to specific pixels. concept-ban based explanation might reveal that the model relies on the presence of 'fur' or 'ears' in an animal image classifier (see Figure TT..
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

CRAFT
ICE
FACE
C1
C2
Figure 1: Comparing concepts extracted by CRAFT 8). ICE [6], and FACE from rabbit images classified by ResNet-34 [14]. C1 and C2 correspond to the top two Sobol-importance concepts for each method. FACE achieves higher faithfulness compared to CRAFT and ICE (discussed in Section 4.2) demonstrating that FACE's extracted concepts better align with the model's true reasoning.
However, existing concept-based methods face significant limitations. Early methods like TCAV [I6 require a manually-labeled dataset for extracting concepts, greatly limiting scalability and practical applicability. Unsupervised methods such ACE [12]. ICE [36), and CRAFT S automate the concept discovery using unsupervised clustering or matrix factorization of encoder activations. Non- negative matrix factorization (NMF) [18) based approaches like CRAFT [8), in particular, have been shown to produce spatially coherent and interpretable concepts [9]. However, all these methods focus solely reconstructing the latent representation and do not account for the behavior of the model's s downstream classifier. As a result, the explanations may appear interpretable and yet misinterpret the original model's true reasoning, resulting in misleading insights.
Figure D Iillustrates this misalignment: CRAFT X attributes the prediction of rabbit to the topmost concept (C1) 'head', while our method, FACE, highlights 'fur (body)' as the most important concept (CI). Although 'head' may be more intuitive to humans, it is fundamentally flawed to assume that the neural network relies on human-aligned features (11.
Thus, faithfulness- how well the explanations align with the actual decision moaionn process of the model- is a critical and often under-exanie examined requirement in concept-based based methods. Moreover, as shown in the identifiability result of Locatello al. [2]). discovering meaningful latent concepts in an unsupervised setting is fundamentally ill- ill-posed without suitable inductive biases. In concept discovery, enforcing consistency between concept- based representations and the model's predictions provides such an inductive bias, helping to disambiguate useful representations from spurious ones.
Our solution: To address these challenges, we propose FACE (Faithful Automatic Concept Extraction), a novel NMF-based concept discovery framework that enforces alignment between concept-based activations and model predictions. FACE augments standard NMF by introducing KL divergence constraint that minimizes the discrepancy between the model's 's predictions on the original and reconstructed activations. This additional regularization ensures that the discovered concepts do not merely compress high-variance directions in latent space, but also closely align with the model's true predictive reasoning. By supervising the factorization process using the model's own predictions, FACE yields explanations that are both semantically interpretable and faithful-cappe capturing the actual features used by the model in making its predictions. We provide theoretical justification for our method in Section 3.3 and empirically validate its effectiveness on ImageNet [7]. COCO [99
and CelebA 20 in Section
Our code is available at
2 Related work
Kim et al. [16] introduced Testing with Concept Activation Vectors (TCAV) explain neural network predictions using human-defined concepts by training a linear classifier activation vectors of specific concepts and random counterexamples. However, TCAV relies on manually curated concept datasets, limiting scalability and introducing human bias. ACE [12 addresses this by segmenting images into superpixels (e.g., via SLIC (I) and clustering segment-level activations to discover concepts automatically. While ACE removes manual concept curation, it often introduces artifacts in concepts and may not align with model's 's actual decision-making ICE (36) replaces clustering with Non- non-nieative matrix factorization (NMF) [18.. offering cleaner concepts but operates at the

---

level of convolutional kernels, discovering localized concepts only. CRAFT 8 improves on ICE by introducing a recursive factorization pipeline that decomposes high-level concepts into sub- concepts. It applies NMF to cropped image activations and proposes a concept scoring method based on Sobol indices 32 While CRAFT improves interpretability and introduces principled way to score concept relevance, it still operates purely on the encoder activations, and the NMF decomposition does not enforce predictive consistency through the classifier. Other recent works explore instance-level concepts using SAM [44 or study concept composition [33]. In parallel to these reconstruction-batt (CRP) and Relevance Maximization (RelMax). Unlike ACE, ICE, or CRAFT, RelMax does not decompose activations but instead selects representative training examples for naturally emerging neurons features by maximizing their relevance during inference. In contrast to these works, our proposed method, FACE, introduces a KL-regularized NMF framework that explicitly aligns discovered concepts with the classifier's predictive behavior. By supervising concept extraction with model predictions, FACE ensures that discovered concepts remain faithful to the model, an aspect overlooked in prior NMF-based approaches.
methods, Achtibat et al. 2 proposed Concept Relevance Propagation
3 Methodology
3.1 Setup
Consider a standard supervised learning setup where classifier f : X' + y maps inputs from an input space X' C 12d to output predictions in y C RR. Without loss of generality, we assume that f can be decomposed into two components: an encoder g X + G and a classifier head h : - Y where G S IRP is an intermediate latent space. The encoder maps inputs to the high-dimensional latent representation, while h maps the latent representations to the output logits, or scores. Consequently, fxx = hxg())). Given a dataset of n input samples X X1 x., E RR" we denote their corresponding latent representations as A g RaXp
Automatic concept discovery methods operate on these latent representations A, typically for a set of inputs from the same class. To identify human-interpretabbe concepts, methods such as clustering [12), or Non-negative matrix factorization (NMF) (8 36 is applied on A to discover a set of interpretable concept vectors. Here, the value of k is typically pre-specified as the number of clusters for seasttrrinn-bseed approaches or the rank of matrix decomposition for matrix factorization based approaches. Once the concepts are discovered, their importance to the prediction is quantified using sensitivity-basse methods such as TCAV 16 or variance decomposition methods such as Sobol indices [8]
In this work, we adopt the NMF-based pipeline with Sobol-based concept importance g (see
Appendix M for discussion Sobol index) due to its demonstrated effectiveness in extracting interpretable and spatially coherent concepts g 9.
3.2 Desiderata of explanations
The goal of explanations is to help end-users understand how neural network makes decision. This information needs to be semantically interpretable so that it's easily understood by the users, however, it should also remain faithful to the underlying model. Below, we outline two important desiderata for explanations:
A critical requirement of any explanation method is that it accurately reflects the model's decision-making process. An explanation is considered faithful if the features or concepts highlights are indeed those that the model uses to make its predictions. However, human intuitions are often poor proxies for model behavior [Л. which motivates the use of evaluation metrics grounded in model performance. We adopt two perturbation-basee metrics from feature attribution literature 225 27 Concept Deletion (C-Del) measures the drop in model accuracy as the most important concepts are progressively removed from the latent representation. A faithful method will
High Faithfulness:
'While FACE is formulated using the penultimate layer to capture high-level, semantically coherent concepts, it is not restricted this layer. For any intermediate layer we simply have to define Ige as the network up to that layer and h as the remaining layers. We focus on the penultimate layer to align with prior works and to target class-level concepts empirically, earlier layers correspond to texture- or edge-like features, whereas deeper layers capture localized, semantically meaningful parts (for details, see [24]
3

---

lead to a steep accuracy drop when key concepts are deleted. We compute C-Del as the area over the accuracy curve where higher scores indicate stronger reliance on the removed concepts, and thus more faithful explanations. 'he second metric, Concept Insertion (C-Ins) measures how rapidly aceuracy recovered when important concepts are reintroduced into a blank representation. high C-Ins score implies that the explanation provides concepts used by the model in prediction.
Semantic interpretability is also contingent on the cognitive simplicity of the explanation. Sparse and focused explanations are easier for users to comprehend and reason about. We measure this property using the sparsity metric of Gini index [3] (C-Gini) computed over the vector of concept importance scores. A higher Gini index corresponds to less complex explanations, where a few dominant concepts explain most of the prediction.
Low Complexity:
See Appendix
I.2 for discussion on evaluation metrics of faithfulness and complexity.
3.2.1 Problem Formulation
Concept-based explanation via Non-negative matrix factorization (NMF) seeks to represent non- negative neural activation as composition small set of interpretable basis vectors. Given an activation matrix A E xR extracted from a neural network's encoder (e.g., penultimate layer activations), classical NMF factorizes it into a non-negative dictionary matrix W E R and coefficient matrix U E I such that the original activation is approximated as A ~ UWT. This typically framed as the following optimization problem:
min U20, W 20
-
I UW 11
where the Frobenius norm penalizes the reconstruction error between the original activations and their low-rank approximation. This formulation ensures that activations are approximately reconstructed using a sparse and additive combination of concept basis vectors.
However, minimizing only the reconstruction loss emphasizes preserving the activation geometry in the encoder space and does not guarantee alignment with the model's predictive behavior. As result, the reconstructed activations UWT may differ significantly in terms downstream predictions through the classifier head h. This can yield concept explanations that appear meaningful to humans, yet fail to capture the actual features the model relies on. Moreover, classical NMF tends to prioritize necessarily predictive or, diasidiicrimiiaaive. These
directions of high variance in A, which factors undermine the faithfulness of the explanations, as the extracted concepts may not align with the model's s recisioonmmaknng process. We discuss the failure cases of standard NMF-based techniques in Section 3.4
To address this limitation, propose Saithfulness-awant aware variant of NMF that explicitly aligns the reconstructed activations with the model's 's predictive behavior. This is achieved by introducing a Kullback-Leible (KL) divergence constraint between the classifier head prediction on and UWT. resulting in the following objective:
min U20.
| UW 1 X (I((((A)I((U"")
Here, lt(-) denotes the classifier head that maps activations to logits, and À > 0 controls the trade- off between reconstruction fidelity and predictive alignment. KL divergence is computed over the coftmax-normalizee logits, enforcing consistency between the model's predictions on the original and concept-based representations. This modification ensures that the learned low-dimensional concept representation U retains both the structure of the encoder activations and the predictive semantics of the classifier, improving the faithfulness of the discovered concepts.
Unlike classical NMF, which can be optimized using multiplicative update rules 17] our formulation introduces a non- Euclideen KL term that depends on the downstream classifier, making multiplicative updates inapplicable. We optimize Eqn. 2 using projected gradient descent, alternately updating U and W while enforeing non-negatyvity constraints via projection. Convergence to a stationary point is guaranteed under mild conditions (see Appendix C. To improve convergence speed and stability, we initialize the factor matrices and W using the Non-negative Double Singular Value Decomposition (NNDSVD) method [ (see Appendix L.I

---

Our formulation in Eqn. 2 can also be viewed through the lens of supervised dictionary learning 12 where the goal is learn dictionary W not only for reconstructing imputs but also for improving downstream task performance. In our case, rather than using explicit class labels, we impose supervision through model prediction alignment, ensuring that the learned concept representations remain faithful to the model's decision-making process.
3.3 Faithfulness Guarantee via KL Divergence Regularization
We provide theoretical justification for incorporating KL divergence regularization term between the classifier head predictions on the original and reconstructed activations. This constraint encourages the reconstructed activation space to preserve the model's predictive behavior, thereby promoting faithful concept extraction.
3.3.1 Reconstruction alone is not faithful
Let A E x=* be the activation matrix obtained from an encoder 9, and let A' UW be its non-negative low-rank approximation obtained via non-negativ matrix factorization (NMF). A first-order Taylor expansion of the fixed classifier head h around UW gives
h h(A') 1(A) VA(AA) (A - A').
where Vh denotes the Jacobian of h at UW T_ This highlights that even when A is close to UW in activation space, the corresponding logits can differ significantly depending on Vh. Therefore, minimizing only the reconstruction error ||A UW [ is insufficient for ensuring predictive
alignment. The following example makes this concrete:
Consider linear classifier head followed by temperature-scaled soft-max:
y = l(z) = Waz, o.(y) = ()
where W,, is fixed and > 0 fixed by training due to weight norm batch-norm. Let A be the true activation and let A' = UWT with a tiny reconstruction error = A so that ||6||2 Ercc < 1. The logit difference is Ay (((A))-((A) o. (-). Although 8 small, the probabilitics can be made arbitrarily large by the fixed gain a and the weight norm Wn. Hence, low Frobenius error reconstruction vector offers no control predictions.
W, ò. and after scaling by a, the predictive distribution
This shows that reconstruction loss alone cannot guarantee faithfulness; must constrain the predictive distribution itself. To address this, our formulation adds the KL divergence regulariza- tion between the poftmax-normalian logits: X (softmaa (oo)))] (U),)) which explicitly penalizes deviations in the predicted distributions.
Let p = softmax (h (h(A) (A)) and q (oftmax(h(AA)) be the predictive distributions before and after KL |q). As direct conse- quence of minimizing this KL divergence, we can bound the difference between the two predictive distributions. By Pinsker's inequality (26), constraining the KL divergence be less than or equal to è guarantees that the total variation distance is bounded:
reconstruction. FACE minimises L(U, w) | U I|:
lp - gll S /2. KL(-II) S 2
(3)
Implication. This result formally establishes that minimizing the KL divergence between predictive distributions directly constrains the deviation in model outputs caused by replacing the original activa- tion with its concept-based approximation UW In contrast, minimizing only the reconstruction loss | UW || does not offer any such guarantee. Note that the value of in Eqn. 3 is not fixed constant but simply the empirical value of KL(pllq). We do not assume a priori bound on but our hyperparameter À from Eqn controls how small & becomes. To make this explicit, we discuss values over range of À in Appendix E
3.3.2 KL divergence regularization leads to local linearity in the concept space
Interpretability methods often seek local linear approximations of complex neural networks that faithfully capture the model's behavior in small neighborhood of interest [3] In our formulation,
5

---

KL divergence regularization between the model predictions on the original and reconstructed activations encourages the local linearity of the classifier h with respect to the concept representation U.
Although the classifier head h is typically linear (e.g., fully connected layer), the final model prediction involves a softmax applied to the logits h( h(A), introducing nonlinearity, This means, even small changes in activations can still lead to disproportionate changes in predicted class probabilities. Consequently, minimizing only the reconstruction error A UW does not guarantee local linearity at the level of predicted distributions.
As discussed in Section 3.3.1 minimizing the KL divergence directly constrains deviations in predictive distributions, as shown by Pinsker's ' incquality [26]: for KL divergence bounded by E, change in prediction is bounded by v thereby ensuring tight control over model prediction.
Implication. KL regularization ensures that the softmax operates in a regime where its nonlinearity is negligible, i.e., it behaves almost linearly around UW This promotes faithful, predictable changes in model output with respect manipulations in the concept representation U. Hence, the discovered concepts form a locally linear and interpretable basis for understanding model behavior.
3.4 Failure Case of Unconstrained NMF
To highlight the limitations of standard NMF-based concept discovery methods such as CRAFT & ICE (36), evaluate their ability to preserve model predictions reconstructed activations. Specifically, analyze classifier performance reconstructed activations Å UWT from ResNet- 34 [14] on ImageNet 77] and COCO [9).. We begin with a set of input images that are correctly classified with 100% accuracy by the original model, ensuring that any deviation in prediction is caused solely by the reconstruction process.
Figure 2a shows that while FACE consistently retains 100% top-1 accuracy across all classes, CRAFT and ICE suffer notable drops, e-g., only 40% accuracy for "Train" (CRAFT). However, even when all methods retain the correct label (e.g.. "English Springer"). Figure 2b shows that CRAFT and ICE incur high KL divergence from the original predictions. This indicates that although the top-1 prediction is preserved, the full output distribution can still shift significantly.
MSE LOSS
accuracy
KL Divergence
CRAFT
ICE
&
FACE (Ours)
Church
English springer
Train
Teddy bear
Umbralla
Church
Train
English springer
Teddy bear
Umbrella
(a) Prediction accuracy on reconstructed activations.
(b) MSE and KL divergence between original and recon- structed activations. Minimizing MSE alone does not ensure prediction faithfulness. High accuracy also does not imply low KL divergence, as seen in CRAFT/ICE on "English Springer."
FACE consistently preserves classifier decisions across classes, while CRAFT ICE degrade certain cases.
Figure 2: Comparison of predictive alignment across concept extraction methods using ResNet- 34 [1]. FACE achieves both accurate and faithful reconstructions, while CRAFT and ICE may preserve top-1 predictions yet diverge in KL-divergence.
we also that FACE occasionally yields marginally higher MSE compared to CRAFT and ICE-an expected outcome, since our method balances the reconstruction loss with KL-reeularizztion to preserve predictive behavior. This highlights key tradeoff: minimizing reconsiruction losy alone is insufficient to ensure prediction faithfulness. Unconstrained NMF may recover latent factors that resemble semantically meaningful patterns but fail to reflect class- discriminative information. By contrast, FACE explicitly enforces predictive consistency through the
In Figure 26

---

KL-divergence term during factorization, ensuring that reconstructed activations preserve not only the correct decision but also the model's class confidences. This results in concept decompositions that are both interpretable and faithful (discussed in Section 4.2) We observe similar trends when
evaluating on MobileNetV2 [28 reported in Appendix F.T
Evaluation
We evaluate FACE across three key dimensions: (1) the quality of the matrix factorization, (2) the
faithfulness of concept-based explanations, and (3) the complexity of the resulting explanations. We compare FACE against two state-of-the-art NMF-based concept discovery methods: ICE [36 and
CRAFT g
We discuss evaluation with clustering based method, ACE 12 in Appendix
Datasets and Models. We evaluate FACE on three datasets of varying semantic granularity: ImageNet 77. COCO [19]. and CelebA [20]. We use ResNet-34 [14] and MobileNetV2 28 as target models for explanation. For ImageNet and COCO, we use publicly available pretrained models. For CelebA, we train ResNet-34 and MobileNetV2 models curated subset of mutually exclusive binary attributes: Black Hair, Blond Hair, Gray Hair, and Wearing Hat (details Appendix
Experimental Setup. All results are averaged over correctly -lasssifie 10,000 samples from 10 different ImageNet classes, 5,000 samples from COCO classes, and 4,000 samples from the selected CelebA attributes. Following prior work (8), we the rank of the matrix decomposition to 25 concepts for all evaluations. We provide implementation details Appendix B
CRAFT
ICE
FACE
Church
Gray hair
Tanch
Figure 3:
Qualitative comparison of top-concept extraction by CRAFT 8 ICE 36 and method,
FACE for four classes (Golf, Church, Gray hair, and Tench) using ResNet-34 14
Qualitative evaluation. We provide sample explanations in Figure 3 where FACE consistently produces semantically meaningful patches. However, since FACE prioritizes faithfulness to the model's 's decision process, the extracted concepts may differ from human intuition. For instance, in the Gray hair class, CRAFT highlights hair, and ICE focuses on the forehead, but FACE reveals that the model actually relies on facial features. This highlights the importance of aligning explanations with model behavior rather than visual plausibility alone. After all, models are not constrained to use human-undee uumman--dddrstaddalee cues; they only use features that minimize loss. See Appendix N for additional examples.
4.1 Quality of matrix factorization
We assess the quality of concept-based factorization along two axes: (1) reconstruction error between original A and reconstructed activations A UW (MSE), and (2) prediction consistency measured as the KL divergence between model predictions A vs. Å (Dx)). See Appendix L.1 for detailed definitions.
As shown in Table I FACE consistently achieves the lowest KI divergence among existing methods on minimizing the -i.-dieeeee (Dxz.) across all datasets and architectures, indicating strong alignment between the model's predictions original and reconstructed activations. This predictive
7

---

Table Quality of matrix factorization using reconstruction enror (MSE), and prediction consistency (Dxr). Values are mean =td across 5 runs. indicates higher is better.
ResNet-34
MobileNetV2
MSE +
Dka.
MSE 4
DKL. L
ICE
0.296 +: 0.001
0.359 + 0.004
0.117 + 0.000
0.469 +: 0.006
ImageNet
CRAFT
0.451 0.004
0.240
0.003
0.191 +: 0.001
0.400 0.002
FACE
0.497 + 0.002
0.220 : 0.000
0.180 = 0.001
0.221 + 0.001
ICE
0.308 + 0.000
0.596 + 0.002
0.111 + 0.000
0.631 + 0.007
COCO
CRAFT
0.457 + 0.002
0.600 +: 0.004
0.192 = 0.000
0.793 + 0.001
FACE
0,462 4: 0.003
0.458 +: 0.001
0.166 + 0.001
0.296 0.000
ICE
0.148 +: 0.000
0.212 + 0.000
0.135 + 0.000
0.121 + 0.000
CelebA
CRAFT
0.498 + 0.010
0.110 + 0.066
0.247 + 0.004
0.160 + 0.097
FACE
0.375 + 0.002
0.021 + 0.001
0.156 + 0.000
0.022 + 0.001
consistency reflects the faithfulness objective encoded in our KL-regularized optimization. While FACE underperforms in reconstruction error (MSE), this trade-off is expected since we minimize both MSE and KL-divergence loss. But, shown in Section 3.4] low reconstruction error alone does not imply predictive fidelity. By explicitly incorporating prediction alignment, FACE prioritizes faithful reconstruction over raw activation proximity.
4.2 Faithfulness and Complexity
Table Comparing faithfulness using Concept Insertion (C-Ins) and Concept Deletion (C-Del), and complexity using Gini-index sparsity (C-Gini). Values are mean = std across 5 runs. f indicates higher better.
ResNet-34
MobileNetV2
C-Ins
C-Del 1
C-Gini |
C-Ins 1
C-Del n
C-Gini
ICE
0.908 0.034
0.484 :: 0.063
0.537 +: 0,071
0
0,020
0.346
0.049
0.605 +: 0.149
ImageNet
CRAFT
0,932
0.001
0.752
+ 0.031
0,835 + 0.031
0,886
0.001
0.646
0.024
0.805
0.041
FACE (Ours)
0.969
0.010
0.891 + 0.011
0.895 + 0
0.974 + 0.003
0.012 0.882
0.947
0.001
0,883
0.632
0.020
0.086
0.007
0.622 0.064
COCO
CRAFT
0.861 = 0.029
0.691
+ 0.029
0.874
+ 0.035
0,764
= 0.036
0.571
0.026
0.874 :: 0.047
FACE (Ours)
0 0.013
0.894
0.010
0.947
0,000
0.974
0.002
0.012 0,905
0.949
0.000
ICE
0,910 = 0.008
0.365
: 0.016
0.662 : 0.087
0.858
0.007
0.385
0.050
0.728 0.032
CelebA
CRAFT
0.953 0.067
0.604
= 0.036
0.901 = 0.026
0,960
0,1
0.592
0.070
0.911 0.028
FACE (Ours)
0,971 0.012
0.635
0.014
0.000
0.001
0,932
We evaluate faithfulness using Concept Insertion (C-Ins) and Concept Deletion (C-Del), and com- plexity using Gini-index sparsity (C-Gini) as discussed in Section[..2 As shown in Table 2 FACE consistently achieves the highest scores across all models and datasets. In particular, FACE signifi- cantly outperforms ICE and CRAFT in C-Del. indicating strong sensitivity of model performance to the removal of discovered concepts. C-Ins improvements further demonstrate that FACE's concept representations align closely with model-relevant features, enabling accurate prediction recovery when only few key concepts are inserted. Notably, while FACE shows large margins in both C-Ins and C-Del on ImageNet and COCO, the gap is narrower on CelebA, especially for C-Del. We attribute this to the lower class complexity in CelebA (4 classes), where model decisions may rely on fewer features. In such cases, inserting few key concepts is often sufficient for recovering accuracy (high C-Ins), but removal may not fully disrupt predictions (lower C-Del). Lastly, the consistently high C-Gini scores suggest that FACE discovers compact, sparse explanations without sacrificing fidelity.
4.3 Ablation on regularization strength
We study the impact of KL regularization strength À in FACE by sweeping À over a wide range from 10-15 to 10 on ImageNet 7. COCO 19). and CelebA 20) datasets using ResNet-34 [14], tracking both classifier accuracy reconstructed activations and faithfulness with Concept Insertion (C-Ins) and Concept Deletion (C-Del).
Figure shows consistent trend across ImageNet and COCO: introducing a small KL. penalty (e.g.. 10 ") significantly improves faithfulness, as reflected by increased C-Del and C-Ins. However, for both datasets, increasing À beyond moderate threshold (c.g., X >> 10²) results in a sharp drop
8

---

MM40I: Value
Value
Meuric Value
Acceracy
C.Dc
Cinss
Clins
0.0
Acouracy
Accuracy
A (0gg scale)
(log scale)
(0gg scale)
(a) ImageNet
(b) COCO
(c) CelebA
Figure Effect KL regularization strength on faithfulness (Concept Insertion (C-Ins) & Concept Deletion (C-Del) and classifier accuracy reconstructed activations across datasets. A small KL penalty improves faithfulness across all datasets. However, large À values degrade performance on high-class datasets (ImageNet, COCO), while CelebA benefits from stronger regularization due to lower class complexity.
in faithfulness and, in the case of ImageNet, also harms classification accuracy. This suggests that excessive KL regularization may over-prioritize prediction alignment while compromising reconstruction fidelity.
Interestingly, observe different behavior CelebA. As shown in Figure cc), faithfulness continues to improve up to much higher À values (around 10%), with no adverse effect on accuracy. We hypothesize that this is due the much smaller number of classes CelebA (4 classes), which makes it casier to optimize KL divergence between the model's output original and reconstructed activations. In contrast, ImageNet and COCO have 1000 and 200 classes respectively, which means aligning entire probability distributions is more difficult. With large X, the optimization struggles to minimize KL effectively across nigh-dimensional distributions, leading worse trade-offs. We also observe a trend of lower C-Del values on CelebA, which suggests that concept removal in CelebA does not change the model confidence dramatically, likely due lower class cardinality.
These findings highlight the importance of dataset-specific calibration of X. We present per-class trends in Appendix G and results using MobileNetV2 28 in Appendix F2
4.4 Ablaioon on Matrix Decomposition Rank
0.00
Mobile salve
Matil Valux
0.6
0.30
C-Del
C-del
decompostiio
Muttic
(a) ImageNet
(b) COCO
(c) CclebA
Figure 5: Effect of decomposition rank (r) on faithfulness (Concepl Insertion (C-Ins), Concept Deletion (C-Del) and sparsity (C-Gini) across datasets. Increasing rank improves both faithfulness and sparsity, with diminishing gains beyond 25.
We analyze the effect of the NMF decomposition rank (r), the number of learned concept di- mensions, on faithfulness (C-Ins & C-Del) and sparsity (C-Gini). We evaluate range of values E (5, 10, ... 50} using ResNet-34 14, with KL regularization coefficient À 10- for ImageNet and COCO and A = 10 for CelebA where reconstructed activation accuracy remains 100% with high C-Del and C-Ins scores (see Figure 4 As shown in Figure we observe that increasing the rank improves faithfulness metrics (C-Ins, C-Del) and sparsity (C-Gini) across all datasets. For both ImageNet and COCO, insertion and deletion scores improve sharply between after which gains plateau. Meanwhile, sparsity improves with higher k, indicating that with more concept dimensions, activations become more selective and structured. On CelebA, we observe similar trend, albeit with lower absolute C-Del values. This suggests that concept removal in CelebA
5 and 25,

---

may impact model confidence less dramatically in CelebA. This is likely due to the simpler decision boundaries and lower class cardinality (4 total classes).
In practice, we use r = 25 similar to prior works 8]. We observe similar trends when evaluating
with MobileNetV2 [88] detailed in Appendix F3 We also present per-class trends in Appendix H
We discuss experiments on crop-size and alternative loss functions in Appendix [.2] and L3
Computational cost. FACE's optimization is lightweight, dominated by single small matrix product UW and a linear head, making tractable even on low-resource hardware. Detailed runtine/memory measurements are provided in Appendix D
Limitations
FACE is currently class-specific and global nature, i.e,, it extracts concepts for an entire class rather than per-instance basis. While this approach aligns with current literature [ 9 12 36], it may affect the granularity of explanations in settings where instance-level interpretabilltyy desired. Our method also relies on fixed hyperparameters (e.g., regularization strength) that require tuning for new datasets. Another limitation is the lack of human-centered evaluation of interpretability. Since the addition of the KL loss changes the representation objective, its effect human interpretability of the learned concepts remains unexplored. We leave such user studies, which are critical to verifying whether FACE indeed improves human understanding of latent spaces, as important future work. FACE is also suitable only to CNN based architecture. Directly applying FACE "as it is" to transformers like ViT is non-trivial. However, we see this as promising future work.
6 Conclusion
In this work, we presented FACE (Faithful Automatic Concept Extraction). a novel framework for concept-based explanations that augments Non-negative matrix factorization (NMF) with a KL divergence constraint to explicitly align concept representations with the model's predictive behavior. Unlike existing unsupervised methods that focus solely on reconstructing latent activations, FACE enforces consistency between model predictions original and reconstructed activations, thereby ensuring the faithfulness of discovered concepts. Through theoretical analysis, we demonstrate that KL regularization bounds predictive deviation and promotes faithful local linearity in the concept space. Our empirical results across three diverse datasets (ImageNet, COCO, and CelebA) show that FACE consistently outperforms existing methods on faithfulness and sparsity.
Acknowledgement
We thank the anonymous reviewers for their valuable comments. This research was supported by Toyota inco Labs through Unrestricted Research Funds. We also acknowledge the authors of CRAFT g for releasing their codebase, which served as useful foundation for this work.
References
[1] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels compared to state-of-the-an superpixel methods. IEEE transactions on pattern analysis and machine intelligence, 34(11):2274-25 2282, 2012.
[2] Reduan Achtibat, Maximilian Dreyer, Ilnaa Fisenbraun, Sebastian Bosse, Thomas Wiegand, Wojeiech Samek, and Sebastian Lapuschkin. From attribution maps to human lunaan-understaddaabee explanations through concept relevance propagation. Nature Machine Intelligence, 5(9): 1006 1019, 2023.
[3] Dipkamal Bhusal, Rosalyn Shin, Ajay Ashok Shewale, Monish Kumar Manikya Veerabhad- ran, Michael Clifford, Sara Rampazzi, and Nidhi Rastogi. Sok: Modeling explainability in security analyties for interpretabillity trustworthines,, and usability. In Proceedings of the 18th International Conference Availability, Reliability and Security pages 1-12, 2023.
10

---

[4] Christos Boutsidis and Efstratios Gallopoulos. Svd based initialization: A head start for nonnegative matrix factorization. Pattern recognition, 41(4):1350-13622 2008.
[5] Prasad Chalasani, Jiefeng Chen, Amrita Roy Chowdhury, Xi Wu, and Somesh Jha. Concise explanations of neural networks using adversarial training. In International Conference on Machine Learning, pages 1383-1391. PM 2020.
[6] Julien Colin, Thomas Fel, Rémi Cadène, and Thomas Serre. What cannot predict, do not understand: A human-centered evaluation framework for explainability methods. Advances in Neural Information Processing Systems, 35:2832-28455 2022.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255. leee, 2009.
[8] Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. Craft: Concept recursive activation factorization for ex- plainability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2711-2721, 2023.
[9] Thomas Fel, Victor Boutin, Louis Béthune, Rémi Cadène, Mazda Moayeri, Léo Andéol, Mathieu Chalvidal, and Thomas Serre. A holistic approach to unifying automatic concept extraction and concept importance estimation. Advances in Neural Information Processing Systems, 36, 2024.
[10] Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE international conference on computer vision, pages 3429-3437, 2017.
[11] Robert Geirhos, Jörn- Förn-He Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning deep neural networks. Nature Machine Intelligence, 2(1 (11):665-673, 2020.
[12] Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept- based explanations. Advances in neural information processing systems, 32, 2019.
[13] Tessa Han, Suraj Srinivas, and Himabindu Lakkaraju. Which explanation should choose? function approximation perspective to characterizing post hoc explanations. Advances in neural information processing systems, 35:5256-5268, 2022.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 770-778, 2016.
[15] Alexandre Janon, Thierry Klein, Agnes Lagnoux, Maëlle Nodet, and Clémentine Prieur. Asymp- totic normality and efficiency of two sobol index estimators. ESAIM: Probability and Statistics, 18:342-364, 2014.
[16] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference machine learning, pages 2668- (668-2677. PMLR, 2018.
[17] Daniel Lee and Sebastian Seung. Algorithms for non-negative matrix factorization. Advances in neural information processing systems, 13, 2000.
[18] Daniel Lee and H Sebastian Seung. Learning the parts objects by non-negative matrix factorization. nature, 401(6755):788-71, 1999.
[19] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6- 6-12, 2014, proceedings, part 13, pages 740-755, Springer, 2014.

---

[20] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep leamnigg face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[21] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 41 14- 4124, PMLR, 2019.
[22] Julien Mairal, Francis Bach, and Jean Ponce. Task-driven dictionary learning. IEEE transactions on pattern analysis and machine intelligence, 34(4):791-80) 2011.
[23] Giang Nguyen, Daeyoung Kim, and Anh Nguyen. The effectiveness of feature attribution methods and its correlation with automatic evaluation scores. Advances in Neural Information Processing Systems, 34:26422- 2021.
[24] Chris Olah, Alexander Mordvintsev. and Ludwig Schubert. Feature visualization. Distill, 2017. doi: 10.239 1033955/iise 1/distill..0.
[25] Vitali Petsiuk, Abir Das, and Kate Saenko. RISE: randomized input sampling for explanation of black-box models. In British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK, September 3-6, 2018, page 151. BMVA Press, 2018. URL http::/bbvcc// org/
contents/papprs/L0062 100peerss116...dd pdf
[26] Mark Pinsker. Information and information stability of random variables and processes. Holden-Day, 1964.
[27] Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus- Robert Müller. Evaluating the visualization of what a deep neural network has learned. IEEE Transactions on Neural Networks and Learning Systems, 28(11):2660-24 2673, 2016.
[28] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear boutlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.
[29] Hua Shen and Ting-Hap Huang. How useful are the machine- -ereerree interpretations to general users? human evaluation guessing the incorrectly predicted labels. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 8, pages 168-172,
2020.
[30] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning, pages 3145-3153. PMLR, 2017.
[31] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings, 2014. Uttp:///://arx org/ abs/ abs//312....////////// 1312. 6034
[32] Ilya M Sobol. Sensitivity estimates for nonlinear mathematical models. Math. Model. Comput. Exp., 1(4):407-414, 1993.
[33] Adam Stein, Aaditya Naik, Yinjun Wu, Mayur Naik, and Eric Wong. Towards compositionality in concept learning. ICML, 2024.
[34] Ao Sun, Pingchuan Ma, Yuanyuan Yuan, and Shuai Wang. Explain any concept: Segment anything meets concept-based explanation. Advances in Neural Information Processing Systems, 36:21826-21840, 2023.
[35] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference Machine Learning, pages 3319-3328. PMLR, 2017.
[36] Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A Ehinger, and Benjamin IP Rubinstein. Invertible concept-based explanations for enn models with non-negative concept activation vectors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11682-11694 11690, 2021.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly articulate the core contribution of the paper: a novel concept-based explanation method, FACE, which augments Non- non-negative Matrix Factorization (NMF) with a KI. divergene constraint to ensure predictive faithfulness. These claims are supported by theoretical guarantees, empirical evaluation on multiple datasets, and comparative analysis with existing methods.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper includes a dedicated Limitations section (Section 5 that clearly discusses the limitations of the proposed method. It highlights the proposed method's re- liance on class-level (rather than instance-level explanations as a limitation for applications requiring fine-gra tne-grained interpretability. Furthermore, the paper acknowledges sensitivity to hyperparameters like regularization strength.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
13

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The paper provides formal proposition stating that minimizing the KL diver- gence between the model's predictions original and reconstructed activations bounds the total variation distance between predictive distributions. The proposition is accompa- nied by proof using Pinsker's inequality. The assumptions are clearly stated, including the definition the predictive distributions and conditions under which KL divergence is minimized. A second theoretical subsection extends this analysis, showing how KL regular- ization promotes faithful local linearity even when the classifier head is linear but followed by softmax. While this proof is more intuitive, it is supported by Taylor expansion and formal reasoning.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper provides complete implementation details for reproducing the main experimental results. This includes clear descriptions of dataset (ImageNet, COCO, and CelebA), model architectures (ResNet-34 and MobileNetV2),, decomposition strategies (NMF with KL regularization), and concept extraction pipelines (e-g.. patching, NNDSVD initialization, projection steps). Setups and hyperparameters for FACE (e.g.. Adam optimizer, learning rate, early stopping criteria) are described Appendix B and additional training setup for CelebA is given in Appendix K Code and scripts are made publicly available at htttt::// anonymous. 4open. science/r/FACC B053/| with sample implementation notebook test-set available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed
14

---

instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The paper provides a public code repository hhttp://githuu com/ dipkamal/FACE that includes full implementation of our proposed FACE method, as well scripts to reproduce the evaluation of FACE and baselines. The datasets used in our experiments (ImageNet, COCO, and CelebA) are publicly available. Additional training details, model decomposition are described in Appendix
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
mublic/guidess (ttlliicguiide//ddeettuiassssss www.eooaiiiiiooonooooooo cisscnyolcyy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 7/nips. cc/ httec/ubbllcggguuee wwwpc//ggieseseeee Sttp:///SoooSSSSsssooieeeee
stiobbmiissiooooooic.. for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
15

---

Justification: The paper specifies all experimental settings necessary to understand and reproduce the results. This includes detailed descriptions of dataset splits (e.g.. 10 ImageNet classes, 5 COCO classes, and 4 CelebA attributes), model architectures (ResNet-34 and MobileNetV2), data preprocessing (e.g., image cropping with sliding windows), matrix decomposition setup (rank selection, NNDSVD initialization), training procedure (Adam op- timizer, learning rate x 10-4, early stopping with e = 10-3), and KL regularization sweep (A E {10-28 ..... 10²n)), These are clearly described in the main paper and Appendix B Additional model training details for CelebA are provided in Appendix K
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All quantitative results reported in the paper include mean and standard deviation computed over five independent runs with different random seeds. These are shown in Tables T and and error bars are clearly labeled using the "+" notation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma eгтоr bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
The core computations for our experiments involve extracting activation features from trained models and running low-rank NMF with KL-regularization for concept extraction. All experiments were performed on local workstation equipped with an NVIDIA TITAN Xp GPU (12 GB VRAM) running CUDA version 12.2. The average runtime for extracting concept representations with FACE per dataset was well under minute. Given that FACE involves backpropagation through relatively shallow classifier
Justification:
16

---

head (i.e., linear layer), the compute requirements are modest and accessible on consumer- grade GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///aaaaaaccccooooo////aaaaa
Answer: [Yes]
We have reviewed the NeurlPS Code of Ethics and ensured that work adheres to its guidelines. The research respects principles of transparency, responsible model development, proper citation and use of external assets, and includes dedicated discussion on broader societal impacts and potential risks. No human subjects or personally identifiable information were involved in our experiments.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We include a dedicated "Broader Impact" in Appendix that discusses both positive and negative societal implications of our method FACE. On the positive side, highlight how faithful concept-based explanations can improve model transparency, trust, and safety in high-stakes applications. On the negative side, we explicitly note that such interpretability techniques could be misused for ad adversarial purposes, such as crafting ad adversarial patches or targeted concept-level attacks.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinfornation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
17

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not involve the release of pretrained generative models, language models, or scraped datasets that pose high risk of misuse. The released code and data pertain to concept-based interpretability for standard vision classifiers and do not require special safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets (ImageNet 77. COCO 19), CelebA (00 and pretrained models (ResNet-34 [4], MobileNet2 (28) used in this work are cited appropriately in the main text. These assets publicly available and widely used in academic research under standard licenses (e.g., ImageNet: non-commercial research use, COCO: CC-BY 4.0, CelebA: for research purposes). For bascline methods (ICE (36), CRAFT (8), cite the original publications and adapt their publicly released code with proper attribution. All asset usage complies with respective licenses and terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pateessspppsiiiiia theode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
20

---

Appendix
Table of Contents
Broader impact
Implementation details: FACE
22
Optimization analysis
D
Runtime and memory footprint
Interpreting and controlling e in practice
Analysis on MobileNetV2
26
Per-class trend: Regularization strength
28
Per-class trend: Rank of matrix decomposition
Ablation studies
NNDSVD initialization
1.2
Crop patch-size
Alternative loss
Comparison with 1CE
34
Model training details for CelebA
Evaluation metries
Concept importance using Sobol-Indices
36
N
Additional qualitative results
37
21

---

A Broader impact
Concept-based explanations aim bridge the gap between deep learning models and human under- standing by representing predictions in terms of semantically meaningful concepts. Our method, FACE, advances this goal by improving the faithfulness of concept-based explanations, ensuring that the extracted concepts not only look interpretable to humans but are actually aligned with the model's decision-making This can empower users in domains such medical diagnosis, scientific discovery, and policy reasoning, where interpretability is a prerequisite for trust and accountability, and domains where incorrect assumptions about what a model "looks at" can have significant consequences.
However, interpretability can also be misused: to justify flawed models, perpetuate biases, or manipulate user trust. While FACE improves alignment between explanations and model decisions, cannot mitigate inherent biases in the model itself. Faithful explanations may faithfully reflect biased reasoning. Thus, FACE should be used as tool for transparency and debugging, for fairness or accountability checks.
substitute
In addition, increasing access to faithful representations of model's internal reasoning can also introduce new attack vectors. Since FACE accurately reflects the model's reasoning through sparse and interpretable concept activations, an adversary could exploit this to reverse-engineer model behavior and identify decision-critice concepts. This can aid in crafting targeted adversarial attacks that manipulate only the most influential concepts, making them more efficient and harder to detect.
B Implementation details: FACE
Similar to existing concept based explanation methods like CRAFT 8 ACE 12 and ICE (36). FACE also provides a global explanation to set of images classified as specific class. So, to explain target class using a target network (e.g. ResNet-34), we first collect correctly classified images using the same network. We use minimum of 1000 correctly classified images of each of the 10 classes (Cassette, Chainsaw, Chruch, English springer, French horn, Garbage truck, Gaspump, Golf, Parachute, Tench) of ImageNet dataset 77. 5 classes (Hotdog. Teddy bear, Train, Umbrella, Zebra) of COCO dataset [9), and 4 classes (Black hair, Blond hair, Gray hair, Wearing hat) of CelebA [20].
Clustering-basee based concept method like ACE [12] generates multi-resolution segmentation of the set of images but this leads artifacts generation. CRAFT 8 overcomes this issue by using image crops. We apply the same approach and generate local crops (patches) from each image using a sliding window approach with kernel size 64. We study the effect of varying patch size in Appendix [.2]
As explained in Section 3.1 a target classifier f X' - y is first decomposed into two components: an encoder g : X' - G and classifier head h - V. where G C R is intermediate latent space. We consider two models for our analysis: ResNet-34 [14] and MobilenetV? (8
For ResNet-34, we define all layers up to the second-last block, and h as the final classifier head:
Listing 1: Model decomposition for ResNet34.
g nn. n.. Smm.entiitilllle (*list (model. ()) [:-2]) (atpattiaallssssttttt wwwettorr.......m..... h lambda x: (torch. mean (x, (2, 3)))
For MobileNetV2, we use:
Listing 2: Model decomposition for MobileNet.
feaeuree
(x) model. classifier wtt.::/iieeraaaaatee.eeeeaooooa.
The local crops (patches) are now passed through the encoder part of the network g(-) to extract activation maps. We apply NMF with KL-divergence regularization the spatially averaged activation matrix A E R"Xp, where n is the number of crops and is the activation dimension. We decompose A = T with U e Raxr and W E Roxr, where is the number of concepts (rank). Both and W are initialized using NNDSVD 4 Our optimization objective is:
min U20. W
- A - UW 1 + KL(M())I((UW)))
(4)
22

---

We optimize this using Adam with a learning rate of 5 10-4 and early stopping when the absolute change in total loss drops below below 10 Non-negativity is enforeed on U and W after each gradient update via in-place clamping. We sweep over À {10-25 ..... 1020) to select the best regularization value per dataset. We use matrix decomposition rank as 25 for experiments but provide ablation study on varying the decomposition rank hyperparameter in Section
Our codebase is available publicly
C Optimization Analysis
Consider a standard supervised learning setup where classifier f : y maps inputs from an input space X' C I2d to output predictions y _ IR. Without loss of generality, we assume that can be decomposed into two components: an encoder X' + 9 and classifier head h : g - y where G S R2 is an intermediate latent space. The encoder g maps inputs to the nigh-dimensional latent representation, while h maps the latent representations to the output logits, or class-scores. Consequently, [x) hx((()). Given a dataset of n input samples X == "x,,..., xn] Rxx we denote their corresponding latent representations as A g(X) Raxp.
standard NMF-based decomposition is framed as the following optimization problem:
min U20, W20
UW 1
where the Frobenius norm penalizes the reconstruction error between the original activations and their low-rank approximation.
FACE augments the standard reconstruction loss with KL divergence regularizer based on classifier head predictions:
L(U, w) - 11 UW 13 X (L(((A))))(UW))
(6)
Standard NMF methods (e.g.. multiplicative updates 177) are not directly applicable due to the KL divergence term, which depends on the downstream classifier h. Instead, we optimize this objective using projected gradient descent with updates:
+++1 ProjR, (Ux - LU,. Wx))
(7)
W k+1 Projæ, (W& V Wx))
(8)
where Projæ; denotes projection onto the non-negative orthant (via elementwise max(0, -)), and > 0 is the learning rate.
Convergence. Let us stack the decision variables as Z [U: W] and denote the feasible set by θ = ((U, W) U, W 2 0}. We assume that:
[[v) Ll( S L7 Zz
Z 1++11
Za. Z++1 E
for some constant Lo 0. Because the first term of Eqn. composition of smooth maps (affine head h and soft-max), Eqn. holds on every bounded subset of θ with explicit L (derived below).
6 is quadratic and the second term is
By the standard descent lemma for -moooh functions, for any Zg, Z E
L(Zx+1) < L(Zx) (VL(Zz), Zx-11 Ze} + Z Zz:
(10)
We have,
Z PRoje, (z 17 VL(z))
(11)
Using the non ex-axaaasiiveess of the projection, ||Proj(x) P S || gives:
Z++1 ZællF S n \l
(12)
23

---

which combined with Eqn. 10 yields
1(Z++1) -(Z))- (1) L n²)
-
(13)
From this, we can see that the loss decreases as long as the tern multiplying the gradient remains positive i.c. If 7 2 the coefficient in Eqn. 13 is positive, so L(Zx) monotonically
non-increasing and bounded below, hence convergent.
"1
Lx n 0
(14)
which gives,
< L
(15)
Initialization. We initialize U W using Non-negative Double SVD (NNDSVD) 4 to ensure non-negativity and leverage structure in which empirically improves convergence speed, stability and quality of explanations (See Appendix I.I
MSF Loss KL Divergence Total
KL Divergence
Loss Valuue
Loss Volue
205
200
200
400
300
630
50
100
256
203
250
300
250
Optimization Steps
Optimization Steps
(a) ResNet-34
(b) MobilenetV2
Figure 6: Loss convergence during KL-regularized NMF for class 'Golf" with stopping criterion for
total loss e 1e-3
Empirical Support. In practice, we observe that the loss decreases steadily and converges within few hundred thousand steps, depending on the dataset under evaluation. See Figure for typical loss trajectory. Notably, the KI. divergence continues to drop even after the MSE almost plateaus. Our optimization stops when the difference between the total losses between two consecutive steps less than le - 3.
24

---

D Runtime and Memory Footprint
During FACE optimization, the encoder g frozen: gradients flow only through the low-rank factors U Raxr, W I and the (typically linear) classifier head h. Each PGD step forms the reconstruction UWT TRex and applies h to obtain logits.
Table 3: FACE runtime and memory on ImageNet (ResNet-34), Mean + std across 5 runs.
Phase
Time (s)
Peak VRAM (GB)
Preprocessing (activations NNDSVD)
5.6 0.2
3.55 0.07
Optimization (PGD U, W:g frozen)
3.55 0.07
We measured wall-clock time and peak VRAM on single NVIDIA TITAN Xp (12GB VRAM, CUDA 12.2) using ResNet-34, rank r 25, and 1500 ImageNet images (classwise run). Preprocess- ing consists of computing encoder activations and NNDSVD initialization; optimization runs PGD on (U, W) with y frozen. Results are provided in Table
Note that FACE runtime scales with the matrix decomposition rank and the number of steps for optimization. In practice, we observed that the loss platcaus early. However, smaller decomposition rank, smaller optimization steps or early stopping approach will yield better runtime. FACE is also run per class; hence, the computational cost scales with the number of images analyzed for that class. For resource-constie constrc-constrainee settings, one can subsample images per class, reduce rank r, or lower iteration steps without changing the method.
E Interpreting and controlling & in practice
By Pinsker's inequality [26]. constraining the KL divergence to less than or equal to & guar- antees that the total variation distance is bounded: ||p 411 S /2 KL(-II-) S V2e. Here, p == voftmax(h(A)) and q softmax (h(Am) are the predictive distributions before and after recon- struction of activation vector. The quantity E in this inequality is not an assumed constant; is the empirical value of (T((p)qq) achieved convergence of FACE objective given by Eqn. 16
min 111 UW 1| (XKL(((A))))(UWW)) U20, 2
(16)
The hyperparameter in Eqn. 16 controls the trade-off between reconstruction prediction alignment and thereby how small becomes. In practice we (i) monitor KL(pp) [Llppq)) pllq)) during optimization, and (ii) select from a broad "flat" region where both the KL term and the empirical |p qll remain small. Table 4 reports the mean KL, the mean é deviation, and the Pinsker upper bound /2 KL across sweep of À (10K ImageNet images; ResNet-34; rank 25). As discussed in ablation study of Section 4.3 large hurt both the KL and reconstruction terms, whereas moderate values keep KL small and the empirical |pp qll1 well within Pinsker's bound.
Table 4: Pinsker bound practice: empirical prediction shift vs. À on 10K ImageNet images (ResNet-34). We report mean (T.(pl/g) mean Jp- qllı, and the Pinsker upper bound /2 KT..
KL (mean)
q/l: (mean)
KL
1.00x 10 10-25
0.302
0.532
0.777
1.00x 10
0.302
0.532
0.777
1.00x
0.302
0.532
0.777
1.00x10
0.302
0.532
0.777
1.00x10
0.420
0.713
0,916
1.00x 10"
0.476
0.781
0,976
1.00x
0.466
0.769
0.966
1.00x
1.744
1.545
1,868
1.0x 10
1.866
1.579
1,322
1.00x 1015
1.867
1.580
1.932
1.00x 20
1,258
1.364
1.586
25

---

Analysis on Vo
Failure case
We investigate the limitations of standard NMF-based concept extraction methods using Mo- bi (28 by analyzing their predictive behavior on reconstructed activations Å UWT As in our main results (Section 3.4), we start with a set of input images that are correctly classified with 100% accuracy by the original network, ensuring that any change in prediction stems solely from reconstruction artifacts.
As shown in Figure 7a FACE consistently retains the correct predictions across all classes, while CRAFT and ICE suffer from performance degradation. For example, CRAFT achieves low accuracy on classes like "Church" and "Train." ICE performs slightly better than CRAFT but still fails to maintain full predictive consistency.
MSE Loss
Accurocy
KL Divergence
CIAFT
KCE
FACE (Ouns)
CRAFT
10
ICE
FAGE
60
Church
English springer
Train
Reddy bear
Umbretta
Church
Train
English springer
Teddy bear
Umbrella
(a) Classifier accuracy on reconstructed activations using FACE, ICE, and CRAFT on MobileNetV2. FACE maintains consistent predictive behavior across all classes, while CRAFT and ICE fail.
(b) MSE and KL divergence on reconstructed activa- tions MobileNetV2. Minimizing MSE alone does prediction faithfulness, divergence CRAFT and ICE.
in higher KL
Figure 7: Comparison of reconstruction accuracy and MSE-KL loss across concept extraction methods.
In Figure 7b we compare the mean squared reconstruction error (MSE) and the KL divergence between predictive distributions from the original and reconstructed activations. Although CRAFT and ICE sometimes achieve lower MSE than FACE, they incur substantially higher KL divergence. This demonstrates key limitation of unconstrained reconstruction: preserving low-level activation geometry alone does not guarantee alignment with the model's decision function. Notably, FACE achieves lower KL divergence despite slightly higher MSE, affirming that our KL-regularized optimization promotes reconstructions faithful to the model's true reasoning. These trends mirror our ResNet-34 [14 results from Section 3.4
F.2 Ablation: Regularization strength
1.0
1.9
C33
Metric Value
Metric Value
Metricc
Cinss Accuracy
C-ins Accuracy
C:Del
Cilns
0.99
Accuracy
12.1
104
A (log scale)
A (log scale)
(log scale)
(a) ImageNet
(b) COCO
(c) CclebA
Figure 8: Effect of KL-regularization strength À on faithfulness (Concept Insertion (C-Ins), Concept Deletion (C-Del)) and accuracy of reconstructed activations for MobileNetV2 on ImageNet, COCO, and CelebA. Small À improves faithfulness, while excessively large values degrade both accuracy and explanation quality (except for CelebA, where higher À remains effective).
26

---

We replicate our KL regularization ablation for MobileNetV2 [8] by sweeping the regularization coefficient over a range values (10-15 10'5) and tracking its effect on reconstructed prediction accuracy and faithfulness metrics: Concept Insertion (C-Ins) and Concept Deletion (C-Del).
As shown in Figure we observe consistent patterns across datasets. On ImageNet and COCO, introducing mild KL regularization (c.g., À E [10-s, 100)) significantly boosts faithfulness, with C-Del and C-Ins improving sharply. However, when À becomes too large, both C-Ins and C-Del degrade, with impact on accuracy for ImageNet. This indicates that excessive KL alignment can distort the latent structure and hurt reconstruction quality.
In contrast, for CelebA, the model remains robust to higher values of A, with faithfulness metrics continuing improve up to À 10 without any noticeable impact accuracy. This mirrors the results in Section 4.3 This behavior likely stems from CelebA's lower output dimensionality (4 classes), which simplifies KI. optimization. In comparison, ImageNet (1000 classes) and COCO (200 classes) require aligning much nigher-dimensional distributions, making it harder to simultaneously minimize KL and reconstruction losses at high À values.
These findings reinforce our conclusion from Section that KL regularization plays a critical role in shaping faithful explanations, but its optimal strength depends both the model architecture and label complexity of the dataset.
F.3 Ablation: Rank matrix decomposition
635
Mettic Vilue
Metric Value
0.5
Cins C-Gini
Metri
CDDel
C.Dei
Cirs
075
C-rrs
C-Gnii
079
C-Gini
Matrax decomposition rank
Matrix decompositinn rank
Matrix decomposition rank
(a) ImageNet
(b) COCO
(c) COCO
Figure 9: Effect of NMF decomposition rank r on faithfulness (Concept Insertion (C-Ins), Concept Deletion (C-Del)) and sparsity (C-Gini) for MobileNetV2 on ImageNet, COCO, and CelebA. Faith- fulness and sparsity improve with increasing rank, stabilizing after 25.
We analyze the impact of the matrix decomposition rank r, the number of learned concept dimensions, on explanation quality using MobileNetV2 (28) across ImageNet, COCO, and CelebA datasets. We evaluate a range of values r 5, 10, 50, fixing the KL regularization strength to À = 10-5 for ImageNet and COCO, and À 10 for CelebA, these values yield 100% accuracy on reconstructed activations. and high C-Ins and C-Del score (See Figure 8)
As shown in Figure 9 we observe consistent trend across all three datasets. Faithfulness metrics: Concept Insertion (C-Ins) and Concept Deletion (C-Del) improve significantly as rank increases from 5 to 25, after which the gains plateau. Sparsity (C-Gini) also improves steadily with increasing Notably, CelebA shows lower absolute C-Del scores than ImageNet and COCO, consistent with our observations in Section 4.4 Nonetheless, the monotonic improvement across metrics confirms that higher-rank decompositions help better capture the model's decision structure, provided the KL alignment is properly regularized.
27

---

G Per class-trend:
Regularization strength
G.1 ImageNet
In Figure 4a we saw that moderate KL regularization (c.g., À E 90 consistently improves faith- fulness scores of concept-based explanations. Accuracy remained largely unaffected in this regime, confirming that regularization does not compromise predictive performance when appropriately tuned. However, increases beyond 102, we saw a sharp drop in all three metries, suggesting that excessive regularization distorts the feature space, leading to degraded model performance and less informative concepts.
To better understand the impact of KL regularization on individual classes, we conduct a class-wise ablation over 10 representative ImageNet classes using ResNet-34 [14], sweeping À across wide range from 10-15 to 10t5_ For each class, we report prediction accuracy on reconstructed activations, along with Concept Insertion (C-Ins) and Concept Deletion (C-Del) as measures of explanation faithfulness.
Mosic VOlLC
Madic
C.Del
C-Del
DeDel
C-Ins
Cars
Cins
scalel
(a) Class: Cassette
(b) Class: Chainsaw
(c) Class: Church
0.81
Metic Vilue
Motric Valle
DeDel
CDeel
Accuracy
Accuracy
2.00
C-Del
Accuracy
2041
124
18
10-4
1244
10.4
124
10
10
12+h
104
10-
10
300 504
0og scalel
2log stalc)
(log stale)
(d) Class: English Springer
(c) Class: French Horn
(f) Class: Garbage Truck
codd Cire
codd
Cids Cire Acowacy
Cid Cate
Cire
Acoracy
Acoracy
Cog scalst
(g) Class: Gaspump
(h) Class: Golf
(i) Class: Parachute
(i) Class: Tench
Figure 10: Class-wise ablation on KL regularization strength (A) across 10 representative ImageNet classes. We show how Concept Insertion (C-Ins), Concept Deletion (C-Del) and accuracy vary with for each class. A small value of is sufficient to improve faithfulness, but large À often leads to degraded performance-hiihligghttn the need for balanced regularization.
As shown in Figure 10 we observe that while the overall trend remains consistent: "low KL regularization improves faithfulness", there is notable class-dependent variability in sensitivity to A.
Classes like Cassette, Chainsaw, and Church exhibit sharp drops in both accuracy and faithfulness for À 2 103, indicating high sensitivity to over-regularization. In contrast, classes such as English Springer, French Horn, and Parachute remain highly stable across wide range of values. Even extremely large regularization values do not cause significant degradation. These classes may have well-separated latent structures, making it easier align predictive behavior without sacrificing reconstruction.
These results further emphasize the need for class/dataset specific tuning of X.
28

---

G.2 COCO
In Figure 4b we saw the aggregate effect of KL regularization on the COCO dataset, evaluating faithfulness and accuracy. Similar to the trend observed for ImageNet, we saw that applying a small regularization strength improves faithfulness without affecting classification accuracy.
To better understand class-specific behavior on COCO, we perform a class-wise ablation of the KL. regularization coefficient À across five representative classes: Hot dog. Teddy Bear, Train, Umbrella, and Zebra using ResNet-34 [1]]. As in previous sections, we sweep over a wide range (10-16 to 10t5) and measure prediction accuracy on reconstructed activations along with Concept Insertion (C-Ins) and Concept Deletion (C-Del).
Muntic Value
MCDIC Volue
Metri Vilue
Cinss
Clirs
Cliis
Accuracy
Accuracy
Accuracy
12
A Dog scale]
Blg stale)
flog scale)
(a) Class: Hot dog
(b) Class: Teddy Bear
(c) Class: Train
C:08
Medic
Memti
C-Ins
C.Del
C-ra
Accuracy
10-1
20th 20-1 Blog
(d) Class: Umbrella
(e) Class: Zebra
Figure 11: Class-wise ablation KL regularization strength (A) across representative COCO classes. We show how Concept Insertion (C-Ins), Concept Deletion (C-Del), and accuracy vary with À for each class. A small value of À is sufficient improve faithfulness, but large À often leads to degraded performance highlighting the need for balanced regularization.
As shown in Figure 11 we observe a consistent pattern across classes where a small KL penalty (e.g., X E [10- 10 enhances faithfulness, but excessive regularization causes sharp performance drops. Hot dog, Teddy Bear, Umbrella classes are particularly sensitive to large Truin class shows more robustness across wider À range. Although dip in C-Del occurs around À = 102, the Concept Insertion (C-Ins) and accuracy remain mostly stable. Zebra class demonstrates resilience to wer.ragulatization. Accuracy stays mear-perfect across all X, and while C-Del fluctuates, it does not suffer severe collapse. This maybe because of the model's 's reliance on clear visual features (c.g.. stripes).
G.3 CelebA
We further conduct a class-wise ablation of KL regularization coefficient across four CelebA classes, Black huir, Blond hair, Gray huir, and Smiling, using ResNet-34 [14]. For each class, we sweep from 10-15 to 1 and report classifier accuracy on reconstructed activations, along with Concept Insertion (C-Ins) and Concept Deletion (C-Del).
Unlike ImageNet and COCO, CelebA classes remain robust over a wider range of with the accuracy curve staying flat up to moderately large values (e.g., À < 10!0). Interestingly, while C-Ins and C-Del exhibit sharp decline for large À in some classes (e.g., Gray Hair, Wearing Har), other classes (e.g., Black Hair) show relatively stable or even slightly improved behavior until very extreme values (e.g., 10.9). This suggests that CelebA's low output dimensionality (4-class prediction) makes easier for KI. minimization to enforce predictive alignment, and hence tolerate stronger regularization without substantial degradation. However, over-regularization eventually leads underfitting the reconstruction space, harming faithfulness.
29

---

Metric value
Metric Value
68-
Cins Accuracy
Gilns Accuracy
Cins
Accuracy
66
10.
18
1.7
1011
A (log scoll)
[log scale)
(log scalo)
(a) Class: Black hair
(b) Class: Blond hair
(c) Class: Gray hair
1.5
0.9
Metric
0.9
C.Del
075
C-ins
Accuracy
31
Xloog scale)
(d) Class: Wearing hat
Class-wise ablation of KL regularization strength (A) on CelebA on Concept Insertion (C- Ins), Concept Deletion (C-Del) and accuracy. Most classes benefit from moderate-high KL alignment, while very high regularization (A N 12 causes sharp degradation in accuracy and faithfulness.
Figure 12:
Implications. Our class-wiss ablations across ImageNet, COCO, and CelebA reveal that the optimal KL regularization strength À is not universal but depends significantly the dataset characteristies, particularly the number of output classes and the complexity of class semantics. Datasets with large number of classes (e.g., ImageNet with 1000 classes) more sensitive high X, likely due the difficulty in aligning dense predictive distributions over many classes. contrast, smaller datasets like CelebA-subset (4 classes) tolerate higher À without significant degradation in accuracy. These findings underscore the importance of dataset-specific tuning of regularization strength when aligning concept representations with model predictions.
30

---

H Per-class trend:
Rank of matrix decomposition
H.1 ImageNet
In Figure Sa. we observed that increasing the rank r improved both Concept Insertion (C-Ins) and Concept Deletion (C-Del), with performance saturating around r 25. The class-wis analysis the Figure 13 using ResNet-34 [14] reinforces these findings across diverse ImageNet categories where we mostly observe that C-Ins and C-Del improving sharply up to k 25, after which gains plateau. In a few classes (e.g., Garbage truck and Tench) exhibits mild fluctuations beyond 25. Nonetheless, across nearly all classes, sparsity measured via C-Gini increases steadily with r. These observations support the choice of = 25 as practical balance between faithfulness and interpretability, aligning with trends the average results.
Marr Value
Netric Valle
Netri Valuee
DDell
C-Ira
G-Ins C-Gni
D-nn
C-Gin
C-Gini
Matrik decompesition rank
Matris decomposkion rark
Matric decompostion ion rank
(a) Class: Cassette
(b) Class: Chainsaw
(c) Class: Church
Maok Valuee
Mc3Fc Value
Metry value.
C:Del
Cnns
Cilrs
ins
C-Cini
C-Gni
C-Gini
Matrik decompesition rank
Matris decomposkion rark
Matric decompostion Lon rank
(d) Class: English Springer
(e) Class: French Horn
(f) Class: Garbage Truck
0.Owl
Cira
decomposit
decompost 190K
Mitr decompost
(g) Class: Gaspump
(h) Class: Golf
(i) Class: Parachute
(j) Class: Tench
Figure 13: Class-wise ablation on rank matrix decomposition (r) across representative ImageNet
classes. We show how Concept Insertion (C-Ins), Concept Deletion (C-Del), and sparsity (C-Gini) vary with rank for each class. We observe that across most classes, faithfulness improves upto r = 25 and plateaus. Sparsity also improves with rank across most classes.
H.2 COCO
In Figure 5b we observed that increasing r' substantially improved both faithfulness and sparsity with improvements saturating around 25. The class-wise breakdown [4gure[4[usin ResNet- 34 [14] confirms this pattern across various classes of COCO dataset. Both Concept Insertion (C-Ins) and Concept Deletion (C-Del) show consistent gains range r 5 to 25, beyond which the improvements plateau. Notably, while the absolute metric values vary slightly across classes, the qualitative behavior is highly consistent, demonstrating that rank r 25 provides a strong balance between faithfulness and interpretability across COCO.
31

---

Mulk Valle
Meark, Value
3.0
Netri vauue
CDe
ins
Ciirs
ins
CGin
C-nni
C-Gini
Matrik decompesition rank
Matris decompostion rark
Matric decompostion rank
(a) Class: Hot dog
(b) Class: Teddy Bear
(c) Class: Train
Medk Value
Netrie Value
Cirrs
Cirs
C-Gri
C-Cini
Matrik decompostiion rank
Matrix descomposiion rank
(d) Class: Umbrella
(c) Class: Zebra
Figure 14: Class-wise ablation rank of matrix decomposition (r) across representative COCO
classes. We show how Concept Insertion (C-Ins), Concept Deletion (C- (C-Del), and sparsity (C-Gini) vary with rank for each class. We observe that across most classes, faithfulness improves upto r 25 and plateaus. Sparsity also improves with rank across most classes.
26.
Mesric Value
C-Del
Metric
Ginii
G-Ins
Cans
C-Gini
C-Gini
65
14
Matrix decommpositon rank
Matrix decompositon rank
Matrix decompositin rank
(a) Class: Black hair
(b) Class: Blond hair
(c) Class: Gray hair
Metric
C-ins
C-Ginl
Matrix decomposition rank
(d) Class: Wearing Hat
Figure 15: Class-wise ablation rank of matrix decomposition (r) across CelebA classes. We show Concept Insertion (C-Ins), Concept Deletion (C-Del), and sparsity (C-Gini) vary with rank for each class. We observe that across classes, faithfulness improves upto 25 and plateaus. Sparsity also improves with rank across most classes.
H.3 CelebA
In Figure 5c we observed similar trend with CelebA where the gain improved till the rank r 25 although the absolute value of the C-Del was low. Fi shoos similar trend for different classes using ResNet-34 [14]. Increasing the rank leads to steady improvements in Concept Insertion (C-Ins) and Gini-based sparsity (C-Gini) across most classes. However, Concept Deletion (C-Del) exhibits overall lower absolute values compared to ImageNet and COCO classes. This indicates that concept removal may impact model confidence less dramatically in CclebA, likely due to the simpler decision
32

---

boundaries and lower class cardinality (4 total classes). Still. the general trend of C-Del increasing and then plateauing holds. Sparsity improves consistently with rank across all classes, showing that increasing the number of concept dimensions yields more localized and disentangled representations.
I Ablation studies
L1 NNDSVD initialization
To enhance the convergence and stability of FACE, we initialize the factor matrices U and W using the Non-negative Double Singular Value Decomposition (NNDSVD) algorithm 4 NNDSVD has been shown to reduce the number of iterations required to reach stable solution and avoid poor local minima by providing structured low-rank approximation that respects the non-negativity constraints.
Lonk Divergence Loss
Diverennce
Shaps
Optimicetion
(a) Random
(b) NNDSVD
Figure 16: Loss convergence in FACE on ImageNet class "Golf" with stopping criterion for total loss
1e-3. NNDSVD initialization leads to faster convergence.
We empirically compare NNDSVD with random non-negative initialization on the ImageNet class "Golf" using ResNet-34 (14). Figure shows the optimization trajectory where observe that compared to random initialization, NNDSVD achieves faster convergence.
Table 5: Effect of initialization method on quality of explanations for class 'Golf", Quality of expla- nations measured with MSE (reconstruction error between original and reconstructed activations), DKI, (KL divergence between model predictions on original and reconstructed activations), C-Ins (faithfulness with Concept Insertion), C-Del (faithfulness with Concept Deletion), C-Gini (sparsity of explanations) and W-Stab (stability of extracted concept activation vector W on different data subset). t and 4. indicate higher and lower are better. NNDSVD provides better reconstruction, and explanation quality.
MSE
DKL +
W-Stab f
C-Ins 1
C-Del f
C-Gini
Random
0.74 + 0.01
1.49 + 0.16
0.64 + 0.01
0.92 0.03
0.39 0.06
0.68
0.00
NNSVD
0.44 + 0.00
0.09 + 0.00
0.84 + 0.00
0.98 0.02
0.92 0.01
0.95
0.00
In Table we summarize the final reconstruction quality, explanation faithfulness, and complexity. We observe that compared to random initialization has significantly lower final loss, both in terms of mean squared error (MSE) and KL divergence (DKL). also leads to better explanation stability, and higher faithfulness and sparsity results.
1.2 Crop patch-size
We study the effect of varying the crop patch size faithfulness and sparsity metrics for both ImageNet and COCO datasets using ResNet-34 [44, as shown Figure [7 We observe that patch sizes larger than 60 yield consistently better performance across all evaluated metrics metrics-including Concept Insertion (C-Ins), Concept Deletion (C-Del), and concept sparsity (C-Gini). However, increasing the patch size leads to coarser concept crops, potentially obscuring fine-grained details that are critical for interpretable explanations. To balance faithfulness with interpretability, we adopt patch size of 64, following the design choices in prior work (8]
33

---

1.09
1.90
Metric Value
Metric Value
0.30
0.59
0.990
C-Del
C-Del
0.55
C-Ins
C-Ins
G-Gini
C-Gini
Accuracy
Accuracy
Crop patch-size
Crop patch-size
(a) ImageNet
(b) COCO
Figure 17: Effect of varying crop patch size on faithfulness metrics: Concept Insertion (C-Ins),
Concept Deletion (C-Del), Complexity with sparsity (C-Gini) and accuracy for ImageNet and COCO using ResNet34.
1.3 Alternative loss functions
We chose forward KL(pllg) between the original and reconstructed predictive distributions because goal is preserve the predictive distribution, not just the logits scale. KL-divergence operates on probabilities and gives the Pinsker bound, directly tying the objective to output deviation. As an ablation study, we swapped KL divergence loss with (i) MSE logits and (ii) Reverse KL (g) ()) on 10K ImageNet images (ResNet-34).
Table 6: Ablation: Alternative loss function to KL divergence. Evaluating faithfulness of explanations using Concept Insertion (C-Ins) and Concept Deletion (C-Del). MSE represents reconstruction error of activation reconstruction and DEL represents prediction consistency. Values are mean across
runs. and indicate higher and lower are better respectively.
Metric/Loss Forward KL (pllq) Reverse KL (qllp)
MSE Logits
C-Ins
0.969
0.959
0.911
C-Del 1
0.891
0.875
0.824
MSE 1
0.497
0.501
0.525
DKL. +
0.220
0.279
0.315
Table shows that forward KI. (pilg) gives the best faithfulness evaluation. MSE-loss preserves top-1 accuracy but allows large distribution shifts and lower faithfulness. Reverse KI. sits in between. Asymmetry does not cause instability in practice because, in our optimization, p is fixed (original outputs) and q is optimized through updating U, W matrices.
J Comparison with ACE [12]
Our main comparison targeted NMF-based methods (ICE 33), CRAFT g to isolate the effect of adding prediction alignment within the same factorization family. By contrast, ACE [12] operates in input space: it segments images into superpixels and clusters them into concepts. As noted by Fel al. [8). superpixel segmentation can introduce boundary artifacts and fragmentation, which often dilute causal signals and negatively impacts faithfulness. While reconstruction-stlye scores (MSE on specific factorization approaches and thus not directly applicable ACE, we nonetheless evaluated ACE on ResNet-34 with 10K ImageNet images; the outcomes reported in Table 7 show substantially lower C-Ins/C-Del than the NMF family, with FACE strongest overall.
latent activations)
K
Model training details for CelebA dataset
For the purpose of evaluating concept-based explanations, we trained two classifiers on subset of the CelebA dataset (20). We selected a set of exclusive attribute attributes E [Black Hair, Blond Hair, Gray Hair, Wearing Hat]. Each attribute treated separate class with corresponding labels (0.
34

---

Table Comparing faithfulness using Concept Insertion (C-Ins) and Concept Deletion (C-Del), and complexity using Gini-index sparsity (C-Gini). Values are mean actoss runs. 1 indicates higher is better respectively.
ImageNet
COCO
Metrics/Method
ACE
ICE
CRAFT
FACE
ACE
ICE
CRAFT
FACE
C-Ins 0.691
0.908
0.932
0.969
0.502
0.883
0.861
0.971
C-Del f
0.510
0.484
0.752
0.891
0.577
0.632
0.691
0.894
C-Gini 0.429
0.537
0.835
0.895
0,482
0.623
0.874
0.947
1, 3). We filtered the images from the CelebA dataset based on the presence of exactly one of the selected attributes.
For model training, we used two backbone architectures pretrained on ImageNet: ResNet-34 [4] and MobileNetV2 (28). We modified the final layers to suit our four-class classification. Both models were trained using the following settings:
Optimizer: Adam
Learning Rate: le-4
Batch Size: 64
Loss Function: Cross Entropy Loss
Training Epochs: 15
for each class. On test set, we obtained accuracy of 96.43% for ResNet-34 and 96.67% for MobileNetV2. Only the images correctly classified by the models were used for subsequent concept extraction and explanation evaluations.
Accuracy was measured on a held-out
L Evaluation metrics
L.1 Evaluation of matrix factorization
1. Reconstruction error (MSE): We define the average reconstruction error in activation space as the Mean Squared Error (MSE) between original encoder activations A and reconstructed activations Á UWT:
MSE - -W" np
where, A and Å Raxp.
2. Prediction consistency (DKI): We measure the prediction consistency as KL divergence loss between the original model predictions h(A) and the model predictions on the recon- structed activation (UW)::
DKL = KI. (sstmmaa/((/))]lloonnat((()))
L.2 Evaluation of concept explanations
1. Faithfulness: We evaluate the faithfulness of concept-based explanations using perturbation- based metrics adapted from the AOPC framework (27), widely used in feature attribution literature.
(a) Concept Deletion (C-Del): Given set of concept importance scores (c;) and batch of concept representations U E x xr we progressively set the top-k most important
concepts to zero in U, compute the reconstructed activations À evaluate classification accuracy on the perturbed representations:
(U @ Ma WT, and
1
Accuracy
arg max l((A))
35

---

where Mj is binary mask that zeroes out the top-k concepts across all spatial locations. The Concept Deletion score is computed the area the accuracy curve:
C-Del
(Accuracy, - AAcuuacy k=1
where is the total number of concepts, and Accuracyo is the accuracy without deletion. Higher C-Del scores imply greater performance degradation upon removing important concepts, indicating higher faithfulness. If accuracy remains unchanged after removing top concepts, the score is low, suggesting the attribution is unfaithful.
(b) Concept Insertion (C-Ins): Complementing deletion, Concept insertion (C-Ins) mea- sures how rapidly accuracy recovers when important concepts are added incrementally. Starting with all-zero concept tensor Uo 0, progressively insert the top-k most important concepts and compute the reconstructed activations:
Á (Uo + M& @ U)W
The model's classification accuracy is evaluated each step k. The Concept Insertion score is then defined as the area under the accuracy curve:
C-Ins = = Accuracy, dk -
In practice, we approximate this integral numerically using the trapezoidal rule. Higher C-Ins scores indicate faster recovery of accuracy from fewer concepts, implying that the explanation identifies truly influential features.
2. Complexity (C-Gini): We use the definition of Gini-index for sparsity to measure complex- ity of explanations, as used by Chalasani et [5]. Given classifier f, a concept- explanation method m, and set of extracted concepts {c1, C2. ... Cr. with corresponding importance scores / (c;,, we define the complexity of m as:
SH (2i - n 1) sorted([(a))
C-Gini
(17)
B==1 sorted(T(c.))
Here, I(c;) represents the absolute importance score of concept c,, is the total number of extracted concepts, sorted (cx)) denotes the concept importance scores sorted in ascending order. C-Gini values lie in between [0, I]. value of indicates perfect sparsity (low complexity), where only one element in the vector (x(x) > 0. The sparsity is zero if all the vectors are equal to some positive value, which means high complexity.
M
Concept importance using Sobol-Indices
To quantify the importance of extracted concepts with respect to a model's prediction, we use Sobol indices 332 instead of TCAV [16] they lead to better estimates of important concepts, as demonstrated by Fel et al. [8]
The Sobol sensitivity index S, measures the contribution of specific concept subset U, to the output variability of the model f(U), where U denotes the full set of input concepts. In particular, S, captures how much of the variance in the model output is attributable the subset Un, thus providing an interpretable metric of concept importance in terms of functional fluctuation.
The sensitivity index S, is formally defined as:
S V(F(UU)) V(B((()))W)) EmCa V(E((U))).) V(f(U))
V(f(U))
where V denotes variance and E denotes expectation. This formulation decomposes the total output variance of s(U) into contributions from each subset of input concepts and their interactions.
Hence, Sobol index for a concept-subset is defined as the proportion of total output variance explained by that subset. In practice, often use the Total Sobol Index S. to assess the overall importance
36

---

of a concept i, which accounts for both its individual contribution and all its interactions with other concepts. The total index is given by:
ST
Σ Sa.
UC
To compute these indices, Fel et al. employs the Jansen estimator [5 along with Quasi- Monte Carlo Sequence (Sobol LP, sequence) to efficiently approximate the necessary variance and expectation terms. We refer the readers to the paper CRAFT [8 for more details.
N Qualitative results
Figure 18 compares concept extraction by CRAFT 81 ICE 33 and FACE across sample examples of ImageNet, COCO and CelebA dataset using ResNet-34.
37

---

CRAFT
ICE
FACE
C1
C2
(a) Class: Black hair
CRAFT
ICE
FACE
C1
C2
(b) Class: Blond hair
CRAFT
FACE
C1
AMAZT
(c) Class: Chainsaw
CRAFT
ICE
FACE
C2
(d) Class: French horn
CRAFT
ICE
FACE
(e) Class: Hotdog
CRAFT
FACE
(0) Class: Teddy Bear
CRAFT
ICE
FACE
C2
(g) Class: Parachute
Figure 18: Concept-based visualizations for different classes.
38


Reviews:
[{"review_id": "7drVyRi1dY", "reviewer": "Reviewer_KKi1", "summary": "This paper presents FACE (Faithful Automatic Concept Extraction), a novel concept-based explanation method for deep neural networks. FACE builds on Non-negative Matrix Factorization (NMF) but introduces a KL-divergence-based regularization term to align concept-based explanations with the model’s actual predictive behavior. By incorporating classifier supervision, the method ensures that reconstructed representations from the learned concepts not only approximate encoder activations but also preserve the classifier’s output distribution. The paper provides theoretical justification that minimizing KL divergence bounds prediction deviation and promotes local linearity in the concept space. Empirical evaluations across ImageNet, COCO, and CelebA datasets show that FACE achieves higher faithfulness and explanation sparsity than state-of-the-art methods (e.g., CRAFT and ICE), while remaining competitive in reconstruction quality.", "questions": "Most of my concerns and questions are addressed in the Weaknesses section.", "limitations": "The authors adequately address the limitations and potential negative societal impact of this work.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths.\n1. The proposed methodology is well-grounded, supported by a clear mathematical formulation and rigorous theoretical justification.\n2. The empirical evaluation is comprehensive, demonstrating the effectiveness of the approach across multiple datasets (ImageNet, COCO, CelebA) and model architectures (ResNet34, MobileNetV2).\n3. This work offers valuable insights into the nature of faithful model interpretation, and the use of KL divergence regularization, especially with its theoretical connection to prediction alignment via total variation bounds, is a particularly notable contribution. The method consistently outperforms existing baselines on key metrics such as faithfulness (C-Ins, C-Del) and sparsity (C-Gini).\n4. The use of Concept Insertion (C-Ins) and Concept Deletion (C-Del) to quantitatively assess the impact of concepts on model predictions is another strength. While related perturbation-based metrics have been explored in prior work, their adaptation to the concept-based explanation setting and direct application to model performance evaluation represents a meaningful contribution.\n\nWeakness.\n1. In Figure 3 and lines 258–259, the authors suggest that FACE reveals the model's reliance on facial features for the \"gray hair\" class. However, if the model is leveraging spurious cues (i.e., shortcuts), such as facial features unrelated to hair color, one would expect the prediction to change when these facial features are altered. It would be helpful to investigate whether such changes in facial features lead to prediction shifts. If confirmed, this would further highlight the effectiveness of FACE in faithfully capturing the model’s true decision-making behavior, distinguishing it from baselines that may align more with human intuition than actual model reasoning.\n\n2. In Proposition 1, the parameter ε is critical to the theoretical claim that KL divergence regularization guarantees predictive faithfulness and local linearity. However, the paper could benefit from a more detailed discussion of how ε behaves in practice. For example, is ε explicitly bounded in real-world applications? If not, how can one reliably assert that the L1 distance between predictive distributions (‖p − q‖₁) is small? Further clarification on how ε is interpreted, estimated, or controlled during training would strengthen the theoretical and practical understanding of the proposed method.", "was_revised": true, "final_justification": "The authors have adequately addressed the concerns I previously raised, and the remaining issues appear to be minor and can be resolved in the camera-ready version. Therefore, I recommend this paper for acceptance.", "rebuttal": "We want to thank the reviewer for their thoughtful and positive assessment of FACE. We appreciate your recognition of our theoretical grounding, empirical breadth, and extensive evaluation. Below, we address the main concerns you raised:\n \n- **On spurious cues leading to change in prediction shifts**: Our goal is precisely to extract such concepts that the model relies on, even if those cues appear spurious for humans. This is the goal of faithfulness-aligned concept extraction method proposed with FACE. And, we already probe whether changing concepts result in prediction shift with our concept deletion (C-Del) test: *when we zero only the FACE-identified top concepts, we measure how the model’s accuracy changes*. Suggested by Reviewer FuWA, we also measured the changes in the logit of the original label by modifying the concepts and computing the logit-based insertion/deletion AUC on 10K ImageNet images (ResNet34). FACE was still able to obtain the highest scores compared to NMF baselines, as shown below.  \n| Metrics/Method    | FACE | CRAFT | ICE |\n| -------- | ------- | ------- | ------- |\n| C-Ins  | 0.9466    | 0.8386 | 0.6403 |\n| C-Del | 0.9472     | 0.8413 | 0.6448 |\n\nTo further emphasize FACE’s ability to expose shortcut behavior, we have run FACE on multiple images attacked with adversarial patches that force a targeted misclassification. FACE consistently isolates the adversarial patch as the top concept with the highest concept importance score. If required, we can include these qualitative experiments in the Appendix. \n\n- **Clarification on how $\\epsilon$ is interpreted from Prop. 1:** In Prop. 1, $\\epsilon$ is not a fixed constant; it is simply the empirical value of KL(p||q) measured between model prediction on the original and reconstructed activation vector. We do not assume a priori bound on $\\epsilon$, but our hyperparameter $\\lambda$ controls how small $\\epsilon$ becomes. To make this explicit, we swept $\\lambda$ over different magnitude on 10k ImageNet images using ResNet34 and report the mean KL and $||p-q|_1$ underneath:  \n\n  \n\n| $\\lambda$ | KL (mean) | $\\|\\|p-q\\|\\|_1$ (mean) | √(2·KL) |\n|--------------|---------------|----------------------|-------------|\n| 1.00E-15     | 0.302         | 0.532                | 0.777       |\n| 1.00E-10     | 0.302         | 0.532                | 0.777       |\n| 1.00E-05     | 0.302         | 0.532                | 0.777       |\n| 1.00E-01     | 0.420         | 0.713                | 0.916       |\n| 1.00E+00     | 0.476         | 0.781                | 0.976       |\n| 1.00E+01     | 0.466         | 0.769                | 0.966       |\n| 1.00E+05     | 1.744         | 1.545                | 1.868       |\n| 1.00E+10     | 1.866         | 1.579                | 1.932       |\n| 1.00E+15     | 1.867         | 1.580                | 1.932       |\n\n\nFor $\\lambda$ used in our main experiment (moderate values), KL is small and the empirical ||p-q||_1 is well within Pinsker’s bound. Extremely large $\\lambda$ hurts both KL and $||p-q||$ as already discussed in our ablation experiments in the main paper. In practice, we monitor KL during optimization and select $\\lambda$ from the large “flat” region where both KL and ∥p−q∥  remain small. We can clarify this in the paper if required."}, {"review_id": "SFgj1jP6PL", "reviewer": "Reviewer_yEUV", "summary": "In this work the authors propose FACE: a variant of NMF that incorporates a KL penalty on classification distributions to encourage reconstructions that also preserve downstream performance. The authors argue that this regularization encourages a more faithful concept extraction that aligns with the behavior of the model, and in the work demonstrate that this is the case through experiments while validating FACE’s competitive performance on traditional concept learning metrics such as sparsity and MSE.", "questions": "* In your convergence proof, Appx C Eqn (8), I think you dropped the Lipschitz Smoothness term. Then you could also explain exactly how sufficiently small $\\nabla$ must be depending on the Lipschitz smoothness of your loss.\n * Do you want your sparsity metric to be high or low? In the table you want it to be high but in the Appendix based on your sparsity definition seems like you want it to be low as you are summing L0 norms.I’m also not super familiar with the Gini metric you are using, so I would make it extremely clear how the sparsity metrics are computed and thus if you want them higher or lower.\n * Also, why would increasing rank improve sparsity? I thought with NMF as you increase rank your decompositions are over more vectors, i.e. a less-sparse decomposition. \n * See weaknesses.\n\nI am very willing to improve my score, but I think some of the above changes should be considered to improve the paper.", "limitations": "This is limited to models with a downstream task, which does not transfer well to generative systems.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths:\n * I really like the concept deletion and insertion metrics as a good causal test of the faithfulness of your explanation. I think these are the most important experiments in the paper. Is it possible to visualize the change in the image as you extract and delete concepts? Or show which concepts are being extracted and deleted in order? I would find that very interesting, at least in an example in the appendix.\n * The experiments validate the efficacy of the method.\n * I like the example of the gray hair and head vs face qualitative example. I would be interested to see if you could fix this “spurious correlation” in the model (for example train on young and old people with gray hair, assuming their faces look very different) and reapply FACE to see if the model is corrected properly.\n * I think the faithfulness of concept explanations is pretty under explored and I liked the methodology used by the authors to test this.\n## Weaknesses:\n * Your proposition is just a direct statement of Pinsker’s inequality. I think the interesting component in your analysis is pointing out via Taylor approximation that even with small reconstruction, we have no bound on the change in logits. I might rewrite the proof to highlight this or drop it as a theoretical statement, similar to how you simply cite Pinkser’s in line 213. It would be interesting to provide a converse result or example where the gradient h(UW^T) is large despite the reconstruction being small.\n * As part of this your conclusion/implication of the proof is not really saying anything: if the KL is bounded, of course the model outputs are bounded, even without converting to TV.\n * Overall, I would rewrite 3.3 and 3.4 to highlight the lack of control we have over model outputs via the taylor expansion; then in 3.4 you can focus on the linear case and point out even in this case the jacobian of the softmax is out of our control and maybe you can construct a counterexample here.\n * Also, given you are modeling your classifier head as linear, you might be able to say something stronger regarding the lipschitzness/convergence of your algorithm.\n * Figure 4 doesn’t actually tell me that the KL divergence penalty is improving faithfulness: you say the metrics increase to $\\lambda = 10^{-5}$ but to me the metrics appear flat. What happens when $\\lambda = 0$? Figure 4 indicates to me that these metrics would not change, which is in disagreement with Table 2 assuming CRAFT is equivalent to $\\lambda = 0$.", "was_revised": true, "final_justification": "The authors satisfiably answered all my questions.", "rebuttal": "We would like to thank the reviewer for their thoughtful review of our work, highlighting the importance of faithfulness-driven explanations and, efficacy of our evaluation. Below, we address your concerns: \n\n- **“Your proposition is just Pinsker’s inequality … the interesting bit is the Taylor part.”** In our work, Pinsker is not presented as a new theorem but as a bridge between our optimization objective (KL on predictive distributions) and a familiar notion of output deviation. The novelty is the placement of this regularizer inside NMF for concept extraction and the empirical demonstration that it improves faithfulness of explanations. We agree that the proposition as currently written can read like “KL small ⇒ prediction-difference small”, which is tautological; and as you have mentioned, “If KL is bounded, of course, the outputs are bounded”. We agree with you that the interesting part is in the first‑order Taylor expansion, which shows that small reconstruction error ∥$A−UW^⊤$∥ does not control the logits because $h$ can have a large Jacobian.  This is exactly why we add the KL term: *to control prediction difference directly*. \nIn the camera‑ready, we will trim the Pinsker statement and move the emphasis to the Taylor observation (“reconstruction alone gives no bound on logits”) and also include converse examples as you suggested, such as: consider a linear head h(z)= $\\alpha$$W$$z$ where $W$ is the learned weight and $\\alpha$ is scaling factor on the logits learned implicitly during model training via weight-norm, batch-norm layers (or, equivalent to temperature of the softmax). Assume that the NMF factorization has made the Frobenius norm small between original A and A’ such that \\delta = $||A’-A||_2$. Now, the logit change = $h(A’)-h(A)$ = $\\alpha$ $W$$\\delta$. Because $W$ is fixed, the product $\\alpha$$W$ can easily amplify the logit change, changing the model prediction.\n\n\n- **Convergence proof: missing Lipschitz term & stepsize condition:** We thank the reviewer for suggesting to extend our existing proof. We have shared the updated version of the convergence underneath:  \n\nRecall our optimization problem for FACE:\nL(U, W) = (1/2) * ||A - U Wᵗ||_F² + λ * KL( h(A) || h(U Wᵗ) )      (C.1)\n\nsubject to *element-wise* non-negativity U ≥ 0,   W ≥ 0. We optimize this using projected gradient descent with a shared stepsize η > 0:  \nUₖ₊₁ = Proj⁺( Uₖ − η ∇ᵤ L(Uₖ, Wₖ) ) ; and Wₖ₊₁ = Proj⁺( Wₖ − η ∇𝓌 L(Uₖ, Wₖ) ).  \n\nLet us stack the decision variables into Z = [U; W], and define the feasible set: Θ = { (U, W) | U ≥ 0, W ≥ 0 }.  \n\nAssume the loss L(Z) is L∇-smooth over bounded subsets of Θ, i.e.,\n|| ∇L(Zₖ) − ∇L(Zₖ₊₁) ||_F ≤ L∇ * || Zₖ − Zₖ₊₁ ||_F            (C.4)\n\n(The quadratic reconstruction error plus the smooth affine‑softmax KL term satisfies this.)\n\nBy the standard descent lemma, for any Zₖ, Zₖ₊₁ ∈ Θ:  \n\nL(Zₖ₊₁) ≤ L(Zₖ) + ⟨∇L(Zₖ), Zₖ₊₁ − Zₖ⟩ + (L∇/2) * ||Zₖ₊₁ − Zₖ||_F²     (C.5)  \n\nUsing non-expansiveness of projection, we get:  \n\n||Zₖ₊₁ − Zₖ||_F ≤ η * ||∇L(Zₖ)||_F                             (C.6)  \n\nInserting (C.6) into (C.5), we obtain:  \n\nL(Zₖ₊₁) ≤ L(Zₖ) − (η − (L∇/2) * η²) * ||∇L(Zₖ)||_F²            (C.7)  \n\nThus, the loss decreases whenever the following condition holds:  \n\nη − (L∇/2) * η² > 0    ⇨    0 < η < 2 / L∇  \n\nSo, for any stepsize:  0 < η < 2 / L∇, the loss sequence { L(Zₖ) } is monotonically non-increasing, bounded below, and convergent.\n\n- **Why would increasing rank improve sparsity?:**  Explanation sparsity (C‑Gini) measures how uneven the importance scores are across the $r$ concepts. A higher Gini means a few concepts dominate, hence a sparser explanation. C‑Gini is computed on the concept importance scores. When $r$ is tiny (few concepts), each concept must lump together many patterns, so their importance is forced to be similar, resulting in low Gini (flat distribution). As $r$ grows, concepts specialize: a handful become truly predictive for the class, while many receive near‑zero importance. The distribution of concept importance scores for the concepts becomes more skewed, so Gini rises. Hence, “rank ↑ ⇒ explanation sparsity ↑”.\n\n- **Sparsity metrics (U‑Spr vs. C‑Gini) and arrows:** We thank the reviewer for catching the confusion. Our current definition of U-Spr does measure the active fraction (density), so lower is better for sparsity. Because our projected gradient descent updates with non‑negativity rarely drive entries exactly to zero, FACE ends up with many very small but non-zero coefficients, yielding higher density than ICE/CRAFT, whose multiplicative updates often drive many entries to zero, so they look “sparser”. However, this property measures the matrix factorization quality; hence, there is no preference for low/high encoding sparsity. For camera-ready, we can either report density explicitly as U-Den (downarrow) or report sparsity as 1-U_Spr. This correction does not change our final conclusion: *FACE still achieves the lowest KL (prediction consistency) and strongest C-Del/C-Ins faithfulness, which is our motivation for FACE*. Our goal is faithful, succinct explanations, not necessarily the sparsest latent code.  \nC‑Gini (explanation sparsity) is computed on the importance vector across concepts, not on U. For example, given a rank of 25, we obtain 25 different concept activation vectors (representing each concept), each with a concept importance score computed using the Sobol-index. Then, C-Gini measures the imbalance in the importance scores assigned to these 25 concepts. The goal in XAI, motivated by comprehensibility of explanations, is that sparser explanations are easily comprehensible. Meaning, if there is one or two highly important concepts among the 25 concepts, then it's much easier to understand how the model made predictions, rather than if the score is distributed evenly across all concepts. C-Gini is inspired by the inequality in the distribution of scores. So, in summary,  higher Gini ⇒ a few concepts dominate ⇒ higher sparsity. This is the sparsity we care about for interpretability.\n\n- **Figure 4 doesn’t show KL helps; what about λ=0? Is CRAFT = λ=0?:**  We have performed evaluation results with $\\lambda$=0 (i.e., reconstruction‑only PGD‑NMF), where we observe a noticeable drop in faithfulness. On 10K ImageNet images (ResNet‑34), we obtain: C‑Ins=0.951, C‑Del​=0.796, C‑Gini=0.848. So the KL term does buy additional faithfulness. The apparent “flatness” in Fig. 4 stems from the saturation region: for small $\\lambda$ (10⁻¹³–10⁻⁵), the KL term is strong enough to keep predictions aligned, so metrics plateau near their maxima. Finally, one point to note is that: CRAFT ≠ $\\lambda$ =0. CRAFT uses multiplicative updates on spatial maps and pure MSE; our $\\lambda$=0 uses projected gradient descent on averaged features. Empirically, their KL and C‑Ins/C‑Del differ (Table 2).\n\n- **Finding shortcut and “fixing” the model:** Thank you for this insightful idea. FACE is intentionally diagnostic: it is built to surface exactly which (possibly spurious) cues the model relies on so that practitioners can decide how to intervene. Repairing the model is orthogonal to (but downstream application of) explanation. Concretely, once FACE highlights a shortcut concept (e.g., a specific facial region for “gray hair” or an adversarial patch), one can employ data-level fixes such as upweight / resample examples, or employ loss-level fixes where we penalize reliance on the offending concept (e.g., add a concept-dropout or gradient regularizer on the FACE-identified dimensions). We are actively exploring these “concept-level debiasing” strategies and will mention this as a promising direction for future work."}, {"review_id": "cXlRgqJ34N", "reviewer": "Reviewer_DAoS", "summary": "The paper introduces a novel method for generating concept-based explanations for deep neural networks. The core of FACE is the integration of a KL divergence regularization term into the Non-negative Matrix Factorization (NMF) framework. This addition is designed to ensure that the extracted concepts are not only interpretable but also faithful to the model's decision-making process. The authors provide a compelling case for their approach, supported by theoretical guarantees and extensive empirical evaluation.", "questions": "1) FACE requires backpropagation through the classifier head to compute the KL divergence, which can be computationally intensive, especially for large models or datasets. Although the paper claims modest compute requirements, suitable for consumer-grade GPUs, this may still pose challenges in resource-constrained environments or real-time applications. A detailed analysis of computational complexity, including runtime and memory usage, would strengthen the paper’s claims.\n\n2) The method assumes a model architecture with a clear separation between an encoder and a classifier head, as seen in ResNet34 and MobileNetV2. Its applicability to other architectures, such as transformers or models with integrated components, is unclear. This limitation restricts FACE’s generalizability across diverse deep learning models, which is a significant concern given the variety of architectures used in practice.\n\nOverall this paper makes an incremental contribution to concept-based explainability by addressing the important problem of faithfulness in automatic concept extraction. The theoretical foundation is sound, the experimental evaluation is comprehensive, and the results clearly demonstrate the method's effectiveness. However, the work suffers from insufficient analysis of failure modes, and significant hyperparameter sensitivity that may limit practical adoption. Hence, I believe the paper would benefit from: (1) more thorough analysis of computational complexity and failure cases, (2) evaluation on a broader range of architectures and domains, (3) human evaluation studies, and (4) development of adaptive hyperparameter selection strategies.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The primary contribution of FACE is its ability to ensure that these concepts are not only interpretable but also faithful to the model’s actual decision-making process, addressing a critical limitation in existing methods where extracted concepts may misrepresent the model’s reasoning. By incorporating a KL divergence term that minimizes the difference between the model's predictions on original and reconstructed data, FACE ensures that the discovered concepts are aligned with the model's predictive behavior.\n\nThe empirical evaluation of FACE is both thorough and convincing. The authors test their method on three diverse datasets (ImageNet, COCO, and CelebA) and two different model architectures (ResNet-34 and MobileNetV2), demonstrating the generalizability of their approach. The inclusion of multiple metrics to evaluate the quality of the matrix factorization, the faithfulness of the explanations, and the complexity of the explanations provides a comprehensive view of FACE's performance. The ablation studies on the regularization strength and the matrix decomposition rank further strengthen the paper by showing the impact of these hyperparameters on the results.\n\nWeaknesses : 1) While the application is novel, the core technical contribution is relatively incremental - essentially adding a KL divergence term to standard NMF optimization. The theoretical analysis, while correct, relies on well-established inequalities (Pinsker's inequality) rather than developing new theoretical insights.\n\n2) The strong alignment with the model’s predictions may lead to concepts that are overly specific to the particular model and dataset used. This could reduce the generalizability of the concepts to other models or domains, limiting their broader applicability. For example, if a model was trained with biases or on a specific data distribution, the extracted concepts might reflect those biases rather than generalizable features.\n\n3) Since FACE relies on NMF, which is inherently linear and suited for extracting additive, parts-based representations, it may struggle to capture nonlinear or higher-order concepts (e.g., “smiling” in face recognition, which involves interactions between multiple facial features). While the classifier head might learn such interactions, the concepts themselves are limited to linear combinations, potentially missing complex relationships critical to the model’s reasoning.\n\n4) FACE requires backpropagation through the classifier head to compute the KL divergence, which can be computationally intensive, especially for large models or datasets. Although the paper claims modest compute requirements, suitable for consumer-grade GPUs, this may still pose challenges in resource-constrained environments or real-time applications. A detailed analysis of computational complexity, including runtime and memory usage, would strengthen the paper’s claims.", "was_revised": true, "final_justification": "I will keep my score for all the reasons that I already provided to the authors.", "rebuttal": "We want to thank the reviewer for their assessment. We appreciate that you understood that the faithfulness issue in concept-extraction is often neglected, and you found our approach and evaluation technically sound. Below we address some of your concerns: \n- **On incremental contribution:** We agree with the reviewer that the building blocks of FACE are classical. However, as the reviewer pointed out, the faithfulness issue in automatic concept extraction is overlooked, and our goal with FACE was to couple matrix decomposition with the classifier’s predictive behavior and make faithfulness an optimization target. We view FACE as a technique that can make an existing family of concept methods faithful to the underlying model.\n\n- **Concepts may reflect biases or specific data distribution used in training:** Yes, and this is intentional. FACE is a post‑hoc, model-faithful explainer: its goal is to reveal exactly what the given model relies on, including spurious or biased cues. This is crucial for auditing, diagnosing shortcut learning, and deciding whether to retrain or intervene on a model. Making concepts appear “more general” at the expense of fidelity would defeat that purpose. We actually ran a test with FACE on images with adversarial patches that force a target misclassification; FACE reliably isolates the patch as the top concept with high importance, exposing the attack vector. We can discuss this in the appendix.\n\n- **On linear/additive NMF:** We acknowledge that FACE, like most post-hoc XAI methods, uses a linear parts-based decomposition. However, with FACE, faithfulness is enforced at the model prediction level. So, if important nonlinear interactions were lost, the KL term would increase, and optimization would adjust the U and W matrices. High C-Del/C-Ins and low KL in practice indicate that a linear basis suffices locally to capture the model’s behavior.\n\n- **Computational cost and complexity:** FACE does not backpropagate through the whole network. The encoder g is frozen; gradients flow only through the low‑rank factors U and W and the (typically linear) classifier head h. Thus, each projected gradient optimization step costs a matrix product $UW^T$ plus a forward/backward pass through $h$. To address the reviewer's concern, we measured the runtime and peak memory on a single NVIDIA TITAN Xp (12 GB VRAM, CUDA 12.2). On 1,500 ImageNet images (model = Resnet34, rank = 25), preprocessing (computing activations + NNDSVD initialization) takes 5.6 ± 0.2 s, whereas the optimization takes 25.7 ± 0.2 s, with a peak memory usage of 3.55 ± 0.07 GB. For resource‑constrained scenarios, we note that FACE can be run on class subsets, with a smaller rank, or fewer steps (our loss plateaus early). We can discuss these computational analyses in the appendix.\n\n- **Applicability to ViTs**: FACE, like other post-hoc concept extraction, assumes a split f=h∘g. Directly applying FACE “as it is” to transformers like ViT is non-trivial, as ViTs lack fixed spatial feature maps, and “head” is often deeper/nonlinear. However, we see this as promising future work and will make this limitation explicit.\n\n- **Failure modes**: In Section 4.3, we discussed a failure case of FACE when an excessively large lambda value is used in high-class datasets like Imagenet and COCO, which degrades both faithfulness and accuracy. In addition, a very small rank can also underfit concepts. Second, as the reviewer pointed out, FACE is currently suitable for architectures where the split of the network into two functions is possible. We can extend the existing limitation by incorporating these points. Also, we have acknowledged lambda sensitivity (hyperparameter of FACE) as a limitation and leave fully automatic selection for future work. However, our current work still provides extensive sweeps and shows broad stable regions. \n\n- **Human evaluation**: The core problem FACE tackles is faithfulness, precisely because human plausibility can diverge from what the model actually optimizes. Therefore, we prioritized model-grounded perturbation tests over human judgment. Human studies are more appropriate for concept-bottleneck or user-facing systems where concepts are meant to match human notions; FACE targets alignment with the model’s internal reasoning. Hence, we refrained from human evaluation in this work. However, we can add this as our limitation."}, {"review_id": "KShNWDCYpY", "reviewer": "Reviewer_FuWA", "summary": "This paper aims to address an identified weak spot in concept-based XAI, which arises from neglecting the alignment of extracted concept representations with the model's inference process. The authors propose FACE (Faithful Automatic Concept Extraction), which elegantly combines the reconstruction loss of a sought NMF solution with a Kullback-Leibler divergence term to guide the optimization process toward minimizing deviations in the model's inference behavior. While the reviewer believes that the paper is a step in the right direction with an already solid foundation, its grounding in related work and the robustness of the evaluation could be improved.", "questions": "*Questions*\n- FACE does not appear to be mathematically limited to application on the penultimate layer of the model, yet the setup may lead the reader to believe so (e.g., the use of terms like \"classifier head,\" see line 87). Have you applied the method to other intermediate layers of the model(s), and compared the results to those of related approaches?\n- The faithfulness evaluations described in Section 3.2 are performed at the granularity of model accuracy. This means you will only observe measurable effects if there are sufficient data points for which the model changes its predicted class. Did you consider measuring (and averaging) changes in output logits for each individual data point instead? This way, the imposed change does not need to be dominant enough to affect the softmax output and aggregate over a large sample to become quantifiable. This more granular and sensitive approach would better reflect the true effect of each ablation step on the model, and would be easier to observe (since it is not obscured by the softmax layer).\n- Equation 2: Did you consider using alternatives to KL divergence as baselines for preserving model inference behavior—e.g., simple MSE (potentially on pre-softmax logits) or sample-wise cross-entropy—which are both symmetric, in contrast to KL divergence?\n- How does the asymmetric nature of KL divergence affect the optimization of the FACE objective?\n- Consider moving the discussion of Figure 2a, currently found at lines 226ff, to Section 4.1, where the relevant context is introduced.\n- Typo in line 723: It should read FACE underperforms, not slightly underperforms.\n- The qualitative results in Figure 3 are discussed in a very subjective manner. To the reviewer, all methods' explanations appear plausible, while FACE might have found a shortcut in the particular facial features it highlights. It would be highly interesting to further investigate the differing concept representation qualities of each method to better understand their respective utilities in uncovering meaningful aspects of model inference behavior.\n- Table 1: U-Spr should include a downward-pointing arrow, as the text describes the metric measures the \"average fraction of active (non-zero) concepts.\" A high value would indicate a dense feature encoding with low sparsity.\n- Table 2: Please unify the number formatting for consistency and clarity.\n- Have you considered directly comparing FACE to ACE in the main paper, given that ACE is mentioned in the appendix? Including it in the main comparison would strengthen the empirical analysis.", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "*Strengths*\n- The proposed solution is simple and elegant, and can be easily combined with various other post-hoc, concept-based explanation approaches.\n- The authors have delivered clean and well-structured code for reproducibility.\n- The use of faithfulness testing through concept insertion and deletion as a quantitative means to estimate concept quality is a good choice.\n- Overall, the paper is well written, and the setup of the experiments is logical, although not fully developed or completed.\n\n\n*Weaknesses*\n- The paper is focused on a narrow range of related work, revolving around direct comparisons with methods ACE, ICE, and CRAFT. This leaves the impression that all works in this area solely focus on reconstruction for concept extraction while completely ignoring the model's use of those features. Since all presented and compared methods aim to communicate the semantics of latent features via examples, the reviewer recommends extending the scope of the related work and the comparative discussion to also include Relevance Maximization (RM) from [Achtibat et al., 2023]. While RM does not attempt to identify influential concepts via reconstruction using NMFs or SAEs and instead relies on naturally emerged neurons (a limitation acknowledged by the authors), it proposes an alternative to the widely-used Activation Maximization (AM) approach. RM selects representative examples for neurons/features, primarily reflecting the model's use of them during inference. As such, RM constitutes a contrasting perspective—no reconstruction but a focus on model behavior vs. reconstruction with limited or no incorporation of model inference—on concept representation, which could optionally be combined with all compared approaches. In the end, RM follows the same goals as FACE and would thus be a valuable addition to the discussion.\n- Figure 1 is missing important information. For one, how were concepts C1 and C2 selected? Furthermore, the representational misalignment evaluation is purely qualitative, and the claim that FACE highlights concepts more aligned with model reasoning than the compared approaches should be supported with quantitative data in the figure.\n- The concept deletion and insertion evaluation is based on model accuracy, which renders the evaluation procedure insensitive to small changes in the model during ablation. See Questions section.\n- Since the primary goal of FACE is to help end-users better understand neural network predictions, and its major effect on interpretability is via the augmentation of presented examples to communicate concepts, a user study is critically missing. This component would elevate the more qualitative aspects of the paper’s evaluation.\n- Lines 178+ and 207+ use inconsistent notations for Jacobian matrices, i.e., $\\nabla$ and $J$.\n- It is unclear whether the 4k to 10k samples per dataset are used for the optimization of each method, whether there is a hold-out set, or whether both training and testing of the concept representation are performed on the same sample set. This information is crucial for interpreting the results in Section 4.3. Based on lines 307ff, the observed results could be explained by the KL term causing the FACE solution to overfit on the training subset—if such a subset exists.\n- In the context of Section 4.2, it is not clearly described how \"concept importance\" is measured. Do the authors directly use activation as a measure of importance (which previous literature, e.g., [Achtibat et al., 2023], shows is not correlated with concept utility), or do they use another method?\n- The results in Figure 4 / Section 4.3 are missing a proper baseline for comparison. For example, if accuracy with COCO remains largely unaffected regardless of the choice of $\\lambda$, does the added KL term in the FACE optimization have any effect at all? How do ICE or CRAFT perform under the same settings?\n\n*Remark*\n- The anonymized code base is not fully anonymized, as the comments at the head of the FACE related files enable the identification of the author(s)", "was_revised": true, "final_justification": "the authors have shown that the flaws listed in my review, leading to a disfavourable rating, can be fixed for a camera ready version of the paper, should it be accepted successfully. Still, AC should check whether the non-anonymized code submitted with the paper will have an impact on the final decision.", "rebuttal": "We sincerely thank the reviewer for their thoughtful review and for highlighting that FACE proposes an elegant solution to addresse a weak spot in concept-based XAI. Below, we clarify some concerns: \n- **Broader context:** We agree with the reviewer that focusing only on ACE/CRAFT/ICE can make it seem as if all automatic concept methods ignore model usage. While our focus was on automatic concept extraction methods that followed similar factorization-based techniques, we will explicitly discuss Relevance Maximization in the related‑work section, highlighting the conceptual contrast to FACE. RM is “no reconstruction + direct model behavior,” while FACE is “reconstruction with explicit prediction alignment via KL.” Both aim to expose what the model actually uses, but through different techniques. \n\n- **Fig. 1 Concepts chosen and quantitative support:** In Fig. 1, concepts C1​, C2​ were simply the top Sobol‑importance concepts for each method. We will state this explicitly in the caption/text. The qualitative “misalignment” claim is supported by our faithfulness metrics (C‑Ins/C‑Del) and KL on outputs (Tables 1–2). We will add the class-specific scores in the caption.\n\n- **User study:** Our primary target is predictive faithfulness, not human plausibility. Human studies are essential when concepts are supposed to match human semantics (e.g., concept-bottleneck models). In contrast, FACE purposefully reveals potentially “non‑plausible” but model‑critical cues (shortcuts, biases). We will make this rationale explicit. A user study, while valuable for downstream interpretability interfaces, is beyond the scope of this faithfulness-focused work. We will add this to our current limitation.\n\n- **Penulimate layer limitation?** FACE is not restricted to the penultimate layer. For any intermediate layer $l$, we simply have to define $g_{\\le \\ell}$​ as the network up to that layer and $h_{>\\ell}$ as the remaining layers. We focused on the penultimate layer to match prior works and, most importantly, to target higher-level, semantically coherent concepts because, as activation merge towards the final layer, more high-level concepts are revealed. We have verified this qualitatively as extracted concepts shift from textures/edges/patterns in early blocks to specific high-level parts (eg, body parts like ears, fur for animals). Quantitatively, faithfulness metrics are the lowest in the earlier layers, consistent with Fel et al. Neurips 2024. We can add this discussion in Appendix. \n\n- **Accuracy granularity vs. logits?** We agree that logit-level signals are more sensitive. Note, however, that our evaluation already includes a distribution-level measure: the KL divergence between original and perturbed predictions, which is computed on the full softmax output, not on accuracy. Still, to directly address the reviewer’s point, we computed additional logit-level insertion/deletion curves on 10K ImageNet images (ResNet34) and got the following scores:\n| Metrics/Method    | FACE | CRAFT | ICE |\n| -------- | ------- | ------- | ------- |\n| C-Ins  | 0.9466    | 0.8386 | 0.6403 |\n| C-Del | 0.9472     | 0.8413 | 0.6448 |\n\nWe can add this additional evaluation in the Appendix for all datasets and models.\n- **Alternative to KL and asymmetry impact:** We chose forward KL(p∥q) between the original and reconstructed predictive distributions because our goal is to preserve the predictive distribution, not just the logits scale. KL operates on probabilities and gives the Pinsker bound used in §3.3, directly tying the objective to output deviation. Still, for completeness, we swapped current-KL loss with (i) MSE on logits and (ii) Reverse KL (q||p) on 10K ImageNet images (Resnet34). Forward KL (p||q) gave the best faithfulness evaluation. MSE preserved top-1 accuracy but allowed large distribution shifts and lower faithfulness. Reverse KL sat in between. Asymmetry did not cause instability in practice because, in our optimization, p is fixed (original outputs) and q is optimized through updating U, W matrices. Table below shows the result. We will add this ablation to the appendix. \n\n| Metrics/Loss    | Original forward KL (p\\|\\|q) | MSE logits loss | Reverse KL (q\\|\\|p) |\n| -------- | ------- | ------- | ------- |\n| C-Ins  | 0.969    | 0.911 | 0.959 |\n| C-Del | 0.891     | 0.824 | 0.875 |\n| MSE | 0.497    | 0.525 | 0.501 |\n| KL | 0.220     | 0.315 | 0.279 |\n\n- **On U-Spr metric:**  We thank the reviewer for catching the ambiguity. Our current definition of U-Spr does measure the active fraction (density), so lower is better for sparsity. Because our projected gradient descent updates with non‑negativity rarely drive entries exactly to zero, FACE ends up with many very small but non-zero coefficients, yielding higher density than ICE/CRAFT, whose multiplicative updates often drive many entries to zero, so they look “sparser”. Two simple fixes are possible (we can apply one in the camera-ready): a) report density explicitly as U-Den (downarrow) or b) report sparsity as 1-U_Spr. This correction does not change our final conclusion: *FACE still achieves the lowest KL (prediction consistency) and strongest C-Del/C-Ins faithfulness. Moreover, the explanation-level sparsity, captured by C-Gini in concept importance, remains the highest in FACE. Our goal is faithful, succinct explanations, not necessarily the sparsest latent code.*\n- **On ACE evaluation:**  We thank the reviewer for the suggestion. Our main comparison targeted NMF-based methods to isolate the effect of adding prediction alignment to the same factorization family. ACE relies on superpixel segmentation and uses a clustering-based approach. As noted by Fel et al. (CRAFT), this can introduce artifacts that degrade faithfulness metrics. While the reconstruction-based metrics are specific to factorization-based techniques, we evaluated ACE on ResNet34 10K ImageNet images, and got the following result: \n| Metrics/Method    | FACE | CRAFT | ICE | ACE |\n| -------- | ------- | ------- | ------- |------- |\n| C-Ins  | 0.969    | 0.0.932 | 0.908 |0.691 |\n| C-Del | 0.891     | 0.752 | 0.484 | 0.510|\n| C-Gini | 0.895 | 0.835 | 0.537 | 0.429|\n\nWe can add ACE analysis on all dataset and models to the appendix.\n- **Subjectivity in Fig. 3 and possible shortcuts:**  We agree that qualitative judgments can be subjective. Our C‑Del test directly probes causality: zeroing FACE’s top “face-feature” concepts in CelebA Gray Hair sharply drops logits/accuracy, while removing low-importance concepts barely moves predictions. We ran an additional experiment to inspect if FACE captures the explicit bias in the dataset. We created adversarial patch images that force misclassification on ImageNet. FACE isolates the patch as the top concept, identifying the shortcut cues used by the model.\n- **Concept importance:** We do not use raw activation magnitudes. Instead, we follow CRAFT and estimate variance-based Sobol importance using Halton sampling. This measures how much each concept contributes to the output variance when masked/unmasked, aligned with model behavior. This is more robust than activation alone. We have mentioned this in Line 100-101 and will make this explicit wherever introduced. \n- **Data usage, splits, overfitting:**  FACE never updates the classifier; it only factorizes fixed latent activations with a low‑rank NMF. Following prior concept‑XAI work (ACE/CRAFT/ICE), we take a class of correctly classified images we want to explain for a given model (this comes from a test-set and not a training set). We then fit U,W on the set of images to explain the class of prediction (e.g., all correctly classified ImageNet “church” images). The computed low-rank matrices U and W are now used to explain the model prediction of “church”. For CelebA, we train the model using a train set and only compute concepts on a test set.\n-  **The results in Figure 4 / Section 4.3:** In the sweep (Fig. 4), once λ exceeds ~10³, both C‑Ins and C‑Del drop sharply, showing the KL term is doing work: too much weight on KL harms the factorization’s ability to isolate causal concepts. Accuracy stays high because the model can still guess the right label, as Accuracy is a step function; large logit drops can occur without flipping the argmax, so accuracy barely moves.  This is exactly the situation where your proposed logit‑based evaluation is more sensitive, and we adopted it. The table below shows the logit-based AUC evaluation: \n| Metrics/Method    | FACE | CRAFT | ICE |\n| -------- | ------- | ------- | ------- |\n| C-Ins  | 0.961    | 0.761 | 0.668 |\n| C-Del | 0.934    | 0.781 | 0.722 |\n- **We will also incorporate other suggestions you mentioned regarding notation, formatting and placement of some sections.**"}]

Author rebuttal:

Meta-review: The paper proposes a method for concept discovery specifically for non-negative matrix factorization, with one additional constraint that encourages similarity between classification outputs between original activation matrix and the reconstructed one. The method is very simple and the paper performed various analyses and proposed metrics like del/insertion -- similar to what has been used for counterfactual analysis in the literature. The reviewers made unanimous recommendations and the AC confirms it.
Decision: Accept (poster)