Title: FACE: Faithful Automatic Concept Extraction
Venue: NeurIPS 2025
Accepted: yes
Keywords: computer vision

faithful interpretability

concept based explanations

interpretability

explainable AI
Abstract: Interpreting deep neural networks through concept-based explanations offers a bridge between low-level features and high-level human-understandable semantics. However, existing automatic concept discovery methods often fail to align these extracted concepts with the model’s true decision-making process, thereby compromising explanation faithfulness. In this work, we propose FACE (Faithful Automatic Concept Extraction), a novel framework that combines Non-negative Matrix Factorization (NMF) with a Kullback-Leibler (KL) divergence regularization term to ensure alignment between the model’s original and concept-based predictions. Unlike prior methods that operate solely on encoder activations, FACE incorporates classifier supervision during concept learning, enforcing predictive consistency and enabling faithful explanations. We provide theoretical guarantees showing that minimizing the KL divergence bounds the deviation in predictive distributions, thereby promoting faithful local linearity in the learned concept space. Systematic evaluations on ImageNet, COCO, and CelebA datasets demonstrate that FACE outperforms existing methods across faithfulness and sparsity metrics.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to address an identified weak spot in concept-based XAI, which arises from neglecting the alignment of extracted concept representations with the model's inference process. The authors propose FACE (Faithful Automatic Concept Extraction), which elegantly combines the reconstruction loss of a sought NMF solution with a Kullback-Leibler divergence term to guide the optimization process toward minimizing deviations in the model's inference behavior. While the reviewer believes that the paper is a step in the right direction with an already solid foundation, its grounding in related work and the robustness of the evaluation could be improved.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths*\n- The proposed solution is simple and elegant, and can be easily combined with various other post-hoc, concept-based explanation approaches.\n- The authors have delivered clean and well-structured code for reproducibility.\n- The use of faithfulness testing through concept insertion and deletion as a quantitative means to estimate concept quality is a good choice.\n- Overall, the paper is well written, and the setup of the experiments is logical, although not fully developed or completed.\n\n\n*Weaknesses*\n- The paper is focused on a narrow range of related work, revolving around direct comparisons with methods ACE, ICE, and CRAFT. This leaves the impression that all works in this area solely focus on reconstruction for concept extraction while completely ignoring the model's use of those features. Since all presented and compared methods aim to communicate the semantics of latent features via examples, the reviewer recommends extending the scope of the related work and the comparative discussion to also include Relevance Maximization (RM) from [Achtibat et al., 2023]. While RM does not attempt to identify influential concepts via reconstruction using NMFs or SAEs and instead relies on naturally emerged neurons (a limitation acknowledged by the authors), it proposes an alternative to the widely-used Activation Maximization (AM) approach. RM selects representative examples for neurons/features, primarily reflecting the model's use of them during inference. As such, RM constitutes a contrasting perspective—no reconstruction but a focus on model behavior vs. reconstruction with limited or no incorporation of model inference—on concept representation, which could optionally be combined with all compared approaches. In the end, RM follows the same goals as FACE and would thus be a valuable addition to the discussion.\n- Figure 1 is missing important information. For one, how were concepts C1 and C2 selected? Furthermore, the representational misalignment evaluation is purely qualitative, and the claim that FACE highlights concepts more aligned with model reasoning than the compared approaches should be supported with quantitative data in the figure.\n- The concept deletion and insertion evaluation is based on model accuracy, which renders the evaluation procedure insensitive to small changes in the model during ablation. See Questions section.\n- Since the primary goal of FACE is to help end-users better understand neural network predictions, and its major effect on interpretability is via the augmentation of presented examples to communicate concepts, a user study is critically missing. This component would elevate the more qualitative aspects of the paper’s evaluation.\n- Lines 178+ and 207+ use inconsistent notations for Jacobian matrices, i.e., $\\nabla$ and $J$.\n- It is unclear whether the 4k to 10k samples per dataset are used for the optimization of each method, whether there is a hold-out set, or whether both training and testing of the concept representation are performed on the same sample set. This information is crucial for interpreting the results in Section 4.3. Based on lines 307ff, the observed results could be explained by the KL term causing the FACE solution to overfit on the training subset—if such a subset exists.\n- In the context of Section 4.2, it is not clearly described how \"concept importance\" is measured. Do the authors directly use activation as a measure of importance (which previous literature, e.g., [Achtibat et al., 2023], shows is not correlated with concept utility), or do they use another method?\n- The results in Figure 4 / Section 4.3 are missing a proper baseline for comparison. For example, if accuracy with COCO remains largely unaffected regardless of the choice of $\\lambda$, does the added KL term in the FACE optimization have any effect at all? How do ICE or CRAFT perform under the same settings?\n\n*Remark*\n- The anonymized code base is not fully anonymized, as the comments at the head of the FACE related files enable the identification of the author(s)\n\nQUESTIONS:\n*Questions*\n- FACE does not appear to be mathematically limited to application on the penultimate layer of the model, yet the setup may lead the reader to believe so (e.g., the use of terms like \"classifier head,\" see line 87). Have you applied the method to other intermediate layers of the model(s), and compared the results to those of related approaches?\n- The faithfulness evaluations described in Section 3.2 are performed at the granularity of model accuracy. This means you will only observe measurable effects if there are sufficient data points for which the model changes its predicted class. Did you consider measuring (and averaging) changes in output logits for each individual data point instead? This way, the imposed change does not need to be dominant enough to affect the softmax output and aggregate over a large sample to become quantifiable. This more granular and sensitive approach would better reflect the true effect of each ablation step on the model, and would be easier to observe (since it is not obscured by the softmax layer).\n- Equation 2: Did you consider using alternatives to KL divergence as baselines for preserving model inference behavior—e.g., simple MSE (potentially on pre-softmax logits) or sample-wise cross-entropy—which are both symmetric, in contrast to KL divergence?\n- How does the asymmetric nature of KL divergence affect the optimization of the FACE objective?\n- Consider moving the discussion of Figure 2a, currently found at lines 226ff, to Section 4.1, where the relevant context is introduced.\n- Typo in line 723: It should read FACE underperforms, not slightly underperforms.\n- The qualitative results in Figure 3 are discussed in a very subjective manner. To the reviewer, all methods' explanations appear plausible, while FACE might have found a shortcut in the particular facial features it highlights. It would be highly interesting to further investigate the differing concept representation qualities of each method to better understand their respective utilities in uncovering meaningful aspects of model inference behavior.\n- Table 1: U-Spr should include a downward-pointing arrow, as the text describes the metric measures the \"average fraction of active (non-zero) concepts.\" A high value would indicate a dense feature encoding with low sparsity.\n- Table 2: Please unify the number formatting for consistency and clarity.\n- Have you considered directly comparing FACE to ACE in the main paper, given that ACE is mentioned in the appendix? Including it in the main comparison would strengthen the empirical analysis.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel method for generating concept-based explanations for deep neural networks. The core of FACE is the integration of a KL divergence regularization term into the Non-negative Matrix Factorization (NMF) framework. This addition is designed to ensure that the extracted concepts are not only interpretable but also faithful to the model's decision-making process. The authors provide a compelling case for their approach, supported by theoretical guarantees and extensive empirical evaluation.\n\nSTRENGTHS AND WEAKNESSES:\nThe primary contribution of FACE is its ability to ensure that these concepts are not only interpretable but also faithful to the model’s actual decision-making process, addressing a critical limitation in existing methods where extracted concepts may misrepresent the model’s reasoning. By incorporating a KL divergence term that minimizes the difference between the model's predictions on original and reconstructed data, FACE ensures that the discovered concepts are aligned with the model's predictive behavior.\n\nThe empirical evaluation of FACE is both thorough and convincing. The authors test their method on three diverse datasets (ImageNet, COCO, and CelebA) and two different model architectures (ResNet-34 and MobileNetV2), demonstrating the generalizability of their approach. The inclusion of multiple metrics to evaluate the quality of the matrix factorization, the faithfulness of the explanations, and the complexity of the explanations provides a comprehensive view of FACE's performance. The ablation studies on the regularization strength and the matrix decomposition rank further strengthen the paper by showing the impact of these hyperparameters on the results.\n\nWeaknesses : 1) While the application is novel, the core technical contribution is relatively incremental - essentially adding a KL divergence term to standard NMF optimization. The theoretical analysis, while correct, relies on well-established inequalities (Pinsker's inequality) rather than developing new theoretical insights.\n\n2) The strong alignment with the model’s predictions may lead to concepts that are overly specific to the particular model and dataset used. This could reduce the generalizability of the concepts to other models or domains, limiting their broader applicability. For example, if a model was trained with biases or on a specific data distribution, the extracted concepts might reflect those biases rather than generalizable features.\n\n3) Since FACE relies on NMF, which is inherently linear and suited for extracting additive, parts-based representations, it may struggle to capture nonlinear or higher-order concepts (e.g., “smiling” in face recognition, which involves interactions between multiple facial features). While the classifier head might learn such interactions, the concepts themselves are limited to linear combinations, potentially missing complex relationships critical to the model’s reasoning.\n\n4) FACE requires backpropagation through the classifier head to compute the KL divergence, which can be computationally intensive, especially for large models or datasets. Although the paper claims modest compute requirements, suitable for consumer-grade GPUs, this may still pose challenges in resource-constrained environments or real-time applications. A detailed analysis of computational complexity, including runtime and memory usage, would strengthen the paper’s claims.\n\nQUESTIONS:\n1) FACE requires backpropagation through the classifier head to compute the KL divergence, which can be computationally intensive, especially for large models or datasets. Although the paper claims modest compute requirements, suitable for consumer-grade GPUs, this may still pose challenges in resource-constrained environments or real-time applications. A detailed analysis of computational complexity, including runtime and memory usage, would strengthen the paper’s claims.\n\n2) The method assumes a model architecture with a clear separation between an encoder and a classifier head, as seen in ResNet34 and MobileNetV2. Its applicability to other architectures, such as transformers or models with integrated components, is unclear. This limitation restricts FACE’s generalizability across diverse deep learning models, which is a significant concern given the variety of architectures used in practice.\n\nOverall this paper makes an incremental contribution to concept-based explainability by addressing the important problem of faithfulness in automatic concept extraction. The theoretical foundation is sound, the experimental evaluation is comprehensive, and the results clearly demonstrate the method's effectiveness. However, the work suffers from insufficient analysis of failure modes, and significant hyperparameter sensitivity that may limit practical adoption. Hence, I believe the paper would benefit from: (1) more thorough analysis of computational complexity and failure cases, (2) evaluation on a broader range of architectures and domains, (3) human evaluation studies, and (4) development of adaptive hyperparameter selection strategies.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work the authors propose FACE: a variant of NMF that incorporates a KL penalty on classification distributions to encourage reconstructions that also preserve downstream performance. The authors argue that this regularization encourages a more faithful concept extraction that aligns with the behavior of the model, and in the work demonstrate that this is the case through experiments while validating FACE’s competitive performance on traditional concept learning metrics such as sparsity and MSE.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n * I really like the concept deletion and insertion metrics as a good causal test of the faithfulness of your explanation. I think these are the most important experiments in the paper. Is it possible to visualize the change in the image as you extract and delete concepts? Or show which concepts are being extracted and deleted in order? I would find that very interesting, at least in an example in the appendix.\n * The experiments validate the efficacy of the method.\n * I like the example of the gray hair and head vs face qualitative example. I would be interested to see if you could fix this “spurious correlation” in the model (for example train on young and old people with gray hair, assuming their faces look very different) and reapply FACE to see if the model is corrected properly.\n * I think the faithfulness of concept explanations is pretty under explored and I liked the methodology used by the authors to test this.\n## Weaknesses:\n * Your proposition is just a direct statement of Pinsker’s inequality. I think the interesting component in your analysis is pointing out via Taylor approximation that even with small reconstruction, we have no bound on the change in logits. I might rewrite the proof to highlight this or drop it as a theoretical statement, similar to how you simply cite Pinkser’s in line 213. It would be interesting to provide a converse result or example where the gradient h(UW^T) is large despite the reconstruction being small.\n * As part of this your conclusion/implication of the proof is not really saying anything: if the KL is bounded, of course the model outputs are bounded, even without converting to TV.\n * Overall, I would rewrite 3.3 and 3.4 to highlight the lack of control we have over model outputs via the taylor expansion; then in 3.4 you can focus on the linear case and point out even in this case the jacobian of the softmax is out of our control and maybe you can construct a counterexample here.\n * Also, given you are modeling your classifier head as linear, you might be able to say something stronger regarding the lipschitzness/convergence of your algorithm.\n * Figure 4 doesn’t actually tell me that the KL divergence penalty is improving faithfulness: you say the metrics increase to $\\lambda = 10^{-5}$ but to me the metrics appear flat. What happens when $\\lambda = 0$? Figure 4 indicates to me that these metrics would not change, which is in disagreement with Table 2 assuming CRAFT is equivalent to $\\lambda = 0$.\n\nQUESTIONS:\n* In your convergence proof, Appx C Eqn (8), I think you dropped the Lipschitz Smoothness term. Then you could also explain exactly how sufficiently small $\\nabla$ must be depending on the Lipschitz smoothness of your loss.\n * Do you want your sparsity metric to be high or low? In the table you want it to be high but in the Appendix based on your sparsity definition seems like you want it to be low as you are summing L0 norms.I’m also not super familiar with the Gini metric you are using, so I would make it extremely clear how the sparsity metrics are computed and thus if you want them higher or lower.\n * Also, why would increasing rank improve sparsity? I thought with NMF as you increase rank your decompositions are over more vectors, i.e. a less-sparse decomposition. \n * See weaknesses.\n\nI am very willing to improve my score, but I think some of the above changes should be considered to improve the paper.\n\nLIMITATIONS:\nThis is limited to models with a downstream task, which does not transfer well to generative systems.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents FACE (Faithful Automatic Concept Extraction), a novel concept-based explanation method for deep neural networks. FACE builds on Non-negative Matrix Factorization (NMF) but introduces a KL-divergence-based regularization term to align concept-based explanations with the model’s actual predictive behavior. By incorporating classifier supervision, the method ensures that reconstructed representations from the learned concepts not only approximate encoder activations but also preserve the classifier’s output distribution. The paper provides theoretical justification that minimizing KL divergence bounds prediction deviation and promotes local linearity in the concept space. Empirical evaluations across ImageNet, COCO, and CelebA datasets show that FACE achieves higher faithfulness and explanation sparsity than state-of-the-art methods (e.g., CRAFT and ICE), while remaining competitive in reconstruction quality.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths.\n1. The proposed methodology is well-grounded, supported by a clear mathematical formulation and rigorous theoretical justification.\n2. The empirical evaluation is comprehensive, demonstrating the effectiveness of the approach across multiple datasets (ImageNet, COCO, CelebA) and model architectures (ResNet34, MobileNetV2).\n3. This work offers valuable insights into the nature of faithful model interpretation, and the use of KL divergence regularization, especially with its theoretical connection to prediction alignment via total variation bounds, is a particularly notable contribution. The method consistently outperforms existing baselines on key metrics such as faithfulness (C-Ins, C-Del) and sparsity (C-Gini).\n4. The use of Concept Insertion (C-Ins) and Concept Deletion (C-Del) to quantitatively assess the impact of concepts on model predictions is another strength. While related perturbation-based metrics have been explored in prior work, their adaptation to the concept-based explanation setting and direct application to model performance evaluation represents a meaningful contribution.\n\nWeakness.\n1. In Figure 3 and lines 258–259, the authors suggest that FACE reveals the model's reliance on facial features for the \"gray hair\" class. However, if the model is leveraging spurious cues (i.e., shortcuts), such as facial features unrelated to hair color, one would expect the prediction to change when these facial features are altered. It would be helpful to investigate whether such changes in facial features lead to prediction shifts. If confirmed, this would further highlight the effectiveness of FACE in faithfully capturing the model’s true decision-making behavior, distinguishing it from baselines that may align more with human intuition than actual model reasoning.\n\n2. In Proposition 1, the parameter ε is critical to the theoretical claim that KL divergence regularization guarantees predictive faithfulness and local linearity. However, the paper could benefit from a more detailed discussion of how ε behaves in practice. For example, is ε explicitly bounded in real-world applications? If not, how can one reliably assert that the L1 distance between predictive distributions (‖p − q‖₁) is small? Further clarification on how ε is interpreted, estimated, or controlled during training would strengthen the theoretical and practical understanding of the proposed method.\n\nQUESTIONS:\nMost of my concerns and questions are addressed in the Weaknesses section.\n\nLIMITATIONS:\nThe authors adequately address the limitations and potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}