Explain Yourself, Briefly! Self-Explaining Neural Networks with Concise Sufficient Reasons

Authors: Shahaf Bassan, Ron Eliav, Shlomit Gur
Primary area: interpretability and explainable AI
Keywords: XAI, explainability, explainable AI, self-explaining neural networks, Formal XAI, sufficient reasons, abductive explanations, interpretability, feature selection

Abstract:
*Minimal sufficient reasons* represent a prevalent form of explanation - the smallest subset of input features which, when held constant at their corresponding values, ensure that the prediction remains unchanged. Previous *post-hoc* methods attempt to obtain such explanations but face two main limitations: (1) Obtaining these subsets poses a computational challenge, leading most scalable methods to converge towards suboptimal, less meaningful subsets; (2) These methods heavily rely on sampling out-of-distribution input assignments, potentially resulting in counterintuitive behaviors. To tackle these limitations, we propose in this work a self-supervised training approach, which we term *sufficient subset training* (SST). Using SST, we train models to generate concise sufficient reasons for their predictions as an integral part of their output. Our results indicate that our framework produces succinct and faithful subsets substantially more efficiently than competing post-hoc methods while maintaining comparable predictive performance.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730317414953

thread_type:
official_review

<thread_object>
creator: reviewer_wp7b

<content>
summary:
This paper aims at the generation of minimal sufficient reasons for model decision explanation. The authors provide theoretical results on the intractability of getting cardinally minimal explanations under specific settings, showing the difficulty of this problem. The authors also propose a new self-supervised approach called sufficient subset training to reduce cost for generating faithful sufficient reasons and reduce sensitivity to OOD examples.

strengths:
* The paper provides rigourous theoretical results on the computational complexity of generating minimal sufficient reasons in different settings. 
* The paper is well-written and easy to follow. 
* The proposed SST method provides an elegant and computationally efficient way to generate a sufficient reason. The experiment results show strong scalibility to larger datasets.

weaknesses:
While the paper presents rigorous theoratical results and a novel empirical method, it's a bit unclear to me how these two contributions are connected. Discussion on how the theoratical analysis motivates the SST method could make the paper more integrated.

questions:
1. In Table 2, how is the faithfulness of SST training using different masking strategy in training? It will be interesting to see if model trained with specific masking stretagy also generalizes to other faithfulness metrics. 
2. In Figure 4, is Cardinality Mask Size (%) axis refering to the percentage of mask size? According to the figure, cardinality goes to 0.5% instead of 50% as discussed in Line 428-429.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730666308544

thread_type:
official_review

<thread_object>
creator: reviewer_gpey

<content>
summary:
The authors propose a novel training framework called Sufficient Subset Training, aimed at generating minimal sufficient reasons as integral outputs of neural networks. 

Unlike post-hoc methods that face computational challenges and out-of-distribution (OOD) concerns, SST directly incorporates the explanation generation process during training. 

This is achieved by adding dual propagation and integrating two additional losses: (i) a faithfulness loss for ensuring sufficiency and (ii) cardinality loss for promoting minimal subsets. The method is validated through experiments on various image and language tasks, demonstrating that it provides concise, faithful, and efficient explanations, outperforming several post-hoc methods at finding minimal subset.

strengths:
- The introduction of SST as a self-explaining mechanism is novel and could be impactful for fields focusing on model interpretability.
- The paper thoroughly defines different sufficiency types and explains how SST addresses them through tailored masking strategies.
- The experiments cover a range of datasets and architectures, showing that SST can be applied across different domains.
- The theoretical analysis on the intractability of obtaining minimal sufficient reasons adds depth to the contribution.

weaknesses:
However, I have identified some issues, both major (**M**) and minor (**m**), that require attention.

**M1** Real Practical Insights Are Limited:  while the theoretical framework and empirical demonstrations are solid, the practical insights into what we learn about the internal workings of neural networks are minimal. The method emphasizes explanation generation without significantly advancing our understanding of how or why models arrive at decisions. Addressing this could bridge the gap between theoretical contributions and **practical interpretability**.

**M2** Validity of Faithfulness Metrics: the paper does not thoroughly discuss the potential limitations of the faithfulness metric used in the evaluation. Faithfulness as defined could lead to explanations that superficially appear sufficient but do not align with internal mechanism or key feature used by the model. Say it otherwise, the model may be using completely different strategy to classify x and xS; zS¯.

**M3** Limited Hyperparameter Discussion: the paper mentions tuning the cardinality loss coefficient but provides no analysis on the impact / visual results of tuning the hyperparameters. What are the results of adjusting \( \tau \), step size \( \alpha \) in robust masking. A sensitivity analysis would add valuable context.

Now for the minors (**m**) issues:

**m1** Related Work Section Is Limited:  
The related work section lacks depth, with insufficient coverage of recent advancements in XAI and self-explaining methods, especially those published in top venues over the last few years.

**m2** Justification for Masking Strategies:  
The choice of certain masking strategies (e.g., baseline and probabilistic) could be better justified. Also, could some baseline lead to better results / scores ? Overall, the rationale for selecting these over alternative approaches should be discussed in more detail.

**m3** Typos and Clarity:  
I havent found a lot of typo, just "probablistic" (should be "probabilistic").

questions:
- Could we incorporate some prior like finding mask in a lower dimensional subspace (7x7 grid of the original image) to allow a smoother explanation and more tractable problem for the model ? I think this would be valuable for both interpretability purpose and theoretical verficiation (it become easier to generate gt).
- How does the method's approach compare when applied to more interpretable models (e.g., decision trees)?
- Can the SST approach provide insights into potential biases in training data by analyzing consistent feature patterns?


**Overall, while the theoretical considerations and framework of the paper are strong, the practical interpretability of models remains underexplored. Despite this, the proposed method is an interesting step toward integrating explanation generation within model training, justifying an acceptance with room for further practical development.**
</content>
</thread_object>

</thread>
<thread>
creation date:
1730939084766

thread_type:
official_review

<thread_object>
creator: reviewer_npb3

<content>
summary:
This paper proposes a new training method called Sufficient Subset training, which proposes learning neural networks with a dual objective that first predicts the label but also predicts a mask of sufficient inputs that is enough to predict the object. They show that training a network this way gives faithful sufficient subset explanations, that are often smaller than those generated with post-hoc methods, and much faster to produce.

strengths:
- Very well written, the description of previous works in terms of 3 definitions is particularly clear and helpful. 
- Interesting theoretical results justifying the use of training time intervention
- Clever and relatively simple method
- Some good results, especially on MNIST

weaknesses:
- Decent loss in performance/overall less impressive results on ImageNet and BERT
- Table 2 and 3 missing sufficiency results of your models on the metric it wasn't trained on. Seems unfair to report baselines on all metrics but your methods only under the best metric.
    - I'm worried the self-explanation might overfit to the specific training scenario and not work well in other settings.
- Having very disjoint input sets looks pretty weird for image explanations, what do you think is the use case for this type of explanation? Overall could use some more discussion on why sufficiency explanations are useful/what is the proposed use case. Have you experimented with favoring more continuous regions?

questions:
- What's stopping the model from learning a "cheating" solution such as it almost always only looks at a certain subset of the inputs and this subset is always the explanation? This would probably not be what we want?
- What is the robustness of your different masking strategies on Table 2 and Table 3 when evaluated in a setting not trained on?
- Is the epsilon ball used too small? Theoretically the robust sufficient reasons case should be harder than baseline etc, but seems like you can learn much smaller explanations with this method. As an extreme case, an adversarially robust model could have a sufficient explanation of size 0 and still be faithful in this metric, but this seems pretty detached from the idea of sufficient explanation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733152160442

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers once again for their valuable feedback and for recognizing the significance of our work.

We have addressed many reviewer concerns in their respective threads. However, we would like to address two general comments raised by reviewers in the general thread due to their importance.

**Practicality aspects of sufficient explanations**

Minimal sufficient explanations are a widely sought-after approach in explainability, with numerous methods proposed to achieve them (e.g., 1-4). The core idea is to identify a minimal subset of features that determine a prediction, allowing one to focus on the essential "reason" for the outcome while excluding redundant features. Some of the figures and demonstrations in our work clearly highlight that, despite the remarkably small size of our generated subsets, the prediction can often be accurately inferred solely from the subset itself, without needing to consider its complement. This level of faithfulness is not always achieved by post-hoc methods like SIS or approaches that produce significantly larger subsets (e.g., Anchors). Moreover, previous work demonstrated that this form of explanation often provides humans with deeper insights into predictions than additive attributions [4].

We thank reviewers GpEy and NPb3 for their insights on human-centric aspects and enhancements of sufficient explanations, such as simplified inputs and broader use cases. While these are valuable directions, they are not unique to SST but broadly relevant to this explanation type. We believe that our work marks a significant step forward by substantially improving the scalability of generating sufficient explanations while enhancing their faithfulness and conciseness. This advancement lays a strong foundation for future efforts aimed at refining the human-level interpretation of these explanations.


**The generalization of different maskings to different sufficiency conditions and the faithfulness metric**

Like many explanation frameworks, sufficient reasons can be defined in various ways due to the inherent challenge of specifying the "missing" complement of a subset $S$. This issue also appears in Shapley values, which permit multiple definitions [5], and in metrics like fidelity [6]. To address this gap, common in many other explainability approaches, we explored and categorized definitions into three forms: baseline, probabilistic, and robust sufficiency. We demonstrated how various masking configurations can adapt to these forms, enabling users to choose a "form" of sufficiency to guide concise subset extraction. Stricter sufficiency yields larger subsets, while looser forms produce smaller ones.

Naturally, training with a specific masking form tailored to a particular definition of sufficiency may not generalize well to others, similar to how adversarial training guaranteeing robustness against $\ell_{\infty}$ attacks does not ensure robustness against $\ell_0$ attacks. However, as noted in our responses to reviewers NPb3 and wp7b, some sufficiency-based maskings can generalize well, especially when they subsume other forms. Moreover, SST can be adapted to include multiple masking criteria across batches. Lastly, as Reviewer GpEy noted, diverse sufficiency definitions also yield varied faithfulness criteria. To address this, our paper explores multiple forms aligned with these definitions.

In conclusion, while we acknowledge the reviewers' concern that the diverse definitions of sufficiency and faithfulness present challenges - such as generalization and evaluation - we see this as a broader issue in post-hoc explanations rather than a limitation specific to SST. We believe that our exploration of many sufficiency definitions and the demonstration of SST's ability to enhance scalability, faithfulness, and conciseness across distinct sufficiency forms highlights its versatility and applicability.

We will focus on enhancing the discussion of these aspects in the final version and will incorporate additional results, as outlined in the individual threads.

Once again, we thank the reviewers for their insightful feedback.

[1] What made you do this? understanding black-box decisions with sufficient input subsets (Carter et al., AI’STATS 2019)

[2] Verix: Towards verified explainability of deep neural networks (Wu et al., Neurips 2023)

[3] Abduction-based explanations for machine learning models (Ignatiev et al., AAAI 2019)

[4] Anchors: High-precision model-agnostic explanations (Ribeiro et al., AAAI 2018)

[5] The many Shapley values for model explanation (Sundararajan et al., ICML 2020)

[6] On the (in) fidelity and sensitivity of explanations (Yeh et al., Neurips 2019)
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes Sufficient Subset Training, a method for training neural networks to generate sufficient reasons for predictions, which combines dual propagation with faithfulness and cardinality losses, to ensure explanations are concise and faithful. The experiments demonstrate the method's scalability, conciseness, and efficiency compared to post-hoc methods.

The strengths of the paper lie in a novel integration of explanation generation into the training process, empowered with a theoretical analysis of sufficient reasons. While the practical interpretability insights appear to be limited, the merits of the paper outweigh its weaknesses.

I agree with the consensus of the reviewers and recommend accepting the paper.