Uncertainty-Aware Counterfactual Explanations using Bayesian Neural Nets

Authors: Ben Batten, Francesco Leofante, Nicola Paoletti, Alessio Lomuscio, Mehran Hosseini
Primary area: interpretability and explainable AI
Keywords: Counterfactual Explanations, Bayesian Neural Networks, BNN

Abstract:
A counterfactual explanation describes the smallest input change required to alter
the prediction of an AI model towards a desired outcome. When using neural net-
works, counterfactuals are obtained using variants of projected gradient descent.
Such counterfactuals have been shown to be brittle and implausible, potentially
jeopardising the explanatory aspects of counterfactuals. Numerous approaches
for obtaining better counterfactuals have been put forward. Even though these
solutions address some of the shortcomings, they often fall short of providing
an all-around solution for robust and plausible counterfactuals. We hypothesise
this is due to the deterministic nature and limitations of neural networks, which
fail to capture the uncertainty of the training data. Bayesian Neural Networks
(BNNs) are a well-known class of probabilistic models that could be used to over-
come these issues; unfortunately, there is currently no framework for developing
counterfactuals for them. In this paper, we fill this gap by proposing a formal
framework to define counterfactuals for BNNs and develop algorithmic solutions
for computing them. We evaluate our framework on a set of commonly used
benchmarks and observe that BNNs produce counterfactuals that are more robust,
plausible, and less costly than deterministic baselines

Decision: reject
Reviewer scores: 5, 1, 5, 5

Reviews and discussion:
<thread>
creation date:
1729099135723

thread_type:
official_review

<thread_object>
creator: reviewer_hqju

<content>
summary:
The authors propose a method for generating counterfactual explanations (CE) for model ensembles. The method is compared against deterministic CE and another baseline based on model ensembles. Experiments are performed on four tabular data sets and MNIST.

strengths:
1. The idea of using Bayesian neural networks (BNNs) for the generation of counterfactual explanations is interesting.
   
2. The manuscript is both well-written and well-structured. The method and experiments are reproducible. I appreciate the level of detail.

3. Empirical results indicate that the proposed method overall performs well on the considered metrics and across a variety of low-dimensional data sets.

weaknesses:
1. The contribution of the proposed method is small. From my understanding, the method only differs from the standard counterfactual explanation method proposed by [1] in that an empirical average over model predictions is considered, rather than just a single model prediction. 
   
2. I do not see how this work is about BNNs. The only BNN feature that the proposed method explicitly takes advantage of is the fact that BNNs allow for sampling multiple models. However, training an ensemble of different models or training a neural network multiple times with different random seeds also yield multiple models.

3. It is not clear to me why this method performs well. I suspect that the choice of the prior over model parameters plays a crucial role in the performance of the proposed method. I cannot find which prior is chosen and why this prior leads to *robust, plausible and less costly* (line 26) counterfactual explanations. Given that the paper claims to be about BNNs, I would appreciate an extensive discussion and evaluation around this aspect. Adressing this point could likely be beneficial in regard to addressing weaknesses 1 and 2 as well.

[1] Wachter, S., Mittelstadt, B., & Russell, C. (2017). Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harv. JL & Tech., 31, 841.

questions:
line 18: It may be that I do not understand this sentence correctly, but Bayesian neural networks are certainly not the only type of neural network that express uncertainty. For instance, a standard classifier with softmax outputs expresses uncertainty in $Y|X$ and it is not a Bayesian neural network. Please consider clarifying or omitting this.

line 25: I would generally suggest replacing the term *counterfactual* with *counterfactual explanation*. The term *counterfactual* is overloaded and could be misleading.

line 59: See comment for line 18.

line 70: MNIST is not a proper vision data set. If the authors want to make a point about vision data sets, please demonstrate experiments on data sets like CIFAR or CelebA. Otherwise, I would suggest removing this claim.

line 71: The experimental results do not support the claim that the method *consistently* produces counterfactual explanations that lie
closer to the data manifold. For instance, the ensemble method performs better on the LOF score on the spambase data set. I would suggest removing such strong claims that lie at odds with the empirical results.

line 188: Equation (7) seem incorrect. I believe the constraint should just be $\mathcal{B}(x_c) = 1$.

line 265: In general, I appreciate the qualitative results. However, I am not sure how well they support the claims. All of the counterfactual explanations shown look quite similar to me. Thus, I suggest moving either Figure 1 or Figure 2 to the appendix.

line 406: I would expect that cross entropy loss from prior work is better suited than the linear loss for classification. Why is the linear loss chosen over cross entropy?

line 481: *we will focus*. The word *will* should likely not be there, given that the related work section is at the end of the manuscript.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730181948749

thread_type:
official_review

<thread_object>
creator: reviewer_ryh5

<content>
summary:
Counterfactual explanations have generated tremendous interest since they provide suggestions on how to change the outcome of a machine learning model with as less change as possible. This paper addresses the problem of finding counterfactuals targeted towards Bayesian neural networks (BNNs) and develops algorithms for this task. They define counterfactuals for the BNN setting and call them probabilistic counterfactuals. They evaluate their framework on some popular benchmark datasets and claim that BNNs produce counterfactuals that are supposedly more robust, plausible, and less costly than the two baselines (naive ensembling and MLP) as demonstrated through experiments.

strengths:
The key strengths of this work are:
1. They propose a definition of counterfactual explanations in the context of Bayesian Neural Networks (BNNs) that they call probabilistic counterfactuals. The approximate BNN output is written as $\tilde{B}(x)=\frac{1}{N} \sum f_w(x)$, and then they seek out algorithms to find the counterfactual for this version of the approximate BNN output. Then, they can perform a gradient-based optimization to find counterfactuals in this setting.
2. Several tabular datasets have been considered. In addition, they also consider a vision dataset - MNIST (admittedly a simple one but it is interesting to see the visualizations). They experimentally show that the generated counterfactuals have additional benefits like being more plausible and supposedly more robust that naive ensembling and MLP.

weaknesses:
1. There is prior work on robust Bayesian recourse: https://arxiv.org/pdf/2206.10833
It would be great to discuss the differences and novelty in connection with this related work.

2. While the paper says that they propose a new technique of generating counterfactuals for BNNs, the optimization strategy is quite similar to generating counterfactuals for ensembles. Also, see related work on counterfactuals under argumentative ensembling: https://arxiv.org/abs/2312.15097 
It would be great to highlight if there is any additional mathematical/optimization nuance in the BNN counterfactual generation strategy, and also discuss how it compares with this related work in terms of technique. In the experiments section, the paper does say that counterfactuals in BNN are like doing ensemble over infinite models but in the practical implementation, how is this realized and is there a major difference?

3. This work has close connections with the robustness of counterfactuals, an extensive body of work in the counterfactual explanations literature [1-4], which has not been compared with. While the paper does cite a survey paper on robustness, there are different facets of robustness in counterfactuals (to model changes and to input perturbations). The relevant works that are closest to this paper should be discussed if not compared with rather than being deferred to the survey. 

For instance, the proposed robustness metric shares close similarities with prior works on robustness [2,3] which also sample in the neighborhood of a counterfactual.

This work also computes the validity of counterfactuals, sharing close similarities with literature on robustness under model changes [1-2]. Particularly, it would be nice to study how the validity is affected when incorporating some of these baselines for robust counterfactuals. While the authors do compute validity, the validity does not seem to be decreasing for all datasets either. 
It would be great to see how alternate robust counterfactual generation mechanisms perform in this BNN setting, and compare their plausibility, robustness, and validity. This work includes a limited number of baselines for comparison. 

4. It would be great to provide intuitions on how and where the improvement in plausibility arises from.

[1] S. Upadhyay, S. Joshi, and H. Lakkaraju, "Towards robust and reliable algorithmic recourse", NeurIPS 2021.
[2] F. Hamman, E. Noorani, S. Mishra, D. Magazzeni and S. Dutta, "Robust counterfactual explanations for neural networks with probabilistic guarantees", ICML 2023.
[3] M. Pawelczyk, T. Datta, J. van-den Heuvel, G. Kasneci and H. Lakkaraju, "Probabilistically robust recourse: Navigating the trade-offs between costs and robustness in algorithmic recourse"

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730336794208

thread_type:
official_review

<thread_object>
creator: reviewer_ucco

<content>
summary:
The papers introduces a techinque for generating counterfactual explanations (CFX) for bayesian neural networks (BNN). The method is based on approximating the gradient update via a sampling scheme to get parameters $w$ from the posterior distribution $p(w|x)$. The authors compare their approach against MLPs and ensemble of MLPs, and their findings show better CFX generation as compared to baselines on various metrics like plausibility and robustness.

strengths:
- To the best of my knowledge there aren't prior works that have explicitly worked on CFX generation for BNNs; contributing to the novelty of the work.

- The paper in general is well-written, with clear motivation, technical details, and experimental resutls.

weaknesses:
- Technical soundness of the work is severly limited. The main results presented in Table 1 and Table 2 do not show the standard deviation/error, which means the findings are not conclusive. Also, the authors do not mention whether the experiments was done for multiple random seeds, so I assume the numbers reported are for a single random seed and denotes the mean over 50 test samples. The experiments must be conducted for *multiple random seeds* to arrive at conclusive results. Even in the current state, the average results over 50 test sample indicate the mean performance is quite similar across methods, and maybe the differences observed are within the standard deviation across them. 

- I don't find the qualitative results in Figure 1 to be convincing, based on one sample (and single random seed) we cannot conclude whether a method is better as compared to others. 

- There is an inherent difference in comparison between MLPs and BNNs in their learning objective, as MLPs are trained using cross-entropy and BNNs are trained using the linear loss. Shouldn't we use the same learning objective for this comparison? 

- Overall, I also question what would be the significance if the authors could conclusively show a result demonstrating that CFX generation is better with BNNs than MLPs. Would BNNs be better than MLPs for the task of prediction? If not, the question of explainability does not arise since the motivation behind CFX is to explain models that would be good at prediction. One could argue that BNNs are more interpretable than MLPs, but the goal with explainability research is to not compare different learning algorithms in terms of how interpretable they are. Rather the goal is to develop techniques that could generate explanations for the same learning algorithm, a typical trend in prior works.

questions:
I have listed my major questions in the weaknesses section above, please refer to them for discussion phase. A minor comment; there is a typo in equation 2, it should be $w_i$.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731088253781

thread_type:
official_review

<thread_object>
creator: reviewer_lpnz

<content>
summary:
This paper proposes a framework for generating counterfactual explanations (CFXs) using Bayesian Neural Networks (BNNs). The authors formally define counterfactual explanations for BNNs, metrics to evaluate the quality of CFXs and provide an algorithm to develop them. Through experimentation the authors show that CFXs generated through BNNs do well on all the metrics compared to point-estimated neural networks or ensembles.

strengths:
- The paper does a good job at explaining the problem from ground up and providing motivation on why counterfactuals are important. 

- Overall I like the presentation of the method via an algorithm. It also does a great job at explaining the metrics through which we measure the quality of counterfactuals.

weaknesses:
- I am little concerned about the novelty of the paper. The paper combines existing ideas of BNNs and counterfactual explanations so I am a bit on the fence here. 

- I think the experimentation could be expanded here. Particularly I am interested in learning how this method scales to larger models and datasets where HMC is not feasible. It'll be great if the authors can use larger models & datasets in their results. 

- BNNs are storage and compute intensive so their practical feasibility is harder and this negatively affects the adoption of the proposed approach . How do the authors envision their method for practical scenarios given the challenges in deploying BNNs for real-world applications?

questions:
- In section 2, there are multiple requirements listed such as sparsity, proximity, actionable, validity, palusibility, etc. Can the authors comment how these are connected and can be consolidated? Since the final experiments only show 3 metrics. 

- In algorithm, for computing the gradients wrt x, do you use grad accumulation?

- Lines 58-59 - "these limitations stem from the deterministic nature of traditional neural networks, which fail to capture the inherent uncertainty in the data" - which limitations are we talking about here?

- In the experiments, there are multiple references to cost, and an assertion that the proposed method leads to counterfactual explanations  with lower cost. However, the authors also say that the cost of getting counterfactual explanations for BNNs is lower than that of ensembles (lines 415-416). Can the authors clarify in detail regarding the costs and why their method leads to lower cost? 

- In eq(7) what is l? And in line 192, what do you mean by `has n units`? It'll be good if the authors use more widely adopted notation for BNNs. See [1] for example.


References

[1] Blundell, C., Cornebise, J., Kavukcuoglu, K. and Wierstra, D., 2015, June. Weight uncertainty in neural network. In International conference on machine learning (pp. 1613-1622). PMLR.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a framework for generating counterfactual explanations using Bayesian neural networks (BNNs). The main claimed practical contributions of the paper concern better plausibility, robustness and cost of generated counterfactuals.  

The main concerns about this paper are its novelty, scalability, practical feasibility, relation to previous work (for example Bayesian recourse, counterfactuals for ensembles) and the experimental evaluation (e.g., variability of the results not reported, unsupported claims on vision data). I believe the authors have not addressed these concerns satisfactorily,  especially when it comes to discussing and comparing to related work and the significance of the results. 4 out 4 reviewers recommend rejection and I agree.