Semi-supervised counterfactual explanations
Authors: SURYA SHRAVAN KUMAR SAJJA
Sumanta Mukherjee
Satyam Dwivedi
Vikas C. Raykar
Venue:  (2021)
Keywords: Semi-supervised feature representation
counterfactual explanations

Abstract: Counterfactual explanations for machine learning models are used to find minimal interventions to the feature values such that the model changes the prediction to a different output or a target output. A valid counterfactual explanation should have likely feature values. Here, we address the challenge of generating counterfactual explanations that lie in the same data distribution as that of the training data and more importantly, they belong to the  target class distribution. This requirement has been addressed through the incorporation of auto-encoder reconstruction loss in the counterfactual search process.  Connecting the output behavior of the classifier to the latent space of the auto-encoder has further improved the speed of the counterfactual search process and the interpretability of the resulting counterfactual explanations. Continuing this line of research, we show further improvement in the interpretability of counterfactual explanations when the auto-encoder is trained in a semi-supervised fashion with class tagged input data. We empirically evaluate our approach on several datasets and show considerable improvement in-terms of several metrics.

Under review as a conference paper at ICLR 2021
SEMI-SUPERVISE COUNTERFACTUAL EXPLANATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Counterfactual explanations for machine learning models are used to find minimal interventions to the feature values such that the model changes the prediction to a different output a target output. A valid counterfactual explanation should have likely feature values. Here, we address the challenge of generating counterfactual explanations that lie in the same data distribution that of the training data and more importantly, they belong to the target class distribution. This requirement has been addressed through the incorporation of auto-encoder reconstruction loss in the counterfactual search process. Connecting the output behavior of the classifier to the latent space of the auto-encoder has further improved the speed of the counterfactual search process and the interpretability of the resulting counterfactual explanations. Continuing this line of research, we show further improvement in the interpretability of counterfactual explanations when the auto-encoder is trained in a semi supervise fashion with class tagged input data. We empirically evaluate our approach several datasets and show considerable improvement in-terms of several metrics.
INTRODUCTION
Recently counterfactual explanations have gained popularity as tools of explainability for Al-enabled systems. A counterfactual explanation of a prediction describes the smallest change to the feature values that changes the prediction the form of a statement like, "You were denied a loan because your annual income was 30, 000. If your income had been 45, 000, you would have been offered loan ". Counterfactual explanations are important in the context of Al-based decision -aaiing systems because they provide the data subjects with meaningful explanations for given decision and the necessary actions to receive a more favorable/desired decision in the future. Application of counterfactual explanations in the areas of financial risk mitigation, medical diagnosis, criminal profiling, and other sensitive socio-economic sectors is increasing and is highly desirable for bias reduction.
predefined output. A counterfactual explanation usually takes
Apart from the challenges of sparsity, feasibility, and actionability, the primary challenge for counter- factual explanations is their interpretability. Higher levels of interpretability lead to higher adoption of Al-enabled decision makinn systems. Higher values of interpretability will improve the trust amongst data subjects Al-enabled decisions. AI models used for decision making are typically black-box models, the reasons can either be the computational and mathematical complexities associated with the model or the proprietary nature of the technology. In this paper, address the challenge of generating counterfactual explanations that are more likely and interpretable. A counterfactual explanation is interpretable if it lies within or close to the model's training data distribution. This problem has been addressed by constraining the search for counterfactuals to lie in the training data distribution. This has been achieved by incorporating an auto-encoder reconstruction loss in the counterfactual search process. However, adhering to training data distribution is not sufficient for the counterfactual explanation to be likely. The counterfactual explanation should also belong to the feature distribution of its target class. To understand this, let us consider an example of predicting the risk of diabetes in individuals as high or low. sparse counterfactual explanation to reduce the risk of diabetes might suggest a decrease in the body mass index (BMI) level for an individual while leaving other features unchanged. The model might predict a low risk based on this change and the features of this individual might still be in the data distribution of the model. However, they will not lie in the data distribution of individuals with low risk of diabetes because of other relevant features of low-risk individuals like glucose tolerance, serum insulin, diabetes pedigree, etc.

---

Under review as a conference paper at ICLR 2021
To address this issue, authors in Van Looveren & Klaise (2019) proposed to connect the output behavior of the classifier to the latent space of the auto-encoder using prototypes. These prototypes guide the counterfactual search process in the latent space and improve the interpretability of the resulting counterfactual explanations. However, the auto-encoder latent space is still unaware of the class tag information. This is highly undesirable, especially when using prototype guided search for counterfactual explanations on the latent space. In this paper, we propose to build a latent space that is aware of the class tag information through joint training of the auto-encoder and the classifier. Thus the counterfactual explanations generated will not only be faithful to the entire training data distribution but also faithful to the data distribution of the target class.
We show that there are considerable improvements in interpretability. sparsity and proximity metrics can be achieved simultaneously, if the auto-encoder trained in a semi-supervised fashion with class tagged input data. Our approach does not rely on the availability of train data used for the black box classifier. It can be easily generalized to a post-hoe explanation method using the semi-supervised learning framework, which relies only on the predictions on the black box model. In the next section we present the related work. Then, in section 3 we present preliminary definitions and approaches necessary to introduce our approach. In section 4 we present our approach and empirically evaluate it in Section 3
2 RELATED WORK
Counterfactual analysis is a concept derived from from causal intervention analysis. Counterfactuals refer to model outputs corresponding to certain imaginary scenarios that we have not observed or cannot observe. Recently Waachter al. (2017) proposed the idea of model agnostic (without opening the black box) counterfactual explanations, through simultaneous minimization of the error between model prediction and the desired counterfactual and distance between original instance and their corresponding counterfactual. This idea has been extended for multiple scenarios by Mahaaan et al. (2019), Ustun et al. (2019) .Pooyadzii et al, (2000) based on the incorporation of feasibility constraints, actionability and diversity of counterfactuals. Authors in Mothilll et al. (2020) proposed framework for generating diverse set of counterfactual explanations based on determinantal point processes. They argue that wide range of suggested changes along with a proximity to the original input improves the chances those changes being adopted by data subjects. Causal constraints of our society do not allow the data subjects reduce their age while increasing their educational qualifications. Such feasibility constraints were addressed by Mahajan et al.) (2019) and Joshi et al. (2019) using a causal framework. Authors in Mahajan al.] (2019) addresses the feasibility of counterfactual explanations through causal rela relationship constraints amongst input features. They present a method that uses structural causal models to generate actionable counterfactuals. Authors in Joshi et al. (2019) propose to characterize data manifold and then provide an optimization framework to search for actionable counterfactual explanation on the data manifold via its latent representation. Authors in Poyiadzi jet al. (2020) address the issues of feasibility and actionability through feasible paths, which are based on the shortest path distances defined via censity--eighted metrics.
An important aspect of counterfactual explanations is their interpretability. A counterfactual explana- tion is more interpretable if it lies within or close to the data distribution of the training data of the black box classifier. To address this issue Dhurandhar et al. (2018) proposed the use of auto-encoders to generate counterfactual explanations which are "close" to the data manifold. They proposed incorporation of an auto-encoder reconstruction loss in counterfactual search process to penalize counterfactual which are not true to the data manifold. This line of research was further extended by Van Looveren & Klaise (2019), they proposed to connect the output behaviour of the classifier the latent space of the auto- encoder using prototypes. These prototypes improved speed of counterfactual search process and the interpretability of the resulting counterfactual explanations.
While Van Looveren & Klaise (019) connects the output behaviour of the classifier to the latent space through prototypes, the latent space still unaware of the class tag information. We propose to build a latent space which is aware of the class tag information through joint training of the auto-encoder and the classifier. Thus the counterfactual explanations generated will not only be faithful the entire training data distribution but also faithful the data distribution of the target class. In post-hoc scenario where access to the training data is not guaranteed, we propose to use the input-output pair data of the black box classifier to jointly train the auto-encoder and classifier in the semi-supervisedd learning framework. Authors in Zhai & Zhang (2016), Gogna et al. (2016) have explored the use semi- supervised auto-encoders for sentiment analysis and analysis of biomedical signal analysis. Authors
2

---

Under review as a conference paper at ICLR 2021
in Haiyan et al. (2015) propose joint framework of representation and supervised learning which guarantees not only the semantics of the original data from representation learning but also fit the training data well via supervised learning, However, as far as our knowledge goes, semi-supervised learning has not been used to generate counterfactual explanations and we experimentally show that semi-supervised learning framework generates more interpretable counterfactual explanations.
3 PRELIMINARIES
Let D = Me be the supervised data set where x; E X' is d-dimensional input feature space for a classifier and y E y 11.2..... E} the set of outputs for a classifier. Throughout this paper we assume the existence of black box classifier h X + 3 trained on DD such that ÿ = 1xx = arg max ply = c D) where ply C x, D) is prediction score/probability for class with an input Based on Wachter et al. (0117) counterfactual explanations can be generated by trading off between prediction loss and sparsity. This is achieved by optimizing a linear combination of the prediction loss (Lggred) and loss of sparsity (Esparsity) L = L--Lgred Laparsiity Prediction loss typically measures the distance between current prediction and the target class, whereas sparsity loss function measures the perturbation from the initial instance Xo with class tag la- This approach generates counterfactual explanations which can reach their target class with a sparse perturbation to the initial instance. However, they need not necessarily respect the input data distribution of the classifier, hence, resulting in unreasonable values for x
Authors in Dhurandhar et al. (2018) addressed this issue through incorporation of La reconstruction error for x evaluated through an autoencoder (AE) trained on the input data X' as Lyecon (x) |x (x)(x))l where AEx represents the auto-encoder trained on entire training dataset X. The auto-encoder loss function L Lescon penalizes counterfactual explanations which do not lie within the data-distribution. However, Van Looveren & Klaise (2019) illustrated that incorporating L Lizcon in may result in counterfactual explanations which hie inside the input data-distribution but they may not be interpretable. To this end, Van Looveren & Klaise (2019) proposes addition of a prototype loss function Lpooo to L to make x more interpretable and improve the counterfactual search process through prototypes in the latent space of auto- encoder. Prooto is the L error between the latent encoding of x and cluster centroid of the target class in the latent space of the encoder defined as proto, (short for target prototype) as lxproto(x, proto,) = || FENCx(x) - proto, where ENCxx represents encoder part of the auto-encoder AEx and E (x) represents the projection of on to the latent space of the auto-encoder. Given atarget t, the corresponding proto, can be defined as ENCx(x())
proto;
(1)
where xf represent the input instances corresponding to the class such that (ENCxx(x)) k=
are the K nearest neighbors of EN (xxx). For applications where target class t is not pre-defined, a suitable replacement for proto, is evaluated by finding the nearest prototype proto, of class j * to to the encoding of Xu. given by j = arg min |ENCx (xu) proto, ll2- Then prototype loss Lpreto can be defined as Isproto (x, proto,)
(x))) proto, 1|2- According to Van Looveren & Klaise (2019) the loss function Lproto explicitly guides the encoding of the counterfactual explanation to the target prototype (or the nearest protoytpe proto ). Thus we have a loss function L given by
L = c- Lpred + Liporsity +7 + + 0 Lyroto
where c, 7 and 0 are syper-parameters tuned globally for each data set. For detailed descriptions of these parameters and their impact on the counterfactual search, we refer the readers to Ltd In this paper, we propose an alternate version of this loss function. The constituent loss functions L Lyecon and I otoo are based on an auto-encoder trained in an unsupervised fashion. In the next section we motivate the use of an auto-encoder trained using class tagged data a semi-supervised fashion.
4 SEMI-SUPERVISED COUNTERFACTUAL EXPLANATIONS
y
Φx
x
ΦD x
X
y
x Z
x
Z.
x
(a) Classification
(b) Autoencoder
(c) Jointly trained model
3

---

Under review as a conference paper at ICLR 2021
For the supervised classification data set D (X.3') machine learning methods learn classifier model h : :X' + y (see figure Ia). This process of learning involves minimizing a loss function of the form: Eentropy = - Σ3 >; I(y = j) log (p(y = j x, D)). Auto-encoder is a neural network framework which learns a latent space representation z E Z for input data x E X' along with an invertible mapping (xxx) (see Ib) in an unsupervised fashion. The subscript X' represents the unsupervised training of Φx and Ox only on the dataset X The un-supervised learning framework tries to learn data compression using continuous map 6x, while minimizing the reconstruction
loss: Eautoenc V Σ N -x!? In this paper we consider only undercomplete autoencoders that produce a lower dimension representation (Z) of an high dimensional space (X)), while the decoder network ensures the reconstruction guarantee (x X (xx()) A traditional undercomplete auto-encoder captures the correlation between the input features for the dimension reduction.
4.1 JOINT TRAINING: SEMI-SUPERVISED LEARNING
For the purpose of generating counterfactual explanations, we propose to use generic neural architecture for undercomplete auto-encoder jointly trained with the classifier model (figure [cp The proposed system would be trained with a joint loss, defined as a linear combination of cross- entropy loss and reconstruction loss Ejoint W1 Eentropy + W22 . Eautoene- This architecture relies on the class tag information y/ for every imput x; used to train the auto-encoder. The subscript D in figure 1c represents the training of ΦD and Φp using dataset X' tagged with classes from y in the spirit of semi-supervised learning. This jointly trained the auto-encoder and the corresponding encoder will be represented by AEp and ENCp. This generic architecture can be implemented in multiple ways based on selection of classifier model h. architecture of neural network for ΦD and weights W1 and W2. Also, if the entire supervised is unavailable and the class tag information is available only for D: = [x;, where i = I..... m < n and un un-tagged data is available as Du = {x:] where m + 1....., ..... n then the joint training approach presented in figure Ic can be generalized to semi-supervised learning framework.
4.2 CHARACTERIZATION
THE LATENT EMBEDDING SPACE
The lower dimension representation (embedding) E Z obtained through joint training using D= (x, Y") is relatively more richer than its unsupervised counterpart trained using X. Thus, we claim that the counterfactual explanations generated through auto-encoders trained on class tagged data will not only be more faithful to the training data distribution but also be more faithful to the target class data distribution and hence more interpretable. We evaluate this claim by generating counterfactual evaluations on several data sets and compare them through suitable metrics. However, to illustrate the motivation behind the proposed approach we consider the German credit data se This data set consists of credit risk for over 1000 individuals. It contains 13 categorical and continuous features. The target variable is binary decision whether borrower will be a defaulter (high r risk) or not (low risk). The counterfactual explanations in this case would typically be the necessary feature changes to make an individual low risk. In figure we try to characterize and visualize the embedding space in two dimensions (Z C IR2) for the German data set. We plot classifier outputs and classification probability contours the latent embedding space z E 2 for the separately trained auto-encoder AEx (henceforth termed as unsupervised auto-encoder) in figures 2a and 2c and the jointly trained auto-encoder AED (henceforth termed as semi-supervised auto-encoder) in figures 2b and 2d.
For a fair comparison between unsupervised and semi-supervised frameworks, we use the same neural network framework for both ox and dp. It can be observed that the unsupervised auto- encoder produces an embedding that does not clearly separate hi gh risk and oo ri sk classes (figure 2ap. In the embedding space, the classification probability contours also overlap significantly (figure 2c) for unsupervised case. For the semi-supervised auto-encoder, the embeddings and their classification probability contours are clearly more separated than their unsupervised counterparts (figures 2b and 2d). Clearly the separation of class clusters is due to inclusion of classification task in the auto-encoder training process. The resulting embedding is indicative of distinct distribution of features between low risk and high risk individuals. Thus, semi-supervised framework learns highly discriminative embeddings and can be utilized to better counterfactual explanations. To do so, we re-define the loss function I, for counterfactual explanation as:
L = c Lyreed + Iisporsity + I.Pccom + 0 T.Proto
https://archi ve. iss. uci...ii.ooo www./////ml...... wwwp////m/atttttteeeeeeeeeeeeeee log+ ogt (german*credi t+data)
4

---

Under review as a conference paper at ICLR 2021
4.0
risk
High rise
high risk
3.0
2.0
8
1.0
0.0
0
0.0
0.5
1.0
2.5
3.0
z1
21
Unsupervised classification
(b) Semi-supervised classification
28
High risc
high
risk
2.0-
4very
1.0
proto
0.0-
0.5
3.0
4.02
1.0
(c) Unsupervised contours
(d) Semi-supervised contours
Figure 2: Characterizing the embedding space
To illustrate the effect Lproto in comparison with LX proto on the counterfactual generation process, we consider data instance in X (the feature space of the German data set) and term its corre- sponding encoding as query E Z. The class tag associated with query is high risk, hence, the target class for counterfactual explanation is low risk. Based on equation [ we evaluated the proto o associated with the class tag low risk and plotted it for both unsupervised (figure 2c) and semi-supervised (figure 2d) scenarios. It can be observed that proto for the semi-supervised scenario falls clearly in the low risk cluster as opposed to proto for the unsupervised. Under such scenarios, the counterfactual explanations generated will be more interpretable because the uncertainty associated with the class tag of the counterfactual is minimized. Apart from higher interpretability, reduction of uncertainty leads more robust counterfactual explanations. Because clear separation between clusters will result in faster counterfactual search towards target proto o without meandering towards other classes. the next section, we experimentally evaluate and compare unsupervised and semi-supervised approaches to generate counterfactual explanations.
EXPERIMENTS AND RESULTS
We show that semi-supervised training has the potential to generate more interpretable counterfactual explanations. However, interpretabilityy
function of several parameters of the joint training framework, parameters used for joint optimization of loss functions in the counterfactual search process, parameters of neural networks used for auto-encoders, and parameters of the models used for classification. Hence, we use common neural network architecture for both unsupervised and semi-supervised frameworks during the comparison. To handle categorical data, we generate categorical embedding based on the architecture proposed by López- Sópez-Sänchez al. (2018) and a corresponding decoding layer for the reverse lookup in our framework. We train the categorical embedding layer to achieve the data distribution faithful embedding definition. The embedding converts the categorical variable to continuous vector space, which is consumed into our auto-encoder (unsupervised and supersupervise)) frameworks.
5

---

Under review as a conference paper at ICLR 2021
5.1 DATASETS USED FOR EXPERIMENTS
For evaluation, we consider German credit data set, adult census data set and breast cancer Wisconsin (diagnostic) data set from Dua & Graff (017), MNIST data set from LeCun & Cortes (2010)
COMPAS data set from Larson et al. (2016 6) and PIMA dataset from Database (2016).
German credit data set: Predict an individual's credit risk using 20 distinct features. The data-set contains over 1000 entries. This data-set contains 13 categorical and 7 continuous features. The target variable a binary decision whether the borrower will be a defaulter or not.
Adult census data This is multivariate data-set based census data, with 12 distinct features. Each feature represents an individual from the census data, where the target variable is binary label representing whether the individual income exceeds $ 50K/yr. The data-set has over 45, 000 entries. In this study, we have ignored the entries with missing values.
MNIST data set p It is a database of 28 x 28 wundtwritten digit images with 60, 000 training, and 10, 000 test examples, with digit label. In this current work, we posed the recognition problem as a binary decision task of detecting odd or even digit.
Compas
#L COMPAS a commercial algorithm used by judges and parole officers for scoring a criminal defendant's likelihood of reoffending. The dataset contains over 10, 000 criminal defendants records in Broward County, Florida, and all 10 features considered by the COMPAS algorithm.
PIMAN This dataset is made available from the National Institute of Diabetes, Digestive, and Kidney Diseases. The objective of the dataset is to diagnostically predict whether a patient has diabetes or not. The dataset consists of several medical predictor variables, viz. BMI, insulin level, age, and so on, and one binary target variable suggesting whether the data is from a diabetes patient or not. All the features of this data-set are continuous in nature.
Breast Cancer Winsconsin tized images of fine needle aspirate (FNA) of breast masses. All features are continuous in nature. The objective of the dataset is to diagnostically predict whether the mass is malignant or benign.
Datase It contains features computed from the digi-
5.2 EXPERIMENTAL SETUP AND DEPENDENCIES
All experiments performed Linux machine running single core, 32 threads, Intel(R) Xeon(R) Gold 6130 @ 20GGGz processor with 256GB RAM. For each data-set, an unsupervised auto-encoder model, and one semi-supervised auto- antoder models trained. These models shared the same neural architecture but trained with different loss functions. The unsupervised auto-encoder trained with reconstruction loss only, while for the semi-supervised scenario the loss function is defined as a linear combination of reconstruction loss, and classification loss. We use Alibi Klaise al. to generate counterfactual explanations for both unsupervised and semi-supervised scenarios. Keras Chollet et al. (2015) library is used to build the corresponding auto-encoders frameworks.
5.3 COUNTERFACTUAL EVALCATION METRICS
For a given query instance x, the counterfactual explanation is xefe We used proximity, sparsity, and interpretability losses from Mothilal et al. (020); Van Looveren & Klaise (2019) as the evaluation metrics for the counterfactual explanations.
Proximity: This metric evaluates the distance between the query point and the counterfactual explanation. The proximity handled separately for continuous and categorical fields.
Ja,il CONT-PROXIMITY h MAD.
cfe q9
CAT-PROXIMITY = Σ Maall + xq..)
The measure of categorical proximity is normalized between [0, 1]. representing a fraction of categorical variables that need to be changed to reach the counterfactual explanation. The continuous proximity is normalized by the median absolute deviation (MAD).
http:
(Diagnostic)

---

Under review as a conference paper at ICLR 2021
Sparsity: Sparsity metric reports the fraction of features changed between the query (Xg). and the
counterfactual explanation (x5/1) and is defined as SPARSITY = 1 - & Ei=1 [ ofe * xq,;). The sparsity is uniformly defined over categorical and continuous features. A good counterfactual explanation desired to have higher sparsity value.
Interpretability; We have used two metrics IM, and M2 proposed by Van Looveren & Klaise (2019) to evaluate interpretability. Both these metrics use class-specific auto-encoders to estimate how much the counterfactual explanation conforms to the new class tag distribution. IM1 measures relative reconstruction error of target class over query class, while I MM2 measures relative improvement in reconstruction error of the target class over nonclass specific reconstruction error. If AEx represents the auto-encoder trained on the entire data set X. and AL, is class-specific autoencoder for the class i. Then the two metrics are described
|| (E, 2 |
[|AE,( AEx(x;
IM
1M,
AE (xg MI t
|x 111 +E
Dataset
S
Poat
Poont
IM1
IM2
SS
SS
U
SS
U
SS
U
SS
0.60
0.65
0.99
0.96
0.11
0.11
0.94
0.94
0.10
0.11
German Credit
0.05
0.04
0.02
0.05
0.11
0.09
0.06
0.06
0.03
0.03
COMPAS
/
0.72
0.64
0.89
0.82
1.34
1.76
2.18
2.97
0.43
0.46
a
0.20
0.27
0.31
0.38
1.05
1.52
3.43
4.45
0.41
0.39
Adult Income
μ
0.88
0.85
1.00
1.00
0.12
0.15
1.30
1.32
0.07
0.07
σ
0.04
0.09
0.00
0.00
0.01
0.01
0.46
0.48
0.05
0.05
μ 0.66
0.59
0.32
0.42
1.43
1.36
0.37
0.39
PIMA
σ
0.20
0.26
0.51
0.58
0.74
0.65
0.44
0.46
Cancer
μ
0.61
0.57
0.29
0.44
1.43
1.16
0.10
0.35
0.07
0.11
0.10
0.12
0.44
0.46
0.02
0.04
Table 1: The top header represents different metrics used for comparison, viz. Sparsity(S), categorical proximity (Poat). continuous proximity (Poont), interpretability metric (LM1), and interpretability metric 2 (2), Each metric is paired with two columns: SS representing the metric obtained using semi-supervised embedding and U representing the metric obtained using classical undercomplete autoencoder. 5.4 RESULTS
We have sampled over 100 instances from each data set and generated the corresponding counter- factual explanations to compare unsupervised (U) and semi-supervised (SS) frameworks using the sparsity. proximity, and interpretability metries as shown in table T A higher value of the sparsity metric indicates that the counterfactual explanation has been obtained by perturbing a lesser number of features. Across various data-sets, the counterfactual explanations produced by the (SS) framework sparser than the solution produced by (U) framework. (SS) framework fares better even for categorical proximity and continuous proximity metrics across all datasets except COMPAS dataset, where continuous proximity of (SS) framework is higher. The interpretability metric 7 TM, compares the reconstruction loss between query class distribution to target class distribution. The comparison yields an improvement in the (SS) framework for adult income and COMPAS data sets, suggesting the counterfactual explanation produced by semi-supervised embedding framework is better explained by the target class distribution. For other datasets IM, is either same for both the frameworks or it's 's slightly high for (SS) framework. Concerning M22, the (SS) framework is consistently better than that (U) framework. The improvement in IM, and I MM2 compared to baseline unsupervised frameworks is not drastically high, however, these results are still important because the marginal improvement in interpretability is happening simultaneously with consistent improvement sparsity. Sparse counterfactual explanations always run the risk of not belonging to the data distribution of the target class.
From figure 2c it is evident that sparse perturbation to the feature space can alter the class outputs either way Hence, a sparse counterfactual should have reduced interpretability. but on the contrary, interpretability results have improved, although marginally. Now we present some individual counterfactual explanations obtained during our experimentation. The feature changes in the counterfactual explanation generated using (SS) framework generally sparser compared to the feature
7

---

Under review as a conference paper at ICLR 2021
Month
Credit Amount
Installment %
Purpose
Decision
U
15 + 20
1778 + 2438
- 2.2
high~>low
SS
15 20
1778 + 2465
U
16 + 21
3 - 4
high->lo
SS
3050 - 3758
A44 +A48
Table 2: Counterfactual generated on German Credit dataset.
changes involved in the counterfactual explanations generated using the (U) framework. Few examples are showed in the table and table 3] Consider the second instance of the counterfactual query from the German Credit data-set, (SS) framework suggests: only by changing the loan application purpose from "Domestic Appliances" (A44) to "Retraining" (A48) the credit risk can go down significantly. This is associated with a corresponding change in the credit amount. The results obtained from the counterfactual of COMPAS data-set show evidence that (SS) framework is capturing the implicit correlation between various features. As an example in the second counterfactual instance (table 3 two features, age, and age-category simultaneously changed in the counterfactual explanation produced using semi-supervised embedding, while instances from unsupervised embedding failed to capture such relation. Contrary to the opinion that sparser counterfactual explanations may neglect to change other correlated variables, (SS) framework makes sure it captures them, thus remaining within target class distribution. The sparser counterfactual explanations generation also indicates that the model can mine out fewer features that maximally influence the classifier decision.
Age
Age category
Prior
Charge
Recidivism
U
25 45 > 245
+ 5
No Yes
SS
2 + 5
U
24
+ 48
+ 7
DL revoked - Robbery
Yes + No
SS
24 38
25 25 45
0 - 12
Table 3: Counterfactual instances for COMPAS data-set.
S8
8
5
g
6
g
2
5
g
9
2
q
6
5
g
Figure 3: Comparing decoded images of proto embeddings generated through unsupervised and semi-supervised frameworks.
Further, we have defined an odd-even digit classification task on the MNIST dataset. this case, embeddings obtained in semi-supervised fashion show clear class separation as in figure 2d We show the decoded images of proto embeddings generated through (U) and (SS) frameworks in figure 3 can be observed that the decoded proto values for the (SS) framework are robust, because for all the three even queries in 3 the digit 9 happens to be the decoded proto. The primary reason being: convergence of prototype guided search the class center in the embedding space. The well-formen digits are positioned around the class centers, while, the visually ambiguous digits are the class boundaries. Thus the resulting proto much stable in the (SS) framework. Embedding generated using unsupervised framework does not show any clear class separation, hence the resulting prototypes are often ill-formed digits, and produces varied decoded proto images, depending on the start query.
CONCLUSION AND FUTURE WORK
We have empirically demonstrated semi-supervised embedding produces sparse counterfactual ex- planations. Sparse counterfactual explanations the risk of not belonging the data distribution of the target class. However, semi-supervised embedding ensures that that guided prototype lies within the target cluster with certain "robustness". For future work, will explore the potential of more rich data representation using semi-supervised Variation Auto Encoders (VAE) and imposing causality and feasibility constraints derive more faithful data embeddings.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
François Chollet et al. Keras.
2015.
Pima Indians Diabetes Database. National institute of diabetes and digestive and kidney diseases.
https: / //www//wwwww...
ans ans-diabes abateeeeattaabase 2016.
Amit Dhurandhar, Pin-Yu Yu Chen, Ronny Luss, Chun-Chen Tu, Paishun Ting, Karthikeyan Shanmugam, and Payel Das. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. In Advances neural information processing systems, pp. 592-603, 2018.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http:://acchii ics.uci.ee www.w./ww.mmmmmcm.cm....
Anupriya Gogna, Angshul Majumdar, and Rabab Ward. Semi-superrvsed stacked label consistent au- toencoder for reconstruction and analysis of biomedical signals. IEEE Transactions Biomedical Fingineering, 64(9)):116-22255 2016.
Wu Haiyan, Yang Haomin, Li Xueming, and Ren Haijun. Semi-supervised autoencoder: A joint approach of representation and classification. In 2015 International Conference on Computational Intelligence and Communication Networks (CICN), pp. 1424-1430. IEEE, 2015.
Shalmali Joshi, Oluwasanmi Koyejo, Warut Vijithenjaronk, Been Kim, and Joydeep Ghosh. Towards realistic individual recourse and actionable explanations in black-box decision making systems. arXiv preprint arXiv: 19970096155 2019.
Janis Klaise, Arnaud Van Looveren, Giovanni Vacanti, and Alexandru Coca. Alibi: Algorithms for monitoring and explaining machine learning models. https://github. com/ Io
alibi
Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin How we analyzed the compas recidivism algorithm. ProPublica (5 2016). 2016.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http:://yamm
lecun. ctt. httpm//xxmmmmiiit..
Daniel Láncz- Jorge Revuelta Herrero, Angélica González Arrieta, and Juan M Corchado. Hybridizing metric learning and case-based reasoning for adaptable clickbait detection. Applied Intelligence, 48(9)22967-2982, 2018.
Seldon Technologies Ltd. alibi documentation release 0.4.0. https::/docs seldon. io/ (bw.looat//LLaa///////taaaee////
Divyat Mahajan, Chenhao Tan, and Amit Sharma. Preserving causal constraints in counterfactual explanations for machine learning classifiers. arXiv preprint arXiv:1912.03277. 2019.
Ramaravind K Mothilal, Amit Sharma, and Chenhao Tan. Explaining machine learning classifiers through diverse counterfactual explanations. In Proceedings the 2020 Conference on Fairness, Accountability, and Transparency, pp. 607-617, 2020.
Rafael Poyiadzi, Kacper Sokol, Raul Santos- Rodriig-. Tijl De Bie, and Peter Flach. Face: Feasible and actionable counterfactual explanations. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pp. 344-350, 2020.
Berk Ustun, Alexander Spangher, and Yang Liu. Actionable recourse in linear classification. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 10-19, 2019.
Arnaud Van Looveren and Janis Klaise. Interpretable counterfactual explanations guided by proto- types. arXiv preprint arXiv:1907.02584 2019.
Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Harv. JL & Tech., 31:841, 2017.
Shuangfei Zhai and Zhongfei Mark Zhang. Semisupervised autoencoder for sentiment analysis. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.


Reviews:
[{"review_id": "iw3_3cqI8O", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new approach for generating counterfactual explanations. Specifically, the presented method optimizing for a counterfactual explanation using a weighted loss function of L_pred, L_sparsity, L_recon, and L_proto, and differs from previous works in the manner in which the latter two losses are computed. In more detail, whereas prior work computes L_recon and L_proto using the reconstruction and latent space distance of *unsupervised* models, the presented method computes these losses using a semi-supervised setup whereby a model is jointly trained to minimize reconstruction and class-conditional loss. The authors conduct experiments on a number of real- and mixed-valued datasets, which is welcome in a field where broad experimentation is historically lacking.\n\nMotivation:\nFurthermore, I believe the work would benefit from a stronger motivation section and justification for the approach. Specifically, why would one not achieve the desired result -- i.e., of ensuring that the CFE is of the target class -- by simply increasing the weight of L_pred. It is peculiar for this not to happen, as counterfactual explanation must be, by definition, from another (desirable) class, and so it seems natural to expect this.\n\nReferences:\nThe authors have missed a number of key citations. Specifically, such terms as \"Actions\", \"Actionability\", \"Feasibility\", and \"Interpretability\" are under-defined, perhaps the authors can refer to a recent survey paper [1] to clarify this terminology. Moreover, in the introduction, the authors suggest that counterfactual explanations are being used for practical applications, and/or for bias reduction, both without citations. The related work section also lacks such relevant citations as [2-5]. Sec 4 describes the \"semi-supervised auto-encoder\" without citing relevant work (as a non-expert in DL architectures, I may be wrong here) [6,7]. Finally, arguments such as  \"interpretability will improve the trust among data subjects\" could be made more concrete if one cited works as [8-11].\n\nOther comments and nits;\n- [sec 1; par 1] \"provide necessary actions to receive a more favorable decision\" is a common misconception in the CFE literature; this was shown not to be the case in [5,12]. perhaps replace with \"provides meaningful explanations as to what features would receive favorable treatment\" (not actions).\n- [sec 1; par 2] it is unclear how \"data distr. of the model\" & \"data distr. of individuals\" are different?\n- [sec 2; par 3] \"input-output pair dat of the black-box\" is unclear where the inputs are sampled from?\n- [sec 3; par 2] the statement \"they may not be interpretable\" is unclear, because (as alluded above) the term interpretable is undefined and one would have to read Van Looveren to understand the meaning here.\n- [sec 3; par 3] x_0 undefined.\n- [sec 3; par 3] L_proto should be L_proto^\\mathcal{X} as above.\n- [sec 3; par 4] are c, gamma, and theta tuned for each \"data set\" or \"data instance\"?\n- [sec 3; par 4] \"Ltd\" reference undefined.\n- [sec 4.2; par 2] \"clearly the separation is due\" is quite strong of a claim; perhaps the training procedure for one model was different? Do different random initializations result in similar trends? Although this observation does not relate to the primary purpose of the paper (which is to then use such distributions in another optimization problem for CFE generation), it may be better to soften such strong claims.\n- [sec 5; par 1] \"reverse lookup\" undefined\n- [sec 5.2; par 1] what criteria were used to select the weights in the training loss for the semi-supervised setting? And how does training affect the resulting CFE?\n- [sec 5.3; par 2] does the cat-proximity loss need \"1 - \"? Are we not aiming to minimize this distance as in cont-proximity? Perhaps I am mistaken.\n\nRefs:\n- [1] Karimi et al., https://arxiv.org/pdf/2010.04050.pdf\n- [2] Guidotti et al. https://arxiv.org/pdf/1805.10820.pdf\n- [3] Sharma et al., https://arxiv.org/pdf/1905.07857.pdf\n- [4] Laugel et al., https://arxiv.org/pdf/1712.08443.pdf\n- [5] Karimi et al., https://arxiv.org/pdf/2002.06278.pdf\n- [6] Oza and Patel, https://arxiv.org/pdf/1904.01198.pdf\n- [7] Rudy and Taylor, https://arxiv.org/pdf/1412.7009.pdf\n- [8] Fjeld et al., https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3518482\n- [9] Lipton, https://arxiv.org/abs/1606.03490\n- [10] Doshi-Velez and Kim, https://arxiv.org/abs/1702.08608\n- [11] Doshi-Velez et al., https://arxiv.org/pdf/1711.01134.pdf\n- [12] Karimi et al., https://arxiv.org/pdf/2006.06831.pdf", "extra_scores": {}, "extra_text": {}}, {"review_id": "iHekrtIIkBr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper continues an emerging line of research to find interpretable (post-hoc) counterfactual explanations of classifier predictions. While prior work has made advances in ensuring that resulting counterfactuals lie in the same data distribution as the original dataset by using auto-encoders, this paper provides a semi-supervised approach in an attempt to also ensure that they lie in the data distribution corresponding to the counterfactual label. The authors show the benefits of their approach on six datasets:  German Credit data, Adult Census data, MNIST, COMPAS, PIMA, and Breast Cancer Wisconsin data.\n\nPros:\n- The paper builds on an interesting and emerging line of research and addresses an important conceptual problem faced by prior work.\n- The authors present comprehensive experiments to show that their method leads to improved results on multiple datasets using multiple metrics, and accompany it with interesting qualitative analysis that further supports the quantitative results. The simplicity of the method would allow practitioners to generalize it to various other datasets.\n- Except for a few issues, the paper is well written and easy to follow\n\nCons:\n- The results are not uniformly in favor of the proposed method when compared to the unsupervised approach applied by prior work, and to the authors’ credit, they acknowledge that in the text.\n- I would like to see more detail in discussion surrounding qualitative results, particularly on MNIST. Furthermore, in Tables 2 and 3, the authors share limited examples, it would be great to see more qualitative analysis in the rebuttal. I also didn’t quite understand what the authors mean by if the digit 9 happens to be the decoded proto for all three even queries presented in Figure 3, that the proto values for the framework are “robust”. Robust in what sense and to what exactly?\n- The proximity metrics aren't very clear and there appears to be some discrepancy. The definitions of continuous and categorical proximity in this paper are different than the ones presented in Mothilal et al. (2020). If we go by their definitions (as the authors claim they do --- Section 5.3, first two lines), one would arrive at the definitions in this paper if we assumed that there is only one categorical variable and one continuous variable, but that is not the case in the datasets used in this paper.\n\nAdditional Questions:\n- Please clarify this from Section 5.4 where you say, “(SS) framework fares better even for categorical proximity and continuous proximity metrics across all datasets except COMPAS dataset, where continuous proximity of (SS) framework is higher.” Based on my interpretation of Table 1, it appears that the performance on COMPAS dataset follows the same trend as on German Credit or Adult Income.\n\nReferences:\nThe paper addresses an important question but at the same time misses on related work done in non-ML fields including Philosophy and Psychology. I recommend that the authors consider discussing some of the works on counterfactual selection from those fields in their related work section. Here are two for a start:\n- Fazelpour, S. Norms in Counterfactual Selection. Philosophy and Phenomenological Research.\n- Byrne, R. M. (2016). Counterfactual thought. Annual review of psychology, 67, 135-157.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_WgSSu--O2q", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "My main concern for this paper is around several factors that I will list from the most important to minor.\n\ni) It is unclear to me when and how generator training should be biased according to the class label, which the authors do by encouraging the latent space to be predictive of the class/target label. I have a few questions regarding this:\n a) Why is it insufficient to train a conditional version of a VAE to create separation in the latent space that guarantees the counterfactual is appropriately separated and lies on the manifold of the target class.\n\nb) This specifically concerns the authors' claim that in the case of post-hoc training, input-output pairs of the model can be used to train the semi-supervised auto-encoder. Now this question is fundamental to me in terms of the goals of the counterfactual explanation. Is the goal to provide explanations in ways that allow us to identify issues with the classifier? Or is the goal of the counterfactual explanation to provide recourse to the end user. In the first case, I would rather see counterfactual explanations that are not in any way or form biased by the generative model, and hoping there is a perfect generative model that captures the entire distribution of the samples this model is supposed to generalize on. Now generating counterfactuals with such a perfect generator might actually give some hope of identifying issues worth debugging in a classifier like a bias toward specific classes etc. If the goal of the counterfactual explanation is to provide recourse, then simply hoping its a likely prototype from the target class is most definitely insufficient. Even if we assumed that it is in-fact sufficient, the post-hoc training on input-output pairs of the classifier will in fact bias the generator heavily and really does not reveal much about the classifier at all. The authors need to strongly consider and clarify what the goal is of their counterfactual explanation, what kinds of biases different methods induce in their explanation, why a conditional generator is not sufficient and potential harms of a jointly trained model. \n\nii) This brings me to my second question around empirical evaluation. \na) Include a conditional baseline\nb) Compare to all the methods that train generator separately, jointly, and semi-supervised model.\nc) what happens if the support of the training data for the generative model is mismatched from the data used for training. \nd) Should counterfactual explanations actually be designed to provide fidelity to the classifier behavior or just be purely interpretable but not of great utility in actually improving the classifier. If the goal is to not improve the classifier but to audit it, I strongly feel simply providing qualitative summaries of sparsity and measures of interpretability in terms of target class probability are insufficient. They are also insufficient to provide recourse. \n\nAll of these factors significantly diminish this contribution. I recommend the authors give critical thought to the goals of counterfactual explanations and the effects of joint vs disjoint training in addition to merely semi-supervised and unsupervised training of the generative model, here an autoencoder. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "WYulqCDGp_r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Given a classifier, this paper proposes to extend the use of auto-encoders to generate interpretable counterfactual explanations by building a latent space that is aware of the class tag information through joint training of the auto-encoder and the classifier. A counterfactual explanation is defined as more interpretable if it lies within or close to the data distribution of the training data of the black box classifier. \nThe experimental part of the paper tests the proposal by using different datasets and some metrics that are adapted to their definition. Experiments are well designed and well performed.\n\nStrong points:\n+ Generating interpretable counterfactual explanations is an open problem that is not only interesting from a theoretical point of view, but also in practical terms.\n+ The paper describes in a fair way alternative methods and correctly identifies their strong and weak points.\n+ The use of a semi-supervised autoencoder seems to be a good ingredient to get explanations that are close to the data distribution of the classifier. \n\nWeak points:\n- The proposed method (implementing a semi-supervised autoencoder) is simple from a technical point of view and does not represent any challenge.\n- The paper proposes the following definition: \"A counterfactual explanation is more interpretable if it lies within or close to the data distribution of the training data of the black box classifier.\" This definition of \"interpretability\" is too simple and it requires some discussion to be accepted.  This definition is not based on causal reasoning and because of this, we cannot assume that this kind of explanation is actionable or even feasible. Moreover, this definition is in contradiction with a causal view (a view that is fairly explained in the paper): counterfactuals are realizations from a probability distribution that is different from that of the classifier.  Lying within or close to the data distribution of the training data of the black box classifier is a property that needs justification.\n\nThis last point is not saying that to lie within or close to the data distribution of the training data is an undesired property for counterfactual explanations, but I miss a sounder motivation. I would suggest some improvements in this line:\n+ I do not think that the concept of \"interpretability\" can be defined independently of a causal model of the world that limits what is feasible and what is not. For this reason, in order to check if the proposed methods represent a clear step towards \"interpretability, a comparison with causal-based counterfactuals is necessary. \n+ Testing counterfactuals is not possible because of the impossibility of ground truth data, but alternative approaches exist, mainly based on some kind of human evaluation. I am missing some experiments in this line.\n\nMinor points:\n- To have a more complete overview of the topic I would recommend to include some papers related to the concept of \"algorithmic recourse\", which identify some limitation of the counterfactual explanation approach.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is a general consensus on the fact that the paper is not yet ready for publication. I encourage the authors to carefully address the detailed concerns raised by the reviewers, which include  among other: i) the incompleteness of the literature overview, which should include the references provided by the reviewers, ii) poor (or bias towards the proposed approach) experimental evaluation, and iii)  a vague treatment of key terms in the interpretability literature  like feasibility (e.g., to make sure that the counterfactual lie in high data density regions). 
Decision: Reject