Debiasing Concept-based Explanations with Causal Analysis
Authors: Mohammad Taha Bahadori
David Heckerman
Venue: ICLR 2021 Poster (2021)
Keywords: Interpretability
Concept-based Explanation

Abstract: Concept-based explanation approach is a popular model interpertability tool because it expresses the reasons for a model's predictions in terms of concepts that are meaningful for the domain experts. In this work, we study the problem of the concepts being correlated with confounding information in the features. We propose a new causal prior graph for modeling the impacts of unobserved variables and a method to remove the impact of confounding information and noise using a two-stage regression technique borrowed from the instrumental variable literature. We also model the completeness of the concepts set and show that our debiasing method works when the concepts are not complete. Our synthetic and real-world experiments demonstrate the success of our method in removing biases and improving the ranking of the concepts in terms of their contribution to the explanation of the predictions.

Published as a conference paper at ICLR 2021
DEBIASING CONCEPT-BASED EXPLANATIONS WITH CAUSAL ANALYSIS
Mohammad Taha Bahadori, David E. Heckerman (bahadorm, heckerma ) (maazon. com
ABSTRACT
Concept-based explanation approach a popular model interpertability tool be- cause it expresses the reasons for a model's 's predictions in terms of concepts that are meaningful for the domain experts. In this work, we study the problem of the concepts being correlated with confounding information in the features. We propose new causal prior graph for modeling the impacts of unobserved variables and method to remove the impact of confounding information and noise using two-stage regression technique borrowed from the instrumental variable literature. We also model the completeness of the concepts set and show that our debiasing method works when the concepts are not complete. Our synthetic and real-world experiments demonstrate the success of method in removing biases and im- proving the ranking of the concepts terms of their contribution to the explanation of the predictions.
INTRODUCTION
Explaining the predictions of neural networks through higher level concepts (Kim al., 2018; Ghor- bani et al., 2019; Brocki & Chung, 2019; Hamidi--Hained et al., 2018) enables model interpretation on data with complex manifold structure such as images. also allows the use of domain knowledge during the explanation process. The concept- pancept-based explanation has been used for medical imaging (Cai et al., 2019), breast cancer histopathology (Graziani et al., 2018), cardiac MRIs (Clough et al., 2019), and meteorology (Sprague et al., 2019).
When the set of concepts carefully selected, we can estimate a model which the discriminative information flow from the feature vectors x through the concept vectors and reach the labels y. To this end, we train two models for prediction of the concept vectors from the features denoted by ē(x) and the labels from the predicted concept vector (C). This estimation process ensures that for each prediction we have the reasons for the prediction stated in terms the predicted concept vector c(x).
However, in reality, noise and confounding information (due to e.g. din-dsrrrmiitiive context) can influence both of the feature and concept vectors, resulting in confounded correlations between them. Figure provides 2011). We train two predictors for the concepts vectors based on features č(x) and labels C(y) and compare the Spearman correlation coefficients between their predictions and the true ordinal value of the concepts. Having concepts for which č(x) is more accurate than č(y) could be due to noise, or due to hidden variables independent of the labels that spuriously correlated and x, leading to undesirable explanations that include confounding noise.
evidence for noise and confounding in the CUB-200-201 dataset (Wah et al.,
In this work, using the Concept Bottleneck Models (CBM) (Koh et al., 2020; Losch al., 2019) we demonstrate a method for removing the counfounding and noise (debiasing) the explanation with concept vectors and extend the results to Testing with Concept Activation Vectors (TCAV) (Kim et al., 2018) technique. We provide a new causal prior graph to account for the confounding information and concept completeness (Ych et al., 2020). We describe the challenges in estimation of our causal prior graph and propose new learning procedure. Our estimation technique defines and predicts debiased concepts such that the predictive information of the features maximally flow through them.
We show that using two-stage regression technique from the instrumental variables literature, can successfully remove the impact of the confounding and noise from the predicted concept vectors. Our proposed procedure has three steps: (1) debias the concept vectors using the labels, (2) predict

---

Published as a conference paper at ICLR 2021
m((x) e)
p(ely).
Spearnnn Correlation
0.5
0.4
0.2
0.1
250
Concegue ids
Figure 1: Spearman correlation coefficients (p) of the predictors of the concepts given features c(x) and labels ē(y) for the 312 concepts in the test partition of the CUB-200-2011 dataset (Wah et al., 2011). 112 concepts can be predicted more accurately with the features rather than the labels. Concept ids in the x-axis are sorted the increasing ((C(y).c) order. We provide the detailed steps to obtain the figure in Section 4.2.
the debiased concept vectors using the features, and (3) use the predict concept vectors in the second step to predict the labels. Optionally, we can also find the residual predictive information in the features that are not in the concepts.
We validate the proposed method using synthetic dataset and the synthetic data, we have access to the ground truth and show that in the presence of confounding and noise, our debiasing procedure improves the accuracy of recovering the true concepts. On the CUB-200-222 200-2011 dataset, we use the RemOve And Retrain (ROAR) framework (Hooker al., 2019) to show that our debiasing procedure ranks the concepts in the order of their explanation more accurately than the regular concept bottleneck models. We also show that we improve the accuracy of CBNs in the prediction of labels using our debiasing technique. Finally, using several examples, also qualitatively show when the debasing helps improve the quality of concept- concept-based explanations.
dataset. On the
METHODOLOGY
Notations. We follow the notation of Goodfellow al. (2016) and denote random vectors by bold font letters and their values by bold symbols æ. The notation p(x) is probability measure on x and dp(x = a) is the infinitesimal probability mass at x = æ. We use (xx) to denote the the prediction of y given In the graphical models, we show the observed and unobserved variables using filled and hollow circles, respectively.
Problem Statement. We assume that during the training phase, we given triplets (æ, Ct, Yi) for = 1,..., n. data points. In addition to the regular features æ and labels we are given a human interpretable concepts vector e for each data point. Each element of the concept vector measures the degree of existence of the corresponding concept in the features. Thus, the concept vector typically have binary or ordinal values. Our goal is to learn to predict y as function of æ and use e for explaining the predictions. Performing in two steps, we first learn a function ē(x) and then learn another function (c(x)) The prediction ē(x) the explanation for our prediction ý. During the test time, only the features are given and the prediction+explaaatiie algorithm predicts both ÿ and с
In this paper, we aim remove the bias and noise components from the estimated concept vector such that it explains the reasons for prediction of the labels more accurately. To this end, we first need to propose new causal prior graph that includes the potential unobserved confounders.
2.1 NEW CAUSAL PRIOR GRAPH FOR CBMS
Figure 2a shows the ideal situation in explanation via high-level concepts. The generative model corresponding to Figure 2a states that for generating each feature x; we first randomly draw the label y:- Given the label, draw the concepts e;. Given the concepts, we generate the features. The
2

---

Published as a conference paper at ICLR 2021
11
(a) The ideal concepts
(b) Our realistic graph
(c) The graph with d(y)
Figure 2: (a) The ideal view of the causal relationships between the features x, concepts c, and labels (b) In a more realistic setting, the unobserved confounding variable impacts both x and c. The shared information between and y go through the discriminative part of the concepts We also model the completeness of the concepts via direct edge from the features to the labels y. (c) When use d E(cly] in place of d and c, eliminate the confounding link
hierarchy in this graph is from nodes with less detailed information (labels) more detailed ones (features, images).
This model in Figure 2a is explanation for the phenomenon in Figure 1, because the noise in generation of the concepts allows the x c- edge to be stronger than the c c-y edge. However, another inon-mutually exclusive) explanation for this phenomenon is the existence of hidden confounders shown in Figure 2b. In this graphical model, u represents the confounders and represents the unconfounded concepts. Note that assume that the confounders and labels y independent when x and c are not observed.
Another phenomenon captured in Figure 2b the lack of concept completeness (Yeh al., 2020). describes the situation when the features, compared to the concepts, have additional predictive information about the labels.
The non-linear structural equations corresponding to the causal prior graph in Figure 2b are as follows
= fi + Edd, c d h(u),,
x = fe(u, d) + fa(y) + Ea,
(3)
for some vector functions h, f1, f2. and f3. We have IL y and IL y. Our definition of in Eq. (2) does restrict u, because simply attribute the difference between and fi (y) function of the latent confounder u and noise.
Our causal prior graph in Figure 2b corresponds to a generative process in which to generate an observed triplet (z,, C., y:) we first draw label ys and confounder us vector independently. Then we draw the discriminative concepts d, based on the label and generate the features æ; jointly based on the concepts, label, and the confounder. Finally, we draw the observed concept vector c; based on the drawn concept and confounder vectors.
Both causal graphs reflect our assumption that the direction of causality is from the labels to concepts and then to the features, + > x, to ensure that u and y are marginally independent in Figure 2b. This direction also correspond to moving from more abstract class labels to concepts to detailed features. During estimation, we the functions in the x + + y direction, because finding the statistical strength of edge does not depend on its direction.
Estimation of the model in Figure 2b is challenging because there are two distinct paths for the information from the labels y to reach the features Our solution is prioritize the bottleneck path and estimate the y - + x, then estimate the residuals of the regression using the y - x direct path. Our two-stage estimation technique ensures that the predictive information of the features maximally flow through the concepts. In the next sections, we focus on the first phase and using two-stage regression technique borrowed from the instrumental variables literature eliminate the noise and confounding in estimation of the d + x link.

---

Published as a conference paper at ICLR 2021
Algorithm Debiased CBMs
Require: Data tuples (z,, C2. y:) for i I....,n.
Estimate a model dyy) = Elcly] using (c;, Mt) pairs.
2: Train a neural network as an estimator for P5
(d x) using (æ;, d:)) pairs.
3: Use pairs (æ,, y:) to estimate function go by fitting J' go ( d dlz;) yi-
4: Compute the debiased explanations E[d[x;]- - 1 Di= < for i = ,,..,, n.
5: return The CBM defined by (pg.90) go) and the debiased explanations.
2.2 INSTRUMENTAL VARIABLES
Background on two-stage regression. In causal inference, instrumental variables (Stock, 2015; Pearl, 2009) denoted by are commonly used find the causal impact of variable x on y when and are jointly influenced by an unobserved confounder u (i.e., x + + y). The key requirement that should be correlated with but independent of the confounding variable (i.e. - - y and z TL u). The commonly used 2-stage least squares first regresses x in terms of z to obtain X followed by regression of y in terms of x. Because of independence between and u, & is also independent of Thus, in the second regression the confounding impact of is eliminated. Our goal to use the two-stage regression trick again to remove the confounding factors impacting features and concept vectors. The instrumental variable technique can be used for eliminating the biases due to the measurement errors (Carroll et al., 2006).
Two-Stage Regression for CBMs. our causal graph in Figure 2b, the label y can be used for the study of the relationship between concepts d and features x. We predict d as function of y and use in place of the concepts in the concept bottleneck models. The graphical model corresponding to this procedure is shown in Figure 2c, where the link u - is eliminated. In particular, given Flccy] II. h(u). This is the basis for the independence relationship Il. u, have d(y)= debiasing method in the next section.
2.3 THE ESTIMATION METHOD
Our estimation uses the observation that in graph 2b the label vector y is a valid instrument for removing the correlations due Combining Eqs. (I) and (2) we have fi(y) huu) + Eg. Taking expectation with respect to p(cly), we have
E[cly] = Elf (y) + (uu) + =aly] = fi()) + E[h(u)] + E[sa].
(4)
The last step is because both and εα are independent of Thus, two terms are constant in terms of x and y and can be eliminated after estimation. Eq. (4) allows us remove the impact of u and
Ea and estimate the denoised and debiased d(y) = E(cly]. We find Efcly] using a neural network trained on (c,, Y:) pairs and them pseudo-observatioon in place of d, Given our debiased prediction for the discriminative concepts d, we can perform the CBMs' two-steps of x -> and + y estimation.
Because we use expected values of in place of during the learning process (i.c., d(y) = E(cly)). the debiased concept vectors have values within the ranges of original concept vectors c. Thus, we do lose the human readability with the debiased concept vectors.
Incorporating Uncertainty in Prediction of Concepts. Our empirical observations show that prediction of the concepts from the features can be highly uncertain. Hence, we present a CBM esti- mator that takes into account the uncertainties in prediction of the concepts. We take the conditional expectation of the labels given features x as follows
Fly| x = F Fo (d)x = /
go go (d) d|x);
(5)
where Pw: (dlxx xx) the probability function, parameterized by Φ. that captures the uncertainty in prediction of labels from features. The go(-) function predicts labels from the debiased concepts.
4

---

Published as a conference paper at ICLR 2021
In summary, we perform the steps in Algorithm to learn debiased CBMs. In Line 3, we approximate the integral using Monte Carlo approach by drawing from the distribution P3( (d|xx) estimated Line 2, see (Hartford et al., 2017). Given the labels' data type, we can use the appropriate loss functions and not limited to the least squares loss. Line 4 removes the constant mean that be due noise or confounding, as we discussed after Eq. (4). In the common case of binary concept vectors, we use the debiased concepts estimated in Line 4 to rank the concepts in the order of their contribution to explanation of the predictions.
3 DISCUSSIONS
Debiasing TCAVs. While presented our debiasing algorithm for the CBMs, we can easily use it to debias the TCAV (Kim et al., 2018) explanations too. We can use the first step to remove the bias due to the confounding and perform TCAV using vectors, instead of e vectors. The TCAV method is attractive, because unlike CBMs, analyzes the existing neural networks and does not need to define a new model.
Measuring Concept Completeness. Our primary objective of modeling the concept incomplete- ness phenomenon in our causal prior graph is to show that our debiasing method does not need the concept completeness assumption to work properly.
we are interested in measuring the degree of completeness of the concepts, we can do it based the definition by Yeh et al. (2020). To this end, we fit a unrestricted neural network (xx) = Fly x) to the to the residuals in the step 3 of our debiasing algorithm. The function (-)) captures the residual information in We compare the improvement in prediction accuracy over the accuracy in step 3 to quantify the degree of concept incompleteness. Because we first predict the labels y using the CBM and then the q(x) to the residuals, we ensure that the predictive information maximally go through the CBM link x + d t
Prior Work on Causal Concept-Bard Based Explanation. Among the existing works on causal concept-based explanation, Goyal et al. (2019) propose different causal prior graph to model the spurious correlations among the concepts and remove them using conditional variational auto- encoders. In contrast, aim at handling noise and spurious correlations between the features and concepts using the labels as instruments. Which work is more appropriate for a problem depending on the assumptions underlying that problem.
The Special Linear Gaussian Case. When the concepts have real continuous values, we might simple multivariate Gaussian distribution model p(d]x) (x ol), a 0. If we further use a linear regression to predict labels from the concepts, we can show that the steps are simplified as follows:
Learn d(y)) by predicting y: - C--
2. Learn d(x)) by predicting x; + d:
3. Learn ((d) by predicting å, + yi-
The above steps show that given the linear Gaussian assumptions, steps 2 and 3 coincide with the sequential bottleneck (Koh et al., 2020) training of CBMs. We only need to change the concepts from C₁ to d we can eliminate the noise and confounding bias.
EXPERIMENTS
Evaluation of explanation algorithms is notoriously difficult. Thus, we first present experiments with synthetic data to show that our debiasing technique improves the explanation accuracy when we know the true explanations. Then, on the CUB-200-222 2019) framework to show that the debiasing improves the explanation accuracy. Finally, using several examples, we identify the circumstances in which the debiasing helps.
dataset, we use the ROAR (Hooker et al.,
5

---

Published as a conference paper at ICLR 2021
0.30
Hegglar
1.0
0.25
Debiasod
Avenge Correlation
0.20
0.9
0.15
Average
0.10
0.05
Regulaz
Dehiased
0.00
0.6
10
10a
110
107
Number of data points
Numbber of data points
Fully Random Design
(b) Orthogonal Confounding Design
Figure 3: Correlation between the estimated concept vectors and the true discriminative concept vectors as the number data points grow. Notice the different ranges of the y-axes and the logarithmic scale of the x-axes.
4.1 SYNTHETIC DATA EXPERIMENTS
We create synthetic dataset according to the following steps:
1. Generate vectors y: IR with elements distributed according to unit normal distribution (0,, 1).
2. Generate n vectors u, IR 100 with elements distributed according to unit normal distribution (0, 1).
3. Generate n vectors ε E distribution \(0, 0.02).
I 100 with elements distributed according to scaled normal
4. Generate n vectors ε,,; 100 with elements distributed according to scaled normal distribution No0, = 0.02).
5. Generate matrices Wi. W2, W3, Wa E R 100x with elements distributed according to scaled normal distribution (00 = 0.1).
6. Compute d, Wiy; Ed,i for 1,...,n.
7. Compute C; d, W2u, for 1,.... n.
8. Compute x; Wad; + W.4; - En,i for i 1,...,7.
In Figure 3a, we plot the correlation between the true unconfounded and noiseless concepts Wy and the estimated concept vectors with the regular two-step procedure (without debiasing) and debiasing method, as a function of sample size n. The results show that the bias due to confounding does vanish as we increase the sample size and our debiasing technique can make the results closer to the true discriminative concepts.
In the ideal case, when the confounding impact is orthogonal the impact of the labels, debiasing algorithm recovers the true concepts more accurately. Figure 3b demonstrates the scenario when W1 Wz, W, in the synthetic data generation. To generate matrices with this constraint, we first generate unitary matrix Q and generate four diagonal matrices A A..... A4 with diagonal elements drawn from Uniform(0.3 (0.1, This choice of the distribution for the diagonal elements caps the condition number of the matrices by 10. To satisfy the orthogonality constraint, 50 diagonal elements of Ag and A4 and last 50 diagonal elements of A₁ to zero. We compute the matrices W, QA,Q* for I..... 4. The orthogonality allows perfect separation of and y impacts and the perfect debiasing by our method as the sample size grows.
set the first
In Figure 6, repeat the synthetic experiments in higher noise regime where the standard deviation of the noise variables e, and E, is set to = 0.1. The results confirm our main claim about our debiasing algorithm.

---

Published as a conference paper at ICLR 2021
Table 1: Mapping the concept annotations to real values.
Annotation
Certainty
Ordinal Score
Numeric Map
Doesn't Exist
definitely
0
Doesn't 't Exist
probably
1
1/6
Doesn't Exist
guessing
2
2/6
Doesn't 't Exist
not visible
3
3/6
Exists
not visible
3
3/6
Exists
guessing
4
1/6
Exists
probably
5
5/6
Exists
definitely
6
4.2 CUB-200-2011 EXPERIMENTS
Dataset and preprocessing. We evaluate the performance of the proposed approach on the CUB- 200-2011 dataset (Wah et al., 2011). The dataset includes 11788 pictures (in 5994/5794 train/test partitions) of 200 different types of birds, annotated both for the bird type and 312 different concepts about each picture. The concept annotations are binary, whether the concept exists or not. However, for each statement, a four-level certainty score has been also assigned: 1: not visible, 2: guessing, 3: probably, and 4: definitely. We combine the binary annotation and the certainty score to create a 7-level ordinal variable the annotation for each image summarized in Table 1. For simplicity, map the 7-level ordinal values to uniformly spaced valued in the [0, 1] interval. We randomly choose 15% of the training set and hold
the validation set.
The result in Figure 1. To compare the association strength between y and with the association strength between x and c we train two predictors of concepts c(x) and ē(y). ). We use Torch Vision's 's pre-trained ResNet152 network (He al., 2016) for prediction of the concepts from the images. Because the labels y have categorical values, ε(y) ) is simply the average concept annotation scores per each class. We use the Spearman correlation to find the association strengths in pairs (č(x), c) and fc(y), c) because the concept annotations are ordinal numbers. The concept ids in the x-axis are sorted in terms of increasing values of p(C(y).c).
The top ten concepts with the largest values of p(č(x), - c) are 'has back color:greenn' 'has upper tail color:green', 'has upper tail color:orange', 'has upper tail color::inkk', 'has back color::ruoou', "has upper tail color:purple', 'has back color:pinkk' "has upper tail color:iridescent' "has back color:pprple', and 'has back color:iridescent', These concepts are all related to color and can be easily confounded by the context of the images.
Training details for Algorithm 1. We model the distribution of the concept logits as Gaussians with means equal the ResNet152's logit outputs and a diagonal covariance matrix. We estimate the variance for each concept by using the logits of the true concept annotation scores that are clamped into [0.05, 0.95] to avoid large logit numbers. In each iteration of the training loop for Line we draw 25 samples from the estimated p(d[x). Predictor of labels from concepts (the function g(-) )) in Eq. (5)) a three-layer feed-forward neural network with hidden layer sizes (312, 312, 200). There skip connection from the input to the penultimate layer. All algorithms are trained with Adam optimization algorithm (Kingma & Ba, 2014).
Quantitative Results. Comparing to the baseline algorithm, our debiasing technique increases the average Spearman correlation between c(x) and c(y) from 0.406 to 0.508. For the above 10 concepts, our algorithm increases the average Spearman correlation from 0.283 0.389. Our debiasing algorithm also improves the generalization in prediction of the image labels. The debiasing also improves the top-5 accuracy of predicting the labels from 39.5% to 49.3%.
To show that our proposed debiasing accurately ranks the concepts in terms of their explanation of the predictions, we use the RemOve And Retrain (ROAR) framework (Hooker et al., 2019). In the ROAR framework, we sort the concept using the scores E[d]æ.] " Σ Eld [z,] in the ascending order. Then, we mask (set to zero) the least explanatory x% of the concepts using the scores and retrain the go (dl function. We perform the procedure for (0, 10, 20, ... 80, 90, 95) and record
7

---

Published as a conference paper at ICLR 2021
1.20
1.15
Nornalized Accureyy
1.10
1.05
1.00
0.95
0.90
Regular Debiased
0.85
0.80
20
60
80
100
Percentage of Masked Concepts
Figure 4: The ROAR evaluation: We mask 3% of the concepts that are identified by the methods as less explanatory of the labels and retrain the go(-) function. We measure the change in the accuracy of predicting the labels y as we increase the masking percentage. For better comparison the trends, we have normalized them by their first data point (x 0%).
Figure 5: Twelve example images where the debiasing helps. A common pattern is that, the image context has either prevented or misled the annotator from accurate annotation of the concepts. From the left to right, the birds 'Brandt Cormorant', 'Pelagic Cormorant', 'Fish Crow', 'Fish Crow', 'Fish Crow", 'Ivory Gull', 'Ivory Gull', 'Green Violetear', 'Green Violetear', 'Cape Glossy Starling', 'Northern Waterthrush', 'Northern Waterthrush'.
the testing top-5 accuracy of predicting the labels y. repeat the ROAR experiments for times and report the average accuracy vary the masking percentage.
Figure shows the ROAR evaluation of the regular and debiased algorithms. Because the debiased normalize both curves by dividing them by
algorithm is more accurate, for casier comparison, their accuracy at masking percentage æ = 0%. An immediate observation is that the plot for debiased algorithm stays above the regular one, which is clear indication of its superior performance in identifying the least explanatory concepts. The results show several additional interesting insights too. First, the prediction of the bird species largely rely sparse of concepts of the concepts and still have a decent accuracy. Second, masking a small percentage of irrelevant concepts reduces the noise in the features and improves the generalization performance of both algorithms. Our debiased algorithm is more successful by being faster finding the noisy features before 20% masking. Finally, the debiased accuracy curve is less steep after 80%. which again indicates its success in finding the most explanatory concepts.
can mask 95%
Qualitative analysis of the results. In Figure 5, we show 12 images for which the d and are significantly different. A common pattern among the examples is that the context of the image does not allow accurate annotations by the annotators. In images 3, 4, 5, 6, 7, 11, and 12 in Figure 5, the color-related concepts listed at the beginning are all to 0.5, indicating that the annotators have failed in annotation. However, our algorithm correctly identifies that for example Ivory Gulls do not have green-colored backs by predicting ē = 0.08 which is closer to 7 = 0.06 than the true 0.5.
Another pattern is the impact of the color of the environment on the accuracy of the annotations. For example, the second image from the left is image of Pelagic cormorant, whose back and upper tail colors are unlikely to be green with per-class average of 0.12 and 0.07, respectively. However, because of the color of the image and the reflections, the annotator has assigned 1.0 to both of "has back color:green' and 'has upper tail color:green' concepts. Our algorithm predicts 0.11 and 0.16 for these two features respectively, which closer to the per-class average.
8

---

Published as a conference paper at ICLR 2021
Table 2: Examples of differences between regular and debiased algorithms in ranking the concepts.
Image
Top 15 Concepts
Dehiased: has throat color:blackk, has head patter::llain has forehead color:bbacck, has breast color:black, has underparts color:black, has nape color:black, has crown color:black, has primary color:black. has bill color;black, has belly colorsblack, has wing color:orange, has hreast patterssooid, has upperparts color:orange, has wing pattem:mulii-oloree
has bill length:about the same as head Regular: has primary blaor, has wing color;black, has threat color:black, has upperparts bolkr, has beeast color:black, has primury color;ibuee, has underpurts color:bback., has belly color:black., has back color:black, has nape color:black, upperparts color:blue, has tail pattem:solid, has crown color:blue, has under tail color::lacck, has forchead color:bluue
Debiased: has primary color:zed. has crown color:red, has forehead color: has throat color:red, has nupe color:red, hreast coloc:edd, has undernarts color:red, has belly color:eed, has forehead color:rufous, has upperparts color:red, has crown color:nufous, has nape color:rufous, has wing paterm:mltiicoooee colooeidd has pri- mary color:rufous, has threat color:rufous
Regular; has underparts color:grey, has breast color: grey, has helly colorggrey, has helly paterm:malti-coloeee has breast aattemmmuliicollee throat color:grey. has upperparts colortred. back patiem::multi colorre, has underparts color.red, has primary coloragrey, has belly culor:red, has throal color:red
has nape color:grey, has bill shorth than bead, has breast color.red,
Debiased: has bill longth:abouu the same as bead, has belly pattemmspotted. has underparts bllor, has sbage::dagger. breast black: has belly color:black, has breast sppter::spoted. has back pat- lemn:spotted has wing patterssspotted has nape color:red, has back color:black, has tail patternsspoteed, has under tail color:black, has upper tail bolor:blackk has primary color:
Regular: hus primary color:bronn, has wing color:brown, has upperpurts color:brown, has color:hrown, has hack color:hrown, has forehead color:brown, has nape color::rown, has thront color:white, has breast pattemmspotted, has under tail bolor, has breast color: has belly color:white, has under- parts color:white., has upper tail color:brown, has breast color:
Debinsed: has shaperaduck like, has hill shape:spatullte, hus size:medium (9- 16 in), has bill length:about the head, has threat color:buff, has underparts color:brown, has breast color:brown, has belly pat- seot:spotted, has crown bolor, has primary color:brown, has belly color: has nape color: Lorehead color:brown, has upperparts color: belly color:purple Regular: has belly color:grey. has underparts color:grey. has breast color:grey, has belly color:pink, has belly color: has underparts color:purple, has belly color:purple. has underparts color:pink. has throat color.grey, has primary color:grey. has belly culor:green, has underparts color:rufous, has underparts color:green, has belly color:irifescent, has forehead color-grey
Debiased:
has threat blakr; has forehead color:black, has crown color:black, has primary color: has nape colorsblack, has breast colorcred, has belly color:white, has underparts color: has underparts color:red, has back color:black, has primary color:white, has hill shape:zone, has breast pattem: calli-- primary color:red, has upperparts color:black
Regular: has nape color:black. has primary color:black, has nape color:rufous, has primary color:red, has wing color:orange, has nape color:red. has breast color:red. has crown bolor: upperparts color:orange, has crown color:eed has underparts color: has upperparts color: has forchead color:black, has throal color:red, has buck color::puple
Debiased: has wing color:bluc, has upperparts color:bluc,, has crown color: has primary bolor, has bill color: has back color:blue, has under tail color;blue, has bill color:red, has upper tail color:blue, has wing pattem:amulii culored, has nupe color:rufius, has crown color:brown. has belly color:browa, hus nape nape color: has underparts color:bown Regular: has primary color:red, has throat color:red, has underparts color:red, has breast color:red, has forehead color:ed, crown color:rufous, has crown color:red, nape colorared, has belly color:red, has primary color:rufous, has throat color: has forchead color:rufous, has wing color:red, has nape coloc:rufous, has underparts color:rufous
In Table 2, we list six examples to show the superior accuracy of the debiased CBM in ranking the concepts in terms of their explanation power. Moreover, in Table 3 in the appendix, we list the top and bottom concepts in terms of their predictive uncertainty in our debiased CBM.
5 CONCLUSIOOS AND FUTURE WORKS
Studying the concept-based explanation techniques, we provided evidences for potential existence of spurious association between the features and concepts due to unobserved latent variables or noise. We proposed a new causal prior graph that models the impact of the noise and latent confounding fron the estimated concepts. We showed that using the labels as instruments, we remove the impact of the context from the explanations. Our experiments showed that our debiasing technique not only improves the quality of the explanations, but also improve the accuracy of predicting labels through the concepts. As future work, we will investigate other two-stage-regression techniques to find the most accurate debiasing method.
9

---

Published as a conference paper at ICLR 2021
REFERENCES
Lennart Brocki and Neo Christopher Chung. Concept Saliency Maps to Visualize Relevant Features in Deep Generative Models. In ICMLA, 2019.
Carrie Cai, Emily Reif, Narayan Hegde, Jason Hipp, Been Kim, Daniel Smilkov, Martin Wattenberg, Feraanda Viegas, Greg S Corrado, Martin C Stumpe, et al. Human-centered tools for coping with imperfect algorithms during medical decision-makinn In CHI, 2019.
Raymond J Carroll, David Ruppert, Leonard A Stefanski, and Ciprian M Crainiceanu, Measurement error in nonlinear models: a modern perspective. CRC press, 2006.
James Clough, Ikka Oksuz, Esther Puyol-Antón, Bram Ruijsink, Andrew P King, and Julia A Schnabel. Global and local interpretability for cardiac MRI classification. In MICCAI, 2019.
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. In NeurlPS, pp. 9273-9282, 2019.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Yash Goyal, Uri Shalit, and Been Kim. Explaining Classifiers with Causal Concept Effect (CaCE). arXiv: 1907.00/65, 2019.
Mara Graziani, Vincent Andrearczyk, and Henning Müller. Regression concept vectors for bidirec- tional explanations in histopathology. In Understanding and Interpreting Machine Learning in Medical Image Computing Applications, pp. 124-132. Springer, 2018
Mandana Hamidi-Haines, Zhongang Qi, Alan Fern, Fuxin Li, and Prasad Tadepalli. Interactive Naming for Explaining Deep Neural Networks: A Formative Study, arXiv: 181200700 2018.
Jason Hartford, Greg Lewis, Kevin Leyton-Brown, Brown, and Matt Taddy. Deep IV: A flexible approach for counterfactual prediction. In ICML, pp. 1414-1423, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. In NearIPS, 2019.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, al. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). In ICML, pp. 2668-200 6678-2677.2 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv: 2014.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept Bottleneck Models. In ICMI,, 2020.
Max Losch, Mario Fritz, and Bernt Schiele. Interpretability beyond classification output: Semantic bottleneck networks. arXiv: 10882, 2019.
Judea Pearl. Causality. Cambridge university press, 2009.
Conner Sprague, Eric Wendoloski, and Ingrid Guch. Interpretable AI for Deep Learning- Based Meteorological Applications. In American Meteorological Society Annual Meeting. AMS, 2019.
James H Stock. Instrumental variables in statistics and econometrics. International Encyclopedia of the Social Behavioral Sciences, 2015.
C. Wah. S. Branson, P. Welinder, Perona, and S. Belongie. The Caltech-UCSD Birds-200- 201-2011 Dataset. Technical Report CONS-TR-201-00 California Institute of Technology. 2011.
Kuah Yeh, Been Kim. Sercan O Arik. Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On Completeness-awar awae Concept-Bard Based Explanations in Deep Neural Networks. In NeurIPS, 2020.
10

---

Published as a conference paper at ICLR 2021
A HIGH-NOISE SYNTHETIC EXPERIMENTS
0.30
Regalar Dehiased
1.0
Regnlar
0.25
Dehiased
0.9
Awrago Convlation
Average Corvelation
0.20
0.15
0.7
0.10
0.6
0.00
0.5
0.00
0.4
10
10'
10'
10
10
10a
10a
10
10a
10a
Number data points
Numbee of data points
(a) Fully Random Design
(b) Orthogonal Confounding Design
Figure 6: Correlation between the estimated concept vectors and the true discriminative concept vectors as the number data points grow. Notice the different ranges of the y-axes and the logarithmic scale of the x-axes.
Certainty Level
Concepts
Most Uncertain
(194) has nape color:blackk, (260) has primary color::black, (164) has forehead color::bacck, (7) has bill shape:alll purposs,, (132) has throat color:black, (305) has crown color:black,, (133) has throat color:white, (150) has bill length:about the same as head, (8) has bill shape::cone, (152) has bill length:sshorter than head
Least Uncertain
(216) has wing shape::tapered-wingg. (215) has wing shape:: (217)
has wing shape::long-wingg (214) has wing shape::ponnedd -ings.. (77) has tail shape:fan-shapee tail, (213) has wing shape:rounded-winggs (79) has tail shape:squared tail, (83) has upper tail color:purple, (82) has upper tail color:iridescent, (75) has tail shape::rounded tail
Table 3: The top 10 most and least uncertain concepts identified by the debiased algorithm.


Reviews:
[{"review_id": "rGlgTQYkrR", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview of the Paper\nThis paper proposes a procedure to 'debias' and account for confounding while using 'concepts' as interpretations for black-box models. The approach proposes a causal graph, and then notes based on the proposed graph that sample labels satisfy conditions required of an instrumental variable. Given an instrumental variable, then they propose a multistage approach to debias the interpretations. In general, this paper provides a useful approach for helping to generate concept explanations that are not confounded with the label or other features. \n\nGeneral Feedback / Quality\nThis paper contains several interesting ideas. I believe this is the first paper to bring the idea of using instrumental variables as debiasing approaches for explanations. Having this said, I cannot recommend the paper in its current form because the exposition makes the paper a tough read. For example, this paper builds on the concept bottleneck approach of Koh et. al. from ICML 2020. However, no contained overview of that approach is given. The only reason I am able to follow the discussion is because I previously read that the Koh et. al. paper and I am generally familiar with instrumental variables. A more damning issue is that the paper does not explicitly set out the question/problem that it seeks to address. As it stands, figure 1 provides a vague motivation, but it is not clearly contextualized. The dataset and the discussion of concepts is not explicitly presented, so one is left to guess how the dataset was processed etc. I have worked with the CUB birds dataset before, so I am familiar with the concepts and data. My takeaway is that the paper requires a substantial rewrite that hand-holds a reader through the work. I document some of the other questions I had in the latter parts of this review. \n\nOriginality\nThe use of instrumental variable methods for explanation debiasing is new and original. The intersection of causal inference and post-hoc interpretability is a new and emerging area, so this work is one of the first to set out this path. There is other work in this area, but these do not address concept bottleneck models. \n\nSignificance\nAs noted above, the main idea and goal of this work is important. Overall, it is important to obtain explanations from models that are not confounded with other variables. It is also likely important for diagnosing bugs. \n\nClarity and Writing\nThe paper is mostly free of typos, but as it stands it is quite difficult to read. Several important concepts are not properly introduced, and a substantial rewrite is likely needed. See the overview for a discussion of some of these issues. \n\nSome other questions/concerns\n\n- Conceptual Confusion: Throughout this paper, it is unclear to how the methods here should be operationalized. Do the authors envision that instead of computing standard TCAVs or concepts, these concepts would first be debiased via the an instrumental variable following the protocol described here? \n\n- The causal graph: I am not an expert on causal inference but I wanted to get the authors' opinion on whether the proposed causal graphs are standard in that literature. Take fig. 2a., the causal graph says that the labels cause the concepts, which then cause the features. However, could this also be the other way? Here is an example: is it the case that calling an object a chair causes the object to have four legs or does an object with four legs cause or induce a label chair? I think the second graph would not fit under the scheme presented, but I am curious if this is the case. Primarily, I don't think the model labels can be used as instruments in such setting, but perhaps the framework is still appropriate? \n\n- The evaluation. I think the paper missed an opportunity the show a wow factor here. From the CUB dataset, it seems like the take away is that the debiased concept interpretations can identify a spurious correlation that a standard concept interpretation cannot? If this is the case, then that is a substantial positive for the approach. Instead of the ROAR evaluation, the authors should have created models with artificial spurious correlations and show that standard concept methods are unhelpful. \n\n- The ROAR approach used in this work is not an appropriate form of evaluation. I agree with the authors that explanation evaluation is difficult but the ROAR metric does not do this. The method retrains a model after masking out some highly important features, and then interprets the retrain model in comparison to the original one. This is not a comparison, especially when the models that are being compared a not convex. There is no guarantee that a retrained model relies on the same set of rules as the original model.  I would suggest the authors create toy settings (with real or toy datasets) where the DGP can be easily manipulated and show the promise of their approach that way. \n\n------\nPost Reponse Update\n-----\nThanks to the authors for their updates. I have updated my score by 1 point here. I believe the exposition in the paper could still be improved at this point. In general, this work provides an interesting use of IV techniques for interpreting black box models.", "extra_scores": {}, "extra_text": {}}, {"review_id": "TwywvSjOV5N", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Abstract: The paper proposed a way to learn unbiased (debiased) concept-based explainable models in the presence of unobserved confounders by the use of labels as instrumental variables. The proposed algorithm has 3 main steps: (1) regresses concept labels from the final labels (2) replace the original concepts with the learnt concepts and learn a model of debiased concepts as a function of features (3) predict label as a function of debiased concepts. The authors show in the experimental section that their training method captures the most salient concepts using the ROAR (Remove and Retrain) evaluation framework much better than the vanilla (non de-biased) approach. \n\nPros: \n- Clarity: The paper is quite well-written and the explanations are clear. \n- Significance/Impact: The ability to explain modeling prediction is quite important and furthermore it can lead to better generalization and robustness\n- Experimental design: I really liked the experimental section, very clear and well-done\n\nCons:\n- The causal graph assumption \n- Motivation: I would have spent a bit more time on explaining the motivation of the work and the scope that it adresses. For example, the causal graph assumption that makes labels to be the causal root of the graph, works well in image classification, but not in learning-to-control applications, such as recommender systems where the label is the reward and is the final consequence of the causal chain.", "extra_scores": {}, "extra_text": {}}, {"review_id": "NatCqAo6gVT", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper studies the causal nature of concept explanations. Specifically, the authors treat labels as instrumental variables to then debias explanations and improve predictive performance as well.\n\nStrengths\n- Figure 2 was helpful for understanding the contributions of this work. However, it is not clear if \\hat_d captures the same concepts that c alone would. If you could motivate using \\hat_d with a pictorial example of where c, the concepts alone, fail that would be helpful.\n- The use of ROAR for concepts is clever and does show the utility of the method, but additional experimentation to show the concepts captured align with human intuition would have been nice. At the minimum, showing how the concepts recovered by the proposed method and CBM differ would be helpful.\n\nWeaknesses\n- While both experiments (synthetic and BIRDs) show the method's utility, it would have been nice to see experiments on other datasets. OAI perhaps like in Koh et al.\n- The connection to Yeh et al. is not clear to me. What is the notion of completeness in the proposed method? \n\nQuestion\n- Can you extract uncertainty estimates for concepts from Equation 5?\n- Can you please explicitly the utility of your method in the linear Gaussian case? It seems as though using \\hat_d for the concepts simply recovers the independent concept case from Koh et al.", "extra_scores": {}, "extra_text": {}}, {"review_id": "BJEN13JNR8K", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The focus of the work is on model interpretability using concept-based explanation. The authors consider the issue of concepts being correlated with confounding information in the features. They propose a causal graph for representing the system and use instrumental variable methods to remove the impact of unobserved confounders. The proposed method is evaluated on synthetic and real data.\n\nI have the following comments:\n\n(1) The authors provide a graphical modelings for the setup of the problem. They assume that: 1. There is an \"unconfounded concept\" generated only from the label, 2. There is no confounding effect on the label.\n\nUnfortunately, no arguments is provided for justifying the conditional independence assumptions in the model. Any edge that is removed from a graphical model implies conditional independency assumptions and they should be carefully justified, specially since the topic of the work is interpretability. This includes the conditional independencies above as well as the way it is assumed that variable c is generated.\n\n(2) The model in this work is different from an IV model: One of the main requirements of the IV framework is exclusion restriction which requires that the effect of the instrument variable on the outcome should be only through the treatment variable. In the proposed model, variable y is also directly connected to variable x. Also, d is not confounded or observed, which again make the model different from IV model. Therefore, the model in this work does not represent the IV model, and y is not a valid IV, although it seems that that was actually not used in the approach. Only an independence assumption is actually used in the approach, which as mentioned above, is not justified.\n\n(3) It is not quite clear what exactly the variable d represents compared to variable c, and what is its interpretation.\n\n(4) In the synthetic simulations, the value of the variance chosen, specially for noises is very small. It is important to see the performance for larger values for the variance of the noises.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper has merits on providing a particular way of understanding a prediction model based on auxiliary data (concepts). I have a generally more positive view of it, aligned with the higher-scoring reviews. However, I feel a bit uncomfortable of framing it as "causal" in the sense it does not aim to provide any causal predictions, but it is more of a smoothing method for capturing signal contaminated with "uninteresting" latent sources - this is more akin to regression with measurement error (see e.g. Carroll, Ruppert and Stefanski's "Nonlinear regression with measurement error") where, like in this paper, different definitions of "instrumental variables" also exist and are different from the causal inference definition. I can see though why we may want to provide a causal interpretation in order to justify particular assumptions, not unlike interesting lines of work from Scholkopf's take on causality. The paper can be strengthened by some further discussion on the assumptions made about additivity on equations (2) and (3), which feel strong and not particularly welcome in many applications.

The proposed title is still a bit clunky, I feel that the two-stage approach is less important than the structural assumptions made, perhaps a title emphasizing the latter rather than the former would be more promising.
Decision: Accept (Poster)