Concept Gradient: Concept-based Interpretation Without Linear Assumption
Authors: Andrew Bai
Chih-Kuan Yeh
Neil Y.C. Lin
Pradeep Kumar Ravikumar
Cho-Jui Hsieh
Venue: ICLR 2023 poster (2023)
Keywords: Interpretability
Concept-based interpretation
XAI

Abstract: Concept-based interpretations of black-box models are often more intuitive for humans to understand. The most widely adopted approach for concept-based, gradient interpretation is Concept Activation Vector (CAV). CAV relies on learning a linear relation between some latent representation of a given model and concepts. The premise of meaningful concepts lying in a linear subspace of model layers is usually implicitly assumed but does not hold true in general. In this work we proposed Concept Gradient (CG), which extends concept-based, gradient interpretation methods to non-linear concept functions. We showed that for a general (potentially non-linear) concept, we can mathematically measure how a small change of concept affects the model’s prediction, which is an extension of gradient-based interpretation to the concept space. We demonstrated empirically that CG outperforms CAV in attributing concept importance on real world datasets and performed case study on a medical dataset. The code is available at github.com/jybai/concept-gradients.

Published as a conference paper at ICLR 2023
CONCEPT GRADIENTS: CONCEPT-BASED INTERPRE- TATION WITHOUT LINEAR ASSUMPTION
Andrew Bai
Chih-Kuan Yeh Department of Computer Science Camegii Mellon University cjyehêcs cmc. edu
Department of Computer Science University of California, Los Angeles andrewba @cs. ucla.edu
Neil Y. C. Lin
Pradeep Ravikumar
Department of Bioengineering University of California, Los Angeles neill feills melliinngg.u. .ucluccla a. edu
Department of Computer Science
Camegie Mellon University pradeepr cmp edu
Cho-Jui Hsieh
Department of Computer Science University of Califormia, Los Angeles chohsi ehê ucla edu
ABSTRACT
Concept-based interpretations of black-box models are often more intuitive than feature-based counterparts for humans understand. The most widely adopted approach for concept-based gradient interpretation is Concept Activation Vector (CAV). CAV relies learning linear relations between some latent representa- tions of given model and concepts. The premise of meaningful concepts lying in linear subspace of model layers is usually implicitly assumed but does not hold true in general. In this work we proposed Concept Gradients (CG), which extends concept-basee based gradient interpretation methods non-linear concept fune- tions. We showed that for a general (potentially non-linear) concept, we can math- ematically measure how a small change of concept affects the model's prediction, which is extension of gradient-based interpretation to the concept space. We demonstrate empirically that CG outperforms CAV in evaluating concept impor- tance real world datasets and perform case study medical dataset. The code is available at g thub...... com/ jybai /conceppt wwwpccepttrriien.t....m.
INTRODUCTION
Explaining the prediction mechanism of machine learning models is important, only for debug- ging and gaining trust, but also for humans to learn and actively interact with them. Many feature attribution methods have been developed attribute importance input features for the prediction of a model (Sundararajan et al. 2017 Zeiler & Fergus, 2014). However, input feature attribution may not be ideal in the case where the input features themselves are not intuitive for humans to un- derstand. It is then desirable to generate explanations with suman-understandaale concepts instead, motivating the need for concept-based explanation. For instance, understand machine leam- ing model that classifies bird images into fine-grained species, attributing importance to high-level concepts such as body color and wing shape explains the predictions better than input features of raw pixel values (see Figure
The most popular approach for concept-based interpretation Concept Activation Vector (CAV) Kim al. 2018). CAV represents concept with vector in some layer of the target model and evaluates the sensitivity of the target model's 's gradient in the concept vector's direction. Many followup works are based CAV and share the same fundamental assumption that concepts represented as a linear function in some layer of the target model (Ghorbani et al.] 2019 [Schrouff al. 2021 This assumption generally does not hold, however, and it limits the application of
be

---

Published as a conference paper at ICLR 2023
Albatross
Rank
Conccgts
CG score
has stze:medruum, [9__16
lengthrabout, head
has bill, shapeshnokod. soabird
1.834
has wing pattorn:solid
1.825
has wing pwwp:rppiittedwwiiggg.
1.609
has shaperduck-llk
has under tall color-grey
14119
has primary culorigrry
1440
has [orebread, culor.while
1.123
has, upper taii color-grey
Figure Comparison of feature-based interpretation heatmap (left: Integrated Gradients) and concept-based importance score (right: Concept Gradients) for the model prediction of "Black footed Albatross". Attribution to high-level concepts is more informative to humans than raw pixels.
concept-based attribution to relatively simple concepts. Another problem is the CAV concept im- portance score. The score defined as the inner product of CAV and the input gradient of the target model. Inner product captures correlation, not causation, but the concept importance scores are often perceived causally and understood as an explanation for the predictions it does.
In this paper, rethink the problem of concept-based explanation and tackle the two weak points of CAV. We relax the linear assumption by modeling concepts with more complex, non-linear functions (e.g. neural networks). To solve the causation problem, extend the idea of taking gradients from feature-based interpretation. Gradient- feature interpretation assigns importance to input features by estimating input gradients, i.e., taking the derivative of model output with respect to input features. Input features corresponding to larger input gradients are considered more important. A question naturally arises: is it possible to extend the notion of input gradients "concept" gradients? Can we take the derivative of model output with respect to post-hoc concepts when the model is not explicitly trained take concepts as inputs?
We answer this question in affirmative and formulate Concept Gradients (CG). CG measures how small changes of concept affect the model's prediction mathematically. Given any target function and (potentially non-linear) concept function, CG first computes the input gradients of both func- tions. CG then combines the two input gradients with the chain rule to imitate taking the derivative of the target with respect to concept through the shared input. The idea is to capture how the target function changes locally according to the concept. If there exists unique function that maps the concept the target function output, CG exactly recovers the gradient of that function. If the map- ping is not unique, CG captures the gradient the mapping function with the minimum gradient norm, which avoids overestimating concept importance. We discover that when the concept func- tion is linear, CG recovers CAV (with a slightly different scaling factor), which explains why CAAV works well in linearly separable cases. We showed in real world datasets that the linear separability assumption of CAV does not always hold and CG consistently outperforms CAV. The average best local recall @ 30 of CG is higher than the best of CAV by 7.9%, while the global recall @ 30 is higher by 21.7%.
PRELIMINARIES
Problem definition In this paper we use f Rd R denote the machine learning model to be explained, E Rd to denote input, and E IR* to denote the label. For an input sample E R.d, concept-based explanation aims explain the prediction f()) based on set of m concepts (cI...., C.. In particular, the goal is to reveal how important is each concept to the prediction. Concepts can be given in different ways, but in the most general forms, we can consider concepts as functions mapping from the input space to the concept space, denoted as Rdd + R. The function can be explicit or implicit. For example, morphological concepts such as cell perimeter and circularity can be given as explicit functions defined for explaining lung nodule prediction models. On the other hand, many concept-baat based explanations the literature consider concepts given as set of examples, which are finite observations from the underlying concept function. We further assume
2

---

Published as a conference paper at ICLR 2023
and 9 differentiable which follows the standard assumption of gradient-based explanation methods.
We define a local concept relevance score R(é; f.g) to represent how concept è affects target class prediction 1 = f f@) on this particular sample &. Local relevance is useful when analyzing individual data samples (e.g. explain what factors lead to bank loan application being denied, which may be different from application to application). Further, by aggregating over all the input instances of class y. Zy - (), (zn; y) we can define global concept relevance score R(Zyi f: g) to represent overall how a concept affects the target class prediction y, Global relevance can be utilized to grasp an overview of what the model considers importance (e.g. good credit score increases bank loan approval). The goal is to calculate concept relevance scores such that the scores reflect the true underlying concept importance, possibly aligned with human intuition.
Recap of Concept Activation Vector (CAV) Concept activation vector is concept-based inter- pretation method proposed by Kim et al. (011)). The idea is to represent concept with a vector and evaluate the alignment between the input gradients of the target model and the vector. In order for the concept to be represented well by vector, the concept labels must be linearly separable in the vector space. The authors implicitly assumed that there exists layer in the target model where concept labels can be linearly separated.
Let ve denote the concept activation vector associated with concept c. The authors define the con- ceptual sensitivity score to measure local concept relevance
RCAV fa; f. v.) == ff(x) (ve/lloel))
(1)
If we view CAV as a linear concept function, ReAv is the (normalized) inner product of the gradients of the target and concept function. The main caveat with the CAV conceptual sensitivity scores is that the underlying concept function is not guaranteed to lie in the linear subspace of some neural activation space. Attempting to fit the concept function with a linear model likely leads to poor results for non-trivial concepts, leading to inaccurate conceptual sensitivity scores.
GC: extending CAV to non-linear concepts Let us consider modeling concepts with general, non-linear function g instead to relax the assumption on linear separability. We define the non-linear generalization of inner product concept-based interpretation, Gradient Correlation (GC), as follows
Roc(E; v.))) = V (Vo())//Vs(++II>
(2)
GC takes the gradients from the target and concept function and calculates the linear correlation of gradients in each dimension in the shared input feature space. Intuitively, if gradients magni- tudes coincide in similar dimensions (large inner product), then the concept is relevant to the target function. CAV is a special case of GC where g is limited a linear function.
The main caveat of GC is the inner product of gradients between the target and concept only yields correlation. In some applications correlation is already sufficient for interpretation, which explains the success of CAV. However, one may be tempted to ask: if a mapping from the concept to the target function output exists, can we retrieve the gradient of the target function output with respect con- cepts to evaluate "causal" interpretation? This is an extension of typical input gradient approaches for Reature-based interpretations (Ancona et al.) [0177] [hrikumar al.) 2017 Sundararajan et al.) 2017. Is it possible to calculate this concept gradient given only access to the gradients Vf(r) and (x))? The answer lies in the shared input feature space of f and g where gradients can be propagated with chain rule.
PROPOSED METHOD
3.1 DEFINITION OF CONCEPT GRADIENTS (CG)
We define the Concept Gradients (CG) to measure how small perturbations on a concept affect the target function output f through gradients:
Vg(z)T
Reo(:; fog(=f.g)) = Vg(z)
[V(())]² vf(x)
(3)
3

---

dr Published as a conference paper at ICLR 2023
where Vg(z)) is the Cooor-Peenroee inverse (pseudo- -nneudo- of Vg(z). Pseudo-inverss is a gen- eralized version of matrix inversion when the inverse does not exist, forms the (unique) inverse mapping from the column space to the space of the original matrix, while leaving all the other spaces untouched. Using pseudo-inverse prevents mis-attribution of importance from other spaces irrelevant to Vg(x:. Conveniently, the pseudo-inverse of Vg((;) is just its normalized transpose since Vg(()) is a d-dimensional vector. Suppose function h that maps the concept to target function uniquely exists,
f(z) h(c) h(g(x))
then CG exactly recovers the derivative of h which is the importance attribution of the target function with respect to concept
dh(c) dh(g(=)) Rea(x; f.9).
(c)
dc
dg(z)
Let us illustrate the intuition CG with a simplified case where y and e are all scalars (k: = d d= 1). Our goal is to represent as a function of e to obtain the derivative dy We can expand the expression with chain rule:
de.
dy dy
dr de
de dr
de
The derivative da the remaining term to resolve. Notice that since r and are scalars, is simply de
de the inverse of de du = gz) assuming de * 0. Thus, concept gradients exactly recovers du de
de
dy - ((xx)
( - (())) f'(x) dr
de
(x)) = RcG(x).
(4)
dc
The pseudo-inverse in extends this derivation to the general case when x, y, and have arbitrary dimensions. Details can be found Appendix B
3.2 IMPLEMENTATION OF CG
Implementing CG is rather simple in practice. Given a target model f interpret, CG can be calculated if the concept model g is also provided by the user. When concepts are given in the form of positive and negative samples by users, instead of explicit concept function, we can learn the concept function from the given samples. Generally, any concept model g sharing the same input representation as would suffice. Empirically, we discovered that the more similar f and g is (in terms of both model architecture and weight), the better the attribution performance. Our hypothesis is since there might be redundant information in the input representation to perform the target prediction, many solutions exist (i.e. many f can perform the prediction equally well). The more similar g utilizes the input information as f, the more aligned the propagation of gradient through the share imput representation is.
We propose simple and straightforward strategy for training concept models similar to f. We train 9 predict concepts by finetuning from the pretrained model The weight initialization from f allows the final converged solution of to be closer to f. We can further constrain g by freezing certain layers during finetuning. The similarity between f and g leads similar utilization of input representation, which benefits importance attribution via gradients. More details regarding the importance of the similarity between and g can be found in Section 3.3
CG can be used to evaluate per-sample (local) and per-class (global) concept relevance score. Fol- lowing TCAV (Kim et al.] (2018), we evaluated global CG by relevance score calculating the propor- tion of positive local CG relevance over a set of same-class samples Z, {(11 v).... (I,, v)}.
Reg(Zyif.g) {(x v) E Z, Reo(:; f.g) 0}l
Global CG relevance score can also be aggregated differently, suitable to the specific case.
3.3 SELECTING LAYER FOR ATTRIBUTION
Similar to CAV, CG can be computed at any layer of a neural network by setting x as the hidden state of neurons in particular layer. The representation of input is relevant attribution, as the
4

---

Published as a conference paper at ICLR 2023
information contained differs between representations. Both methods face the challenge of properly selecting layer to perform calculation.
For feed-forward neural network model, information irrelevant to the target task is removed when propagating through the layers (i.e. feature extraction). Let us denote the representation æ in the 1th layer of the target model f as ft--- We hypothesized that the optimal layer I" for performing CG is where the representation f- contains minimally necessary and sufficient information to predict concept labels. Intuitively, the representation of Iy needs to contain sufficient information to correctly predict concepts to ensure the concept gradients (g(x) are accurate. On the other hand, there redundant information in zz that can be utilized to predict the target y and concept c, then g may not rely on the same information as f. which causes misalignment in gradients Vg(z) and (x)) leading to underestimation of concept importance.
The algorithm for selecting the optimal layer to perform CG is simple. The model is initialized with weights from and all the weights are initially frozen. Starting from the last layer, we unfreeze the layer weights and finetune 9 to predict concepts. We train until the model converges and evaluate the concept prediction accuracy on a holdout validation set. The next step is to unfreeze the previous layer and repeat the whole process until the concept prediction accuracy saturates and no longer improves as more layers are unfrozen. We have then found the optimal layer for CG as well as the concept model g.
3.4 CONNECTIONS BETWEEN CAV, GC, AND CG
In the special case when g(x) = ve x is a linear function (the assumption of CAV). we have Rea v (=)//ll This is almost identical conceptual sensitivity score in Eq except a slightly different normalization term where CAV normalizes the inner product by 1/ e/l Fur- thermore. the sign of CG and CAV will be identical which explains why CAV is capable of retrieving important concepts under the linearly separable case.
Here we use simple example demonstrate that the normalization term could be important in some special cases. Consider f as the following network with two-dimensional input [xo, æ1]:
20 21
100
0
hg
ho
0.01 0
0
x 21
y 0.1%0 21:
(5)
0
1
h1
h1
and cg = Dp. C = x1. Then we know since y = 0. 0. + 21 = 0.1x% + 31, the contribution of
dy = 0.1, dy = 1 and it's easy to verify that CG will should be 10 times larger than cp- In fact, deg
des
correctly obtain the gradient no matter which layer is chosen for computing (3) However, the results will be wrong when different normalization term is used when computing concept explanation the hidden layer h. Since co- = 100hg, C1 = h1 y 10hg + hr, we have
For concept eot v = dcp = [100, 0 dy
= [10, 1], v" u/llull = 10, 10,uTe//lell 2/|lu||== 0.1
dh
dh
dog
[0, 1)7, dy
For concept c:: v dh.
dh = [10, 1), TT
TTu //||o]| = u, //1/0/12 2 1.
(6)
Therefore, the normalization term used in CAV (in red color) will lead to a conclusion that co C1. while CG (in blue color) will correctly get the actual gradient and conclude > cp. This is mainly because CG formally derived from the actual gradient, as will discuss below. In contrast, CAV is based on the intuition of correlation in the form of gradient inner product and not the exact chain-rule gradient, so its attribution is subject to per-dimensioe scaling.
Although the normalization term can be important in some special cases, in practice we do not find the attribution results to be much different with different normalization terms in empirical studies. We compared different methods of calculating CG (including different normalization schemes) em- pirically in Section [ We hypothesis such extreme per-dimensional scaling in input features are less common in well-trained neural networks. Thus, in practice if the concept can be accurately modeled by a linear function, CAV might be a good approximation for concept gradients. When linear separability assumption does not hold, GC might be a good approximation for concept gradi- ents. But in general only CG recovers the concept gradients when feature scaling conditions are not ideal.
5

---

Published as a conference paper at ICLR 2023
Inception v3
Resnet50
Vgg16 bn
0.8
0.9
0.8
0.8
0.7
0.7
0.6
0.6
0.6
*
0.5
0.5
0.4
0.4
0.4
0.3
0.3
0.3
Mixedd 6c
Misedd 6d
Mited, Be
Minod, 7a
Mixed, 7b
Mixed, 76
layer3.3
layer3.4
layer3.5 5
layord
Jayerd.
byyer..2
feabures
foaturea 14
fontures
featuren 34
classileer
classiffers
classifiord
Local Recall@30 (CG)
Global Recall@30 (CG)
Local Recall@30 (CAV)
Global Recall@30 (CAV)
Figure 2: CUB concept recalls for different imput representations in various layers and architectures (left to right, deep to shallow layers). CG consistently performs better than CAV locally and globally.
4 EXPERIMENTAL RESULTS
4.1 QUANTITATIVE ANALYSIS
In this experiment, our goal is to quantitatively benchmark how well CG is capable of correctly retrieving relevant concepts in setting where the ground truth concept importance is available. In this case, the ground truth concept importance consists of human annotations of how relevant concept is to the model prediction. We can evaluate the quality of concept importance attribution by treating the task as a retrieval problem. Specifically given a data sample and its target class prediction, the concept importance given by CG is used to retrieve the most relevant concepts to the prediction. good importance attribution method should assign highest concept importance to concepts relevant to the class.
We benchmarked local (per-sample) and global (per-class) concept importance attribution sepa- rately. Given an input sample (æi, W:). ). its ground truth relevant concept set C; and the top attributed local important concept set C, the local recall @ defined as |Cinc.]
Given set of data samples from the same class Σ,, {(I1,,), .)................... (#m, and their ground truth relevant concept sets {Či,..., Čr.), we can obtain the class-wise ground truth concept set G2 by majority voting. We can also obtain the top k attributed global important concept set Cy. The global
recall@k @k is then defined as je,ncy)
The local attribution performance is measured by the average local recall@k (over samples) on holdout testing set. The global attribution performance is measured by the average global recall@k (over classes) on holdout testing set.
Dataset. We experimented on age classification. consists of 11k bird images, 200 bird classes, and 312 binary attributes. These attributes are descriptors of bird parts (e.g. bill shape, breast pattern, eye color) that can be used classification. We followed experimental setting and preprocessing in (Koh et al. 2020) where attributes with few samples are filtered out leaving 112 attributes as concepts for interpretation.
(Wah et al. 2011), a dataset for fine-grained bird im-
Evaluation. For CG, we finetuned the target model f for fine-grained bird classification on the concept labels to obtain the concept model g. We also evaluated CAV on the corresponding model layers for comparison. All the details regarding model training for reproducibility can be found in Appendix J.2 The local and global recalls for all models and layers plotted in Fig 2 The layers in the x-axis are shallow to deep from left to right.
Tables and compares the best result of CAV and CG of the three different model architectures. CG consistently outperforms CAV in concept importance attribution (recall) since the non- -oneinnarity in CG captures concepts better (higher concept accuracy).

---

Published as a conference paper at ICLR 2023
Table 1: Local concept importance attribution comparison CUB
Concept
Local (per-sample)
Model
Method
Accuracy
R@30
R 40
R R@50
CAV
0.709
0.556
0.664
0.745
Inception v3
local-CG
0.791
0.612
0.718
0.790
CAV
0.727
0.557
0.663
0.740
Resnet50
local-CG
0.793
0.627
0.726
0.794
CAV
0.703
0.525
0.628
0.708
16 bn
local-CG
0.793
0.637
0.733
0,800
Table 2: Global concept importance attribution comparison on CUB
Concept
Global (per-class)
Model
Method
Accuracy
R 30
R@40
R@
TCAV
0.709
0.619
0.742
0.822
Inception v3
global-CG
0.791
0.790
0.894
0.944
TCAV
0.727
0.592
0.692
0.775
Resnet50
global-CG
0.793
0.827
0.915
0.951
TCAV
0.703
0.628
0.747
0.832
1g bn
global-CG
0.793
0.872
0.935
0.961
Ablation study model layers. We performed an ablation study on performing attribution with different input representations, as given by different model layers. We observe that for every input representation in every model, CG consistently outperforms CAV in concept importance attribution, both locally and globally. For CG, the performance trend peaks in the penultimate layer represen- tation. While finetuning more layers may lead to better concept prediction, it does not necessarily translate into better attribution. There a trade-off between the concept model capturing the con- cept well (finetuning more layers better) and utilizing the input more similarly to the target model (finetuning less layers better). In this case, the input representation sufficient information capture the concept labels. For CAV, the performance is generally better in deeper layers since concepts are more linearly separable in the latter layers. Unlike the non-linearity of CG, CAV is unable to exploit the more concept information contained in the representation of earlier layers.
the penultimate layer retains
Ablation study on normalization sehemes. We compared different methods of calculating the concept gradients with various gradient normalization schemes. We ran all experiments on Incep- tion v3 with finetuned layers Mixed_ 7b+. The results are presented in Table 3 Recall that CG is exactly inner product normalized with squared concept norm, IP represents pure inner product between Vz(z) and Vg(x) without normalization, GC with normalized Vg(x). and cosine with both normalized V fxr) and Vg(x). We observe that all methods performs equally well. This sup- ports the argument that the gradient norms trained neural network are well-behaved and it less common to encounter cases where normalization influences the attribution results significantly.
Table 3: Comparison CG with different normalization schemes
Local
Global
Scheme
Formula
R@ 30
R 4400
R @ 50
Φ 30
R @@40
R@ 50
Volay"
CG
V fx)
0.612
0.718
0.790
0.783
0.907
0.949
[
IP
Vg(z)*
V (x)
0.601
0.718
0.801
0.783
0.907
0.949
GC
Vola)"
(x())
0.610
0.720
0.799
0.783
0.907
0.949
V9(z)]
Cosine
Vo(a)"
((())
0.610
0.720
0.799
0.783
0.907
0.949
4.2 QUALITATIVE ANALYSIS
The purpose of this experiment is provide intuition and serve sanity check by visualizing instances and how CG works.
Dataset. We conducted the experiment the Animals with Attributes (AwA2) dataset Xian et al. 2018), an image classification dataset with 37k animal images, 50 animal classes, and 85 binary attributes for each class. These concepts cover wide range of semantics, from low-level colors and

---

Published as a conference paper at ICLR 2023
Uipess
Figure 3: Visualization of instances with highest CG attributed importance (AwA2 validation set) for each concept (top instance the top classes per concept). CG is capable of handling low level (colors), middle level (textures), and high level (body components) concepts simultaneously.
textures, to high-level abstract descriptions (e.g. "smart", "domestic"). We further filtered out 60 concepts that are visible in the input images to perform interpretation.
Evaluation. The evaluation is performed on the validation set. visualizes the instances with the highest CG importance attribution for 6 selected concepts, filtering out samples from the same class (top instance the top 3 classes). The concepts are selected to represent different levels of semantics. The top contains colors (low-Ievel), the middle contains textures (medium- level), and the bottom row contains body components (high-level). Observe that CG is capable of handling different levels of semantics simultaneously well, owing the expressiveness of non-linear concept model Additionally, we presented randomly sampled instances from the validation set and listed top-10 most important concepts attributed by CG (see the appendix). We intentionally avoided curating the visualization samples to demonstrate the true importance attribution perfor- mance of CG. The most important concepts for each instance passed the sanity check. There no contradictory concept-class pairings and importance is attributed to concepts existent in the images.
4.3 CASE STUDY MORTALITY RISK MYOCARDIAL INFARCTION COMPLICATIONS
The purpose of the case study is to demonstrate the effectiveness of CG in critical domain applica- tions beyond classification tasks on natural natural-image data.
Dataset. We experimented the Myocardial infarction complications database. The database sists of 1,700 entries of patient data with 112 fields input tabular features, complication fields for prediction, and field of lethal outcome also for prediction. The input features measurements taken when the patients were admitted to the hospital due to myocardial infarction as well as past medical records. The target models predict lethal outcome given the 112 input fields. The concept models predict the complications given the same 112 input fields.
Evaluation. Our goal is interpret the lethal outcome with the complications and compare interpretation with existing literature regarding how each complication affects the risk of death. We expect good interpretations
assign high relevance to complications that pose high mortality risk. Table shows the global CG scores (aggregated by averaging local CG) as well as the excerpted description of the complication mortality risk in the existing medical literature. The severity of descriptions in the medical literature is largely aligned with the CG scores. The highest risk compli- cations are attributed the most importance (e.g. relapse of MI) while the lower risk complications are attributed the least importance (e.g. post-infarction angina). This supports CG an effective method for real-life practical use for interpreting models in critical domains. We also provided TCAV scores (aggregated by averaging local CAV) for comparison, which is largely aligned with CG with few exceptions (c.g. chronic heart failure) where the CAV interpretation deviate from litcrature. The full table can by found in Appendix T.3
RELATED WORK
Our work belongs to post-hoe concept-based explanations. While training with self interpetrrtt
models (Bouchacourt & Denoyer 2019 Chen al. 2019 Lee al. 2019 Wang & Rudin 2015)
8

---

Published as a conference paper at ICLR 2023
Table 4: Mortality risk attribution with respect to subset of myocardial infarction complications and comparison with existing medical literature
Complication Relapse of MI
CG
TCAV
Excerpted mortality risk description from medical literature Recurrent infarction causes the most deaths following myocar- dial infarction with left ventricular dysfunction. (Om et al.) 2005)
3.47
2.55
Chronic heart failure
3.27
-1.26
The mortality rate in a group of patients with class III and IV heart failure is about 40% per year, and half of the deaths sudden. (Bigger 1987)
Myocardial
1.62
6.52
Myocardial rupture is relatively rare and usually fatal compli- cation of myocardial infarction (MI). (Shamshad et al. (0T00)
rupture
Ventricular
0.91
1.90
Patients developing VF in the setting of acute MI are at higher
fibrillation
risk of in-hospital mortality. (Bougouin et al. (2014)
Dressler
0.32
-2.85
The prognosis for patients with DS is typically considered to be quite good. (Leib al. (2017)
syndrome
Post-infarction angina
-1.40
-2.85
After adjustment, angina was only weakly associated with cardiovascular death, myocardial infarction, or stroke. (Eisen)
et al. 2016)
is recommended if the use case allows, often times concepts are not known beforehand which makes the practicality of post-hoe explanations more widely adoptable. Other classes of post-
hoc explanations include featured-based explanations (Zintgraf et al.) 2017 Petsiuk et al., 2018
Dabkowski & Gal 2017 Shrikumar et al. [2017] Sundararajan et al.| [20 counterfactual explana-
DhuranDhur et al. 2018 Hendricks et al., 2018 8 van der Waa et al. 2018 Goyal et al. 2019b
Joshi et al. 2019 Poyiadzi et al. [2200] Hsieh et al. 2021), , and sample- based explanations Bien &
Tibshirani 2011 Kim et al.
& Liang 2017 Yeh et al.] 2018 Pruthi et al. 2020 Khanna 2018). 8). Our work considers the gradient from prediction to concepts, which is in spirit con-
et al.
nected to feature explanations which considers the gradient from prediction feature inputs (Zeiler
& Fergus 2014 Ancona et al.) 2017).
Concept-based explanations aims to provide human-centered explanations which answer the ques- tion "does this human understandable concept relates to the model prediction?". Some follows-up for concept-based explanations include when are concept sufficient to explain model (Ych al. 2019). computing interventions on concepts for post-hoc models (Goyal et al.] 2019a) and self- interpretable models (Koh et al.] (020), combining concept with other feature attributions (Schrouff et al. 202T). unsupervised discovery of concepts (Ghorbani et al. 2019 Yeh et al., 2020; Ghan- deharioun et al. 202T), and debiasing concepts (Bahadori & Heckerman 2020). The most similar work to ours the work Chen et al. (2020), which is motivated by the issue of CAV that concept does not necessarily lie in the linear subspace of some activation layer. They address this problem by training ceff-interpretable model and limits the concepts to be whitened. On the contrary, work address the non-linear concept problem of CAV in the post-hoe setting by learning a non-linear concept component and connects to the activation space via chain rule. Another work that considers nonlinear modeling of concepts is TCAR (Crabbé & van der Schaar (022). They modeled concepts with non-linear kernel functions and calculate relevance score via the concept function output mag- nitude. However, merely considering the function output ignores interactions between the target and concept function and potentially leads explaining spurious correlations.
CONCLUSION
We revisited the fundamental assumptions of CAV, one of the most popular concept- based, gradient interpretation methods. We tackled the problem from mathematical standpoint and proposed CG to directly evaluate the gradient of given target model with respect to concepts. Our insight ex- plains the success of CAV, which is a linear special case of CG. Empirical experiments demonstrated that CG outperforms CAV on real datasets and is useful in interpreting models for critical domain applications. Currently CG depends on the representation of input. Devising an input representation invariant method is interesting future direction. CG also requires user-specified concepts to func- tion. Integrating automatic novel concepts discovery mechanisms into the method may be useful in applications with insufficient domain knowledge.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work is supported in part by NSF under IIS-2008173, IIS-2048280, an Okawa research grant, NIH NIGMS MIRA (1R35GM146735), UCLA California NanoSystems Institute and the Noble Family Innovation Fund. would also like to thank Karen who provided immense emotional support and offered insightful suggestions.
REFERENCES
Marco Ancona, Enea Ccolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-bawe brasee attribution methods for deep neural networks. arXiv preprint arXiv:1711.66100 2017.
Mohammad Taha Bahadori and David Heckerman. Debiasing concept-based explanations with causal analysis. In International Conference on Learning Representations, 2020.
E. J. Benjamin, P. A. Wolf, R. B. D' D'gostino, H. Silbershatz, W. B. Kannel, and D. Levy. Impact of atrial fibrillation on the risk of death: the Framingham Heart Study. Circulation, 98(10)994 -522, Sep 1998.
Jacob Bien and Robert Tibshirani. Prototype selection for interpretable classification. The Annals of Applied Statistics, pp. 2403-2424, 2011.
J. T. Bigger. Why patients with congestive heart failure die: arrhythmias and sudden cardiac death. Circulation, 75(5 Pt 2):28-35, May 1987.
Diane Bouchacourt and Ludovic Denoyer. Educe: Explaining model decisions through unsupervised concepts extraction. arXiv preprint arXiv: /905. 1852, 2019.
W. Bougouin, E. Marijon, E. Puymirat, P. Defaye, D. S. Celermajer, Y. Le Heuzey, Boveda, S. Kacet, P. Mabo, C. Barnay, A. Da Costa, C. Deharo, J. C. Daubert, J. Ferrières, T. Simon, and N. Danchin, Incidence sudden cardiac death after ventricular fibrillation complicating acute myocardial infarction: 5-year cause-of-death analysis of the FAST-MI 2005 registry. Eur Heart J, 35()): 16-122, Jan 2014.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan Su. This looks like that: deep learning for interpretable image recognition. In Advances in Neural Information 2019.
Processing Systems, pp.
Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence. 2(12):772-782, 2020.
Jeremy M Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. arXiv preprint arXiv:
2019.
Jonathan Crabbé and Mihaela van der Schaar. Concept activation regions: A generalized framework for concept-basee explanations. In Advances Neural Information Processing Systems, 2022.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In Advances in Neural Information Processing Systems, pp. 6967-6976. NeurlPS, 2017.
Amit Dhurandhar, Pin-Yu Chen, Ronny Luss Chun-Chen Tu, Paishun Ting, Karthikeyan Shan- mugam, and Payel Das. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. In Advances Neural Information Processing Systems, pp. NeurIPS, 2018.
A. Eisen, D. L. Bhatt, P. G. Steg, K. Eagle, S. Goto, J. Guo, S. Smith, E. M. Ohman, and B. M. Scirica. Angina and Future Cardiovascular Events in Stable Patients With Coronary Artery Dis- case: Insights From the Reduction Atherothrombosis for Continued Health (REACH) Registry. J Am Heart Assoc, 5(10), 09 2016.
10

---

Published as a conference paper at ICLR 2023
Asma Ghandeharioun, Been Kim, Chun-Liang Li, Brendan Jou, Brian Boff, and Rosalind W Pi- card. Dissect: Disentangled simultaneous explanations via concept traversals. arXiv preprint arXiv:2105.15164, 2021.
Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept-based explanations. In Advances in Neural Information Processing Systems, 2019.
Yash Goyal, Amir Feder, Uri Shalit, and Been Kim. Explaining classifiers with causal concept effect (cace). arXiv preprint arXiv:
2019a.
Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. In International Conference on Machine Learning, pp. 2376-2388 ICML, 2019b.
P. Harikrishnan, T. Gupta, C. Palaniswamy, D. Kolte, S. Khera, M. Mujib, W. Aronow, C. Ahn, S. Sule, D. Jain, A. Ahmed, H. A. Cooper, Jacobson, Iwai, W. H. Frishman, D. L. Bhatt, G. C. Fonarow, and J. A. Panza. Complete Heart Block Complicating ST-Segment Elevation Myocardial Infarction: Temporal Trends and Association With In- -osstalt Outcomes. JACC Clin Electrophystol, (66):529-536 Dec 2015.
Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, and Zeynep Akata. Grounding visual explana- tions. In ECCV. ECCV, 2018.
Cheng-Yu Yu Hsieh, Chih-Kuan Yeh, Xuanqing Liu, Pradeep Kumar Ravikumar, Seungyeon Kim, Sanjiv Kumar, and Cho-Jui Hsieh. Evaluations and methods for explanation through robust- ness analysis. In International Conference on Learning Representations. ICLR, 2021. URL https:// optpreiieeeeeeeeeeeemmee net www.:eewwwweeetoouueuiieoo. het:/oorum1G-444aaCCCC
S. Joshi, O. Koyejo, Warut D. Vijitbenjaronk, Been Kim, and Joydeep Ghosh. Towards realistic individual recourse and actionable explanations in black-box decision making systems. ArXiv, 0bs/1907.09611 2019.
Rajiv Khanna, Been Kim, Joydeep Ghosh, and Oluwasanmi Koyejo. Interpreting black box predic- tions using fisher kernels. arXiv preprint arXiv:1810.10116 pp. 3382-33222 -390,, 2018
Been Kim, Rajiv Khanna, and Oluwasanmi Koyejo. Examples are not enough, learn to criticize! criticism for interpretability. In Advances in Neural Information Processing Systems, pp. 2280- 2288. NeurlPS, 2016.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International Conference on Machine Learning, pp.
ICML, 2018.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning, pp. 1894. ICML, 2017.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. arXiv preprint arXiv:2007.04611 04611, 2020.
B. A. Koplan and W. G. Stevenson. Ventricular tachycardia and sudden cardiac death. Mayo Clin Proc, 84(3):289-297, Mar 2009.
Guang-He Lee, Wengong Jin, David Alvarez-Melis, and Tommi Jaakkola. Functional transparency for structured data: game- theoretic approach. In International Conference on Machine Learn- ing. pp. 3723-3733. PMLR, 2019.
Ari D Leib, Lisa A Foris, Tran Nguyen, and Karam Khaddour. Dressler syndrome. 2017.
C. Medi, J. M. Kalman, and S. B. Freedman. Supraventricular tachycardia. Med Aust, 190(5): 255-260, Mar 2009.
S. Orn, J. G. Cleland, M. Romo, J. Kjekshus, and K. Dickstein. Recurrent infarction causes the most deaths following myocardial infarction with left ventricular dysfunction. Am J Med, 118(7): 752-758, Jul 2005.
11

---

Published as a conference paper at ICLR 2023
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv: 2018.
Rafael Poyiadzi, Kacper Sokol, Raúl Contos-Rodriguee D. Bie, and Peter A. Flach. Face: Feasi- ble and actionable counterfactual explanations. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 2020.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33, 2020.
A. Roguin, D. Behar, H. Ben Ami, S. A. Reisner, S. Edelstein, S. Linn, and Y. Edoute. Long-term prognosis of acute pulmonary oedema-an ominous outcome. Eur J Heart Fail, 2(2):137-144, Jun 2000.
Jessica Schrouff, Sebastien Baur, Shaobo Hou, Diana Mincu, Eric Loreaux, Ralph Blanes, James Wexler, Alan Karthikesalingam, and Been Kim. Best of both worlds: local and global explana- tions with human- uuman-undeestandable concepts. arXiv preprint arXiv:2106.086411 2021.
F. Shamshad, S. Kenchaiah, P. V. Finn, J. Soler-Soler, J. McMurray, H. J. Velazquez, A. P. Mag- gioni, R. M. Califf, K. Swedberg, L. Kober, Y. Belenkov, S. Varshavsky, M. A. Pfeffer, and S. D. Solomon. Fatal myocardial rupture after acute myocardial infarction complicated by heart fail- ure, left ventricular dysfunction, or both: the VALsartan In Acute myocardial iNfarcTion Trial (VALIANT). Am Heart 160(1):145-1511 151, Jul 2010.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. International Conference on Machine Learning, 2017.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825, 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference Machine Learning, pp. 3319-3328. PMLR, 2017.
Jasper van der Waa, Marcel Robeer, Jurriaan van Diggelen, Matthieu Brinkhuis, and Mark Neerinex. Contrastive Explanations with Local Foil Trees. In 20/8 Workshop on Human Interpretability in Machine Learning (WHI). WHI, 2018.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-uesd birds-200-205 2011 dataset. Technical Report SNNS-RR-001--00 California Institute of Technology. 2011.
Fulton Wang and Cynthia Rudin. Falling rule lists. In Artificial Intelligence and Statisties, pp- 1013-1022, 2015.
Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning- comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis 2018
and machine intelligence, 41(9):2251-266
Chih-Kual Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar, Representer point selection for explaining deep neural networks. In Advances in Neural Information Processing Systems, pp. 9291-9301. NeurlPS, 2018
Chih-Kual Yeh, Cheng- Yu Hsieh, Arun Sai Suggala, David I. Inouye, and Pradeep Ravikumar, On the (in)fidelity and sensitivity of explanations. In NeurIPS, volume 10976, 2019.
pp. 10965-
Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-aware concept-based explanations in deep neural networks. Advances in Neural Information Processing Systems. 33, 2020.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In
European conference on computer vision, pp.
Springer, 2014.
Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint arXiv: 2017.
12

---

Published as a conference paper at ICLR 2023
A LIMITATIONS
CG is a gradient-based interpretation methods, which is can only be applied to differentiable white- box models. Bradient-based methods convey how small changes in the input affect the output via gradients. Larger changes in the input requires intervention-ban based causal analysis to predict how the output is affected. As a modification upon CAV, CG also requires users to specify concepts with sufficient representative data samples or close-form concept functions. Sufficient amount of data is more important for CG to prevent the nonlinear function to overfit. Automatic discovery of novel concepts requires introducing other tools. Finally, CG requires fitting non-linees concept models if the concept is provided in the form representative data samples. This might be computationally intensive for complex non-linear models (e.g. neural networks). The quality of interpretation highly depends on how accurately the concept is captured by the concept model.
B GENERAL CG FOR MULTIPLE CONCEPTS
Here we derive CG in the general case when the concept function maps E Rd to m concepts g: Rd Rm Recall the definition of CG:
Reo f.9) = Vg(x)
Note that when computing the gradient of multivariate function such as f. we follow the convention that V fz() E Rdx where the (vf())) = dx,
(z) And Vg(z)* E Rm is the pseudo-inverse of Vg(x). Rc)(x) will thus be an m x k matrix and its (i,j) element measures the contribution of concept label j.
Let us first consider the scenario where Vg(x) is invertible. In this case, there exists an unique function g (c) mapping c x locally around By chain rule CG($) is equivalent to the derivative of y with respect to c:
Oy dc c==
0f(g dc
c=c Vg- (OVT66 -VV()) CG())
Now let us consider the scenario when Vg(x) is not invertible. For simplicity, we assume (g(x) has full column rank, which implies is surjective (this is always true when m = 1). Otherwise we can directly constrain e within the row space of Vg(z) and the arguments remain valid.
Our goal is to construct mapping from to and analyze the gradient. However, there are infinitely many functions from to & that can locally inverse g(x) since the dimension of concept (m) could be much smaller than input dimension d. Despite an infinite number of choices, show the gradient of such function always follows a particular form:
Theorem 1. Consider a particular point with = g()). Let : Rm - Rdd be a smooth and differentiable function mapping to and satisfy g(h(c)) locally within the e-ball around è, then the gradient of will take the form of
+ Vh(c) = Vg( +G1.
where any row vector of G belongs to null (V(l(()(zz)"))) (null space of Vg(an)²).
The proof of the theorem is deferred to the appendix. Intuitively, this implies the gradient will take particular form in the space of Vg(z)" while being arbitrary in its null space since any change in the null space cannot locally affect c. We can verify that g(x) is locally unchanged in the null space of Vg(x')
g(x + G1)
Vg(x) = g(x).
This is saying there are multiple choices in Ar to achieve the same effect on c, since any additional perturbation in null (u((())(=)²*)) won't make any change to locally. For example, when we want to change the concept of "color" in an image by Ar, we can have Ar only including the minimal change (e.g., only changing the color), or have Ar including change of color and any arbitrary change to another orthogonal factor (c.g., shape). For concept-based attribution. it is natural to consider the minimal space of x that cover e, which corresponds setting G as 0 in 7
13

---

Published as a conference paper at ICLR 2023
represent y function of locally by y = p(c) = f(h(c)) near By chain rule we then have
If we pick such h then Vh(ê) = Vg(ā)
Vp(ê) va()) Va(+) (c)
In summary, although there are infinitely many functions from c to y since the input space has a larger dimension than the concept space. if we consider subspace in such that change of will affect (locally) and ignore the orthogonal space that is irrelevant to c. then any function mapping from to y through this space will have gradient equal to Concept Gradients defined in
B. SHOULD WE EXPLAIN MULTIPLE CONCEPTS JOINTLY OR INDIVIDUALLY?
When attributing the prediction to multiple concepts, our flexible framework enables two options: 1) treating each concept independently using single-concept attribution formulation (??) 2) Combining all the concepts together into E IR" and run CG. Intuitively, option 2 takes the correlations between concepts into account while option does not. When both concepts A and B are important for prediction but concept A is slightly more important than concept B, option 1 will identify both of them be important while option 2 may attribute the prediction concept B only. For example, when y = Ho + while option will produce negative score for Co. We further compared these two options on real datasets. The results are presented in Table 5 We verified that empirically applying pseudo-inverse individually (independently) for each concept better aligned with the attributes labeled by humans.
= To, C1 = Zo + 0.1z1, option 1 will identify both concepts to be important,
Table 5: Comparison of different CG pseudo-inverss calculations
Local
Global
Method
R@30
R@40
R@50
R@ 30
R@40
R@50
Joint
0.345
0.440
0.530
0.409
0.511
0.607
Independent
0.612
0.718
0.790
0.783
0.907
0.949
C EXAMPLE FOR DEMONSTRATING DIFFERENT CG CALCULATION METHODS
We use a simple toy example to explain the difference between option and 2. Assume y = xp + Ø1 and two concepts co Zo, zo + 0.1z1, Since these relationships are all linear, concept gradients will be invariant to reference points. The results computed by option I and are
Option I (individually apply (??)): Reg. 1, R Rog. R 1.01
Option 2 (apply CG jointly by (3: Reo.y = -,, R... = 10
At the first glance, might be counterintuitive as to why the contribution of co is negative to with option 2. But in fact there exists unique function y -9cp + 10c; mapping jointly considering two concepts), which leads to negative contribution of co. This shows that when considering two concepts together as a joint function, the gradient is trying to capture the effect of one concept with respect to others, which may non-intuitive to human.
(when
D EXAMPLE FOR SELECTING PSEUDO-INVERSE INSTEAD OF OTHER INVERSE MATRICES
For the calculation of CG, multiple inverse matrices exist when Vg(xr) is not invertible. We gave high level example in Section B of why makes sense to apply the pseudo-inverse. In this example, provide concrete example with numbers to further elaborate.
Suppose the raw input x has two dimensions: #1 indicates the feature "color" and 22 indicates the feature "shape". There's 's only one single concept corresponds "color". target model is y = fz) = 21 F2-
g(x) 21. The
Let's consider particular input å [1, 0] and 1. Clearly small perturbation to "color" can lead to linear change to output with slope 1, so the concept attribution score should be 1. Now let's follow the derivation of CG in this example to show why choose the minimum-norm solution.
14

---

Published as a conference paper at ICLR 2023
Centered at é, there be infinite number of (linear) mappings from to 2, but all of them will follow the form according to Theorem !I
h(c) = h(ě) + Vh(O)(c-
Vh(c) = Vg(+)* + G1
In our example, be written explicitly as follows:
h(c) [1. 0] + ([1.0] + [o, 1)
[c. a(c - 1)], Va
This corresponds to Theorem I, where Vg( = [1. 0] and G is the set of [0, af with any a. These functions all set æ1 c but will æ2 a(c 1) with arbitrary a. Conceptually, this is saying there are infinite number of ways to form the mapping from "color" (c) to input (r): we always change "color" according to but we can arbitrarily change the other orthogonal feature "shape" (I:2) according to color. Concept Gradients is trying to see how the change of affect output while keeping all the other orthogonal factors fixed. Therefore, we want to find the mapping that has minimal perturbation to when changing c, corresponding to keeping all other orthogonal factors fixed. This the reason to choose the minimal norm solution pseudo inverse.
In this example, our method sets 0 and will lead to h(c) = [1, 0). If is set to some non-7ero values in the pseudo-inverse, it will lead to the following score:
af((e)) = ([1, [1, -1]) =
dc which will give the wrong attribution score for any + O.
E PROOF
Theorem 2. Let h R. -> Rdd be a smooth and differentiable function mapping e to and satisfy g(h(c)) locally within an c-ball around co- Then the gradient of h will take the form of
Vh(co) = Vg(zo)* + g1.
(8)
where g1 in the null space of gg(xx)".
Proof. Assume h. m R is function mapping e to By Taylor expansion have
g(x') = g(x) + Vg(z)" -2) O(I=' =113)
(9)
h(c)) = M(c) + (d O(Il cll3)
(10)
where x' -((c)) and x h(c). Therefore
g(x') g(x) + (g(x) - M(c)) + 0(1((())))))))))))))))))(()))0 - k(c))
=g(x)= + Vg(z)" Vh(c)) + - -cl + - - h(c))))).
The last two terms are both Ollcc 2112) since Vz(z)) is constant and h is smooth (thus locally Lipschitz), so
c- Vg(a)² Vh(c) (d' c) c) O cll²) Therefore, Vh(c)"VV(ceT = which means Vh(c) Vg(z)) + gg..
(11)
E. DEFINITION OF CONCEPT GRADIENTS
Given input space X' and two differentiable functions f X' + ) and X - C, where y is the target label space and C is the concept label space. We define the concept gradients of E to attribute the prediction of the model to the concepts:
ReG (a) Vs(x) Vg(x)
(12)
15

---

Published as a conference paper at ICLR 2023
where Vg(z)² denotes the pseudo inverse of Vg(z). Let f(z) y and g(z) E C. Essentially concept gradients approximate gradient-based saliency of to c, H'(c), via chain rule. For the case where X, y/ and C are all scalar fields, the concept gradients exactly recover f'(c) if g'(x:) o for all æ E X.
Given input space X' and two differentiable functions f x ) and g & C, where y is the target label space and the concept label space. Suppose there exist an unknown differentiable h g. Let f', g'. and hí' denote the first-order derivatives. We define
function h C - the concept gradients of x E X as
Rc)(x) = Vf(x). Vg(z)*
where Vg(z)) denotes the pseudo-inverse of Vg(z). Let y f(x) E y and c g(x) E C. Essentially concept gradients approximate gradient-based saliency of to c, l'(c,, via chain rule. For the where Y. and C are all scalar fields, the concept gradients exactly recover (c()) if g' (I)) f for all a E X,
(((c) dy dy dr dy dc
-
f(())
Rea(1)
de dr
de
dr
g(xx)
We now generalize X' to n-dimensional vector space. Since f, 9. and h are differentiable, we can perform Taylor expansion around x and
f(x') f(x) Vf(x) - 2) O((x' - x)²)
(13)
s(x') g(x) Vo(z)(z' +) +((d' x))
(14)
h(d') = ((c) + ('(c)c' c) +Odd - -c)
(15)
Let us denote Ar = x - x. We can plug Eq 14 into Eq
15
h(g(x')) h(c) M -
e)2)
(c) [g()) 0(A -0) +0((g(z') -))
hc))
Vg(z)Az+0(A4)) +0((g(2)) -c)3)
(()) (V(())
(1)))f(z)) vf(z)Ax O(A²3) ~ vz(z)Ar
h(g(z')) h((c) f(x') f(x)
N((c) (Vo(z)Az) ~ (f(x)Ar
Let us denote the set of right inverses for Vg(æ) (z)
(z) = (Vg(z)* + I Vgl (Vg(z).g1) = 0}
By definition for all 9f E (r),
M((c) Ax Vf(x) g. Ar
16

---

Published as a conference paper at ICLR 2023
If Vg(2)) is invertible, (z) [Vg(z)*} and the equality exactly holds
[((c) - Vf(x) Vg(x)* - vf(x) Vg(x)
If Vg(ar) is not invertible, g. is not unique and infinitely many right inverses exist for Vg(x). In this case, how does the selection of right inverse relate to interpretation? For interpretation, the goal is attribute small change Ae via 9, small change Ar. Non-invertibility of Vg(z)
implies there are many ways we could perform the attribution. Consider the alignment between some 9,
Vg(x)) of and Vg(xr)
Vg(z) (Vg(z) +9) Vi(+)] (V)(t)t + of
Vg(z) Vg(z) [v(())| Vo(e)T
Vg(z) Vg(z)' [v((2] VV
We argue that the best interpretation is when the attribution is faithful to the relation y between X' and C, i.e. when 9, 2 is best aligned with Vg(x). Observe that Vg(z)1 is in the same direction as Vg((r) On the other hand, any right inverse with 91 & 0 is attributing some proportion of de to da direction that is orthogonal to (g(x)) Thus, the best choice for g, is Vg(a)'
E.2 CONNECTION WITH CAV
CAV is special case of concept gradients with g restricted to linear functions. Let denote the concept activation vector associated with concept c. CAV defines the conceptual sensitivity S as the inner product of the input gradient and concept activation vector,
RcAv = ff(x)
|ell
g is restricted to linear functions,
for some constant bias be. Concept gradients is equivalent to CAV conceptual sensitivity normalized by the norm of the concept activation vector,
ReAv(*)
(xo(2) f((x) Vg(z)² ff(x). Vf(x))
| ||
Thus, if the concept can be accurately modeled by linear function, CAV is capable of retrieving the concept gradients. However, in general the linear separability assumption does not hold. In contrast, concept gradients consider general function classes for g, which better captures the relation between X' and C. Given accurately modeling the concept with g is necessary condition for correct interpretation, concept gradients is superior CAV.
F ALTERNATIVE PERSPECTIVE FOR LAYER SELECTION
We can also view layer selection in typical bias-variance tradeoff perspective. If selected later layer to evaluate CG, we are biased towards using representation of x that is optimized for predicting the target y, not c. However, since the information consists the representation is less, we also enjoy the benefit of less variance. On the other hand, if we selected an earlier layer to evaluate CG, then we suffer less from the bias (towards y) but is penalized with higher variance due to abundance of information. The optimal layer is where the representation of x is capable of predicting the concept (minimized bias) while no redundant information is available (minimized variance).
We verified the bias-variance tradeoff hypothesis with experiments. More bias (with respect to target labels) in later layers is confirmed with the observation that finetuning more layers yields higher concept prediction accuracy (see Fig 4 Less variance in later layers is confirmed with the experiment below. We repeated the CUB experiments on the Inception v3 model with 5 different random seeds and evaluated the variance of the gradient Vg(x) over repeated trials, averaged over all data points. Specifically, the gradients for models finetuned starting from different layers evaluated on the same layer (Mixed 6c) for fair comparison. The results are shown in Fig 5 and confirmed the variance hypothesis.
17

---

Published as a conference paper at ICLR 2023
Inception v3
Resnet50
VGG16
0.96
0.975
0.96
0.950
0.94
0.92
0.92
0.900
0.90
0.875
0.80
-X-
recall@30
recall@
re
concept accuracy
concept accuracy
concept accuracy
Mixed 7b+
Mixed 7a*
Mized, Ba
Mixed 6d
Mixad, Gc+
fe*
Jayord 2+
loyork.1
layer4.00
Jager3.55 3
leyer3 4+
Jayer3.3 3
classifee8+
claxsifier:3
classfiir0+
featur features 34
fontures 24
fonturea 14+
feabores 7
Mixed,
Figure 4: Concept prediction accuracy and concept attribution recall when finetuning starting from
different layers of the model. Finetuning more layers leads to higher concept prediction accuracy.
average gradient variance
5
3-
2
Ec+
Mixed 7c+
Mixed 7b+
Mixed 7
Mixed 6e
Figure 5: Variance of gradients finetuning starting from different layers. The variance is higher when finetuning starting from earlier layers.
G ABLATION STUDY ON CONCEPT MODEL CONFIGURATION
Experiments in Section 4. and 4.2 shares the same model architecture for the target and concept model. The two design choices here are 1) using the same model architecture and 2) warm-starting the training of concept models with target model weights. The choices are rather straightforward since both models need to share the same input feature representation for the gradients with respect the input layer to be meaningful. Nevertheless, we conducted an ablation study on the CUB experiment to verify how using different model architectures different weight initialization for the concept model affects interpretation. In this case, CG needs to be evaluated in the input layer, the only layer where the feature representation is shared between target and concept models.
The model architecture alabltion study results are presented in Table The CG scores are all evalu- ated in the input layer. Evidently, using the same model architecture for both the target and concept models is crucial for good interpretation quality with CG. The degradation of interpretation when using different architectures may be caused by mismatched usage of input feature representation be- tween models. The concept model weight initialization results are presented Table 7 Using target model weights more similar the pretrained task is to the downstream task is, the better the finetuned performance. In this case, the concept prediction accuracy suggests the target model task of bird classification is
initialization outperforms using ImageNet pretrained weights significantly. The
18

---

Published as a conference paper at ICLR 2023
better pretraining task for predicting bird concepts, allowing the concepts to be better captured by the concept function (higher accuracy). This naturally leads to better interpretation results.
Table Ablation study on concept model architecture
Target model
Concept model
R 30
R@40
R@50.
Inception v3
Inception v3
0.872
0.931
0.957
Inception
Resnet50
0.535
0.620
0.684
Inception v3
VGG16
0.486
0.561
0.619
Resnet50
Resnet50)
0.933
0.976
0.988
Resnet50
Inception v3
0.623
0.705
0.757
Resnet50
VGGI6
0.554
0.631
0.692
VGG16
VGG16
0.955
0.986
0.994
VGG16
Inception v3 Resnet50
0.525
0.606
0.664
VGG16
0.609
0.698
0.758
Table 7: Ablation study on concept model weight initialization Weight initialization ImageNet pretrained target model pretrained
Concept accuracy
R@30
R 40
5@50.
0.916
0.577
0.670
0.739
0.972 0.872
0.931
0.957
H ABLATION STUDY ON THE EXPRESSIVENESS OF CONCEPT MODEL CLASS
In most of this study, the concept model is implemented by finetuning the target model to predict the concepts instead. When using an input representation in deeper layers, the portion of finetuned model would be relatively less, yielding simpler concept model. Our hypothesis is input repre- sentation in deeper layers generally contains higher level semantics and thus only requires simpler models to capture the concept.
To test this hypothesis, we explored whether using a more expressive function class for the concept model benefits the attribution result. We focused the ResNet architecture and constructed cept models of different complexities. We experimented on input representations in deeper layers (layer4. and layer4. 2) where the original concept model is simpler and may not be ex- pressiveness enough to capture the concepts. To increase the complexity of ResNet model, we duplicated ResNet residual blocks. The more times block is duplicated, the deeper and more expressive model is.
Table the results. The number duplication represents how many time the selected residual blocks duplicated for the concept model. As increasing the complexity of the model does not lead to better concept prediction accuracy, the attribution performance (recall) is not improved. This confirmed our hypothesis for using simpler (more complex) models for deeper (shallower) layer representations.
Table 8: Comparing concept models of different expressiveness. Increasing model expressiveness does not lead increase in accuracy and thus does not translate to better attribution performance. Residual Block
Number of Concept Duplication
Local (per-sample)
Global (per-class)
Accuracy
R@30
R@40
R@50
R@30
R@
R@ 50
0.794
0.614
0.718
0.786
0.808
0.909
0.948
2
0.795
0.605
0.705
0.775
0.810
0.901
0.944
layer4.1
3
0.791
0.597
0.697
0.767
0.811
0.904
0.944
4
0.791
0.594
0.693
0.762
0.820
0.906
0.943
1
0.793
0.627
0.726
0.794
0.827
0.915
0.951
2
0.793
0.622
0.720
0.788
0.838
0.919
0.955
layer4.2
3
0.794
0.613
0.710
0.777
0.838
0.918
0.952
4
0.793
0.613
0.712
0.781
0.857
0.931
0.964
19

---

Published as a conference paper at ICLR 2023
Inception v3
Resnet50
Vgg16 6 bn
0.80
0.75
0.8
0.70
0.65
0.70
0.60
0.63
0.35
0.60
0.6
0.50
0.55
Mixd
Mixed 6d
Mired
Mixed 7a
Mixed,
Mixed 7c
fc
Joyer3.3
Jayer3
Jayer3
Jayerd.0
layert 1
layort
features
featurns
features. 24
features 34
classitlen0
classifier3
classifier.
-X-
Local Recall@30 (CG)
Global Recal (CG)
-X- Local Recall@30 (SmoothCG)
Global Recall@ 30 (SmoothCG)
Figure 6: Comparing CUB concept recalls between CG and SmoothCG (stdev=0.01, n=8) for dif- ferent input representations in various layers and architectures (left to right, deep to shallow layers). SmoothCG performs similarly to CG with marginal improvement in shallow layers.
I To SMOOTH OR NOT TO SMOOTH?
It has been shown that relying gradient saliency maps may yield misleading interpretation due to the high non- linearity in deep neural networks. There are many works on improving gradient-base based interpretations with respect the input features (Smilkov et al.) 2017: Sundararajan et al.) 2017 Shrikumar et al. 2017). Since CG also relies on input gradients for calculation, is important to examine whether CG suffers from the same issues. In this ablation study, we compared SmoothCG, a variant of CG, that remedies inaccurate gradients.
Inspired by SmoothGrad (Smilkov al., 2017), we proposed SmoothCG, where the attribution is averaged over the neighborhood D near data sample.
SmoothCG(z) = E CG(x cc) c>D
Smoothing over the neighborhood mitigates inaccurate interpretation caused by abrupt changes in input gradients in highly non- non-linear functions. It also shown that randomized smoothing improves the robustness (Cohen et al., (201)). In practice, we take finite samples near an input instance with predetermined perturbation distribution (e.g. Gaussian) and average their CG scores.
Tigure/]shows the concept recall comparison between CG and SmoothCG. SmoothCG calculated by smoothing over samples per instance, adding perturbations sampled from (0 0.01). In gen- eral the performances are similar. CG performs marginally better in deeper layers while CmoothCG in shallower ones, likely because gradients propagating deeper layers are less accurate and thus benefit from the smoothing, However, the difference in performance is not consistent nor significant enough to justify the addition computation cost of SmoothCG.
EXPERIMENT DETAILS
J.1 ANIMAL WITH ATTRIBUTES (AwA2)
Data preprocessing Since the original task is proposed for zero-shot classification, the class labels in the default training and validation set disjoint. To construct a typical classification task, we combined all data together then performed 80:20 split for the new training and validation set. During training, the input images are augmented by random color jittering, horizontal flipping, and resizing, then cropped to the default input resolution of the model architecture (299 for Inception v3, 224 for others). During evaluation, the input images are resized and center cropped to the input resolution.
The attribute labels provided in the dataset contains both concrete and abstract concepts. Some abstract concepts cannot be identified merely by looking at the input image (e.g. new world vs
20

---

Published as a conference paper at ICLR 2023
old world). We filtered out 25 attributes that are not identifiable via the input image and used the remaining 60 attributes for interpretation.
Training We trained the target model ff with Inception v3 architecture with ImageNet pretrained weights (excluding the final fully connected layer). We optimized with Adam (learning rate of 0.001, betal of 0.9, and beta2 of 0.999) with weight decay 0.0004 and schedule the learning rate decay by 0.1 every 15 epochs until the learning rate reaches 0.00001. We trained the model for maximum of 200 epochs and early stopped the validation accuracy had not improved for 50 consecutive epochs. The validation accuracy of the trained target model is 0.947.
We trained the concept model g by finetuning different parts of f (freezing different layer model weights). We reweighted the loss for positive and negative class to balance class proportions. We optimized with Adam (learning rate of 0,01, betal of 0.9, and beta2 of 0.999) with weight decay 0.00004 and schedule the learning rate decay by 0.1 every 25 epochs until the learning rate reaches 0.00001. We trained the model a maximum of 200 epochs and early stopped if the validation accuracy had not improved for 50 consecutive epochs.
Evaluation Visualization is conducted on the validation set. Finetuning from Mi xed. a is the latest layer that still predicted the concepts well. According to the layer selection guideline, we selected xed_7a to evaluate CG. We computed CG with the individual inverse method (??). ZigZshoww random samples from the validation set and their top 10 rated concepts. These samples are intentionally randomly sampled as opposed to intentionally curated to provide an intuition of the true effectiveness of CG. In general, the retrieved highest ranked concepts are relevant with the input image. In terms of sanity check, there are no contradictions in the concepts (e.g. furry never assigned an whales).
1. tail
1. walks
1. quadrapedal
2. strong
2. furry
2. ground
fish
3. ground
3. walks
toughskin
4. strong
4. timid
qua drapeaa
5. Turry
hairless
6. black
6. tail
flippers
7. white
7. black
agility
paws
8.
toughskin big
quadrapedal
ground
furry
Turry
ground
agility
3. agility
mountains
4. quadrapedal
4. forest
bulbous
5. walks
5. walkss
hooves
6. tail
6. ground
vegetation
7. fields
7. quadrapedal
8. fields
8. brown
8. lean
9. while
9. Forest
9. strong
10. walks
10. paws
10. tail
1. tail
1. quadrapedal
1. brown
2. big
2. furry
2. ground
arctic
3. tail
3. tail
blue
4. ground
4. Limid
black
5. furry
flippers
loughskin
gray brown
7.
8.
gray
9. timid
9. black
10. strong
10. fierce
10. gray
furry
1. furry
1. quadrapedal
walks
2. black
2. walks
lean
3. brown
3. ground
agilitt
4. quadrapedal
4. Strong
white
5. walks
5. tail
tail
6. strong
6. big
quadrapedal
7. agiiity
7. furry
timid
8. vegetation
8. plains
meat
9. bulbous
10. paws
small
toughskin
Figure 7: Visualization of randomly sampled instances (AwA2 validation set) and the most important concepts associated with their respective CG attribution (top 10 concept, 60 in total).
J.2 CALTECH-UCSD BIRDS-200-201 (CUB)
Data preprocessing Similar to the AWA2 experiments, the input images are augmented by random color jittering, horizontal flipping, and resizing, then cropped the default input resolution of the
21

---

Published as a conference paper at ICLR 2023
Table 9: Finetune layers for architectures
Architecture Inception v3
Finetuned layers
fct, Mixed_7ct, Mixed_7bt, Mixed.7a+, Mixed.6et, Mixed,dd,, Mixed fct. 2ayer4.2t, layer4.1f, 18yer4.0t, 1ayer3. layer3. 4+, 1ayer3.3* classifier.6t
Resnet50
VGGT6
classifier.0 3+, features. 34+,
classifier.0t, features. features.14* features.
model architecture (299 for Inception v3, 224 for others) during training. During evaluation, the input images are resized and center cropped to the input resolution. followed (Koh et al. (020) procedure of removing attributes with insufficient data samples. A total of 112 attributes remains for conducting interpretation. The class attribute labels are assigned as instance labels, i.c., instances from the same class share the same attribute labels.
Training We trained the target model f with three different CNN architectures: Inception v3, Resnet50, and VGG16 (with batch normalization), each with ImageNet pretrained weights (exclud- ing the final fully connected layer). We searched for hyperparameters over a range of learning rates (0.01, 0.001), learning rate schedules (decaying by 0.1 for every 15, 20, 25 epochs until reaching 0.0001), and weight decay (0.0004, 0.00004). We optimized with the SGD optimizer. We trained the model for maximum of 200 epochs and early stopped if the validation accuracy had not im- proved for 50 consecutive epochs. The validation accuracy of the trained target model is 0.797, 0.764, and 0.782 for the three models, respectively.
We trained the concept model by finetuning different parts of f (freezing different layer model weights). The different layers we started to finetuned from for each model architecture is listed in Table 9 The plus sign in the table represents all layers after the specified layer are all finetumed while all layers prior to the specified layer have their weights kept frozen. We reweighted the loss for positive and negative class to balance class proportions. We searched for the hyperparameters and trained the model same as the target model f.
Evaluation Evaluation is conducted on the testing set. CG and CAV are evaluated in the layer prior to finetuning. We evaluated the recalls for & = 30, 40, 50 and generally the recall trend consistent for all Ks. These thresholds are chosen since the number of concepts with positive labels for each instance is in the range of 30 to 40.
J.3 MYOCARDIAL INFARCTION COMPLICATIONS
22

---

Published as a conference paper at ICLR 2023
Table 10: Mortality risk attribution with respect to myocardial infarction complications and com-
parison with existing medical literature
Complication
CG
TCAV
Excerpted mortality risk description from medical literature Recurrent infarction causes the most deaths following myocar- dial infarction with left ventricular dysfunction. (Om et al.,
Relapse of MI
3.47
2.55
2005)
Chronic
3.27
-1.26
The mortality rate in group of patients with class III and IV heart failure is about 40% per year, and half of the deaths are sudden. (Bigger 1987)
heart failure
Atrial
2.29
1.11
AF increases the risk of death by I.5-fold in men and 1.9-fold women. (Benjamin al. 1998)
fibrillation
Myocardial
1.62
6.52
Myocardial rupture is relatively rare and usually fatal compli-
rupture
cation myocardial infarction (MI). (Shamshad et al.) 2010) Pulmonary oedema in patients with acute MI hospitaliize
Pulmonary edema
1.51
2.21
coronary care units was reported to be associated with a high
mortality 38-57%. (Roguin al. (2000)
Ventricular fibrillation
0.91
1,90
Patients developing VF in the setting of acute MI are at higher
risk in-hospital mortality. (Bougouin et al. 2014)
Third-degree AV block
0.69
1.71
In patients with CHB complicating STEMI, there was change in risk-adjusted in-hospital mortality during the study period. (Harikrishnan et al.) 2015)
Ventricular tachycardia
0.51
-0.19
Ventricular tachycardia is most commonly associated with is- chemic heart disease or other forms of structural heart disease that are associated with risk of sudden death. (Koplan & Stevenson 2209))
Dressler syndrome
0.32
-2.85
The prognosis for patients with DS is typically considered to be quite good. (Leib et al.) (017)
Supraventricular
0.24
-0.36
Although SVT is usually not Tife-threatening, many patients
tachycardia
suffer recurrent symptoms that have a major impact on their quality life. (Medi et al.) 20009)
fost-infarction
-1.40
-2.85
After adjustment, angina only weakly associated with car- diovascular death, myocardial infarction, or stroke, but signif- icantly associated with heart failure, cardiovascular hospital-
angina
ization, and coronary revascularization. [Eisen et al.) 2016)
23


Reviews:
[{"review_id": "kTgdWT2s8lt", "reviewer": "Reviewer_txu2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a novel concept-based explanation method: Concept Gradient (CG). In contrast to previous work, they use non-linear concept functions and show how the standard Concept Activation Vector (CAV) approach can be generalized to this setting.\nExperimental results show superior performance of CG over CAV.", "summary_of_the_review": "Overall I think the paper presents a nice contribution and I would lean toward publication at ICLR.", "strength_and_weaknesses": "### Strengths\nThe idea is convincing and the theoretical motivation seems sound. \nI do appreciate the toy examples that highlight where CAV might fail although it does not seem relevant in practice.\nExperiments span different data sets and classifiers and a comparison to CAV shows significantly better performance of the proposed CG method, especially for global explanations.\n\n### Weaknesses\nThe local concept recalls in figure 1 do not look that convincing for CAV or CG. Table 4 could include CAV values for comparison. \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fHxcTPePQ6i", "reviewer": "Reviewer_zoZR", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work extends the CAV approach for concept-based explainability to\ndeal with concepts that cannot be expressed as linear combinations of\ninput (or latent) features. \n", "summary_of_the_review": "Overall, I think this is a reasonable contribution to the XAI field,\nfixing a clear deficiency of existing solutions for concept-based\npost-hoc explanations.  My main concern is on the limitations of\nconcept-based post-hoc explanation itself. However I do not think this\nis a reason to disregard this research line and this work in\nparticular.  Rather, I would like the authors to discuss in more\ndetail the pros and cons of post-hoc interpretation with respect to\nconcept-based models. The authors did briefly mention this in the\nrelated work section, but I think a deeper analysis is needed. For\ninstance, the suboptimal performance of concept gradient computed\njointly over concepts hints at the lack of orthogonality /\ndisentanglement between \"concepts\" learned by the network, something\nthat the research on concept-based models is actively addressing.", "strength_and_weaknesses": "PROs\n\n- The work addresses a clear limitation of the CAV approach, that severely affects its practical applicability.\n\n- The derivation is sound\n\n_ The experimental evaluation is extensive and convincingly supports the advantage of the approach wrt to CAV.\n\nCONs\n\n- A critique one could make to this line of research is that if\n  concepts are available, one should aim at directly enforcing models\n  to use them (the concept-based learning models) rather than testing\n  their alignment post-hoc. \n\n- The complete version of the approach, jointly computing concept\n  gradient for all concepts, seems to provide worse results in terms\n  of alignment with concept importance as assessed by humans. To me\n  this again hints at a limitation of this research line (see the\n  summary of the review).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IQnhmgHLGRu", "reviewer": "Reviewer_pw8j", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper propose a method for explaining intermediate activations of a neural network using concepts. In a nutshell, f(x) and g(x) are the networks that take activations x from an intermediate layer as input and respectively predict class and concept. The relevance of a concept c to a class y is measured using the proposed Concept Gradient (CG) score, which is based on the gradient of f and g with respect to x. CG score is calculated as normalized dot product of the two gradients. The proposed method is compared with TCAV (Kim 2018) on two datasets (CUB and AWA). The proposed method is also applied to a third dataset but the results are not compared with TCAV or any other method. ", "summary_of_the_review": "I have some questions that I hope the authors can clarify.", "strength_and_weaknesses": "+The method is simple. \n+The provided formulations about gradient of class with respect to concept are insightful. Although, eventually it all reduces to a simple dot product of two gradient vectors. \n\n\n-Why finetuning the same network for g? Specially for deeper layers, g becomes very shallow and simple. Why should one not use a more complex model for g?\n\n-Concept accuracy in Table 1 and 2 are very close suggesting that a linear model is not as bad as one may think (0.72 vs 0.79 in Res50). Which layer is used for generating the results in Table 1 and 2? \n\n-TCAV does not simply use directional derivative which is analogous to the proposed local CG. Instead, they define TCAV_Q as “the fraction of k-class inputs whose l-layer activation vector was positively influenced by concept C”. In contrast the proposed global CG is defined based on sum of sign(local CG). What is the advantage of this definition over the definition of TCAV_Q score?  \n\n-TCAV performs a statistical significance testing and uses only the results that pass this test (Kim 2018, section 3.5). Did you do this step? \n\n-When comparing to TCAV, did you rank the concepts according to TCAV_Q score from “relative” CAV (Kim 2018, section 3.6) which is the recommended way of using the method when assessing relative importance of multiple concepts? Please clarify these details in the paper.\n\n-Simply using gradient to measure feature importance is widely admitted to be misleading and that’s why many gradient-based methods have been trying to generating more faithful saliency maps (Adebayo 2018). CG score is purely based on gradient. Does CG suffer from similar issues? \n See Adebayo , et al Sanity Checks for Saliency Maps, Neurips 2018.\n\n-In the experiments on the third dataset about mortality risk, have you had the results reviewed by a medical expert? Since I, and likely other reviewers, are unable to comment on the validity of these results, I was hoping you could provide TCAV_Q scores too. Having both CG and TCAV_Q scores side by side would enable us to at least see if there are any disagreement between the two method and if CG has any advantage over TCAV_Q on this dataset.\n\n- CG is defined to measure how small perturbations on each concept affect the label prediction. CG essentially is dot product of the two gradient vector (after a normalization which is shown not to be important). The dot product is a symmetric operation. So it also measures how small perturbations on a predicted class affect the concept prediction. Right? \n\nTypo: Pp 15 Appendix D para. 2 Fig ??\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "96t3UOix9_G", "reviewer": "Reviewer_UPnS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a concept-based explanation called Concept Gradient (CG), which is a non-linear extension of Concept Activation Vector (CAV).\nThe idea of CG is as follows.\nSuppose the outcome of the model $f$ is given as $y = f(x)$.\nSimilarity, there is another model $g$ predicting the concepts $c = g(x)$.\nThe authors then proposed to use the sensitivity of the outcome with respect to the concept change $\\frac{\\partial y}{\\partial c}$ as the contribution of the concept $c$ to the model's outcome $y$ for the input $x$.\nTo compute $\\frac{\\partial y}{\\partial c}$, the authors introduced chain rule $\\frac{\\partial y}{\\partial c} = \\frac{\\partial x}{\\partial c} \\frac{\\partial y}{\\partial x} \\approx \\left(\\frac{\\partial c}{\\partial x}\\right)^\\dagger \\frac{\\partial y}{\\partial x}$ where $\\dagger$ denotes pseudo-inverse.\nIn particular, the authors reported that treating each individual concept separately induced better results.\nIn such a case, we have $\\frac{\\partial y}{\\partial c} \\approx \\frac{1}{\\left\\|\\frac{\\partial c}{\\partial x}\\right\\|^2} \\frac{\\partial c}{\\partial x}^\\top \\frac{\\partial y}{\\partial x}$.\nIn the experiments, the authors reported that CG is better at identifying true relevant concepts compared to CAV.", "summary_of_the_review": "I think the proposed score based on the inner product $\\frac{\\partial c}{\\partial x}^\\top \\frac{\\partial y}{\\partial x}$ is novel and will be effective in practice.\nHowever, its justification is questionable.\nThere may be some other ways to justify the score.", "strength_and_weaknesses": "### Strength\nThe proposed score $\\frac{1}{\\left\\|\\frac{\\partial c}{\\partial x}\\right\\|^2} \\frac{\\partial c}{\\partial x}^\\top \\frac{\\partial y}{\\partial x}$ seems to be a reasonable one for quantifying the correlation between the model's outcome and the concept.\nEssentially, $\\frac{\\partial c}{\\partial x}^\\top \\frac{\\partial y}{\\partial x}$ is away from zero when both $y = f(x)$ and $c = g(x)$ changes simultaneously for moving $x \\to x + \\Delta x$ for a certain choice of $\\Delta x$.\nThis is intuitive in a sense that moving $x$ to a direction changing $y$ also changes the strength of the concept $c$ simultaneously, indicating that $y$ and $c$ are correlated.\n\n### Weakness\nThe essential weakness of the paper is on the justification of the score $\\frac{1}{\\left\\|\\frac{\\partial c}{\\partial x}\\right\\|^2} \\frac{\\partial c}{\\partial x}^\\top \\frac{\\partial y}{\\partial x}$.\nIn the paper, the authors first adopted the sensitivity $\\frac{\\partial y}{\\partial c}$ as the ideal ground truth, and then derived its estimate $\\frac{1}{\\left\\|\\frac{\\partial c}{\\partial x}\\right\\|^2} \\frac{\\partial c}{\\partial x}^\\top \\frac{\\partial y}{\\partial x}$ by using pseudo-inverse.\nThis justification is questionable in two ways.\nFirst, $\\frac{\\partial y}{\\partial c}$ is not well-defined in general unless $g$ has inverse mapping.\nSecond, the choice of pseudo-inverse has a large degree of freedom.\nIn the paper, the authors adopted the pseudo-inverse with the minimum norm, i.e., Moore-Penrose inverse.\nIt is not very clear to me whether these questionable steps are required to justify the score $\\frac{1}{\\left\\|\\frac{\\partial c}{\\partial x}\\right\\|^2} \\frac{\\partial c}{\\partial x}^\\top \\frac{\\partial y}{\\partial x}$.\nIndeed, as the authors reported in the results, treating multiple concepts together leads to inferior performance.\nThis finding implies that the form of the inner product $\\frac{\\partial c}{\\partial x}^\\top \\frac{\\partial y}{\\partial x}$ will be a more appropriate expression than $\\left(\\frac{\\partial c}{\\partial x}\\right)^\\dagger \\frac{\\partial y}{\\partial x}$ obtained from these questionable steps.\nAs I pointed out in Strength, it would be more natural to interpret the score as a (scaled) correlation between $f$ and $g$.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster