Human-interpretable model explainability on high-dimensional data
Authors: Damien de Mijolla
Christopher Frye
Markus Kunesch
John Mansir
Ilya Feige
Venue:  (2021)

Abstract: The importance of explainability in machine learning continues to grow, as both neural-network architectures and the data they model become increasingly complex. Unique challenges arise when a model's input features become high-dimensional: on one hand, principled model-agnostic approaches to explainability become too computationally expensive; on the other, more efficient explainability algorithms lack natural interpretations for general users. In this work, we introduce a framework for human-interpretable explainability on high-dimensional data, consisting of two modules. First, we apply a semantically-meaningful latent representation, both to reduce the raw dimensionality of the data, and to ensure its human interpretability. These latent features can be learnt, e.g. explicitly as disentangled representations or implicitly through image-to-image translation, or they can be based on any computable quantities the user chooses. Second, we adapt the Shapley paradigm for model-agnostic explainability to operate on these latent features. This leads to interpretable model explanations that are both theoretically controlled and computationally tractable. We benchmark our approach on synthetic data and demonstrate its effectiveness on several image-classification tasks.

Under review as a conference paper at ICLR 2021
HUMAN-INTERPRETABLE MODEL EXPLAINABILITY ON HIGH-DIMENSIONAL DATA
An Anonymous authors Paper under double-bling review
ABSTRACT
The importance of explainability in machine learning continues to grow, as both neural-network architectures and the data they model become increasingly plex. Unique challenges arise when a model's input features become high di- mensional: on one hand, principled model-agnostic approaches to explainability become too computationally expensive; on the other, more efficient explainability algorithms lack natural interpretations for general users. In this work, we in- troduce a framework for numan-interpretable explainability on nigh-dimensional data, consisting of two modules. First, we apply semantically-menningg representation, both to reduce the raw dimensionality of the data, and to ensure its human interpretability. These latent features can be learnt, e.g. explicitly as disen- tangled representations can be based on any computable quantities the user chooses. Second, we adapt the Shapley paradigm for tures. This leads to interpretable model explanations that are both theoretically- controlled and computationallyy-raataai traataabee........ We benchmark our approach syn- thetic data and demonstrate its effectiveness on several image-classificatin tasks.
latent
implicitly through image-to-image translation, or they
explainability operate these latent fea-
INTRODUCTION
The explainability of Al systems is important, both for model development and model assurance. This importance continues to rise as Al models and the data on which they are trained be- come ever more complex. Moreover, methods for Al explainability must be adapted maintain the human-interpretabiliiy of explanations in the regime of highly complex data.
Many explainability methods exist in the literature. Model-specific techniques refer the internal structure of a model in formulating explanations (Chen & Guestrin 2016 Shrikumar et al.) 2017). while model-agnostic methods are based solely on input-output relationships and treat the model as a black-box (Breiman 2001 Ribeiro et al. 2016). Model-agnostic methods offer wide applicability and, importantly, fix a common language for explanations across different model types.
The Shapley framework for model- aagnosigc explainability stands out, due to its theoretically princi- pled foundation and incorporation of interaction effects between the data's features (Shapley [953 Lundberg & Lee 2017). The Shapley framework has been used for explainability in machine learn-
ing for years (Lipovetsky & Conklin 2001 Kononenko et al. 2010 Strumbelj & Kononenko 2014
Datta al. 2016). Unfortunately, the combinatorics required to capture interaction effects make
Shapley values computationally intensive and thus ill-suited for nigh-dimensional data.
More computationaally-fffttii efficciiet methods have been developed explain model predictions on high- dimensional data. Gradient- and perturbation-basee methods measure model prediction'' 's sensitiv- ity to each of its raw input features (Selvaraju et al.) 2020 /hou al. 2016 Zintgraf et al.) 2017).
Other methods estimate the mutual information between input features and the model's prediction (Chnn et al.) 22118| Schulz et al. 2020), or generate counterfactual feature values that change the model's 's prediction (Chang et al. 2019 Goyal et al. 2019 Wang & Vasconcelos 2020). See Fig. for explanations produced by several of these methods (with details given in Sec. 3.5
When intricately understood by the practitioner, these methods for model explainability can be use- ful, e.g. for model development. However, many alternative methods exist to achieve broadly the same goal (i.e. monitor how outputs change inputs vary) with alternative design choices that

---

Under review as a conference paper at ICLR 2021
Saliency Map
Gredients
Ablation
Occlusion
((x)
semantic representation
apply explainabiity to (((())
Pixel-based explanations of model trained to predict the attractiveness label CelebA
Figure 1:
Figure 2: Our proposed framework for semantic explainability.
make their explanations uncomparable general user: e-g. the distinct explanations in Fig. de- scribe the same model prediction. Ideally, set of axioms (agreed upon or debated) would constrain space of explanations, thus leading to framework of curated methods that the user can choose from based on which axioms are relevant to the application.
further challenge on nigh-dimensional data the sheer complexity of an explanation: in the methods described above, explanations have the same dimensionality as the data itself. Moreover, the importance of raw input features (e.g. pixels) are not individually meaningful to the user. Even when structured patterns emerge in explanation (e.g. in Fig. [ this is not sufficient to answer higeer-level questions. For example, did the subject's 's protected attributes (e.g. gender, age, or eth- nicity) have any influence on the model's decision?
In this work, we develop methods for explaining predictions terms of a digestible number of semantically meaningful concepts. We provide several options for transforming from the high- dimensional raw features to lower-dimensional latent space, which allow varying levels of user control. Regardless of the method used, transformation basis is useful step, if explanations are to satisfy experts and non-experts alike.
low-dimensional human-interpretable
Once a of semantic latent features is selected, obtain quantitative information about why certain model prediction was made. Fortunately, since the set of latent features low-dimensional by construction, Shapley-based approach becomes once again viable. In this work, we develop a method to apply Shapley explainability at the level of semantic latent features, thus providing a mecoretically-controe explainability high-dimensional data. Our main contributions are:
choose an explainability algorithm to
model-agnostic foundation for
We introduce an approach to model explainability on high-dimensional data that involves encod- ing the raw input features into digestible number of semantically meaningful latent features. We develop procedure to apply Shapley explainability in this context, obtaining Shapley values that describe the siggh-imensional model's dependence on each semantic latent feature.
We demonstrate methods to extract semantic features for the explanations: Fourier transforms. disentangled representations, and image-to-image translation. We benchmark our approach on dSprites with known latent space and showcase its effectiveness in computer vision tasks such as MNIST, CIFAR-10, ImageNet, Describable Textures, and CelebA
2 SEMANTIC SHAPLEY EXPLAINABILITY
In this section, we present a simple modular framework for obtaining meaningful low-dimensional explanations of model predictions on figh-dimensional data. The framework contains modules: (i) mechanism for transforming from the high-dimensional space of raw model inputs to low- dimensional space of semantic latent features, and (ii) an algorithm for generating explanations of the model's predictions in terms of these semantic features. See Fig. 2
We will begin by describing module (ii) Sec. 2.1 where we will show how adapt Shapley explainability to latent features. Then we will describe several options for module (i) in Sec. 2.2
2.1 SHAPLEY VALUES FOR LATENT FEATURES
1953 were developed in cooperative game theory to distribute the value v(N) earned by team 1.2....,7 n) among its players. The Shapley value (i) represents
Shapley values (Shapley
2

---

Under review as a conference paper at ICLR 2021
the marginal value added by player upon joining the team, averaged over all orderings in which the team can form. In particular,
S| |S| 1)! n!
(1(
[u(S (2}) e(5)
(I)
SCNVO
where bSS) represents the value that coalition S obtains without the rest of their teammates. Shapley values are the unique attribution method satisfying 4 natural axioms (Shapley) 1953). For example, they sum to the total value earned: E; ((N) v(O), and they are symmetric if two players and j are functionally interchangeable: Φe (1) = (n()). Shapley values thus serve as a well-founded explanation of an output (the earned value) in terms of inputs (the players).
The method can be adapted to explain the output of a machine learning model by interpreting the model's input features
(....... Fn,) as the players of a game. Consider a classification task, and let fg(2) (x) be the model's predicted probability that data point æ belongs to class y. To apply Shapley explainability, one must define a value function representing the model's expected output given only a subset of the input features Tg. The most common choice is
(=) = Ey(e') [fy Uz'z)]
(2)
where p(x') is the distribution from which the data is drawn, is the complement of S, and æg Liz's represents the spliced data point with in-coalition features from x and out-of-coalition features from r². Then, inserting the value function of Eq. (2) into the definition of Eq. (I) one obtains Shapley values (,,(,))() (i) representing the portion of the prediction (x) attributable to feature 24.
The Shapley values presented above provide local explanation of the model's behaviour on an individual data point. For a global explanation, local values can be aggregated (Frye et al. 2020
t (i) = Epte.s (1)
where p(z, y) is the joint distribution from which the labelled data is drawn. This aggregation preserves the Shapley axioms and is motivated by the sum rule:
(0)
= Ep(rav) [f) - "p(a''
(4)
which can be interpreted as the model accuracy above class-balance baseline. Global Shapley values thus represent the portion of model accuracy attributable to each feature.
To adapt the Shapley framework to latent features, suppose (as in Fig. 2 that mapping exists to transform the raw model inputs x into a semantically meaningful representation 2(x). and that (approximate) inverse mapping 2 ã exists as well. Then we can obtain explanation of the model prediction (y) in terms of the latent features z(x) by applying Shapley explainability instead to the function fy( the point z(x). To be precise, we define a value function
(5) = Eppce fo f0 (2s(()) ( uzz((")))
(5)
which represents the marginalisation of (z(2)) over out-of-coalition features zg. Here 2s(x) is the in-coalition slice of 2(x), and 23(2²) is the out-of-coalition slice corresponding to a different data point. These get spliced together in latent space before transforming back model-input-space and feeding into the model. Inserting the value function of Eq. (5) into the definition of Eq. (T) produces semantic Shapley values that explain fy in terms of latent features z;.
2.2 LANDSCAPE OF SEMANTIC REPRESENTATIONS
A wide variety of methods exist to transform from the high- dighndimensional set of raw model inputs to an alternative set of features that offer semantic insight into the data being modelled. In this section, we consider several options for this semantic component of our approach to explainability.
2.2.1 FOURIER TRANSFORMS
Despite their high dimensionality, dixel-based explanations as in Fig. T manage to convey meaning to their consumers through the local structure of images. A central claim of our work, however, is
3

---

Under review as a conference paper at ICLR 2021
that such meaning remains limited and incomplete. One way to complement the location information of pixel-based explanations is with frequency- based explanations via Fourier transforms.
If an image consists of value 2; for each pixel (Cooley & Tukey 1965) consists of a value 25 for each Fourier mode & = L..... n. Each mode corresponds to set of frequencies in the horizontal and vertical directions, ranging from 0 (uniform imagery) to 1/2 (one oscillation every 2 pixels). As the inverse Fourier transform #(2(x)) = x
........ n, then its discrete Fourier transform
exists, we can obtain Shapley values for the Fourier modes of an image using Eq. (5).
We can also aggregate Fourier-based Shapley values into frequency bins to reduce the complexity of the calculation and lower the dimensionality of the explanation. Such frequency- explanations can offer insight into whether a model's decisions are based on shapes or textures (see Sec. 3.2) and whether model is robust to adversarial examples (see Sec. 3.1.
2.2.2 DISENTANGLED REPRESENTATIONS
The goal of disentangled representation learning (Bengio et al. 2013 Schmidhuber 1991) is to learn a mapping from data set's 's raw digensiinnii features lower-dimensional basis of semanti- cally meaningful factors of variation. In an image, these factors of variation might correspond to the subject's position and orientation, their emotional state, the lighting, and the setting. Disentangled representations are highly aligned with goal of achieving semantic explanations.
Different approaches offer varying levels of control over which semantic features are learnt in the representation. Unsupervised disentanglement extracts factors of variation directly from the data.
Such methods are often based on variational inference (Higgins et al., 2017: Burgess et al.) 2018
Kim & Mnih 2018 Chen et al., (0186) though other approaches exist (Chen al.] 2016)
and often seek a factorised latent representation. Supervised disentanglement (Schmidhuber 1991
Lample al. 2017) allows the practitioner to specify a subset of the sought-after semantic features by labelling them - in full or in part (Locatello et al. 2020)- - in the data. Such methods then involve learning representation that factorises the specified and unspecified factors of variation.
Disentangled representations based on variational autoencoders (Kingma & Welling 2014) include an encoder xxx) and decoder z(z), thus fitting neatly into our framework (Fig. 2 for semantic explainability. The value function of Eq. (5) then leads to Shapley values that explain model's predictions terms of the disentangled factors of variation underlying the data. We will demonstrate this for unsupervised (Sec. 3.3 and supervised (Sec. 3.4) disentanglement with experiments.
2.2.3 IMAGE-TO-IMAGE TRANSLATION
In image-to-image translation (Tsola et al. 2017), one is not interested in directly extracting semantic factors of variation, but instead in transforming images by selectively modifying underlying seman- tic features. Generative adversarial methods can accomplish this goal without passing through an explicit compressed representation (Zhu et al. 2017). Other adversarial methods allow the user to selectively perturb the semantic attributes of an image (e.g. hair colour or gender) for attributes that are labelled in the data set (Choi et 2018 He al. 2019 Liu et al. 2019a).
Image-to-image translation methods can straightforwardly be incorporated into our framework for semantic explainability. To do so, one inserts the result of the translation + a ((=s(1) U =g(x')), which corresponds to the modification of semantic attributes 2g (z)) + 2g (æ'). into the value func- We demonstrate this in Sec. 3.5 below.
tion of Eq. 55)
RESULTS
Here demonstrate the practical utility of our approach on variety of data sets and for diverse set of semantic representations. We use pre-trained models for the semantic representations to show that our method can be applied with existing tools. See App. A for full experimental details.
3.1 C1FAR-10
We begin by applying our explainability framework using the Fourier transform as the semantic representation. We do this on CIFAR-10 (Krizhevsky al. 2009) and investigate whether sensitivity
4

---

Under review as a conference paper at ICLR 2021
Natural data
Non-robust model
Robust model
1D
Comparison
Adversarial data
Local Shapley value
Ron-teuu ueuur nose home
Neviotun rodel
Global Shapley velue
Ree more
Sdbelt mede
Global
Shapley values in Fourier space
||
frequency
frequencyy
||
Figure 3: Fourier-space model explanations on CIFAR-I0 shedding light on adversarial sensitivity.
to adversarial examples (Szegedy et al. [0014] Goodfellow al., 2015 Nguyen et al. 2015) is linked
to dependence on high-frequency (i.e. small length-scaler fluctuations. We considered two classifiers on CIFAR-I0 10: robust model (Madry et al. 2018 [Egstrom et al. 2019) trained to be insensitive to adversarial perturbations, and a non-robust model (He et al. 2016 trained naturally. We computed
semantic Shapley values according Sec. 2.2.1
The first row of Fig. 3 shows the semantic explanations that result for a single image. The Shapley values in Fourier space show that, this particular image, the non-robust model is sensitive to high-frequency features (less detectable by the human eye), while the robust prediction is based exclusively low-frequency information near the Fourier-space origin (large-scale structure). The 1-dimensional comparison, in which frequency modes were binned according to Euclidean norm, allows for quantitative comparison that confirms this effect.
We also computed the adversarial perturbation of the local image in Fig. 3 using projected gradient descent (Nocedal& & Wright, 2006) and 8/255 (separately for each model). Such adversarial perturbations do not significantly alter the image to the human eye, and the robust model's predic- tion accordingly remains unchanged; however, the non-robust model's prediction is perturbed from "horse" to "ship". Fig. 3 explains the model decisions these adversarially perturbed images, showing that the non-robust model's mistake is due to high-frequency features.
The second row of Fig. shows global semantic explanations for these models, which correspond to aggregating local explanations across the data set. We see that the trends found above for one particular image hold in general throughout the data. Our framework for semantic explainability thus leads to the interesting result that adversarially robust classifiers are less sensitive to high- giggffreyeency information. See App. A.2 for similar results ImageNet (Deng al. 2009
3.2 DESCRIBABLE TEXTURES
Here we apply our framework again using the Fourier transform to explain whether model's predictions are based on shapes or textures. Shapes tend to correspond to extended objects and thus sit at the lower end of the frequency spectrum. Textures correspond to small-scale patterns and thus occupy the higher end of the frequency spectrum, often with distinctive peaks that represent periodic structure. We explore this question by explaining ResNet-50 (He et al. 2016) trained on the Describable Textures Dataset (Cimpoi et al. 2014).
Fig. showw randomly drawn "banded" and "woven" images from the data set, well as their pixel- based explanations computed with integrated gradients (Sundararajan et al.) 2017). At the level of pixels, it is difficult judge what qualities (e.g. colours, shapes, textures) drive the model's pre- diction. However, the frequency- based explanations in Fig, 4show clear peaks at high frequencies corresponding to regular patterns in each image. See App. A.3 A.3 for additional examples.
5

---

Under review as a conference paper at ICLR 2021
"Banded"
Pixel explanation
Semantic Shapley values
"Woven"
Pixal explanation
Semantic Shapley values
banded
inugo
02 33 11
frequency
|| frequency
Figure 4: Frequency- explanations on Describable Textures, showing sensitivity to periodicity.
3.3 DSPRITES
Here we apply our explainability framework using unsupervised disentangled representations to ex- tract the semantic features. We do this on dSprites (Matthey et al. 2017), synthetic images of sprites (see Fig. 5ap generated from known latent space with 5 dimensions: shape, scale, orientation, and horizontal and vertical positions. This experiment serves benchmark of our approach.
In this experiment, we explain a rules- model that classifies sprites according to the number of white pixels in the top half of the image (with 6 classes corresponding to 6 latent scales). We articulate the explanation in terms of the unsupervised disentangled representation (cf. Sec. 2.2.2)
of a 8-TCVAE (Chen et al. 2018b), using a publicly available model (Dubois) 2019
Fig. Sa shows semantic Shapley values for this model. Globally, the model relies the sprite's vertical position and scale, while generally ignoring its shape, orientation, and horizontal position. model we laid out above. Locally, the model classifies the sprite as = 0, as it has no white pixels in the top half of the image. The vertical position of the sprite is the main driver of this decision. In fact, the sprite's scale (maximum) would tend to indicate a different class, so the scale receives a negative local Shapley value in this case.
This is indeed consistent with the
This experiment validates our framework for semantic explainability, the modelling task and factors variation in the data are fully understood and consistent with our results. Moreover, this example showcases an explanation that differentiates between shape and scale semantic features that cannot be distinguished in a pixel-based explanation.
3.4 MNIST
Here we apply our framework again using disentangled representations to binary classifier on MNIST (LeCun & Cortes) 2010). In particular, we explain a rules-based model that detects the presence of geometric hole, which exists when all black pixels are not contiguous. We use the unsupervised disentangled representation a Join pre-trained on MNIST (Dupont (2018), which accommodates combination of continuous (e.g. handwriting) and discrete (e.g. digit value) latent dimensions. Supervised methods also exist for disentangling the digit value the latent representation (Makhzani al. 2016).
Fig. 5b shows semantic Shapley values for the rules- model. The global values show that the digit value has the most bearing the model's general behaviour; indeed, some digits (0, 6, 9) almost always contain a geometric hole while others (1, 3, 5, 7) almost always do not. The global values also show sensitivity to writing style (e.g. 4) and stroke thickness (which can close hole) as expected. The local values in Fig. 5b roughly in line with the global trends.
This example demonstrates that latent disentanglement is powerful tool for understanding model decisions at semantic level. These results also highlight that explanations are only as good as the chosen semantic representation. One can see the latent traversals of Fig. 15b in this representa- tion, semantic concepts are not perfectly disentangled: stroke thickness and hole size are entangled, and writing style mixes 7's and 9's. This can lead to a small corruption of the explanation.
3.5 CELEBA
Here we apply our explainability framework using image-to-in -o-gge translation as the implicit se- mantic representation. We do this CelebA (Liu et al. 2015) demonstrate, real world data,

---

Under review as a conference paper at ICLR 2021
Glsbal roplaration Local oxplanzion yeb
Gobe sradion Local exclanatinnal
0.5
on
9
6.3
Shapley
Shapley
hortbrrtall poston
ortentation
shape
oniontation
winng
9
9
9
9
7
0
9
9
9
9
9
/
Latent traversale
9
9
9
9
9
2
9
9
9
9
9
3
9
9
9
9
9
:
9
9
8
9
9
9
(b)
Figure 5: Model explanations in terms of disentangled latent features on (a) dSprites and (b) MNIST.
that our method elucidates patterns model behaviour that are missed by other methods. In partic- ular, we train a CNN to predict the labelled attractiveness of individuals in the data. We choose this admittedly banal label because influenced by a variety other higher level attributes, including sensitive characteristics (e.g. gender, age, skin tone) that should not influence fair model predictions. Pixel-based explanations (e.g. Fig. D provide no insight into these important issues.
We explain the model's predictions in terms of other semantic attributes also labelled in the data. We do so according to Sec. [.2.3] using the implicit representation of an STGAN (Liu et al. (20 pre-trained on these other attributes (Liu et al. 2019b). Fig. 6 shows the resulting semantic Shapley values explaining the model's prediction y for an individual in the data. The "unlabelled" bar represents remaining factors of variation not captured by labelled attributes.
Globally, the model makes significant use of age, eyeglasses, and blond hair. This reflects stereotyp- ical western-cultural conceptions of beauty, not surprising in a database of celebrities. The attribute with the largest influence is gender, result of bias in the data set: women are more than twice as likely to be labelled attractive men. Locally, the subject's gender (male) thus increases the ten- dency for the model to predict = 0. The subject's age (young) receives a negative local Shapley value, as this correlates instead with y (opposite the model's prediction) in the data. Hair colour, smile, and eyebrows also played role in the model prediction for this individual.
Interestingly, explanation captures dependence on features barely perceptible the human eye. For example, while traversals of the "young" attribute are hardly noticeable for this individual, this feature significantly impacts the model prediction. Model evaluations on traversals of this feature confirm that this is genuine pattern exploited by the model and not an aberration in the explanation. Figs. 9 [ in App. [A.6] provide explicit comparisons between Shapley explanations and model evaluations over a latent-space grid, as well as additional validation of our CelebA experiment.
4 RELATED WORK
In this work, we have focused on explaining model predictions terms of semantic latent fea- tures rather than the model's 's raw inputs. In related work, natural language processing is leveraged to produce interpretable model explanations textually (Hendricks et al.) 20]8a]b). Other methods (Kim al. 2018 Zhou al. 2018) learn to associate patterns of neuron activations with seman-
7

---

Under review as a conference paper at ICLR 2021
Global explanation
0.20
Local explanation of y=0 on:
Shapley value
0.10
0.05
-0.05
mustache
beard
hair
not black
bar
bushy
eyebrows
rot bald
moune alighely 6p01
hair not
hair
not bloni
no
oyeglasses
young
inrabbelle
not
Latent traversals
Figure 6: Semantic explanations of model predicting the attractiveness label in CelebA
tic concepts. However, each of these requires large sets of annotated data for training barrier to widespread application - whereas we offer both unsupervised and analytic options for seman- tic explainability in our approach. Orthogonal efforts exist train high-capacity models that intrinsically interpretable (Alvarez-Melis & Jaakkola, 2018 Chen et al. 2019).
Other related works leverage generative models to produce explanations. Several methods generate counterfactual model inputs (e.g. images) with the features that led to the model prediction accen- tuated (Singla al., 2020 Samangouei al., [018 Arrieta & Ser 2020). In contrast our work, such explanations are pixel-based Techniques developed by Liu et al. (2019c) generate counterfac- tual images that cross class boundaries while remaining nearby the original image. When employed using a disentangled generative model, such methods can assign importance to latent features; how- ever, they aim to minimise an ad-hoc pixel-based distance metric, whereas semantic latent features generally control large-scale changes in an image.
recent workshop (Singh et al. 2 showed that a change of basis model explanations can offer insights in cosmology applications. Complementary to this work, we develop this idea in general, offer several alternatives for the basis change, and benchmark with extensive experiments.
Recently released work-in progress bust models on the frequency modes of an image's ' Fourier transform. While taking a somewhat dif- ferent approach (e.g. explaining based on "Occluded Frequencies") the study finds results generally consistent with ours in Sec. 3.1 adversarial sensitivity primarily a high-frequency phenomenon.
worr-in-prooreess (Wang et al. 2020) studies the dependence of adversarially
CONCLUSION
In this work, we introduced an approach to model explainability on high- nigh-dimensional data, in which explanations are articulated in terms of digestible set of semantic latent features. We adapted the Shapley paradigm to this task, in order to attribute model's 's prediction to the latent fea- tures underlying its input data. These two developments form principled, flexible framework for interreitterreetbale explainability on complex models. To demonstrate its flexibility, we high- lighted Fourier transforms, latent disentanglement, and image-to-image translation as options for the semantic representation that offer varying levels of user control. We benchmarked our method on synthetic data, where the underlying latent features controlled, and demonstrated its effectiveness in an extensive set of experiments using off thels pretrained models. We hope this framework will find wide applicability and offer practitioners new way to probe their models.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
D. Alvarez-Melis and T. S. Jaakkola. Towards robust interpretability with self-explaining neural networks. In Advances Neural Information Processing Systems, 2018.
A. Barredo Arrieta and J. Del Ser. Plausible counterfactuals: Auditing deep learning classifiers with realistic adversarial examples, 2020. [arXiv: 2003. 11323].
Y. Bengio, A. C. Courville, and Vincent. Representation learning: review and new perspectives. IEEE Transactions Pattern Analysis and Machine Intelligence, 2013.
L. Breiman. Random forests. Machine Learning, 2001.
C. P. Burgess, Higgins, A. Pal, L. Matthey, N. Watters, G. Desjardins, and A. Lerchner. Under- standing disentangling in 3-vae, 2018. [arXiv: 1804. 03599).
C. Chang, E. Creager, A. Goldenberg, and D. Duvenaud. Explaining image classifiers by counter- factual generation. In International Conference on Learning Representations, 2019.
C. Chen, O. Li, D. Tao, Barnett, C. Rudin, and J. Su. This looks like that: Deep learning for interpretable image recognition. In Advances in Neural Information Processing Systems, 2019.
J. Chen, L. Song, M. Wainwright, and M. Jordan. Learning to explain: An information-theoreti perspective on model interpretation. In International Conference on Machine Learning, 2018 18a.
T. Chen and C. Guestrin. Xgboost: A scalable tree boosting system. In International Conference Knowledge Discovery and Data Mining, 2016.
T. Q. Chen, X. Li, R. B. Grosse. and D. Duvenaud. Isolating sources of disentanglement in varia- tional autoencoders. In Advances Neural Information Processing Systems, 201 8b.
X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems, 2016.
Y. Choi, M. Choi, M. Kim, J. Ha, S. Kim, and J. Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. Conference on Computer Vision and Pattern Recognition, 2018.
M. Cimpoi, Maji, I. Kokkinos, S. Mohamed, and Vedaldi. Describing textures in the wild. In Conference Computer Vision and Puttern Recognition, 2014.
Cooley and J. Tukey, An algorithm for the machine calculation of complex Fourier series. Math- ematics of Computation, 1965.
A. Datta, S. Sen, and Y. Zick. Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems. In JEEE Symposium on Security and Privacy, 2016.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Lcrge-Scale Hierarchical Image Database. Conference on Computer Vision and Pattern Recognition, 2009.
Y. Dubois. Disentangling VAEs, 2019. [github. com. hetrr/YannDDubss /wwssniiinnnnnnnnnnn -va).
E. Dupont. Learning disentangled joint continuous and discrete representations. In Advances in Neural Information Processing Systems, 2018.
I. Engstrom, A. Ilyas, S. Santurkar, and D. Tsipras.
Robustness, 2019.
[github. com Catiuu..coo//arrylaaaa Wwwaaaaalrrrbusttaaaaaaaaaaaaaee
C. Frye, D. de Mijolla, L. Cowton, M. Stanley, and I. Feige. Shapley- explainability on the data manifold, 2020. [arXiv:2006. 01272].
J. Goodfellow, Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. In International Conference Learning Representations, 2015.

---

Under review as a conference paper at ICLR 2021
Y. Goyal, Z. Wu, J. Ernst, D. Batra, D. Parikh, and S. Lee. Counterfactual visual explanations. In International Conference on Machine Learning, 2019.
K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. Conference on Computer Vision and Pattern Recognition, 2016.
Z. He, W. Zuo, M. Kan, Shan, and X. Chen. Attgan: Facial attribute editing by only changing what you want. IEEE Transactions on Image Processing, 2019.
L. A. Hendricks, R. Hu, T. Darrell, and Z. Akata. Grounding visual explanations. In European Conference on Computer Vision, 2018a.
L. A. Hendricks, R. Hu, Darrell, and Akata. Generating counterfactual explanations with natural language, 201 8b. [arXiv: 6. 09809].
I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. 3-vae: Learning basic visual concepts with constrained variational framework. In International Conference Learning Representations, 2017.
P. Isola, Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. In Conference on Computer Vision and Pattern Recognition, 2017.
B. Kim, M. Wattenberg, J. Gilmer, C. J. Cai, J. Wexler, F. B. Vićgas, and R. Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors. In International Conference on Machine Learning, 2018
H. Kim and A. Mnih. Disentangling by factorising. In International Conference Muchine Learn- ing, 2018
D. P. Kingma and M. Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
I. Kononenko et al. An efficient explanation of individual classifications using game theory. Journal of Machine Learning Research, 2010.
A. Krizhevsky, V. Nair, and G. Hinton. Canadian Institute For Advanced Research, 2009. [cs. ccs.toronto.o wwtteedti/eeeeeeeeeeeeeeaeaaaett http:/iiaaaraaeeeee html].
G. Lample, N. Zeghidour, Usunier, A. Bordes, L. Denoyer, and M. Ranzato. Fader networks: Ma- nipulating images by sliding attributes. In Advances in Neural Information Processing Systems, 2017.
Y. LeCun and Cortes. MNIST database, 2010. [yann. lecun. hetm//xxbbmmmiiste
Lipovetsky and M. Conklin. Analysis regression in game theory approach. Applied Stochastic Models in Business and Industry, 2001.
M. Liu, Y. Ding, M. Xia, X. Liu, E. Ding, W. Zuo, and S. Wen. STGAN: A unified selective transfer network for arbitrary image attribute editing. In Conference on Computer Vision and Pattern Recognition, 2019a.
M. Liu, Y. Ding, M. Xia, X. Liu, E. Ding, W. Zuo, and S. Wen. STGAN, 2019b. [github. com/ honm//mmmi//TTAN
S. Liu, B. Kailkhura, D. Loveland, and Y. Han. Generative counterfactual introspection for explain- able deep learning. Global Conference on Signal and Information Processing. 2019c.
Z. Liu, Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In International Conference Computer Vision, 2015.
F. Locatello, B. Poole, G. Rätsch, B. Schölkopf, O. Bachem, and M. Tschannen. Weakly-supervised disentanglement without compromises, 2020. [arXiv:2002 02886].
M. Lundberg and S. Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems. 2017.
10

---

Under review as a conference paper at ICLR 2021
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
A. Makhzani, J. Shlens, N. Jaitly, and I. J. Goodfellow. Adversarial autoencoders, In International Conference on Learning Representations, 2016.
L. Matthey, Higgins D. Hassabis, and A. Lerchner. dSprites: Disentanglement testing sprites dataset, 2017. Igithub. com/ deepmind/ http//dsppssaatteeeeeeeee
A. M. Nguyen, Yosinski, and J. Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Conference Computer Vision and Pattern Recogni- tion, 2015.
J. Nocedal and S. J. Wright. Numerical Optimization. Springer, 2006.
T. Oliphant. A guide to NumPy, 2006.
M. T. Ribeiro, S. Singh, and C. Guestrin. "Why should trust you?": Explaining the predictions of any classifier. In Special Interest Group on Knowledge Discovery and Data Mining, 2016.
P. Samangouei, A. Saeedi, L. Nakagawa, and N. Silberman. ExplainGAN: Model explanation via decision boundary crossing transformations. In European Conference on Computer Vision, 2018.
J. Schmidhuber. Learning factorial codes by predictability minimization. Technical report, Univer- sity of Colorado at Boulder, 1991.
K. Schulz, L. Sixt, F. Tombari, and T. Landgraf. Restricting the flow: Information bottlenecks for attribution. In International Conference on Learning Representations, 2020.
R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-CAM: Vi- sual explanations from deep networks via gradient-based localization. International Journal of Computer Vision, 2020.
L. S. Shapley. A value for n-person games. In Contribution to the theory of games. 1953.
A. Shrikumar, P. Greenside, and A. Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning, 2017.
C. Singh, W. Ha. F. Lanusse, V. Boehm, J. Liu, and B. Yu. Transformation importance with appli- cations to cosmology. In ICLR Workshop on Fundamental Science the era of AI, 2020.
S. Singla, B. Pollack, J. Chen, and K. Batmanghelich. Explanation by progressive exaggeration. In International Conference Learning Representations, 2020.
E. Strumbelj and I. Kononenko. Explaining prediction models and individual predictions with fea- ture contributions. Knowledge and Information Systems, 2014.
M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, 2017.
C. Szegedy, W. Zaremba, I. Sutskever, Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
Wang and N. Vasconcelos. SCOUT: Self-aware discriminant counterfactual explanations, 2020. [arXiv: 2004 07769]
Z. Wang, Y. Yang. A. Shrivastava, V. Rawal, and Z. Ding. Towards frequency- based explanation for robust CNN, 2020. 2005iv:2005. 03141].
B. Zhou, A. Khosla, Å. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discrimi- native localization. Conference on Computer Vision and Pattern Recognition, 2016.
B. Zhou, Y. Sun, D. Bau, and A. Torralba. Interpretable basis decomposition for visual explanation.
In European Conference Computer Vision, 2018.

---

Under review as a conference paper at ICLR 2021
J. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In International Conference on Computer Vision, 2017.
L. M. Zintgraf, S. Cohen, Adel, and M. Welling. Visualizing deep neural network decisions: Prediction difference analysis. In International Conference on Learning Representations, 2017.
A DETAILS OF EXPERIMENTS AND ADDITIONAL RESULTS
Below we provide numerical details and supplementary results for the experiments presented in this paper. For each experiment, we describe both the semantic representation employed as well as the Shapley calculation and its statistical uncertainty.
First we mention the pixel-based explanations that appear in Figs. I and (and Fig. IT beloww. We produced these using the Captum PyTorch package [g thub. com pytorch/ captum] We refer the reader to the package documentation for additional details regarding our chosen explanation methods. Explanations using integrated gradients, occlusion, and feature ablation were calculated using a velo-valued baseline.
A.1 CIFAR-10
For our experiments CIFAR-I0 (Krizhevsky al., 2009), we explained the predictions of pre- trained robust and non-robust ResNet-50's s that are publicly available (Engstrom et al. 2019). The robust model particular was adversarially (pre-)traaiee) with respect to Eoo perturbations norm e = 8 3/255. For the semantic representation, we used the FFT (Cooley & Tukey 1965) functionality of NumPy (Oliphant 2006).
We computed local Shapley values according to Sec. 2.1 using 10k Monte-Carke samples from the validation set estimate the local Shapley value for each pixel. The colour plots Fig. displla the means of these Monte-Carlo estimations. (We treated the RGB components of each Fourier mode as a single feature zę these calculations. We similarly grouped together conpeae-cojuggtte modes, Fg and zgz, as these provide redundant information for real-valued images.) The global Shapley values were similarly computed with 10k samples.
To produce the Id-comparison plots of Fig. 3 aggregated Shapley values from the colour plots, binning Shapley values according to the &z norm of the corresponding 2d-frequencies. The shaded bands around the curves display the standard error of the mean in the Monte Carlo sampling dis- cussed above. (These uncertainty bands are very narrow.)
In the final column of Fig. + explain model predictions adversarially perturbed data. We computed these perturbations according to Engstrom et al. (2019. In particular, we performed projected gradient descent with respect to an for norm of 8/255, using step size of 0.01 and 100 iterations towards targeted, randomly-drawn incorrect class. Each (robust and non-robust) model was evaluated separate of direct adversarial attacks.
A.2 RESTRICTED IMAGENET
Here we present supplementary results on Restricted ImageNet (Engstrom et al. 2019), confirming that the trends found on CIFAR-10 also exist for higher-dimensional images. particular, we explain the predictions of pre-trained robust and non-robust ResNet-50's (Engstrom et al.) 2019); robust model was adversarially trained with lo perturbations of norm e = 4/255.
In Fig. 7 we present results analogous to those in Fig. In the first row, we show frequency- based Shapley values for each model's prediction on local image. There we see that the robust model primarily depends low-frequency information in the image. By contrast, the non-robust model sensitive to nigher-frequence features. This sensitivity leads to the false classification of the adversarially perturbed image of the primate an insect. In the second row, show that this trend persists across the data set: the global Shapley values aggregated over many images display trends similar to those just described for individual image.
For this calculation, we treated all the Fourier modes that fall into the same frequency bin as a single feature in the Shapley caleulation. This reduced the 224 224 Fourier modes down to 25
12

---

Under review as a conference paper at ICLR 2021
On natural data
On adversarial data
Honertur model primute Fobust mode: prosue
Nen-edbut model Inacct Rapul batrate
Local Shaplyy volue
Local
24
02
Global Shapley value
Rebust model
model
3.70
Global
2.5
015
211
froquency
[| froquencyy
Figure 7: Frequency- based explanations on Restricted ImageNet, providing further evidence that adversarial sensitivity is linked to high-frequency modes.
"Spiralled"
"Perforated"
"Stratified"
Local Shapley values
spirailed Image
0.10
image
8:20
stredifed Image
0.64
0.23
0.22
0.04
0.32
6/99
0.38
0.00
6.5
0.7
05
67
frequency)
||
[requeeccy
frequency
Figure 8: Additional frequency-based explanations on the Describable Textures data set.
aggregate features, thus enabling a very tractable calculation that preserves the primary information- of-interest in the explanation. All other numerical details of our experiment Restricted ImageNet are identical to those on CIFAR-10, 10, but replacing 8/255 - 4/255 for the adversarial perturbations.
A.3 DESCRIBABLE TEXTURES
For Describable Textures Cimpoi et al. 2014), we explained a ResNet-101 (He al. (016) trained to predict the 47 textures classes. Our model optimised using Adam and obtained top-1 accuracy of 53.5% and top-5 accuracy of 83.1% % on a held-out test set.
Explanations in Fig. were computed according the same procedure outlined in App. A.2 In par- ticular, 10k Monte Carlo samples were used to estimate the local Shapley value each frequency bin, and narrow uncertainty bands display the standard error of the mean in each bin. Fig. 8 pro- vides additional local explanations of the model on randomly-drawe "spiralled", "perforated" and "stratified" images. These show clear dependence specific high-frequency patterns in the images.
13

---

Under review as a conference paper at ICLR 2021
Latent traversais
na
mustache
beard
hair not black
no
bangs
skin not pale
bushy syebrows
not bald
mouth siightty open
hair
not brown
hair not blond
Bunoy
maee
Figure 9: Model dependence each labelled attribute in CelebA. Below each image in the latent traversals, we show the model's corresponding output. The model was trained to predict the "attrac- tive" label in the data. Its dependence here is consistent with the Shapley explanation in Fig. 6
A.4 DSPRITES
For our experiments on dSprites (Matthey et al., (017). we explained the output of a rules-based model. The model counts the number of white pixels in the top half of the image in order classify images into classes. The chite-pixel-counte naturally fall into clusters because the synthetic dSprites images generated according 6 underlying scales.
For the semantic representation, we employed a 3-TCVAE (Burgess et al. 2018), obtaining a pub- licly available pre-trained network (Dubois 2019). We applied a rotation matrix to the latent repre- sentation to align translationddimessions with the horizontal and vertical axes.
We computed explanations for Fig- 5a according to Sec. 2.1 Each Shapley value estimated using 10k Monte Carlo samples: each bar heights represents the mean, and each error bar displays the standard error of the mean.
A.S MNIST
For our experiments on MNIST (LeCun Cortes 2010), we explained the output of rules-based model that checks for the presence of a geometric hole in an image. Such a hole exists if there is black pixel that is not path-connected the image's 's perimeter, restricting paths that only pass through other black pixels. For example, "0" has a hole, because the black pixels at the centre are disconnected from the black pixels near the perimeter, but "1" does not. practice, we detected holes using SciPy's s binary fill ho les function.
For the semantic representation, we employed JointVAE (Dupont 2018), using the pre-trained network available at the paper's associated repository. Explanations in Fig. 55 were computed using 10k Monte Carlo samples to estimate the means and standard errors.
A.6 CELEBA
For our experiments on CelebA (Liu et al. 2015), we pre-processed the data according to the proce- dure described by Liu et al. 2019a). This entails cropping the central 170 170 region of the images and using a bicubic linear interpolation to resize them to 128 x 128. We explained the predictions of CNN trained to predict the "attractive" label, which exhibits 51 49 class balance in the data. Our CNN (with 3 convolutional and fully-connected layers) was optimised using Adam and the cross-entropy loss achieve 78. 7% accuracy held-out test set.
14

---

Under review as a conference paper at ICLR 2021
Global explanation Local explanation of y-1
Shapley value
-0.1
mustache
hair
not black"
bangs
skin not pale
eyebrows not bushy
not bald
mouth closed
hair not brown
hair not blond
no
young
male
uniabelle
Latent traversals
Figure 10: Semantic Shapley explanation of a model predicting the "attractive" label in CelebA. Additional local explanation to complement the one shown in Fig. of the main text.
For the semantic representation, we employed STGAN (Liu et al. 2019a), using the pre-trained network available at the paper's 's associated repository. This network was trained to selectively mod- ify the labelled attributes listed in Fig. 6 Our Shapley explanations also include an "unlabelled" dimension: this is meant to represent all remaining factors of variation distinguishing one image from another that are not captured by the labelled attributes. In practice, this unlabelled dimension is simply the image's ' index in the data set. To vary image along this unlabelled dimension, one simply draws other images from the data set while using the STGAN to hold their labelled attributes hair colour, age, gender, etc. fixed. Shapley values for CelebA were computed using 5k Monte Carlo samples to estimate the mean and standard error for each feature.
In Fig. 9 we provide results that validate the hierarchy of Shapley values that appears in Fig. 6 Below each image in the latent traversals of Fig. 9 we display the CNN's predicted probability that y 1. This shows that the model indeed depends strongly on the rightmost features while remaining relatively insensitive to the leftmost features. While the Shapley explanation of the CNN's prediction also takes into account feature interactions that are not shown here, the model evaluations in Fig. provide a qualitative check on the Shapley values.
Fig. provides an additional explanation of the CNN evaluated on a different image in the data set. The model predicts 1 for this individual, and this is attributed in large part to the unlabelled fac- tors of variation. The subject's gender (male) anti-correlates with the model's prediction (attractive), while his age (young) is a positive predictor of the model's 'ss behaviour.
As a further sanity check on our CclebA explanations, we present supplementary results here ex- plaining model that predicts the "mouth slightly open" label, which exhibits a 48 52 class balance in the data. Training our CNN architecture on this task resulted in 91.8% accuracy on the held-out test set. The pixel-based explanations of this classifier are shown in Fig. [I Since (in contrast to the "attractive" label) this is a visually-localiee prediction task, the pixel-based explanations are sen- sible, highlighting the region surrounding the subject's 's mouth. Furthermore, the semantic Shapley explanation of this classifier (Fig. 12 attributes the prediction primarily to the "mouth slightly open" latent feature. Since this is indeed the label that the model was trained to predict, this validates our setup for computing semantic explanations on CelebA.
15

---

Under review as a conference paper at ICLR 2021
Saiency Kep
Int. Gradients
Feat. Ablation
Occlusion
-0.2
Figure 11: Pixel-based explanations of classifier that predicts "mouth slightly open" label in CelebA
Global explanation Local explanation of y-1
Shapley value
0.3
0.0
noc baid
skin not pale
eyaglases
hair not blond
hair not brown
hair not black
beard
young
mouth aiggiit open
urtabeelled
Latent traversais
Figure 12: Semantic explanations of classifier that predicts "mouth slightly open" in CelebA
16


Reviews:
[{"review_id": "tF0nAU-Qns7", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposes an approach to generate semantic explanations for high-dimensional data. The proposed approach consists of two modules -- the first module transforms the high-dimensional raw data into lower-dimensional semantic latent space and the second module applies Shapely explainability to this lower-dimensional latent space to generate explanations in terms of semantic concepts. The approach has been applied to six different datasets.\n\n—————————————————————————————————————————————————————————————\n\n\nStrengths:\n\nS1) The paper is very well written and easy to understand.\n\nS2) The proposed approach is modular consisting of 2 modules. This allows the flexibility to replace each module with more efficient/advantageous methods as research progresses in future.\n\nS3) The paper proposes three different ways of transforming high-dimensional raw data to lower-dimensional latent space, over which Shapely explainability can be applied to generate explanations in terms of semantic concepts.\n\nS4) The paper shows results for six different datasets.\n\n—————————————————————————————————————————————————————————————\n\n\nWeaknesses:\n\nW1) The authors should compare their approach (methodologically as well as experimentally) to other concept-based explanations for high-dimensional data such as (Kim et al., 2018), (Ghorbani et al., 2019) and (Goyal et al., 2019). The related work claims that (Kim et al., 2018) requires large sets of annotated data. I disagree. (Kim et al., 2018) only requires a few images describing the concept you want to measure the importance of. This is significantly less than the number of annotations required in the image-to-image translation experiment in the paper where the complete dataset needs to be annotated. In addition, (Kim et al., 2018) allows the flexibility to consider any given semantic concept for explanation while the proposed approach is limited either to semantic concepts captured by frequency information, or to semantic concepts automatically discovered by representation learning, or to concepts annotated in the complete dataset. (Ghorbani et al., 2019) also overcomes the issue of needing annotations by discovering useful concepts from the data itself. What advantages does the proposed approach offer over these existing methods?\n\nW2) Faithfulness of the explanations with the pretrained classifier. The methods of disentangled representation and image-to-image translation require training another network to learn a lower-dimensional representation. This runs the risk of encoding some biases of its own. If we find some concerns with the explanations, we cannot infer if the concerns are with the trained classifier or the newly trained network, potentially making the explanations useless.\n\nW3) In the 2-module approach proposed in the paper, the second module can theoretically be any explainability approach for low-dimensional data. What is the reason that the authors decide to use Shapely instead of other works such as (Breiman, 2001) or (Ribeiro et al., 2016)?\n\nW4) Among the three ways of transforming the high-dimensional data to low-dimensional latent space, what criteria should be used by a user to decide which method to use? Or, in other words, what are the advantages and disadvantages of each of these methods which might make them more or less suitable for certain tasks/datasets/applications?\n\nW5) The paper uses the phrase “human-interpretable explainability”. What other type of explainability could be possible if it’s not human-interpretable? I think the paper might benefit with more precise definitions of these terms in the paper.\n\n\nReferences mentioned above which are not present in the main paper:\n\n(Ghorbani et al., 2019) Amirata Ghorbani, James Wexler, James Zou, Been Kim. Towards Automatic Concept-based Explanations. NeurIPS 2019.\n\n(Goyal et al., 2019) Yash Goyal, Amir Feder, Uri Shalit, Been Kim. Explaining Classifiers with Causal Concept Effect (CaCE). ArXiv 2019.\n\n\n\n\n—————————————————————————————————————————————————————————————— —————————————————————————————————————————————————————————————— \n\nUpdate after rebuttal: I thank the authors for their responses to all my questions. However, I believe that these answers need to be justified experimentally in order for the paper’s contributions to be significant for acceptance. In particular, I still have two major concerns. 1) the faithfulness of the proposed approach. I think that the authors’ answer that their method is less at risk to biases than other methods needs to be demonstrated with at least a simple experiment. 2) Shapely values over other methods. I think the authors need to back up their argument for using Shapely value explanations over other methods by comparing experimentally with other methods such as CaCE or even raw gradients. In addition, I think the paper would benefit a lot by including a significant discussion on the advantages and disadvantages of each of the three ways of transforming the high-dimensional data to low-dimensional latent space which might make them more or less suitable for certain tasks/datasets/applications. Because of these concerns, I am keeping my original rating.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HAuPemRQtig", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper develops a Shapley value approach to explanation that uses low-dimensional latent features to explain the original input. In high-dimensional settings, Shapley values can be computationally intractable; as such, the authors adapt the characteristic (aka value) function $v(\\cdot)$ to consider coalitions defined in latent space. This helps ensure that only feasible and plausible (i.e., semantically meaningful) perturbations are made. They consider three different latent encodings: Fourier Transforms, Disentanglement, and image-to-image translation (to isolate factors of variation).\n\nStrengths\n- The paper tackles a difficult problem in practice: applying Shapley explanations to high dimensional data. The authors do a fine job of motivating this problem and their subsequent solution. \n- The paper is quite thorough in its experimentation as it experiments with CIFAR-10, Describable Textures, dSprites, CelebA, and MNIST. \n\nWeaknesses\n- I would have strongly preferred to see a computational analysis of how much more tractable moving to latent space makes the Shapley value. It may be an exponential time reduction if z is chosen to be log(d) for example.\n-- Is there a sensible algorithm (or heuristic) for selecting the number of latent factors when encoding? Perhaps there is a tradeoff between the size of z and the explanation quality... \n\nQuestions\n- Can you comment on if the semantic space needs to be the same space that the discriminative model itself operates in? If you just train an end-to-end model, f, on (x,y) pairs, how can we be sure that the generative, latent factors captured by z are actually used by f? \n- Can you comment on the relationship between the Shapley values directly on the input data versus the Shapley values on the latent factors, if any? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Fr_NKfSzAFh", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: I think that what is presented is a promising method for human interpretation of high-dimensional models. However, the experiments feel too much like toy examples, without rigorous attempts to validate the resulting feature attributions or to compare to other ways of getting concept-level attributions.\n\n**Update**: After reading the rebuttal, I am maintaining my previous score. I believe that, given the related work reviewers have mentioned, the paper most likely requires some experimental comparisons with these other methods. The rebuttal arguments are good, but not convincing enough that I can believe in this method's superiority without seeing a comparison with, e.g., CaCE.\n\nObjective: Explain the output of high-dimensional ML models in a human-interpretable way by using Shapley values on semantically meaningful latent features.\n\nStrengths:\n* This is an extremely important problem; the shortcomings of e.g. pixel-based methods for images are important and well-noted.\n* The proposed method is a promising way to attribute to (and in some cases, learn) semantically meaningful latent features.\n* dSprites serves as a good ground truth where the generative process is known, and serves to validate some of the human-interpretable patterns found.\n\nWeaknesses: \nMy overall concern is that while the experimental results are interesting, they are not thorough and don't demonstrate the marginal value of this method relative to other possible methods in the space; for example, concept bottleneck or causal concept effect models.\n* I didn't see a reference to Koh et al's \"Concept Bottleneck Models\" (https://arxiv.org/abs/2007.04612), which do supervised learning of semantic concepts then train a classifier on top of those concepts, which is very similar/relevant work.\n* Similarly I didn't see a reference to Goyal et al's \"Explaining Classifiers with Causal Concept Effect (CaCE)\", which trains a VAE to learn a meaningful latent space and report causal effects of modifications to the latent variables.\nThe existence of this prior work sharpens some specific questions I had when reading the paper:\n* For the non-Fourier tasks, why use a VAE latent space? It's impossible to know that VAE latent dimensions (when they do look like they correspond to a concept) correspond *only* to that concept. The bar charts stating \"most importance goes to vertical position/digit identity\" seem overconfident, since that is only our guess of what the latent dimension means. In contrast, a concept bottleneck model has dimensions with clear meaning (they were trained with supervision). This is acknowledged to some degree in the text.\n* Also, for the non-Fourier tasks, why use the Shapley framework? CaCE is a very similar approach, using a VAE to get explanations in terms of an interpretable latent space ,but doesn't use Shapley. What are the pros and cons of each approach? Also, given a meaningful latent space, why not use gradient-based methods (i.e., integrated gradients) to attribute to it? \nOverall, I don't think this method needs to blow the others out of the water; there are legitimate counterpoints to be made -- for example, concept bottleneck models require concept labels, though I believe such labels are available in the CelebA experiments. But a thoughtful discussion of the relationship of this work to methods like concept bottlenecks or CaCE is essential -- and not present in the current version.\nOther weaknesses:\n* The quality of the resulting explanations is not assessed rigorously. The fact that the dSprites example picks up on vertical position and the MNIST example picks up on digit value is useful, but many interpretability papers look more exhaustively, across the dataset, at what happens to the model output when the features reported as important are at least perturbed or ablated, or if a model is retrained with the features altered. I would particularly like to see such a comparison with concept bottleneck and/or CaCE: if this method (Shapley with VAE) is the right way to do things, perhaps it will outperform the other methods at predicting how much certain concept shifts will affect model output or model retraining.\n* The Fourier examples are interesting but feel a bit like toy examples to me because they are a specific case with a convenient invertible feature map and mostly recapitulate known properties of adversarial examples (they rely on high-frequency patterns).\n* Minor comment: The \"landscape of semantic representations\" could be shortened, potentially leaving room for more comparisons to other methods/benchmarks of the attributions.", "extra_scores": {}, "extra_text": {}}, {"review_id": "AbYdGY91jZT", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a technique for interpreting the results of a neural net in human-readable terms.\n\nThe basic idea is natural and simple: (1) find a small number of coordinates for the feature space that correspond to human-interpretable concepts; (2) use Shapley values to assign credit to these coordinates.\n\nStrong points: While there is work related to this idea (which the authors cite appropriately), the overall technique is new. The authors do a good job of backing up the theoretical idea with a set of experiments that seem to produce useful results. (The adversarial example scenario and the celebrity data set use case were especially easy to understand.) The exposition is also very clear.\n\nWeak points: Shapley values are not the only way to exploit human-interpretable coordinates (although they especially are made feasible by the reduction in number of features). The paper might be even stronger if it compared other methods. However, I wouldn't reject the paper on this basis. \n\nI recommend acceptance. This is a solid paper and is likely to be well-cited.\n\nAdditional feedback / suggestions for improvement:\n\n* I found the math notation a little confusing. There are three general types of objects: (a) the different spaces where representations live (feature space vs. the interpretable space), (b) elements of those spaces, and (c) functions between those spaces. Certain symbols seemed to do double duty, for instance $\\tilde{x}$ seemed to float between these meanings. I'd recommend clear distinctions and contrasting notation for each type of object.\n\n* The authors talk about an \"approximate inverse\" $\\tilde{x}$. It's not really an inverse, maybe more like a pseudoinverse (or maybe a left or right inverse, depending on notation). Perhaps consider alternate terminology.\n\n* When $\\tilde{x}$ is differentiable, one could immediately apply other interpretability techniques besides Shapley values (for instance, take a gradient of the class score with respect to the interpretable coordinate space). Ideally, it would be nice to see how these look, so we could compare how much of the clarity of results is due to good coordinates, and how much to use of Shapley values. This would be particularly straightforward in the first Fourier transform example.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an approach to model explainability on high-dimensional data by: (1) first mapping inputs to a smaller set of intelligible latent features, and then (2) applying the Shapley method to this set of latent features. Several methods are considered for (1), and empirical results are examined across several settings.

Reviewers were mixed in their views - one reviewer was in favor of acceptance, and three were against. 

Some concerns were addressed by authors in the discussion period but remaining issues include:
concerns over the faithfulness of the approach; 
missing comparisons to other related methods such as CaCE; and 
a desire for more in-depth discussion of the pros and cons of the different methods considered for (1).
Decision: Reject