Contrastive Corpus Attribution for Explaining Representations
Authors: Chris Lin
Hugh Chen
Chanwoo Kim
Su-In Lee
Venue: ICLR 2023 poster (2023)
Keywords: explainable artificial intelligence
interpretable machine learning
feature attributions
contrastive explanations

Abstract: Despite the widespread use of unsupervised models, very few methods are designed to explain them. Most explanation methods explain a scalar model output. However, unsupervised models output representation vectors, the elements of which are not good candidates to explain because they lack semantic meaning. To bridge this gap, recent works defined a scalar explanation output: a dot product-based similarity in the representation space to the sample being explained (i.e., an explicand). Although this enabled explanations of unsupervised models, the interpretation of this approach can still be opaque because similarity to the explicand's representation may not be meaningful to humans. To address this, we propose contrastive corpus similarity, a novel and semantically meaningful scalar explanation output based on a reference corpus and a contrasting foil set of samples. We demonstrate that contrastive corpus similarity is compatible with many post-hoc feature attribution methods to generate COntrastive COrpus Attributions (COCOA) and quantitatively verify that features important to the corpus are identified. We showcase the utility of COCOA in two ways: (i) we draw insights by explaining augmentations of the same image in a contrastive learning setting (SimCLR); and (ii) we perform zero-shot object localization by explaining the similarity of image representations to jointly learned text representations (CLIP).

Published as a conference paper at ICLR 2023
CONTRASTIVE CORPUS ATTRIBUTION FOR EXPLAIN- ING REPRESENTATIONS
Chris Lin', Hugh Chen", Chanwoo Kim, Su-In Lee
Paul G. Allen School of Computer Science & Engineering
University of Washington Seattle, WA 98195, USA {clin25, hughcher, chanwkim, suinlee]8cs washington. wwaaiieeeess.wahhiggoo...c.
edu
ABSTRACT
Despite the widespread use of unsupervised models, very few methods are de- signed to explain them. Most explanation methods explain a scalar model out- put. However, unsupervised models output representation vectors, the elements of which are not good candidates to explain because they lack semantic mean- ing. To bridge this gap, recent works defined scalar explanation output: dot product-based similarity in the representation space to the sample being explained (i.e., explicand). Although this enabled explanations of unsupervised models, the interpretation of this approach can still be opaque because similarity to the explicand's representation may not be meaningful humans. To address this, we propose contrastive corpus similarity, a novel and semantically meaningful scalar explanation output based a reference corpus and contrasting foil set of sam- ples. We demonstrate that contrastive corpus similarity is compatible with many post-hoc feature attribution methods generate COntrastive COrpus Attributions (COCOA) and quantitatively verify that features important to the corpus are iden- tifie.. We showcase the utility of COCOA in two ways: we draw insights by explaining augmentations of the same image in a contrastive learning setting (SimCLR); and (ii) we perform zero-shot object localization by explaining the similarity of image representations to jointly learned text representations (CLIP).
INTRODUCTION
Machine learning models based on deep neural networks are increasingly used in a diverse set of tasks including chess (Silver et al. 2018), protein folding (Jumper al. (021), and language trans- lation (Jean et al. 2014). The majority of neural networks have many parameters, which impede humans from understanding them (Lipton 2018). To address this, many tools have been developed to understand supervised models in terms of their prediction (Lundberg & Lee 2017 Wachter et al.) 2017). In this supervised setting, the model maps features to labels (f X V), and explanations aim to understand the model's prediction of a label of interest. These explanations are interpretable, because the label of interest (c.g., mortality, an image class) is meaningful to humans (Figure
In contrast, models trained in unsupervised settings map features to representations (f X H). Existing supervised explanation methods can be applied to understand an individual element (h:) in the representation space, but such explanations are not useful to humans unless h; has natural semantic meaning. Unfortunately, the meaning of individual elements in the representation space is unknown in general. One possible solution is enforce representations have semantic meaning as in Koh et al. (2020), but this approach requires concept labels for every single training sample, which typically impractical. Another solution is enforce learned representations to be disentangled as in Tran et al. (2017) and then manually identify semantically meaningful elements to explain, but this approach is not host-hoe and requires potentially undesirable modifications the training process.
Related work. Rather than explain a single element in the representation, approaches based on explaining the representation as a whole have recently been proposed, including RELAX (Wick- strøm et al., (202I) and label-free feature importance (Crabbé & van der Schaar 2022) (Figure
"Equal contribution.

---

Published as a conference paper at ICLR 2023
(additional related work in Appendix A). These approaches both aim to identify features in the plicand (the sample to explain) that, when removed, point the altered representation away from the explicand's 's original representation.
Although RELAX and label-free feature importance successfully extend existing explanation tech- niques to unsupervised setting, they have two major limitations. First, they only consider similar- ity to the explicand's representation; however, there are a variety of other meaningful questions that can be asked by examining similarity other samples' representations. Examples include asking, "Why is my explicand similar to dog images?" or "How is my rotation augmented image similar to my original image?". Second, RELAX and label-free importance find features which increase similarity to the explicand in representation space from any direction, but in practice some of these directions may not be meaningful. Instead, just as human perception often explains by comparing against a contrastive counterpart foil) (Kahneman & Miller 1986 Lipton 1990, Miller 2019), we may wish to find features that move toward the explicand relative to an explicit "foil". As an example, RELAX and label-free importance may identify features which increase similarity to dog explicand image relative to other dog images or even cat images; however, they may also iden- tify features which increase similarity relative to noise in the representation space corresponding to unmeaningful -itro--distribution samples. In contrast, we can use foil samples to ask specific questions such as, "What features increase similarity my explicand relative to cat images?".
Contribution. (1) To address the limitations of prior works on explaining unsupervised models, introduce COntrastive COrpus Attribution (COCOA), which allows users to choose corpus and foil samples in order to ask, "What features make my explicand's representation similar to my cor- pus, but dissimilar my foil?" (Figure [t). (2) We apply COCOA to representations learned by self-supervised contrastive learning model and observe class-preserving features in image augmen- tations. (3) We perform object localization by explaining a mixed modality model with COCOA.
Motivation. Unsupervised models are prevalent and can learn effective representations for down- stream classification tasks. Notable examples include contrastive learning (Chen et al., (2020) and self-supervised learning (Grill et al. 20200. Despite their widespread use and applicability. unsu- pervised models are largely opaque. Explaining them can help researchers understand and therefore better develop and compare representation learning methods (Wickstrøm et al.] 2021] Crabbé & van der Schaar 2022). In deployment, explanations can help users better monitor and debug these
models (Bhatt al. 2020).
Moreover, COCOA is beneficial even in a supervised setting. Existing feature attribution methods only explain the classes the model has been trained to predict, so they only explain classes which are fully labeled in the training set. Instead, COCOA only requires a few class labels after the training process is complete, so it can be used more flexibly. For instance, if we train a supervised model on CIFAR-10 (Krizhevsky et al.] 2009), existing methods can only explain the ten classes the model was trained on. Instead, we can collect new samples from unseen class, and apply COCOA to representation layer of the trained model to understand this new class.
2 NOTATION
We consider an arbitrary input space X' and output space Z. Given a model f X + Z and an expli- cand xx e X to be explained, local feature attribution methods assign score to each input feature based on the feature's importance to a scalar explanation target. A model's intermediate or final out- put usually not a scalar. Hence, feature attribution methods require an explanation targer function Ts X + H that transforms model's behavior into explanation target. The subscript indi- cates that a model is considered a fixed parameter of an explanation target function, and denotes an arbitrary number of additional parameters. To make this concrete, consider the following example with a classifier feloss X + (0, 1]C, where C' is the number of classes.
Example 2.1. Given an explicand x E X, let the explanation target be the predicted probability of the explicund's predicted class. Then the explanation larget function is
Vrolaase xn (x)
pelass
max c Jelass (m=))))))))
for all x E X. where f. freass() (-) denotes the ith element of felass() Here, the explanation target function has the additional subscript x. to indicate that the explicand is a fixed parameter.
2

---

Published as a conference paper at ICLR 2023
Let R denote the set of functions that map from X' to R. Formally, local feature attribution method Φ Rd x X' - TRl takes an explanation target function 7s. and an explicand ar as inputs. The attribution method returns ( x²) E | feature importance scores. For k 1, .... (x',, let 2; denote the kth feature x and ox ("if... z²) the corresponding feature importance æi We demonstrate this definition of local feature attribution methods with the following example.
Example 2.2. Given an explicand x", the above classifier folass, and the explanation target function Example 2.1 consider Vanilla Gradients (Simonyan el al. 2013) as the local feature attribution method. The feature importance scores of xf are computed as
@(Tigetan .e: x Vaygetaas ar (x)
polass se arg max ju
faglar fglave (wx" (x)
[0]
Faeture Altritution for Supertied Medels
[b)
Coperands-Cantii Feature Adt butoon for Unsupervised Models
(5)
Contraatoo Dorpus
Fecture Adtriuution for
Unsupervied Models
Abnornel
Nornal Cases
Figure 1: Illustration of feature attribution approaches in different settings. (a) Feature attribution for a supervised model (e.g., an X-ray image classifier for bone abnormality) identifies features that, when removed, decrease the predicted probability of a class of interest (e.g., abnormal). (b) Explicand-centric feature attribution (i.e., RELAX and label-free feature importance) for an unsu- pervised model identifies important features that, when removed, make the representation dissimilar to the explicand's original representation. (c) Contrastive corpus attribution for an unsupervised model identifies important features that, when removed, make the representation similar a foil set (e.g., normal cases) and dissimilar to corpus set (e.g., abnormal cases).
APPROACH
In this section, motivate and propose novel explanation target function for local feature attri- butions in the setting of explaining representations learned by an encoder model. We also describe how to combine the proposed explanation target function with existing attribution methods and how to interpret the obtained attribution scores. All proofs are in Appendix H
3.1 DEFINING CONTRASTIVE CORPUS SIMILARITY AS AN EXPLANATION TARGET FUNCTION
We first define the notion of similarity between representations. Without loss of generality, suppose the representation space is Rd for some integer > 0.
Definition 3.1 (Representation similarity). Let f X - Rd be a representation encoder. For all x, x' E X, the representation similarity between æ and x' based on
f f(z')
81 x')
(1)
[ [[(''
where ||-I| denotes the Fuclidean of vector.

---

Published as a conference paper at ICLR 2023
We note that the above representation similarity is the cosine similarity of representations, Motiva- tions for this choice are in Appendix B
With Definition 3.1, the explanation target functions for RELAX and label-frre feature importance respectively (Wickstrøm et al, 2021 Crabbé & van der Schaar 2022 Intuitively, these explanation target functions identity ex- plicand features that, when removed or perturbed, point the altered representation away from the explicand's original representation.
are Ye (.) s sy(-, x²) and Yi (-) sy(,,
As mentioned in Section 1 we address the limitations of prior works by (i) relaxing the repre- sentation reference for feature attribution to any corpus set of samples; and (ii) enabling explicit specification of foil to answer contrastive questions. For generality, we define the foil to be distribution of inputs, which can also remove the extra burden of identifying particular foil samples. Nevertheless, we acknowledge that a fixed set of foil samples can indeed be selected for tailored use cases. Overall, our proposed improvements lead contrastive corpus similarity as defined below. Definition 3.2 (Contrastive corpus similarity). Let C C X be finite set of corpus samples and Doit be a foil distribution X. The contrastive corpus similarity any x E X to C. in contrast to D foul is
J.f.C.Doo (x)
Σ sy(x, x0) BeonDDooul (ss
Given set of foil samples F = (x(11) (,,,),,,m.))))) (mm),, where x(1), .... x (m) hud fost the empirical esti- mator for the contrastive corpus similarity is
(1.C.=(x)
Σ (1( EC
Σ 5 x1))
(3)
|C|
The unbiasedness of and a concentration bound for the empirical estimator are stated in Appendix Practically, these analyses allow us to choose an empirical foil set size with theoretical justification.
3.2 GENERATING AND INTERPRETING FEATURE ATTRIBUTIONS
As Example 2.2 given a feature attribution method & IR x X + R which takes an explanation target function Ys.- and an explicand æf as inputs, COCOA computes attributions by explaining the empirical contrastive corpus similarity:
(4) In practice, this amounts to wrapping the representation encoder with the empirical estimator for contrastive corpus similarity and applying feature attribution method. In particular, there are two classes of feature attribution methods which can be used to explain contrastive corpus similarity and other similarity-based explanation target functions. The first are removal-based methods (e.g.. RISE, KernelSHAP) and the second are gradient-basee methods (e.g., Vanilla Gradients, Integrated Gradients). Since the explanation target functions and are differentiable with respect to the inputs, we can directly apply these methods without mod- ification. Finally, it is worth noting that methods which satisfy desirable properties (Sundararajan jtt al. 2017: (Sundararajan & Najmil (020)) in the supervised feature attribution setting also satisfy analogous properties for similarity based explanation targets (Appendix D).
consider in this paper can always be evaluated
To interpret feature attribution results from COCOA, consider the contrastive direction toward the corpus representation mean and away from the foil representation mean. Features with high COCOA scores have large impact on the explicand's 's representation direction pointing along this contrastive direction. This interpretation of COCOA is formalized in the following proposition.
Proposition 3.3. The empirical estimator of contrastive corpus similarity is the dot product of the input's representation direction and the empirical contrastive direction. That is,
(1.0.(()))
(1
[
[(x))
C
Σ #FEC
[f(x))
fxx
1 m m
[f(+)))
f(x(11)
(5)
Proposition E.1 is analogous statement for contrastive corpus similarity. An extension of Propo- sition 3.3 and Proposition E. general kernel functions is in Appendix This extension implies that the directional interpretation of COCOA generally holds for kernel-based similarity functions.
4

---

Published as a conference paper at ICLR 2023
4 EXPERIMENTS
In this section, we quantitatively evaluate whether important features identified by COCOA are in- deed related to the corpus and foil, using multiple datasets, encoders, and feature attribution methods (Section 4.1). Furthermore, we demonstrate the utility of COCOA by its application to understand- ing image data augmentations (Section 4.2) and mixed modality object localization (Section 4.3
4.1 QUANTITATIVE EVALUATION
To evaluate COCOA across different representation learning models and datasets, we apply them to (i) SimCLR, a contrastive self-supervised model (Chen et al. 2020), trained on ImageNet (Rus- sakovsky et al. 2013); (ii) SimSiam, non-contrastive self-supervised model (Chen & He 2021 trained on CIFAR- 10 (Krizhevsky et al. 2209); and (iii) representations extracted from the penul-
timate layer of ResNet18 (He et al.
2016), trained on the abnormal-vs--normal musculoskeletal
X-ray dataset MURA (Rajpurkar et al.
2017)
Setup. consider four explanation targets: representation similiarity, contrastive similarity, corpus similarity, and contrastive corpus similarity; in our tables, these correspond to the methods: label- free, contrastive label-free, corpus, and COCOA, respectively. To conduct ablation studies for the components of COCOA, we introduce contrastive similarity, which is contrastive corpus similarity when the corpus similarity term is replaced with similarity to an explicand; and corpus similarity, which is contrastive corpus similarity without a foil similarity term. We use cosine similarity in all methods for the main text results (analogous results with dot product similarity are in Appendix M)
To evaluate the feature attributions for given explicand image x² TR and explanation tar- get, we compute feature attributions using either Integrated Gradients (Sundararajan et al.) 2017). GradientSHAP (Smilkov et 2017 [Eioon et al.) (022)) or RISE (Petsiuk et al. 2018). Then, we average the attributions across the channel dimension in order to use insertion and deletion metrics (Petsiuk et al.) 2018). For deletion, we start with the original explicand and "remove" the M most important pixels by masking them with a mask m E 1) h.,0. {0,1) and a blurred version of the expli- cand "t E ". We then evaluate the impact of each masked image with an evaluation measure 7 @ x + (1 m) @ )." In the supervised setting, this evaluation measure is typically the pre- dicted probability of class of interest. If we plot the number of removed features on the x-axis and the evaluation measure on the y-axis, we would expect the curve to drop sharply initially, leading to low area under the curve. For insertion, which "adds" important features, we would expect a high area under the curve. In the setting of explaining representations, we measure (i) the contrastive corpus similarity, which calculated with the same corpus and foil sets used for explanations: and (ii) the corpus majority prediction, which is the predicted probability of the majority class based on each corpus sample's 's prediction, where the prediction from a downstream linear classifier trained to predict classes based on representations
We evaluate and average our insertion and deletion metrics for 250 explicands drawn for ImageNet and CIFAR-I0 and 50 explicands for MURA, all from held-out sets. We consider both scenarios in which explicands and the corpus are in the same class and from different classes. For our experi- ments, 100 training samples are randomly selected to be the corpus set for each class in CIFAR- 10 and MURA. For ImageNet, 100 corpus samples are drawn from each of the 10 classes from Ima- geNette (Howard. 2019) instead of all 1,000 classes for computational feasibility. Foil samples randomly drawn from each training set. Because ResNets output non-negative representations from ReLU activations, we can apply Equation (C.3) and set the foil size to 1,500 based on d 0.01 and ε = 0.05 in Propasition C.2 We find that, even with a corpus size = 1. COCOA can have good performance, and results with corpus size 2 20 are similar to each other (Appendix D). Varying the foil size does not seem to impact COCOA performance (Appendix K). Additional experiment details are in Appendix[] [
Results. First, our evaluations show that COCOA consistently has the strongest performance in contrastive corpus similarity insertion and deletion metrics across all feature attribution methods, models, and datasets (Appendix [.. These metrics imply that COCOA generates feature attributions that successfully describe whether pixels in the explicand make it more similar to the corpus and
We always mask out all channels for given pixel together. Except for MURA, where we use the final layer of the jointly trained network.
5

---

Published as a conference paper at ICLR 2023
Table 1: Insertion and deletion metries of corpus majority probability when explicands belong to the corpus class. Means (95% confidence intervals) across 5 experiment runs are reported. Higher insertion and lower deletion values indicate better performance, respectively. Each method is a combination of feature attribution method and an explanation target function (e.g.. COCOA under RISE corresponds to feature attributions computed by RISE for the contrastive corpus similarity). Performance results of random attributions (last row) are included as benchmarks.
Anrihution Method
Imagenet SimCIR
CIFAR- & SimSiam
MURA & ResNet
Insertion (1)
Deletion (1)
Insertion (*)
Delction (4)
Insertion
Deletion (L)
Integrated Gradients
Label-Frree
0.362 (0.005)
0.136 (0.004)
0.403 (0.010)
0.249 (0.007) 0.631 (0.040)
0.513 (0.027)
Contrastive Label-Free
0.377 (0.005)
0.125 (0.003)
0.401 (0.011)
0.243 (0.007)
0.690 (0.019)
0.453 (0.025)
Corpus
0.377 (0.005)
0.147 (0.002)
0.355 (0.014)
0.249 (0.010)
0.653 (0.014)
0,500 (0.039)
COCOA
0.422 (0.006)
0.119 (0.003)
0.386 (0.012)
0.230 (0.011)
0.807 (0.013)
0.330 (0.030)
Gradient SHAP
Label-Fres Free
0.409 (0.004)
0.131 (0.001)
0.500 (0.008)
0.244 (0.013)
0.691 (0.038)
0.523 (0.033)
Contrastive Label-Free
0.411 (0.003)
0.127 (0.002)
0,500 (0.009)
0.238 (0.012)
0.697 (0.037)
0,510 (0.018)
Corpus
0.421 (0.006)
0.136 (0.001)
0.478 (0.008)
0,242 (0.009)
0.729 (0.024)
0,494 (0.037)
COCOA
0,445 (0.003)
0.123 (0.002)
0.508 (0.007)
0.211 (0.008)
0.788 (0.030)
0.419 (0.013)
RISE
Label- Free (RELAX)
0.396 (0.005)
0.160 (0.005)
0.630 (0.005)
0,283 (0.004)
0.704 (0.022)
0.600 (0.012)
Contrastive Label-Free
0.424 (0.008)
0.141 (0.005)
0.632 (0.008)
0.279 (0.004)
0.730 (0.019)
0.534 (0.015)
Corpus
0.394 (0.010)
0.166 (0.003)
0.588 (0.005)
0.314 (0.006)
0.701 (0.019)
0.617 (0.026)
COCOA
0.456 (0.009)
0.126 (0.001)
0.663 (0.006)
0,256 (0.006)
0.840 (0.009)
0.415 (0.025)
Random
0.269 (0.003)
0.268 (0.002)
0.329 (0.013)
0.329 (0.010)
0.624 (0.018)
0.629 (0.018)
Table 2: Insertion and deletion metries of corpus majority probability when explicands do not belong to the corpus class. Means (95% confidence intervals) across experiment runs are reported. Higher insertion and lower deletion values indicate better performance, respectively. Each method is combination of a feature attribution method and an explanation target function (e.g.. COCOA under RISE corresponds to feature attributions computed by RISE for the contrastive corpus similarity). Performance results of random attributions (last row) are included as benchmarks.
Attribation Method
Imagenct & SimCLR
CIFAR- 10 & SimSiam
MURA & ResNet
Insertion (f)
Delction (L)
Insertion (f)
Deletion (.)
Insertion (f)
Deletion (.)
Integrated Gradients
Label-Free
3.53c-04 1.06-004
4.44o-04 = 1.01c-04
0.061 (0.004)
0.079 (0.004)
0.394 (0.028)
0.481 (0.031)
Contrastive Label-Free
3.36x-04
1.13e-04
4.44e-04 1: 1.00c-04
0.062 (0.004)
0.082 (0.004)
0.354 (0.022)
0.518 (0.027)
Corpus
1.09c-03
2.85--04
2.26c-04 = 5.31c-05
0.094 (0.003)
0.066 (0.003)
0.609 (0.017)
0.262 (0.029)
COCOA
1.69c-03
5.07e-0M4
1.55e404
2.21e-05
0.099 (0,004)
0.059 (0.005)
0.647 (0.017)
0,213 (0,030)
Gradient SHAP
Label-Free Contrastive Label-Free
2.05c-04 5.96c-05
5.55c-04 = 1.11c-04
0.054 (0.004)
(0.004)
0.362 (0.031)
0.469 (0.021)
2.02c-04 + 5.06e-05
5. 10c-004 = 9.65e-05
0.053 (0.002)
0.080 (0.004)
0,361 (0.022)
0.477 (0.018) 8)
Corpus
8.03c-04 1664-044
2.65e-04 =: 5.44c-05
0.106 (0.006)
0.055 (0.003)
0.5449 (0.020)
0.325 (0.019)
COCOA
1.56e-03 + 4.322-04
1.67e-04= = 5.66e-05
0.122 (0.008)
0.045 (0.003)
0.592 (0.025)
0.236 (0.012)
RISE
Laee (RELAX)
3.70c-04 1: 7.34e-05
5.37e-04 :: 1.23e-04
0.039 (0.003)
0,080 (0.005)
0.330 (0.016)
0.433 (0.030)
Contrastive Label-Froo
3,32o-04
1.140-04
5,84c-04 = 8,24c-05
0.040 (0.003)
0.081 (0.005)
0.307 (0.018)
0.472 (0.027)
Corpus
4.79c-04 9.68e-05
3.87e-04 1 8.41e-05
0,086 (0.005)
0,043 (0.003)
0.497 (0.051)
0,269 (0.020)
COCOA
9.51c-04 2.58c-04
3.60e-04
= 1.40-004
0.107 (0.007)
0.031 (0.002)
0,590 (0.027)
0.181 (0.014)
Random
4.87e 04 9.50e-05
5.03e 04 1 9.73e
0.070 (0.003)
0.070 (0.004)
0.406 (0.013)
0.407 (0.016)
dissimilar to the foil based on cosine similarity. Although this is an important sanity check, it is perhaps unsurprising that COCOA performs strongly in this evaluation metric, because COCOA's explanation target is exactly the same as this evaluation metric.
Next, in Tables 1 and COCOA has the strongest performance in corpus majority probability inser- tion and deletion metrics across nearly all feature attribution methods, models, and datasets. This is arguably a more interesting evaluation compared contrastive corpus similarity, given that COCOA is not explicitly designed to perform this task. Evaluation using the corpus majority predicted prob- ability is semantically meaningful, because it tests whether features with high attributions move an

---

Published as a conference paper at ICLR 2023
explicand representation closer to the corpus opposed to other classes across decision boundary. Furthermore, evaluation with a linear classifier has direct implications for downstream classification tasks and follows the widely used linear evaluation protocol for representation learning (Bachman eet al., 2019, Kolesnikov et al.] 2019 Chen et al.) 2020). In Table [ we evaluate each method in the setting where explicands are from the same class as the corpus. This enables us to compare to label-free attribution methods, which can still perform strongly on these metrics, because large changes in the explicand representation likely correlate with changes in the predicted probability of the explicand's s class. Here, we can see that COCOA consistently outperforms label-free attribution methods and also benefits from the combination of a corpus and foil set. Importantly, in Table 2 corpus-based attribution methods perform strongly especially when the corpus is from different class compared to the explicand. This is significant, because it implies that we can use COCOA to generate explanations that answer questions such as, "What pixels in my dog image make my rep- resentation similar to cat images?", which was previously impossible using label-free approaches.
4.2 UNDERSTANDING DATA AUGMENTATIONS IN SIMCLR
In this section, we aim to visualize coamenttationinvarrinn features that preserve class labels in SimCLR. Because class labels are associated with original images, we use COCOA find features that make the representation of an augmented image similar to that of the original image.
SimCLR is a self-supervised contrastive learning method that pushes images from the same class close together and images from different classes far apart. This is achieved by its contrastive objec- tive, which maximizes the similarity between representations from two augmentations of the same image and minimizes the similarity between representations from different images. Empirically, linear classifiers trained on representations learned by SimCLR perform comparably to strong su- pervised baselines (Chen et al. 2020). Theoretical analyses of SimCLR performance rely on the idea that data augmentations preserve the class label of an image (Arora et al. [2199] HaoChen et al.) 2021). Here, show which input features are associated with label preservation, which has not been previously shown.
Setup. The same SimCLR model and its downstream classifier for ImageNet from Section 4.1 used here. For each image, we investigate augmentations used to train SimCLR including flipping, cropping, flipping with cropping, grayscaling, and color jittering. We also consider cutout and 90 rotation, which are not included in SimCLR training. Class prediction based on the linear classifier is computed for the original version and all augmentations of each image to check class preservation. With each original and augmented image as the explicand, the original image as the corpus, and 1,500 random images from the ImageNet training set as the foil, COCOA is paired with RISE to visually identify class-preserving features. The RISE parameters are the same as in Section 4.1
Results. We first note that the original images and the augmented images after flipping, cropping, flipping with cropping, grayscaling, and color jittering yield correct class predictions (Figure 2 However, cutout and rotated images, which are not part of SimCLR training, are less robust and can result in misclassifications. Because class labels are generated based on the original images, impor- tant features identified by COCOA with each original image as the corpus are class-preserving for correctly classified augmentations. Qualitatively, the class-preserving region of the English springer seems to be its face, as the face is highlighted in all augmentations with the correct classification and the original image (Figure 2).. Contrarily, in the rotated image, which is misclassified as llama, COCOA highlights the body of the English springer instead of its face. Similarly, the piping in the French horn is the preservrsg region (Figure p).. When the piping is cut out, the image is misclassified, and COCOA highlights only the face of the performer. Finally, the co-occurrence of the parachute and parachuter is the important feature for an augmented image to have the correct classification of parachute (Figure 2b) Additional results are in Appendix O
4.3 MIXED MODALITY OBJECT LOCALIZATION
Here, we showcase the utility of COCOA by performing zero-shot object localization based on mixed modality representations as in Gadre et al. (2022). In this experiment, we work with CLIP which trains text and image encoder on paired data: images with associated captions (Radford) ct 202T). To apply COCOA, a necessary prerequisite is a set of corpus examples. These examples not necessarily hard to gather, since small sets of corpus examples can be sufficient (Appendix
7

---

Published as a conference paper at ICLR 2023
Original
Flip
Crop
Flip Grop
Grayscale
Color Jitter
Cutout
Rotation
scriger
Sygtn
Aoreiia Englan Sptaee
Fred
Pecitten
Prode hae
Peritte
aeteee goa
(c)
paachuee
Figure 2: Original version and augmentations of images with their class predictions (top row), along with the corresponding COCOA attributions (red for higher values and blue for lower values) with each original image the corpus and random images as the foil (bottom row); for (a) an English springer, (b) a French horn, and (c) parachute. Cutout and rotation are not included in SimCLR training.
However, if our image representations generated via the CLIP image encoder, circumvent the need to gather a set of corpus images by instead explaining similarity of our explicand image to corpus caption of interest. This enables us easily ask semantically meaningful questions such as, "What in this image makes its representation similar to a woman?".
Setup. In this experiment, fimage Vimage + IRd, Both encoders map their respective input domains into a common repre- sentation space such that the representations of caption and image pair have high cosine similarity when the caption is descriptive of the paired image. We use the original implementation of CLIF and model ViT-B/32. The feature attributions are computed using RISE with the same parameters as in Section 4.1 except with 20,000 masks.
have two CLIP representation encoders frezt Xeentt Rd and
Here, we aim to explain an explicand image xf Pimage For label-free importance, the explanation target is the representation similarity based on the image encoder: Tfsmage are (+). Then, we use an adapted version of the COCOA explanation target, which uses text corpus set Ctext C. ptext and text foil set Flext C plext and maps from images to contrastive corpus similarity f flene CCeess Froms (x) pimmage RR defined as follows:
Σ
gimage (x) (o))gee())))) prext(x"
1
Σ TE
pimage (x) (
(6)
Chese
famage
ftort ftert(x*)]
Flent
famage x
ftent (x
20t ECtent
In partieular, the corpus set consists of single caption C C- (x=) of the following form, x "This is photo of P", where P is manually chosen. Then, the foil set is either (i) single caption photo of Q"., where Q manually chosen or (ii) many
F == {x/} of the form, x "Thhs captions F = {æ! } of the forms, x "TTis a photo of a Q₂" where Q; E Coifar100 P each of the 100 classes in CIFAR-100 100 excluding the corpus class P. Each explicand image is public domain image from homphttps::/ wwww http::/wwwwwpppwll..epp/ com/ that the CLIP model is unlikely to have been trained of
https: thtbb thub. cwm/ com/openai /CLIP
With the exception the astronaut image, which is available in the sci k c--imaae mage package.
8

---

Published as a conference paper at ICLR 2023
Expt icand
Label-rse butiona
COCOA
COCOA
Expl icand
Label free attriuutioe
COCOA NS
COCOA
J856e chor000
atrbee dlter
Deg va
56 dtg
Eydte eParino
CUT Cheri0
nourtai w. riud
doad mostan
Figure 3: Visualization of feature attributions. (a) Importance of a corpus text compared to many foil texts (based on CIFAR-100 100 classes). (b) Importance of corpus text compared to foil text. (a)-(b) The leftmost column is the original explicand, the second column is label-free attributions, and the third and fourth columns are COCOA attributions using P vs. Q as the notation for the corpus and foil text sets (details in Section 4.3). Each feature attribution is overlaid on the original image such that blue is the minimum value and red is the maximum value in given attribution.
Results. We visualize the label-free and COCOA attributions for several explicands in Figure 3 First, we find that label-free attributions highlight a variety regions the explicand image. AT- though these pixels contribute similarity to the explicand's representation, with the exception of the astronaut image, the important pixels are largely not localized and therefore hard to interpret. For instance, in the mountain image we cannot say whether the mountain cloud is the most important to the explicand's 's representation, because the most positively and negatively important pixels are distributed across both objects.
Instead, we find that using COCOA, we can ask specific questions. For instance, we may ask, "Compared many other classes, what pixels in this image look like p?" (Figure We evaluate a number of these questions in the first three images. In the astronaut image, we can see the woman's 's face and the rocket are localized. Likewise the image with both a bicyele and car, we can localize either the bicycle or the car. In the zebra image, we are able to localize the zebras by using corpus texts that aim to identify "zebra" or "stripes". Finally, in the last three images we find that we can formulate specific contrastive questions that ask, "What pixels in this image look like P rather than Q?" (Figure (b). Doing so we successfully identify a bee rather than flower, a dog rather than a cat, and a mountain rather than clouds (and vice versa respectively).
DISCUSSION
In this work, we introduce COCOA, a method to explain unsupervised models based on similarity in representation space. We theoretically analyze our method and demonstrate its compatibility with many feature attribution methods. We quantitatively validate COCOA in several datasets and on contrastive self-supervised, non-contrastive self-super and supervised models. Then, use COCOA to better understand data augmentations in self-supervised setting and to perform zero-shot object localization.
Although COCOA is already powerful technique to understand representations, there are many interesting future research directions. First, we can adapt automatic concept discovery methods (Ghorbani 2019 Yeh et al., (2020) to automatically identify homogeneous corpus sets. Second, we can use other explanation methods for our novel contrastive corpus similarity explanation target. This includes counterfactual explanation methods (Verma et al. 2020), which only require gradients to optimize a counterfactual objective. Finally, a fundamental limitation of COCOA is that it cannot explain corpus sets if the model has not learned anything about the given corpus. Therefore, valuable future direction is to investigate complementary methods to detect whether the model's representation contains information about corpus prior to application of COCOA.

---

Published as a conference paper at ICLR 2023
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, lan Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018.
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.00229, 2019.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
Dominika Basaj, Witold Oleszkiewicz, Igor Sieradzki, Michal Górszczak, Barbara Rychalska, Tomasz Trzcińssii, and Bartosz Zieliński. Explaining self-supervised image representations with visual probing. Freiburg. Germany: International Joint Conferences on Artificial Intelligence, 2021.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on paltern analysis and machine intelligence, 1788 2013.
Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José MF Moura, and Peter Eckersley. Explainable machine learning in deployment. In Proceedings of the 2020 conference on fairness, accountability, and transparency, pp. 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Dan Ciregan, Ueli Meier, and Jürgen Schmidhuber. Multi-column deep neural networks for image classification. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3642- 3649. IEEE, 2012.
Ian Covert, Scott M Lundberg, and Su-In Lee. Explaining by removing: A unified framework for model explanation. Mach. Learn. Res., 22:209-1, 2021.
Jonathan Crabbé and Mihaela van der Schaar. Label-free explainability for unsupervised models. arXiv preprint arXhv:2203.01923 2022.
Li Deng, Geoffrey Hinton, and Brian Kingsbury. New types of deep neural network learning for speech recognition and related applications: An overview. In 2013 IEEE international conference on acousties, speech and signal processing, pp. 8599-8603 IEEE, 2013.
Gabriel Erion, Joseph D Janizek, Pascal Sturmfels, Scott M Lundberg, and Su-In Lee. Improving performance of deep learning models with axiomatic altribution priors and expected gradients. Nature machine intelligence, 3(7):620- 2021.
Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, and Shuran Song. Clip wheels: Zero-shot object navigation object localization and exploration. arXiv preprint (eXiv:2203.1042) 2022.
Benyamin Ghojogh, Ali Ghodsi, Fakhri Karray, and Mark Crowley. Reproducing kernel hilbert space, mercer's theorem, eigenfunctions, nystr " om method, and use of kernels in machine learn- ing: Tutorial and survey. arXiv preprint arXiv:2106 08443, 2021.
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. Advances in Neural Information Processing Systems, 32, 2019.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 1803:07033008 2018.
10

---

Published as a conference paper at ICLR 2023
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in neural information processing systems, 33:21271-21284. 2020.
Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. Advances Neural Information Processing Systems, 34:5000-5011. 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Wassily Hoeffding.
Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58:13-30, 1963.
Jeremy Howard. Imagenette. URL: https://github. comffasstai/mageeeett. 2019.
Alon Jacovi, Swabha Swayamdipta, Shauli Ravfogel, Yanai Elazar, Yejin Choi, and Yoav Goldberg. Contrastive explanations for model interpretability. arXiv preprint arXiv:2103.01378, 2021.
Sebastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. On using very large target vocabulary for neural machine translation. arXiv preprint arXiv:1412.2000 2014.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger,
Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-588) 2021.
Daniel Kahneman and Dale T Miller. Norm theory: Comparing reality to its alternatives. Psycho- logical review, 93(2):136, 1986.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learming, pp. 2668-2677. PMLR, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14/2.6900, 2014.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885-1894. PMLR, 2017.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. International Conference on Machine Learning, pp- $338-5348. PMLR, 2020.
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang. Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, et al. Captum: A unified and generic model interpretability library for pytorch. arXiv preprint 27X.v:2009.07896 2020.
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self supersupe visual repre- sentation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1920-1929, 2019.
Satyapriya Krishna, Tessa Han, Alex Gu, Javin Pombra, Shahin Jabbari, Steven Wu, and Himabindu Lakkaraju. The disagreement problem in explainable machine learning: A practitioner's perspec- tive. arXiv preprint arXiv:2202.01602 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Peter Lipton. Contrastive explanation. Royal Institute of Philosophy Supplement, 27:247-266, 1990. doi: 10 010077/113522661000511331

---

Published as a conference paper at ICLR 2023
Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of inter- pretability is both important and slippery. Queue, 16(3):31-57,201 2018.
Scott M Lundberg and Su-In Lee. A unified approach interpreting model predictions. Advances in neural information processing systems, 30, 2017.
James Mercer. Xvi. functions of positive and negative type, and their connection the theory of integral equations. Philosophical transactions of the royal society of London. Series A, containing papers of mathematical or physical character,
1909.
Tim Miller, Explanation in artificial intelligence: Insights from the social sciences. Artificial intel- ligence, 267:1-38, 2019.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6707-6717, 2020.
Vitali Petsiuk, Abir Das. and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:/806.005 1806.0741,, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pp. 8748-8763. PMLR, 2021.
Pranav Rajpurkar, Jeremy Irvin, Aarti Bagul, Daisy Ding, Tony Duan, Hershel Mchta, Brandon Yang. Kaylie Zhu, Dillon Laird, Robyn L Ball, et al. Mura: Large dataset for abnormality detec- tion in musculoskeletal radiographs. arXiv preprint 172.06677 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-253 2015.
David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre. Dharshan Kumaran, Thore Graepel, al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 140- 1144, 2018
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:7312.6033 2013.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1006.03825 2017.
Mukund Sundararajan and Amir Najmi. The many shapley values for model explanation. In Inter- national conference on machine learning, pp. 9269-9278. PMLR, 2020.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-3328, PMLR, 2017.
Luan Tran, Xi Yin, and Xiaoming Liu. Disentangled representation learning gan for pose-invariant face recognition. In Proceedings of the IEEE conference on computer vision and puttern recog- nition, pp. 1415-1424, 2017.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Sahil Verma, John Dickerson, and Keegan Hines. Counterfactual explanations for machine learning: A review. arXiv preprint 200: 2020.
Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Harv. JL & Tech,, 31:841, 2017.
12

---

Published as a conference paper at ICLR 2023
Kristoffer K Wickstrøm, Daniel J Trosten, Sigurd Løkse, Karl Øyvind Mikalsen, Michael C Kampffmeyer, and Robert Jenssen. Relax: Representation learning explainability. arXiv preprint arXiv:2112.10161 2021.
Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-wware concept-based explanations in deep neural networks. Advances in Neural Information Processing Systems, 13:20554-20555 2020.
Jure Zbontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: learning via redundancy reduction, In International Conference on Machine Learning, pp. 12310- 12320. PMLR, 2021.
Self-supervised
13

---

Published as a conference paper at ICLR 2023
A ADDITIONAL RELATED WORK
Representation learning. COCOA can be used to explain any representation learning model. Rep- resentation learning aims to automatically discover latent dimensions that are valuable to down- stream tasks (e.g., classification) (Bengio el al. 2013). It has been applied to great effect within natural language processing (Deng et al. 2013 Van Den Oord et al., 2017) and computer vision (Ciregan et al. 2012; Gidaris et al. 2018) and encompasses both supervised and unsupervised mod- els ( : + 3). For supervised models the output space corresponds to known labels (2 = Y), whereas for unsupervised models the outputs are latent representations (2 = H). Within unsu- pervised representation learning, self-supervised learning leverages unlabelled data to formulate learning tasks (Grill et al. 2020, Misra & Maaten 2020). One particularly successful instantiation of self-supervised learning is contrastive learning, which optimizes the similarity between different views of the same sample and dissimilarity to random samples (Chen et al. 2020: Radford et al., 2021
Representation-space explanations. COCOA explains representation (latent) space. An early post-hoc explanation method based on representation spaces is TCAV (Kim et al. 2018) which defines concepts based the representation space and produces global explanation of whether given class depends on a concept of interest. A number of approaches similarly define concepts in the representation space and build upon TCAV with automatic concept discovery (Ghorbani et al.) 2019 Yeh et al. 2020). Finally, [asaj et al. (2021) build upon TCAV to define visual probes to validate whether representation encodes property of interest. These approaches detect whether given concept is important to the model, whereas our approach aims to understand which features make a representation similar to a given concept captured by a corpus.
Supervised explanations. COCOA builds upon existing supervised explanation methods to explain an unsupervised setting. The majority of explanation methods operate on supervised models and aim to explain a scalar output, often the model's predicted probability for specific class. These methods vary in the unit of explanation (i.e., the way in which they describe what was important) and the explanation target (i.e., the model behavior they aim to explain). The unit of explanation for example attributions and counterfactual attributions are samples which are either important to or modify the explanation target (Koh & Liang 2017 Wachter al. 2017). In contrast, the unit of explanation of feature attributions or concept attributions are importances which quantify how features or concepts contribute to the explanation target (Lundberg & Lee, 2017 Kim al. 2018).
In this paper, we focus on feature attribution methods, which are the most widely-used explanation technique, but note that is possible to apply many other explanation techniques to understand our new explanation target. It has been shown that different feature attribution methods can generate disagreeing results (Krishna et al. 2022). so we empirically evaluate contrastive corpus similarity and other explanation target functions with number of varied attribution methods. In particular, we compute local feature attributions using three popular methods. The first is Integrated Gradi- ents (Sundararajan et al., 2017). a gradient-based feature attribution method, which accumulates gradients between a baseline and an explicand. The second is GradienSHAFF which is a SHAP- related method that combines SmoothGrad (Smilkov et al.] 2017) and Expected Gradients (Erion) et al. 2021). a generalization of Integrated Gradients. Third, we use RISE (Petsiuk et al. 2018), remeval-based explanation method (Covert et al. 2021). which computes feature attributions by randomly masking portions of an explicand.
Unsupervised explanations. COCOA explicitly aims to provide explanations for unsupervised models. There is largely a deficit of explanation methods suitable to this task and we are aware of only two recent methods that address this problem: RELAX (Wickstrøm et al., 2021) and label-free explainability (Crabbé & van der Schaar (022). Both methods define an explana- tion target function based on similarity to the explicand, where RELAX uses the cosine kernel and label-free explainability uses the dot prod- uct (7/ (x) ffx)) fz))). Then, both approaches utilize pre-existing local feature attribution methods to explain their respective explanation targets, where RELAX specfically utilizes RISE, and label-free explainability is an agnostic wrapper function. A final distinction between the two papers that label-free explainability additionally provides example attributions which aim to identify im- portant training examples. While COCOA, RELAX, and label-free explainability are motivated to
(x) = (f(x)T (()()))(() [x)]]
htttss:/ ai /api /gradi ent__hap.hhhte htm1 htm]
14

---

Published as a conference paper at ICLR 2023
explain unsupervised models, they all can be applied to explain representations in general, whether obtained through unsupervised or supervised learning.
Contrastive explanations. COCOA makes unsupervised explanations more interpretable by mak- ing them contrastive. Contrastive explanations aim to understand an event P (the fact) by contrasting it to another event Q (the foil) (Jacovi et al. 2021 Social science studies recognize that contrastive explanations align with human intuition and reduce an explanation 's complexity by ignoring factors common to both the fact and the foil (Kahneman Miller 1986, Lipton 1990 Miller 2019). One strategy to obtain contrastive explanations is to simply define an explanation target function which
describes a contrastive model behavior (e.g., the difference between the model's prediction for the fact and the foil) (Jacovi et al. 2021).
B MOTIVATIONS FOR COSINE SIMILARITY AS THE REPRESENTATION SIMILARITY
The representation similarity defined in Definition 3.1 the cosine similarity of representations. There are multiple motivations for this choice. First, this representation similarity is used in RELAX as the explanation target function (Wickstrøm et al. 2021). Similarly, label-free feature importance uses the dot product without normalization (Crabbé & der Schaar 2022). Hence, the choice of cosine similarity facilitates comparisons with prior works. Second, many self-supervised models trained with the cosine similarity or the related cross-correlation in their loss functions (Chen eet al., 22200 Grill et al., 2020; Chen & He 2021 Radford et al.] 2021; Zbontar et al., 2021). so cosine similarity
natural choice for measuring similarity between representations learned by these models. Finally, other similarity measures, such as the Gaussian kernel function, may contain parameters that impact explanation outcomes, but our knowledge there are no principled ways of tuning these parameters for the purpose of generating explanations.
C SOME PROPERTIES OF THE EMPIRICAL ESTIMATOR OF CONTRASTIVE CORPUS SIMILARITY
Proposition C.I. The estimator (f.)).((:)) is an unbiased estimator "f,C, (z)).
Proposition C.2. For any ô 1) and = > 0, if the sample size m in (1),, satisfies
22g(2
m 2
(C.1)
then
P(1)f.e.(1)) TF.C (x)
(C.2)
If it is further assumed that I maps from X to the set of non-negative vectors I& (e.g. neural network with a ReLU output layer), then the same concentration bound is achieved with
log (2/5)
m A
(C.3)
2e2
Proposition C.2 allows us choose an empirical foil sample size with theoretical justification.
D FEATURE ATTRIBUTION PROPERTIES
There are a multitude of feature attribution methods. order to compare these methods, researchers have debated their respective merits in terms of the properties they satisfy (Sundararajan & Najmi, 2020). Most feature attribution methods satisfy certain properties with respect to the model output (explanation target) they explain. Methods which satisfy a property when the explanation target is the predicted probability of a particular class will also satisfy a related property for other explanation targets such representation similarity contrastive corpus similarity.
As an example, we may consider the property of completeness, which is satisfied by Integrated Gradients, KernelSHAP, DeepL.FFT, and number of other explanation methods, Methods which
15

---

Published as a conference paper at ICLR 2023
satisfy completeness generate feature attributions which sum up to the explanation target function applied to the explicand minus baseline value by:
Σ "o((),,;=")) (5.- bo-
(D.1)
For attribution methods that use single baseline (e.g., Integrated Gradients, DeepLIFT), the base- line value is equal to the explanation target function applied to a baseline of the user's choosing (i.e., bu = yf. (x=)). Therefore, supervised feature attributions up to the difference between the predicted probability of class of interest for the explicand x" and baseline x":
S (Tprcie,,,, x²) Vysinse (x²) potnes (xx").
(D.2)
Analogously, in the unsupervised explanation setting, COCOA feature attributions will sum up to the difference in the contrastive corpus similarity between the explicand and the baseline:
de (e(1)..FF x²) = (1f.c.((x)) - ((,),,(x")).
(D.3)
Note that analogous completeness properties may be shown for representation similarity, contrastive similarity, and corpus similarity.
E PROPOSITION FOR INTERPRETING COCOA WITH RESPECT TO THE CONTRASTIVE CORPUS SIMILARITY
Proposition E.1. Contrastive corpus similarity the dot product of the input's 'su representation di- rection and the contrastive direction. That is,
f(x))
T )(
f(x²)
f(x²)
Doo (x))
E DD Dooit
(E.1)
[[](
C
[(()))]]
#PEC
F EXTENSION OF PROPOSITION 3.3 AND PROPOSITION E.1 TO GENERAL KERNEL FUNCTIONS
Let another kernel function R. Rd R, such that 8y s(f(), be the represen- tation similarity measure instead of the cosine similarity. Then, in another latent feature space, the contrastive direction points toward the corpus's mean representation and away from the foil's mean representation. Formally, we have the following proposition.
Proposition F.1. Suppose the function s(...) is a continuous positive semi-definite kernel on compact set Z C Rdd, and f maps from X to Z. Let Lz(Z) denote the set of functions over Z that are square integrable. Additionally, suppose that the integral operator T,, defined as (T.9)(-) 12 s(,, ())(() for all g E L2( is positive semi-definite:
/
s(z, 2)g(2) 2 0.
(F.1)
Let
Σ
s(f(2), f(x")) - (s)
(F22
C
3'EC
and
m m Σ {((())
(5,C.7(x) C
Σ {(1(x),((x))
(F3)
3
Then
(....Dooa = (
Σ ((((x() Eg D (e) [f(())))))
(F4)
#CEC
16

---

Published as a conference paper at ICLR 2023
and
v(((e) - D
( E
- Ev((())))
((c.((a)=(((a)))
(F.5)
F' where vb 2 + D is a feature map from the representation space to the feature space D induced by the kernel function.
G CONTRASTIVE CORPUS SIMILARITY WHEN THE CORPUS CONSISTS OF MULTIPLE SUB-CORPORA
Here, we consider the scenario where a corpus consists of multiple groups (i.e., sub-corpora) with different characteristics. this setting, the contrastive corpus similarity and its empirical estima- tor are weighted average of the contrastive sub-corpus similarities and weighted average of the empirical estimators contrastive sub-corpus similarities, respectively. Importantly, the weight cor- responding to each sub-corpus is proportional to its size. This insight is formalized in the following proposition.
Proposition G.1. Suppose corpus consists of disjoint sub-corpora Cx = I,.., for some integer K > O. That is, C = = Cs. and C, n ne, = for all i i j. Then the contrastive corpus similarity with respect the corpus C is weighted average of contrastive corpus similarities with respect to the sub-corpora:
K
C+ C
M.C.DD fost (x)
Dr.Cu. (z).
(G.1)
Furthermore, the empirical estimator of contrastive corpus similarity with respect the corpus C is weighted average of empirical estimators of contrastive corpus similarity with respect to the sub-corpora:
Cal C
11,0, (x) Σ
(1.)..()))
(G.2)
H PROOFS
H.1 PROOF PROPOSITION C.1
Proof. We have
1
[x),,,(())]
Σ
(2, (1))
(1)))
.n (m) Dosl
C
20 mn
12 sou
EC
1=1
(H.1)
Σ
8 (x) ) Emg-D 1s5 x²)] = 7f.C.Doou (x),
C
(H.2)
where (H.1) follows from the linearity of expectation, and (H.2) from the fact that (1) .... x (m)
identically distributed from foil-
H.2 PROOF OF PROPOSITION
2
We first state Hoeffding's inequality.
Lemma H.1. Let Xi...., Xn, be independent random variables such that a; S X₁ S by almost surely. Consider S, X₁ Xx,, then
2
P(SS, - E[Sa]| 2) < 2 exp
(H.3)
E 1 (b: a;)²
17

---

Published as a conference paper at ICLR 2023
Proof. See Hoeffding
1963)
We now proceed to prove Proposition C.2
Proof. First, have
85( x1))
(2)
Em
DD DD
(s) x²)] 2( (H.4)
and note that
m 8 (2c, 2011)
RewwDroulsyte xx)] (1)
(H.5)
(on (m DD Dost
m
Because the cosine similarity is bounded: -1 87(3 it follows that -1/m 8)((,)/m
1/m. Applying the Hoeffding's inequality, we obtain
28a
P(())...=0)) - yon (x)) 2 s 2exp
(H.6)
Li=1 (2/m)²
Th
2exp
(H.7)
2
8²
2 llog(2/0)
S 2exp
6.
(H.8)
2
e²
is further assumed that f maps from X' to Rd, then 0 S ()(:)))
I and hence sf(;;))mm
1/m. Applying the Hoeffding's inequality with m 2 log(2/6)
yields
2=
P((ff,c,(x)) Joy. (x)) 2 2) S 2exp
(H.9)
2i=1 (1/m)²
exxpp( 1app-2ee2.0 m)
(H.10)
(2222 2 log(2/8)
< exp
(H.11)
2=a
H.3 PROOF OF PROPOSITION 3.3 AND PROPOSITION E.1
Proof. By the definition of f1 and linear algebra, we have
(x))
Σ
f(x)
((x)
1
f(x)
)
f(x(1)
(H.12)
C
[[f(
[[fx))]|
[[f(z)]
[[(())]]
DrEc
f(x)
f(x²)
f(x)
(a(l)
C
Σ
[[
[[(()))
[f(x)))]|
(H.13)
E.EC
f(x) )
(1d)
Σ f(x9)
m
f(x(1))
(11((=))
[[f(z))
m Σ ([(((()1))
(H.14)
#FEc
By the definition of Ju (w), linear algebra, and the linearity of expectation, we have
1
f(x) )
f(x))
EE (11/(ee) )
T f(x")
YJ,C,
|e
[[f(x)]
[[f(==)]
[[[(x*)]
(H.15)
-(15(e))
f(x)
)
T
f(x09
( f(x) ) EEo
[(xx")
(1f(())])
[Cl
Σ
[lf(
[|]
||
(H.16)
T (f()) )
m(x²)
[[f(x())1 1).
fax²))
Σ
[[(()))]
[l
E "unn Doul
(H.17)
18

---

Published as a conference paper at ICLR 2023
H.4 PROOF OF PROPOSITION F.1
Proof. By Mercer's Theorem Merrer 1909) (see Ghojogh et al. (2021) for an accessible survey
relevant to machine learning), we have
fos, (x)
Σ
(((()))
Ba (
(H.18)
[C] EFEC
=V((())))
Σ (((EE9)
( (f
(H.19)
C
= =(f(x)))
1
Σ (f((x'))
BernD ( (Vf(2')))
(H.20)
C
EC
Similarly, we have
)fx),=(()) C
Σ
(((()) - 1 - ((f()))((((E
(H.21)
EFEC
=((((0))
Σ ((((E9)) - ~((()) 1
Σ ((((((()))
(H.22)
C
SCER
1 Σ (f((x)) - Σ {((xo(1))
-
(H.23)
C
3ree
H.5 PROOF OF PROPOSITION G.1
Proof. Because the sub-corpora are disjoint sets, we have
Do.t (x)
x)(
BonnDDoolss
(H.24)
C
cC
C+
()(0,x0) - SonwDroulss(a,
(H.25)
CC
[Ccl
1
81(2, x9)
E B (x, x")]
(H.26)
|C|
|CA|
1Ccl
(x,
EennD (s (20, x²"]
(H.27)
[CI
CC
w'Eli
Ckk
Yf.Cs. Dooc C..Ce.Doo (x).
(H.28)
19

---

Published as a conference paper at ICLR 2023
Similarly,
1
1 m
(x))
sy 2))
sf sy (x) x (
(H.29)
C
m
EC2
K
C
(x
8 f (t. x0
(H.30)
ECC
==1
[Cx|
K
|Ckl
m
81(2,
Σ t=1
8 (2x, x1
(H.31)
CI
Ckkl
CC
m
#CE
k=
8 $1(
1
Σ
m
s (x, x(.1)
(H.32)
|Cl
m
K
|Ckl
(H.33)
ICl
I EXPERIMENT DETAILS
Code is available attppps//ggtthbb..mm///neeeeells www.pleepppaaaabllllaaaaa
1.1 DATASETS
ImageNet. The ImageNet ILSVRC dataset contains 1.2 million labeled training images and 50,000 labeled validation images over 1,000 object classes (Russakovsky et al. (201)). Since the SimCLR model and its downstream linear classifier trained and tuned with only the training set, the validation set as a held-out set for quantitative evaluation (Section 4.1) and understanding data augmentations (Section 4.2).. For computational feasibility, subset of 10 easily classified classes called ImageNette (including tench, English springer, cassette player, chainsaw, church, French horn, garbage truck, gas pump, golf ball, and parachute) (Howard 2019) are used for quantitative evaluation.
CIFAR-10. The CIFAR-10 dataset consists of 50,000 labeled training images and 10.000 labeled test images of size 32 x 32 over 10 classes. The test set is used for quantitative evaluation.
MURA. The MURA (MUsculoskeletal RAdiographs) dataset contains 36,808 training radiograph images from 13,457 musculoskeletal studies and 3,197 validation images from 1,199 studies. Each study and its associated images are labeled by radiologists as either normal abnormal. We further split the official training images into an 80% subset for our model training and 20% subset for hyperparameter tuning. The official validation set is held out and used for quantitative evaluation.
I.2 MODELS
SimCLR. We obtained the weights of SimCLRvI (with a ResNer50 backbone) and its downstream linear classifier, pre-trained with ImageNet, from https: google-research/ simcir by [Cen et al. (2020). The pre-trained SimCLR model and linear classifier were converted to the PyTorch format following the conversion in https: //g thub. com/tonylinsi mi w converter as recommended in the SimCLR authors' official doc- umentation.
com/
SimSiam. We trained a SimSiam model (with a ResNet18 backbone) for CIFAR- 10 following the training procedure and hyperparameters outlined in Chen & He (202)) using the implementation in hattp:: Sitopl-acc-on-Ciree
com/Re za wat.aaaaaaaaaaaaa Stts/mmmSii S am 91. The downstream linear classifier trained with stochastic gra- dient descent with a learning rate of 30.0 and a batch size of 256 to achieve top-1 accuracy of 92.06% the CIFAR-10 test set.
20

---

Published as a conference paper at ICLR 2023
ResNet classifier. We trained ResNet18 classifier for normal abnormal radiograph images with an 80% training subset and tuned hyperparameters with the other 20%. We randomly augmented the data during training by horizontal flipping. vertical flipping, and rotation between -30" and 30%. The classifier was trained with an Adam optimizer (Kingma & Ba 2014), with a batch size of 256, learning rate decay of 0.1 every 10 steps, and weight decay of 0.001. The initial learning rate was tuned over {0.1, 0.01, 0.001, 0.0001} to identify the optimal initial learning rate of 0.001. The trained ResNet18 classifier achieves an accuracy of 80.54% on the held-out official validation set.
I.3 FEATURE ATTRIBUTION METHODS
Integrated Gradients. Integrated Gradients (Sundararajan et al. 2017) with 50 steps for the Riem- man approximation of integral were run in this work. The implementation in the Captum package was used (Kokhlikyan et al. 2020). The baseline values correspond to the explicand image with blurring.
GradientSHAP. GradientSHAP with 50 random Gaussian noise samples having a standard devi- ation of 0.2 for the gradient expectation estimation was run. The implementation in the Captum package was used (Kokhlikyan et al. 2020). The baseline values correspond the explicand image with blurring.
RISE. RISE (Petsiuk al. 2018) with 5000 random masks generated with masking probability of 0.5 was run. For CIFAR- 10, each binary mask before upsampling had size 4 x 4. For ImageNet and MURA, each initial binary mask was 7 7. The baseline values for replacing masked pixels correspond to the explicand image with blurring.
J CORPUS SIZE SENSITIVITY ANALYSIS
Figure 4: Insertion and deletion metrics of contrastive corpus similarity for COCOA with corpus size = 5, 20, 50, 100, 200. The foil size is fixed at 1500. Performance results of random attributions plotted as benchmarks. Means and 95% confidence intervals across 5 experiment runs shown.
21

---

Published as a conference paper at ICLR 2023
333ee
Figure 5: Insertion and deletion metrics of corpus majority probability for COCOA with corpus size = 1,5, 5, 20, 50, 100, 200. The foil size is fixed at 1500. Performance results of random attributions are plotted as benchmarks. Means and 95% confidence intervals across 5 experiment runs are shown.
K FOIL SIZE SENSITIVITY ANALYSIS
Contrative orra did wnyy tpg loard 118 copas canl
Conzratiie capussmdort sortrrtyyyy Seciicann arE Eot OVE CIaTl
CoOM
May Me Ondeed
Figure 6: Insertion and deletion metrics of contrastive corpus similarity for COCOA with foil size 100, 500, 1500, 2500, 5000. The corpus size is fixed at 100. Performance results of random attributions are plotted as benchmarks. Means and 95% confidence intervals across 5 experiment runs are shown.
22

---

Published as a conference paper at ICLR 2023
Prdiites pobubiity crooo molery prod of cando DroNa camt
Prokosdpobell of COoO male pICC Sopioancc In dool
000M
COOOA
Figure 7: Insertion and deletion metrics of corpus majority probability for COCOA with foil size = 100, 500, 1500, 2500, 5000. The corpus size is fixed at 100. Performance results of random attributions are plotted benchmarks. Means and 95% confidence intervals across 5 experiment runs are shown.
L SANITY CHECK RESULTS
Table 3: Insertion and deletion metrics of contrastive corpus similarity when explicands belong to the corpus class. Means (95% confidence intervals) across 5 experiment runs are reported. Higher insertion and lower deletion values indicate better performance, respectively. Each method is combination of feature attribution method and an explanation target function (c.g., COCOA under RISE corresponds to feature attributions computed by RISE for the contrastive corpus similarity). Performance results of random attributions (last row) are included as benchmarks.
Attribution Method
Imagenet & SimCLR
CIFAR 10 & SimSiam
MURA & ResNet
Insertion (t)
Deletion (})
Insertion (t)
Deletion (L)
Insertion (*)
Delction (L)
Integrated Gradients
Label-Free
0.154 (0.003)
0.076 (0.001)
0.037 (0.002)
0.021 (0.002)
0.046 (0.010)
9.77e-03 + 1.15e-02
Contrastive Label-Free
0.161 (0.003)
0.071 (0.002)
0.036 (0.003)
0.020 (0.002)
0.058 (0.007)
-4.66--03 = 1.20c-02
Corpus
0.157 (0.002)
0.081 (0.002)
0.033 (0.002)
0.020 (0.002)
0.051 (0.007)
9.39c-03 1.30c-02
COCOA
0.172 (0.002)
0.067 (0.002)
0.037 (0.002)
0,018 (0.002)
0.091 (0.006)
-0,036 (0.013)
Gradient SHAP
Label-Free
0.171 (0.004)
0.067 (0.001)
0.048 (0.002)
0.019 (0.002)
0.053 (0.013)
0.017 (0.010)
Contrustive Label-Free
0.173 (0.004)
0.064 (0.001)
0.048 (0.002)
0.019 (0.002)
0.056 (0.013)
0.011 (0.007)
Corpus
0.172 (0.004)
0.071 (0.001)
0.047 (0.002)
0.019 (0.002)
0.064 (0.011)
9.77c-03 + 1 1.26e-02
COCOA
0.181 (0.004)
0.062 (0.001)
0.051 (0.002)
0.015 (0.002)
0.081 (0.010)
-0.014 (0.007)
RISE
Label-Free (RELAX)
0.175 (0.003)
0.091 (0.002)
0.061 (0.002)
0.023 (0.002)
0.049 (0.008)
0.017 (0.002)
Contrustive Label-Free
0.184 (0.003)
0.085 (0.002)
0.062 (0.002)
0.023 (0.002)
0.055 (0.006)
-1 1.08%-03 3.02c-03
Corpus
0.173 (0.003)
0.094 (0.003)
0.059 (0.002)
0.025 (0.002)
0.044 (0.010)
0.026 (0.005)
COCOA
0.195 (0.003)
0,080 (0.002)
0.068 (0.002)
0.019 (0.002)
0.083 (0.0666
-0,028 (0.088)
Random
0.115 (0.002)
0.114 (0.002)
0,028 (0.002)
0.029 (0.002)
0.038 (0.009)
0.039 (0.009)
23

---

Published as a conference paper at ICLR 2023
Table 4: Insertion and deletion metrics of contrastive corpus similarity when explicands do not belong to the corpus class. Means (95% confidence intervals) across 5 experiment runs are reported. Higher insertion and lower deletion values indicate better performance, respectively. Each method is combination of a feature attribution method and an explanation target function (e.g., COCOA under RISE corresponds to feature attributions computed by RISE for the contrastive corpus similarity). Performance results of random attributions (last row) are included as benchmarks.
Attiiuation Mathol
Imagenet & SunCLR
CIFAR-IO & SinSiam
MURA & ResNet
Insersion (1)
Deletion (1)
Inserdion (1)
Deletion (1)
Insertion (1)
Deletion (J)
Ireggaaed Goadients
Label-Pize
0.11
0301 (0:001)
-5.34.-13
1146c-03
-3.32e-113
1136c-43
0.120 (0,009)
-1.50c-0 8.7Nc-03
Concastive Label Free
0.011
(0.001)
0.817 (0.001)
5.377
1.500
33221e
0.025 (0.025)
0:012
Corpus
044-003
-00020 (0.001)
-2.9c-04
1.79--03
-6.17c-133
1.47-43
0:400 (8.005)
<00K2 (0:009)
COCOA
5.246-03
1.5
.0,128 (0.001)
1.46-1133
1.74c-113
-8.796013
1.50c-83
0.048 (1.006)
.0.176 (0.009)
Gradient SHAP
Fabe--Fore
1:33-00
-0006
-A1 Re-03-3-
1.57-08
-0.057
500-03 100-0202
Contrasiie Lild-Frre
-8.87-433-
1.35--03
-0.07
146:08
-3066-33
1.56-03
-0.035
1.66-01 38c-83
Corpus
Liite 3.9te-03 1.76e-03
6.20e -0.012 (0.002)
0:013
0.041
COCOA
.0007 (0,001)
565e-13
2.03e-033
0,002
4.067 (0.005)
RISE:
Label (RELAN)
2.877 05
05
4.05% 05
1,44g 05
6.14e
1.31e 05
2.18% 03
1.53e 05
0,047
0.1124
Conmastive Lobel Free
2.87e 03
1.32e 03
4.15e 05
1,55e 05
6.20e
03
1.38 05
2.16c 03
1,57 05
0.054 (0.006)
0.017 (0.005)
1.088-03-
1.72c-03
-5.33.405
1.25-00
1.32--13 1.75c-113
-8.70-433
-8882-033
6.31c-03
-0.063
COCOA
8.26e-03
2.17e-413
40.055
7.19c-13
1.45e-113
-0.034
0.015 (00007)
-0.159
Random
0,013 (0.001)
0,013 (0:001)
4. (0e 05
1.59 05
-4.07 03
1.46e 05
0.021 (0.007)
0.021 (0:007)
M ADDITIONAL QUANTITATIVE EVALUATION RESULTS WITH DOT PRODUCT SIMILARITY
Table Insertion and deletion metrics of contrastive corpus similarity when explicands belong to the corpus class. All explanation target functions are based on dot product instead of cosine sim- ilarity. Means (95% confidence intervals) across 5 experiment runs are reported. Higher insertion and lower deletion values indicate better performance, respectively. Each method is combination of a feature attribution method and an explanation target function (c.g., COCOA under RISE corre- sponds to feature attributions computed by RISE for the contrastive corpus similarity). Performance results of random attributions (last row) are included as benchmarks.
Attribution Method
Imagenet& & SimCLR
CIFAR-10 10 & SimSiam
MURA & ResNet
Insertion (t)
Deletion (})
Insettion (t)
Deletion (L)
Insertion (*)
Delction (L)
Integrated Gradients
Label-Free
0.153 (0.002) 0.076 (0.002)
0.031 (0.002)
0.020 (0.002)
0.058 (0.008)
-1.40e-03
1.08e-02
Contrastive Label-Free
0.157 (0.002)
0.072 (0.002)
0.032 (0.003)
0.019 (0.002)
0.058 (0.007)
-3.8ee-03 = 1.29c-02
Corpus
0.157 (0.002)
0.076 (0.002)
0.031 (0.002)
0.020 (0.002)
0.079 (0.008)
-0.018 (0.010)
COCOA
0.168 (0.002)
0.068 (0.002)
0.036 (0.002)
0.017 (0.002)
0.091 (0.006)
-0,038 (0.013)
Gradient SHAP
Label-Free
0.170 (0.003)
0.067 (0.001)
0.045 (0.002)
0.019 (0.002)
0.055 (0.013)
0.013 (0.006)
Contrustive Label-Free
0.174 (0.004)
0.064 (0.001)
0.046 (0.002)
0.018 (0.002)
0.056 (0.013)
0.011 (0.005)
Corpus
0.172 (0.003)
0.068 (0.001)
0.042 (0.002)
0.019 (0.002)
0.073 (0.012)
2.12c-04 2 5, 3.80c-03
COCOA
0.181 (0.003)
0.061 (0.001)
0.050 (0.002)
0.015 (0.002)
0.082 (0.010)
-0.014 (0.007)
RISE
Label-Free
0,180 (0.003)
0.087 (0.002)
0.061 (0.002)
0.024 (0.002)
0.053 (0.006)
7.95c-03 +: 4,51e-03
Contrustive Label-Free
0. 187 (0.004)
0.083 (0.002)
0.062 (0.002)
0.023 (0.002)
0.056 (0.006)
-3. -3.166-444 2.35c-03
Corpus
0.181 (0.003)
0.088 (0.002)
0.058 (0.002)
0.027 (0.002)
0.062 (0.006)
4. 4.62c-03 + 6.23c-03
COCOA
0.196 (0.004)
0.078 (0.002)
0.067 (0.002)
0,020 (0.002)
0.084 (0.066)
-0,028 (0.M07)
Random
0.115 (0.002)
0.114 (0.002)
0,028 (0.002)
0.029 (0.002)
0.038 (0.009)
0.039 (0.009)
24

---

Published as a conference paper at ICLR 2023
Table 6: Insertion and deletion metrics of contrastive corpus similarity when explicands do not belong to the corpus class. All explanation target functions are based on dot product instead of cosine similarity. Means (95% confidence intervals) across 5 experiment runs are reported. Higher insertion and lower deletion values indicate better performance, respectively. Each method is combination of feature attribution method and an explanation target function (e.g., COCOA under RISE corresponds to feature attributions computed by RISE for the contrastive corpus similarity). Performance results of random attributions (last row) are included as benchmarks.
Attribttion Method
Imagonet SimCER
CIFAR-10 SimSium
MURA RosNot
Insertion (1)
Deletion
Insertion
Deletion
Insertion
Deletion
Integrated Graadicuts
Lahel-Free
-0.112 (0001)
-0015 (0.001)
-4.86-03
1.36--33
-3,48e-011
1.48--03
.0,99 (0 (07)
0.017 (0.011)
Conrrstive Label-Free
-0.012
-0016 (0. 00.00)
-..1ee03
1.36-0
-3.49-003
1,40- 00
-0,037 (0.005)
0.015 (0.008)
Corpus
-7.55e033 # 1.288-03
-0018 10.0011
-2.55e-03
1.46-US
-3.201e113
1.41e-0s
0.033 10.008)
-0.027 (01007)
CDOOA
4.98c-13 1.51
-4028 (0.001)
1.39--03
1.90--13
-.656-133
1,54x-03
0.851 (0.006)
-1,0766 (0.007)
Gradient SHAP
Free
-9.41e-03 #: 1.33e-133
4006 (0.0011
-6.1le.33
1.40e-U5
-3,20le-133
1.62-00
-0,440 (0.0855
5.79g-113 = L.U2e-412
Controtive Label-Froe
-8.91-031+ + 1.35o-03
-0017 (0.001)
-6.33e003-
11376-03
-3.07-033
1 1.00-03
-0.836 (0 (07)
1.000-03 + 9. 9,376-03
Corpus
-..25 03 1.Me 03
0019 (0.001)
-1.79 03
1.600
6,33e-03
1,31e 00
0.019 (0.012)
0,050 (0.00))
COCOA
3.70c-113 1.776-45
40026 (0.001)
5.13--03
1.722-113
-0.012
0.032
-0,068 (0.005)
RISE
Free
33066 1.4de
4.03e
1.43e 0
66.33 03
1.37e-03
2.31e-03
1.55
(0.006)
0:019
Conmmstive Label-Fres
9.86--3
1.30--13
4.4 4.44c-113
1.56--03
-6.16e-33
13336-83
-2.226-413
1.56e-03
.0,55 (0.007)
-0.038 (8.002)
Corpus
8. 8.766-04 1646-03
-5.5 3.55e-03
1.4e-00
-1.02e-03
134--0
-.855-03
1,44-00
-0.018 (0.00))
-0.057 (0.005)
COCOA
8.57e-00 2.10c-03
40015
(0.001)
7.07e-03
1: 1.31e-03
-0.014 (0.001)
0.015 (0.006)
-0,085 (0.005)
Random
(0.01)
-0.13 (0.001)
-4. 10-00
1.56--03
-4.07-033
= 1.46c-03
-0.031 (0.007)
-0.021 (02))000))) (0.007)
Table 7: Insertion and deletion metrics of corpus majority probability when explicands belong to the corpus class. All explanation target functions are based on dot product instead of cosine sim- ilarity. Means (95% % confidence intervals) across 5 experiment runs are reported. Higher insertion and lower deletion values indicate better performance, respectively. Each method is combination of a feature attribution method and an explanation target function (c.g., COCOA under RISE corre- sponds feature attributions computed by RISE for the contrastive corpus similarity). Performance results of random attributions (last row) are included as benchmarks.
Attribution Method
Imagenet & SimCLR
CIFAR-10 & SimSSam
MURA & ResNet
Insertion (t)
Deletion (1)
Insertion (*)
Deletion (L)
Insertion (f)
Deletion ())
Integruted Gradients
Label- Freee-Free
0.364 (0.004)
0.125 (0.003)
0.343 (0.014)
0.237 (0.012)
0.696 (0.020)
0.451 (0.018)
Contrastive Label-Free
0.372 (0.005)
0.119 (0.003)
0.354 (0.017)
0.234 (0.012)
0.690 (0.022)
0.452 (0.029)
Corpus
0.384 (0.004)
0.126 (0.002)
0.346 (0.012)
0.240 (0.010)
0.767 (0.010)
0.392 (0.018)
COCOA
0.415 (0.006)
0.113 (0.002)
0.379 (0.012)
0,222 (0.009)
0.806 (0.012)
0.325 (0.031)
Gradient SHAP
Label-Free Contrastive Label-Free
0.408 (0.004)
0.130 (0.002)
0.475 (0.011)
0.235 (0.013)
0.699 (0.040)
0.515 (0.018)
0.413 (0.003)
0.126 (0.002)
0.487 (0.011)
0.229 (0.012)
0.700 (0.041)
0,510 (0.019)
Corpus
0.423 (0.004)
0.130 (0.002)
0.444 (0.011)
0.234 (0.008)
0.762 (0.035)
0.468 (0.016)
COCOA
0.444 (0.006)
0.122 (0.002)
0.502 (0.008)
0.207 (0.008)
0.789 (0.031)
0.417 (0.013)
RISF
Label-Free
0.414 (0.007)
0.145 (0.003)
0.627 (0.007)
0.288 (0.004)
0.724 (0.019)
0.562 (0.010)
Contrustive Label-Free
0.434 (0.008)
0.134 (0.004)
0.633 (0.008)
0,280 (0.005)
0.737 (0.021)
0.533 (0.011)
Corpus
0.417 (0.010)
0.145 (0.005)
0.592 (0.005)
0.315 (0.006)
0.765 (0.011)
0.542 (0.019)
COCOA
0.465 (0.009)
0.120 (0.004)
0.664 (0.007)
0.257 (0.006)
0.844 (0.006)
0.412 (0.025)
Random
0.269 (0.003)
0.268 (0.002)
0.329 (0.013)
0.329 (0.010)
0.624 (0.018)
0.629 (0.018)
25

---

Published as a conference paper at ICLR 2023
Table 8: Insertion and deletion metrics of corpus majority probability when explicands do not be- long to the corpus class. All explanation target functions are based on dot product instead of cosine similarity. Means (95% confidence intervals) across 5 experiment runs are reported. Higher insertion and lower deletion values indicate better performance, respectively. Each method is combination of feature attribution method and an explanation target function (e.g., COCOA under RISE corresponds to feature attributions computed by RISE for the contrastive corpus similarity). Performance results of random attributions (last row) are included as benchmarks.
Attribution Method
Imagenet & SimCI R
CIFAR-I 10 & SimSiam
MURA & ResNet
Insertion (f)
Delction (L)
Insertion
Deletion
Insertion
Deletion
Inlegrated Gradients
Label-Free
2.74c-04 9.69s-05
5.42o-04 = 1.12c-04
0.066 (0.003)
0.078 (0.004)
0.352 (0.024)
0.522 (0.029)
Contrastive Label-Free
2.75e-04 + 1.00=-04
5.00e-04
1.07-04
0.064 (0.004)
0.078 (0.004)
0.347 (0.024)
0.529 (0.023)
Corpus
5.60e-04
2.54s-04
4.13c-04 = 5000--5
0.079 (0.004)
0.070 (0.004)
0.529 (0.018)
0.381 (0.026)
COCOA
1,58e-03 4.99c-04
1.64-444
2.97e-05
0.100 (0.0005
0.060 (0.005)
0.657 (0.017)
0.210 (0.0244
Gradient SHAP
Label-Free
1.88c-04 + 4.255-00
6.01c-04 = 1.13c-04
0.055 (0.005)
0.082 (0.006)
0.343 (0:016)
0,488 (0.020)
Contrastive Label-Free
2.03e-04 + 3.73e-05
5. 5.30c-04 + 7, 7,78e-05
0.053 (0. (0.004)
0.082 (0. (0.005)
0.353 (0.019)
0.478 (0.01 9)
Corpus
4.31c-04
L.08-004
4.56c-04 = 9.36c-05
0,081 (0:008)
0.065 (0.006)
0.548 (0:036)
0.303 (0.012)
COCOA
1.43e-03
3.30e-04
1.70c-04 6.64e-05
0.119 (0.006)
0.047 (0.002)
0.592 (0.025)
0.234 (0.014)
RISE
Label-Fiee
3.64c-04 + 8.02c-05
5.40c-04 = 1.11c-04
0.040 (0.003)
0.079 (0.006)
0.311 (0.022)
0,450 (0.033)
Contrastive Label-Free
3,23c-04
4.88-00
6.25c-04
1.01c-04
0.040 (0.003)
0.080 (0.005)
0,303 (0.026)
0,466 (0.024)
Corpus
5.19e-04 + 1.06e-04
5.69c-04
2.08e 04
0.070 (0.004)
0.056 (0.003)
0.452 (0.029)
0.294 (0.020)
COCOA
8.65c-04
1.57-004
3.06e-04
5.43e-05
0.105 (0.005)
0.032 (0.002)
0.593 (0.028)
0.177 (0.016)
Random
4.87c-04 9.50e-05
5.03e-04 +: 9.73e-05
0,070 (0.003)
0,070 (0.004)
0.406 (0.013)
0.407 (0.016)
N QUANTITATIVE EVALUATION RESULTS WITH MODEL PARAMETER RANDOMIZATION.
has been shown that feature attribution methods, especially those based on gradients, can be undesirably invariant model parameter randomization, suggesting that the attributions are not related to the trained models (Adebayo et al. 2018). We performed additional experiments to show that our empirical evaluation results do not suffer from this issue. We generated COCOA scores with randomized models, where the parameters are randomized by sampling from truncated normal distributions (as in [Adebayo et al. (2018)). and then evaluated the performance of these COCOA scores. As shown in Tables 92 the performance results of COCOA with randomized models are different from and worse than the performance of COCOA with trained models. These results suggest that COCOA performance is indeed sensitive to model parameter randomization and is related to model parameters.
Table 9: COCOA Insertion and deletion metrics of contrastive corpus similarity when explicands belong to the corpus class. Model parameters of randomized models were randomly sampled from truncated normal distributions. Means (95% confidence intervals) across 5 experiment runs are reported. Higher insertion and lower deletion values indicate better performance, respectively, Per- formance results of random attributions (last row) are included as benchmarks.
Attribution Method
Imagenet& & SimCLR
CIFAR-1O SimSiam
MURA ResNet
Insertion (t)
Deletion (1)
Insertion (?) Deletion (L)
Insertion (1) Deletion (L)
Integrated Gradients
COCOA (trained model)
0.172 (0.002)
0.067 (0.002)
0.037 (0.0022
0.018 (0.0M2)
0.091 (0.006)
-0.036 (0.013)
COCOA (randomized model)
0.148 (0.003)
0.147 (0.002)
0.025 (0.003)
0.026 (0.003)
0.040 (0.009)
0.038 (0.010)
Gradient SHAP
COCOA (trained model)
0,181 (0.004)
0.062 (0.001)
0.051 (0.0022)
0.015 (0.IM12)
0.081 (0.010)
-0.014 (0.007)
COCOA (randomized model)
148 (0.003)
0.147 (0.002)
0.026 (0.002)
0.026 (0.003)
0.041 (0.008)
0.039 (0.01 1)
RISE
COCOA (truined model)
0.195 (0.003)
0.080 (0.002)
0.068 (0.002)
0.019 (0.042)
0.083 (0.006)
-0,028 (0.008)
COCOA (randomized model)
0. (0.004)
0.149 (0.003)
0.042 (0.002)
0.041 (0.002)
0.033 (0.006)
0.036 (0.005)
Random
0.115 (0.002)
0.114 (0.002)
0.028 (0.002)
0.029 (0.002)
0,038 (0.009)
0.039 (0.009)
26

---

Published as a conference paper at ICLR 2023
Table 10: COCOA Insertion and deletion metrics of contrastive corpus similarity when explicands do not belong to the corpus class. Model parameters of randomized models were randomly sampled from truncated normal distributions. Means (95% confidence intervals) across experiment runs are reported. Higher insertion and lower deletion values indicate better performance, respectively. Performance results of random attributions (last row) are included as benchmarks.
Attiluttoon Method
Imageeett SinCLR
CIFAR-10 SimSica
MURA ResNet
Insertion (f)
Delction (1)
Insetiion
Deletion
Insertion
Deletion (J)
Integrated Gradients
(trined model)
5.24c-03 1.50e-03
4.028 (9,001)
147e00
1.76-00
-8.79-003
L.5e-03
0,48 (0,006)
-0,07 (0,009)
COCOA (andomized model)
-1.986-03 1446-633
5456--1
1.62-48
-400c-03
1646-03
-3.95-00
1.70--08
.0.000 (0.010)
-0.013 (0.00)
Gradiest SHAP
COCOA (trained model)
1.76e-03
4,027 (0.001)
5.66-033 E 202--03
0.012 (0.02))
0.032 (0.010)
-0.667 (0J055)
CDOOA (mndomized model)
-1 -1,900-03 + 1.376-63
4 4.422-01
1,75-403
-3.91--33
1 1.66-88
-3822-03
1.7e-00
-0.014 (0.089)
-0.017 (0,09)
RISF:
COCOA (trained model)
8.36c-03 #: 2.17e-035
.0015 (0,001)
7.19c-03 + 143e41
00014 (0,001)
0,015 (0.007)
-0.089 (0J003)
COCOA (aaddomized model)
-2.08--03 + 1436-83
4 4.422-03
1.64--B3
-3.576-03
1 40c-03
4.6c-033
1616-03
-0.040 (0.008
-0.036 (0,005
Random
(0.01))0.00)
-00133 (0.001)
-4. -4,10--33 59c-03
-4 0776-00 1.46--03
-0.021 (00007)
-0.021 (0.007)
Table 11: COCOA Insertion and deletion metrics of corpus majority probability when explicands belong to the corpus class. Model parameters of randomized models were randomly sampled from truncated normal distributions. Means (95% confidence intervals) across 5 experiment runs are
reported. Higher insertion and lower deletion values indicate better performance, respectively. Per- formance results of random attributions (last row) are included as benchmarks.
Attribution Method
Imagenet & SimCLR
CIFAR- 10 & SimSiam
MURA & ResNet
Insertion (1)
Deletion ())
Insertion (f)
Delction (L)
Insertion (f)
Deletion (.)
Integrated Gradients
COCOA (trained model)
0.422 (0.006)
0.119 (0.003)
0.386 (0.012)
0.230 (0.011)
0.807 (0.013)
0.330 (0.030)
COCOA (randomized model)
0.321 (0.005)
0.317 (0.007)
0.299 (0.011)
0.303 (0.009)
0.603 (0.017)
0.598 (0.019)
Gradient SHAP
COCOA (trained model)
0.445 (0.003)
0.123 (0.002)
0.508 (0,007)
0.211 (0,008)
0.788 (0.030)
0.419 (0.013)
COCOA (randomized model)
0.321 (0.005)
0.315 (0.007)
0.299 (0.008)
0.305 (0.009)
0.622 (0.021)
0.613 (0.019)
RISE
COCOA (trained model)
0.456 (0.009)
0.126 (0.001)
0.663 (0.0066
0.256 (0.006)
0.840 (0.049)
0.415 (0.025)
COCOA (candomized model)
0,321 (0.008)
0.322 (0.00S)
0.455 (0.008)
0.433 (0.015)
0.649 (0.025)
0.662 (0.007)
Random
0.269 (0.0003 0.268 (0.002)
0.329 (0.013)
0.329 (0.010)
0.624 (0.018)
0.629 (0.018)
Table 12: COCOA Insertion and deletion metrics of corpus majority probability when explicands do not belong to the corpus class. Model parameters of randomized models were randomly sampled from truncated normal distributions. Means (95% confidence intervals) across experiment runs are reported. Higher insertion and lower deletion values indicate better performance, respectively. Performance results of random attributions (last row) are included as benchmarks.
Attribution Method
Imagenet& & SimCLR
CIFAR-10 SimSiam
MURA & ResNet
Insertion (f)
Deletion (L)
Insertion (1)
Deletion (.)
Insertion (?)
Deletion (L)
Integrated Gradients
COCOA (trained model)
1.69c-03 5.07e-04
1.55e-04
2.21e-05
0.099 (0.004)
0.059 (0.005)
0.647 (0.017)
0.213 (0.030)
COCOA (randomized model)
5.52e-04
9.43e-05
4.54c 1444e-44...
1 3.29e-05 05
0.072 (0.004)
0.072 (0.005)
0.434 (0.018)
0.424 (0.006)
Gradient SHAP
COCOA (trained model)
1.56e-03 4.32e-04
1.67e-14 5.666-05
0.122 (0.008)
0.045 (0.003)
0.592 (0.025)
0.236 (0.012)
COCOA (randomized model)
5.38c-04
9.44e-05
0.58c-04 + 4.48c-05
0.071 (0.004)
0.073 (0.005)
01.427 (0.022)
0.415 (0.016)
RISE
COCOA (trained model)
9.51e-04 + 2.58c-04
3.60k-0M
1.40e-04
0.107 (0.007)
0.031 (0.002)
0.590 (0.027)
0 (0.014)
COCOA (raddoizzd model)
5.30<-04
1.14c-04
4,59c-04
4.66c-05
0.070 (0.003)
0.057 (0.004)
0.364 (0.035)
0.375 (0.015)
Random
4.87c-04 + 9,50c-05
5.03e-04 = 9.73c-05
0.070 (0.003)
0.070 (0.004)
0.406 (0.013)
0.407 (0.016)
27

---

Published as a conference paper at ICLR 2023
O ADDITIONAL RESULTS FOR UNDERSTANDING DATA AUGMENTATIONS IN SIMCLR
Original
Flip
Crop
Flip Crop
Grayscale
Color Jitter
Cutout
Rotation
Prediction::
Prediction
Predicioon:
Prediction:
Prediction
Pred cilor: spetiger
Preciction: acrtooer
Prediciion parer
English spstiper
Enilken
springe
English anttaer
Ceocten sprtiwer
Predictions Eoglish serrggr
Prediction: Enggish apringer
Pred clion:
Prediction:: Erglish springgr
Prediction trighan sprippr
Pred clae:: gragta sprrigrr
Preckesin:: Eng sortogur
Prediction:
ragar
Predictions Engiiih ppr nger
Prediction: Englinh serroer
Predicion: Enn apringer
Brediction: Exglih spenger
Predict Englin sotinger
Prediction Engiih spriggr
Prediction: English springer
Predicion:
Figure 8: Original version and augmentations of English springer images with their class predictions (top rows), along with the corresponding COCOA attributions (red for higher values and blue for lower values) with each original image as the corpus and random images as the foil (bottom rows). Cutout and rotation are not included in SimCLR training.
28

---

Published as a conference paper at ICLR 2023
Original
Flip
Crop
Flip Crop
Grayscale
Color Jitter
Cutout
Rotation
Predictin: merch hom
Prediction French hom
Reach homm
Feench From
French nom
Fitech hom
notebook
Frach hem
French nom
French horm
Frewen hom
foench homn
French hrom
Trombree
bassonn
connet
Fredittinn Freec homm
Repdiction: Fieech hom
Prosict Neetch hom
Predictoon meech hom
Prediction suding chair
Prediction Frechh nom
Fronch foom
French frm
Figure 9: Original version and augmentations of French horn images with their class predictions (top rows), along with the corresponding COCOA attributions (red for higher values and blue for lower values) with each original image as the corpus and random images as the foil (bottom rows). Cutout and rotation not included in SimCLR training.
29

---

Published as a conference paper at ICLR 2023
Original
Flip
Crop
Flip Crop
Grayscale
Color Jitter
Cutout
Rotation
Predicton:
Prefiction
Prediction:
Predicion:
Prediciion
Predittion: parsrtede
Prediction:
Fredettion:
parerity
parartura
parachite
Predetion:
Predicion:
Predict on carachute
Prediction:
Pred clor
parachute
parachute
Pradiction:
Prediction:
Pred ctlon
Prediction:
Predictioss
Prodictin:
Prediction:
Prediction:
Figure 10: Original version and augmentations of parachute images with their class predictions (top rows), along with the corresponding COCOA attributions (red for higher values and blue for lower values) with each original image as the corpus and random images as the foil (bottom rows). Cutout and rotation not included in SimCLR training.
30


Reviews:
[{"review_id": "otJM4DlD9F", "reviewer": "Reviewer_9VMz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This study indicates contrastive document similarity as a scalar explanation output for unsupervised methods. A corpora of samples collected and foil are used in the process (negative samples or distribution). It calculates the average cosine similarity between an input's encoded representation and those in the corpus, subtracting the average similarity with foil representations, for the given input. The suggested method can find pertinent input features when used in conjunction with current feature attribution methods.", "summary_of_the_review": "The article proposes a simple, efficacious, and potentially useful method for generating attribute inferences for unsupervised models that really are different from each other.", "strength_and_weaknesses": "Strength\n1. The approach is simple, and appears to work well, as demonstrated by the excellent results.\n2. The figures hopefully demonstrate how the method might be employed and what sort of impacts it could have.\n\nLimitation\n1. More often than not, the author to labeled the data to determine what the corpus and foil are.\n2. If the feature representations of the dataset are heterogeneous in nature (high commonality with some corpus examples and low similarity with others), the proposed model might start giving a scoring rate that is excessively limited.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "egVeoyFtZNO", "reviewer": "Reviewer_Y75o", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes contrastive corpus similarity as a scalar explanation output for unsupervised models. The method relies on a corpus (positive samples) and foil (negative samples or distribution). For a given input, it computes the average cosine similarity between its encoded representation and those of the corpus and subtracts the average similarity with the foil representations. Combined with existing feature attribution methods, the proposed approach can identify relevant input features.", "summary_of_the_review": "The paper proposes a simple, effective and potentially impactful method to generate contrastive feature attributions for unsupervised models.", "strength_and_weaknesses": "Strengths\n\nThe approach is quite simple, yet effective as demonstrated by the strong results.\n\nThe figures clearly help demonstrate the potential impact and applications of the approach.\n\nWeaknesses\n\nTo identify the corpus and foil, it is sometimes necessary to have some labeled data.\n\nIt is unclear why the results do not include the performance of RELAX.\n\nIf the feature representations of the corpus are multimodal (i.e. you get high similarity with some corpus examples, but low similarity with others), the proposed method might arguably assign too low of a score.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EVEO9ib2TVj", "reviewer": "Reviewer_chxC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for post-hoc explanations of unsupervised methods. At the heart of this proposed idea is the *contrastive corpus similarity* which employs a corpus set of samples (e.g. abnormal cases in an illustrative task of classifying bone abnormality) and a foil set of samples (e.g. normal cases for the same task). The proposed method uses the notion of contrastive corpus similarity to generate Contrastive Corpus Attributions (COCOA). COCOA works by identifying the features that, when removed, make the representation of an explicand (a sample to be explained) similar to the foil set and dissimilar to the corpus set (hence the term `contrastive' in the name). \n\nThe authors evaluate COCOA to establish whether the identified features are indeed related to corpus and foil samples using different models (SimCLR, SimSiam etc.), datasets, and feature attribution methods. They also demonstrate the utility towards understanding image data augmentation and mixed modality object localization. ", "summary_of_the_review": "The proposed idea of generating explanations for unsupervised methods is simple and elegant. Incorporating the notion of contrastive changes to the representation (towards a foil cluster and away from a corpus cluster) in response to changes in the feature space is interesting. The writing is clear and demonstrates notational discipline. \n\nHowever, my issues with this paper is that it fails in contextualizing how the proposed idea and empirical evaluation do not suffer from well-established issues with the underlying attribution methods used in buildding the empirical evidence in support of the proposed method. (Especially because the paper seems aware of the discourse in the \"The Mythos of Model Interpretability\" paper). I have added other references to help the authors address this issue. Nevertheless, this prevents me from recommending a higher pre-rebuttal score. ", "strength_and_weaknesses": "### Strengths\n\n1. Writing and notational discipline: The idea is clearly described, and the notations introduced are well-explained. I cannot find any errors in the notations. \n\n2. Simplicity: The idea of using contrastive similarity towards a foil set and away from a corpus set is simple and elegant. \n\n3. Extensive experiments: The experiments are comprehensive, or as extensive as can be expected within one paper. The sanity checking in the Appendix is also appreciated. \n\n### Weaknesses\n\n1. Conceptual Concerns and situating within the broader concerns surrounding XAI (Explainable AI)\n\nThe paper early-on cites the paper on \"The Mythos of Model Interpretability\" by Lipton to effectively state that deep learning methods can be black boxes. However, I believe it fails to contextualize the proposed method within the larger discourse on XAI methods established in that paper. Specifically, the use of attribution maps (or saliency maps or heat maps as alternatively known) has been shown to suffer from several issues. As Barredo Arrieta et al. point out, “there is absolutely no consistency behind what is known as saliency maps, salient masks, heatmaps, neuron activations, attribution, and other approaches alike” [R1, Sec. 5.3]. Also see [R2, R3] for a discussion on the issues with saliency/attribution maps. \n\nThe experiments still uses these and other SHAP based methods known to have issues to support their argument. (See [R4] for links to other papers discussing how SHAP can produce inconsistent and unstable explanations). \n\nI'd like to hear the authors' thoughts on why, broadly, their empirical argument which is built upon these methods will not suffer from similar issues. I don't see in the experiments how the proposed method would implicitly be robust to these issues surrounding instability since the method is inherently dependent on having a well-separated concept of foil and corpus samples (or at least that these samples need to already have a human understandable semantic meaning which is subject to biases in the human perception). \n\nOne of the core issues of the current state of XAI is how the methods largely depend on biases and interpretation of the observer, and it would be good to establish how this method is not adding to the same issue. \n\n\n[R1] Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. - Barredo Arrieta et al. \n[R2]  Sanity checks for saliency maps. - Adebayo et al. \n[R3] Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning - Atrey et al. \n[R4] The Disagreement Problem in Explainable Machine Learning: A Practitioner’s Perspective - Krishna et al. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster