Measuring the Interpretability of Unsupervised Representations via Quantized Reversed Probing
Authors: Iro Laina
Yuki M Asano
Andrea Vedaldi
Venue: ICLR 2022 Poster (2022)
Keywords: Representation learning
Computer vision
Interpretability

Abstract: Self-supervised visual representation learning has recently attracted significant research interest. While a common way to evaluate self-supervised representations is through transfer to various downstream tasks, we instead investigate the problem of measuring their interpretability, i.e. understanding the semantics encoded in raw representations. We formulate the latter as estimating the mutual information between the representation and a space of manually labelled concepts. To quantify this we introduce a decoding bottleneck: information must be captured by simple predictors, mapping concepts to clusters in representation space. This approach, which we call reverse linear probing, provides a single number sensitive to the semanticity of the representation. This measure is also able to detect when the representation contains combinations of concepts (e.g., "red apple'') instead of just individual attributes ("red'' and "apple'' independently). Finally, we propose to use supervised classifiers to automatically label large datasets in order to enrich the space of concepts used for probing. We use our method to evaluate a large number of self-supervised representations, ranking them by interpretability, highlight the differences that emerge compared to the standard evaluation with linear probes and discuss several qualitative insights. Code at: https://github.com/iro-cp/ssl-qrp.

Published as a conference paper at ICLR 2022
MEASURING THE INTERPRETABILITY OF UNSUPER- VISED REPRESENTATIONS VIA QUANTIZED REVERSE PROBING
Iro Laina University of Oxford
Yuki M. Asano
Andrea Vedaldi
University of Amsterdam y n. aasanonuuva...
University of Oxford
i iro. Lainafeng. ox ac. uk
veda @edaldi@robot. ox ac uk
ABSTRACT
Self-supervised visual representation learning has recently attracted significant re- search interest. While a common way to evaluate self-supervised representations is through transfer to various downstream tasks, we instead investigate the prob- lem of measuring their interpretability, i.e. understanding the semantics encoded raw representations. We formulate the latter as estimating the mutual information between the representation and a space manually labelled concepts. To quantify this introduce a decoding bottleneck: information must be captured by simple predictors, mapping concepts to clusters in representation space. This approach, which call reverse linear probing, provides a single number sensitive the se- manticity of the representation. This measure is also able to detect when the rep- resentation contains combinations of concepts (e.g., "red apple") instead of just individual attributes ("red" and "apple" independently). Finally, we propose to use supervised classifiers to automatically label large datasets in order enrich the space of concepts used for probing. We use our method to evaluate large number of self-supervised representations, ranking them by interpretability. highlight the differences that emerge compared to the standard evaluation with linear probes and discuss several qualitative insights. Code at: https: /ttps://ghn httpps//githubs com/ firo op/ I-1p-...
INTRODUCTION
Relying on black-box models such as deep networks comes sometimes with significant methodolog- ical and ethical challenges. This is particularly true for unsupervised and self-supervised models which are learned without human supervision. While these models perform increasingly well in downstream applications, often outperforming supervised counterparts, there is very little under- standing of what they learn, making their real-life deployment risky.
In this paper, thus consider the problem of characterizing the meaning of data representations, with a particular focus on unsupervised and self-supervised representations of images. Given a rep- resentation mapping images to representation vectors [(x) E RD, our goal is to find whether these vectors contain human-interpretable information. This is usually done by finding a relationship correspondence between f(x) and human-provinn provided descriptions y(x:) of the images, essentially translating the information from the representation space to a concept space. Consider for example the popular linear probing method (Alain and Bengio, 2017). Given large dataset I' of images with corresponding manual annotations y(z), one learns linear classifiers (probes) to map the fea- ture vectors (xx to labels y(r), measuring the resulting classification accuracy. If the predictor is accurate, then one can argue that the representation captures the corresponding concept y.
One possible issue with this type of approaches is that one can only discover in the representation concepts that are represented the available data annotations. In order to maximize the semantic coverage of the analysis, it is thus customary to combine annotations for several types of attributes, such as object classes, textures, colour, etc. (Bau et al., 2017). this case, is a vector of image attributes, and one compute several linear probes predict each individual attribute И4 By doing so, however, attributes are treated independently, which may be unsatisfactory. order to obtain single number assessing the overall semanticity of representation, one must combine

---

Published as a conference paper at ICLR 2022
the prediction accuracies of several independent classifiers, and there is no natural way of doing so (e.g.. a simple average would not account for the different complexity of the different prediction tasks). Furthermore, the representation might be predictive of combinations of attributes, i.e. it might understand the concept of "red apple" without necessarily understanding the individual concepts, "red" and "apple". While it is in principle possible test any combination of attributes via linear probing, most attribute combinations are too rare to generate significant statistics for this analysis.
We propose a complementary assessment strategy to address these shortcomings. In contrast to linear probes, we start by considering the reverse prediction problem, mapping label vectors y(r) to representation vectors f(x). A key advantage that the entire attribute vector is used for this mapping, which, as we show later, accounts for attribute combinations more effectively.
Next, we consider the challenge of deriving a quantity that allows to compare representations based on the performance of these reverse predictors. Obviously a simple metric such as the average Le prediction error is meaningless as its magnitude would be affected by irrelevant factors such as the scale of the representation vectors. To solve this problem in principled manner, we consider instead the mutual information between the concepts and quantized representation vectors. This approach, which we justify from the viewpoint of information theory, essentially measures whether the representation groups the data in a human- interan-tterre manner.
We use our approach to evaluate a large number of self- supervveed and supervised image represen- tations. Importantly, we show that (a) all methods capture interpretable concepts that extend well beyond the underlying label distribution (e.g., ImageNet labels, which are not used for training). (b) while some clusters that form in the representation space are purely semantic (object-centric), others carry information about scenery, material, textures, or combined concepts, and (c) context matters. We also show that more performant methods recover more of the original label distribution, i.e. learn better "ImageNet concepts", and rely less on lower-level concepts. Quantitatively, we observe that our interpretability measure results in a similar but not identical ranking for state-of-the-art methods with clustering- based approaches generally producing more interpretable representations.
2 RELATED WORK
Self-supervised representation learning (SSL) In this paper, we focus on self-supervised meth- ods that learn representations from image data. Early sulf-super learning approaches devise nu- merous pretext tasks - colorization, predicting image rotations or solving image puzzles- - to learn useful representations from unlabeled images (Gidaris et al., 2018; Pathak al., 2016; Noroozi and Favaro, 2016; Doersch et al., 2015; Larsson et al., 2016; Zhang et al., 2017; 2016). More recent approaches follow the contrastive learning paradigm (Chen et al., 2020a: Frankle et al., 2020; He et al., 2020; Chen et al., 2020b; Hénaff et al., 2019; Oord et al., 2018; Misra and Maaten, 2020; Tian al., 2019; 2020; Wu et al., 2018), where the goal is to discriminate instances belonging to the same image from negative samples. Later methods take closer look at the effect of negatives samples (Mitrovic et al., 2020; Robinson et al., 2020; Chuang et al., 2020; Kalantidis et al., 2020) or eliminate the need for negatives altogether (Zbontar et al., 2021; Chen and He, 2020; Grill et al., 2020; Caron et al., 2021), while others consider nearest neighbors (Dwibedi et al., 2021; Assran et al., 2021). Although several studies aim to explain why constrastive learning works (Arora et al., 2019; Wang and Isola, 2020; Tschannen et al., 2020; Tsai et al., 2020; Purushwalkam and Gupta, 2020), in most cases, the focus lies on measuring empirical improvements in downstream tasks.
Clustering A related part of literature deals with unsupervised image clustering. Early approaches include using autoencoders (Hinton and Salakhutdinov, 2006), agglomerative clustering (Bautista et al., 2016) and partially ordered sets of hand-crafted features (Bautista et al., 2017). More recent methods combine representation learning with clustering, using mutual information (Ji et al., 2019; Hu et al., 2017), K-means (Caron et al., 2018; Zhan et al., 2020), prototypes (Li et al., 2021) or optimal transport (Asano et al., 2020; Caron et al., 2020). Other methods build on strong feature representations and, at second stage, cluster and further refine the network (Yan et al., 2020; Van Gansbeke et al., 2020; Lee et al., 2019). These methods produce pseudo-labels which can be used to evaluate representations by measuring the correlation to the ground-truth labels.
Evaluation and analysis of SSL There are two main directions in evaluating self-supervised learning methods. The first one uses pre-trained representations as initialization for subsequent
2

---

Published as a conference paper at ICLR 2022
supervised tasks, e.g., for object detection. The second approach is to train linear classifier on the representation space which is kept frozen after pre-training. Some concerns have been voiced re- garding the generality of ImageNet linear classification accuracy the main metric to evaluate rep- resentations (Kotar et al., 2021) and, as a result, several benchmarking suites with various datasets and tasks have been proposed (Zhai et al., 2019; Goyal al., 2019; Van Horn et al., 2021; Kotar et al., 2021). Complementary to this, clustering in the frozen feature space has also been proposed as an evaluation metric (Zheltonozhskii et al., 2020; Sariyildiz et al., 2020). In addition, downstream dataset (Ericsson et al., 2021) and pretraining dataset (Zhao et al., 2021; Cole et al., 2021) depen- dencies have been evaluated. Finally, recent investigations of self-supervised feature spaces aim to understand separability (Sehwag et al., 2020), concept generalization (Sariyildiz et al., 2020) and the effect of balanced vs. long-tailed training data (Kang et al., 2020).
Interpretability Although a large amount of work has studied feature representations in Convo- lutional Neural Networks (CNNs), it is heavily focused on models trained with full supervision. Zeiler and Fergus (2014) and Zhou et al. (2014) analyze networks by visualizing the most activating patches, while activation maximization methods (Mahendran and Vedaldi, 2016;;;; Nguyen et al., 2017; 2016; Olah et al., 2017; Simonyan et al., 2014) generate inputs to activate specific neurons.
Another line of work focuses on understanding what information is present in intermediate repre- sentations of CNNs, usually mapping activations to high-level concepts. Alain and Bengio (2017) introduce linear probes as a means understanding the dynamics of intermediate layers. by pre- dicting the target labels from these layers. Escorcia et al. (2015) also use linear formulation to study the relationship of mid-level features and visual attributes, while Oramas al. (2019) adopt this to predict task-specific classes from the features (similar to Alain and Bengio (2017)), select relevant features and generate a visual explanation. Similarly, Zhou et al. (2018) decompose feature vectors into set of elementary and interpretable components and show decomposed Grad-CAM heat maps (Selvaraju et al., 2017) for concepts that contribute to the prediction. Furthermore, Kim et al. (2018) relate feature vectors and human-interpretee amples for each concept and Bau et al. (2017) propose to quantify the interpretability of individual units by measuring the overlap between each unit and set of densely annotated concepts, Finally, Ghorbani et al. (2019) propose a method to automatically assign concepts to image segments and Yeh al. (2020) analyze the completeness of these concepts for explaining the CNN.
concepts using set of user-puu suer--rovdded ex-
With the exception of (Laina et al., 2020; Bau et al., 2017; Fong and Vedaldi, 2018), relatively little work has been done to understand the emergence of interpretable visual concepts in self-supervised representations specifically. In particular, Laina et al. (2020) quantify the learnability and describa- bility of unsupervised image groupings, by measuring how successful humans are at understanding the learned concept from set of provided examples. Instead, our approach does not depend on human input and is thus easily scalable to wide range of methods and number of classes.
3 METHOD
We interested in measuring the semanticity of data representations. A representation f is map f:X + RRD that encodes data samples x E X' as vectors f(x) E P. In this paper, we assume that the data are images X a and f is deep neural network, but other choices are possible.
The semantic content of an image x can be summarized by obtaining a label or description v(x) E y of the image from a human annotator. If the representation f(z) captures the meaning of the image well, then it should be predictive of the description y(z). The mutual information [f v(z)) provides a natural measure of predictivity, and thus tempting to use this quantity as a measure of the overall semanticity of the representation. Note that, due to the data processing inequality (T()(z), y()) & T(x, v()))), the information is maximized by observing the raw image, i.e. by the identity representation f(z) useful representation must preserve information while also making it easier to decode and act on it.
Since data processing cannot increase information content,
There are many possible definitions of what constitutes "easy decoding". Common in literature is the transfer through a simple predictor (e.g., linear) to new tasks. Here we propose a definition that stems from the interpretation of differential entropy as the limit of discrete entropy for quantized versions of the corresponding variables (Cover and Thomas, 2006). In our case, the description y(ø) is already discrete, but the representation vectors are continuous. We thus propose discretize (vector-
3

---

Published as a conference paper at ICLR 2022
Feature extraction and quantization
Concept extraction
Reverse linear probing
Expert
Frozen Encoder
Mx
Expert
ha
Expert
Image collection
Image collection
Overview of our approach. (I) we evaluate pre-trained SSL models on an image collec- tion, extracting and quantizing feature vectors to obtain clusters (=mm--") (2) we label the image data with a diverse set of concepts ( (=иава (BB) from expert models trained with supervision on
Figure I:
external data sources, and (3) we train a linear model hig to map concepts to clusters, measuring the mutual information between the representation and human-interpretable concepts.
quantize) the representation and to use its information as a measure of semanticity. Intuitively, in the limit of infinitely-fine quantization, this quantity reduces (up to a normalizing term) the mutual information above. For finite number of clusters, however, a large value of information means that the representation groups images in a way that makes sense to a human observer. our case, the quantization amounts to grouping images based on the -is-dintance between their representation vectors, i.e. clustering.
Formally, we use vector quantization algorithm such K-means (Lloyd, 1982) to learn quan- .......,,
tizer function μ.K : DR + {1, (1....,Ks K} for the representation vectors, and estimate the mutual information (px(f(r)), by rewriting it as:
(1) In the above equation, the first term denotes the entropy of the cluster assignments. In practice given a sample dataset A' of images, we first run K-means to compute the quantizer μ.K. and then compute the frequency cluster assignments fK fx(x), x X' calculate the entropy. The second term in Eq. (1) is the conditional entropy:
((fk(r),
(x)x - H(fx(z) ((x)), fx(x)=xx(f{)
HS (I:) v(*)) En(-Inp(f((e y(x))) S Ep[- (fx(())) v(x))),
where p is the true joint distribution between variables Sx (x) and y(x).. While this is difficult to compute, we can upper-bound it by considering an auxiliary posterior distribution & interpreting the conditional entropy as the cross-entropy loss of predictor hg y K, parametrized by 0 that maps labels y(x) to clusters fK (I). The gap can be minimized by learning the parameters
In practice, learning the predictor may result in overfitting, which would lead to over-optimistic estimate the mutual information. We address this issue by learning the predictor he on a subset L' C X' of the data and evaluating its cross-entropy loss on the remaining subset X' - X.. Importantly, we consider a linear predictor (probe) for he. to further reduce the risk of overtitting. We refer to this as quantized reverse probing, as it maps semantic concepts to quantized data representations.
Number of clusters The number of clusters K controls the size of the bottleneck. Mutual in- increases, we show the Appendix (Fig. 6); when comparing representations, it is thus important to fix a value of K and use it consistently. For added convenience, in practice we report a normalised value of information (NMI).
formation increases monotonically
Obtaining labelled data via automatic experts Similar prior work on interpretation, our method requires dataset x E X' of images equipped with manually-provided labels y(x) E y Because we do not know a-priori which concepts may be captured by given representation f(x) under analysis, these labels should provide good coverage for number of different concepts (e.g., textures, object types, parts, etc.). A good example of such dataset is Broden (Bau et al., 2017).
Because it would be cumbersome, and maybe even infeasible, to annotate any target dataset with all present visual concepts, a cost-effective way to increase the coverage of the label space is to predict the labels y()) automatically via battery of expert classifiers learned using full supervision. The noise these automated predictions is small compared to the noise in the correlation between the visual representations under study.
concepts and the unsupervised/seff-superee
4

---

Published as a conference paper at ICLR 2022
40
50
60
44
65
(a) All attributes are linearly separable (Acc[/t] Acc[/a]= = 100%).
(b) Color attribute not linearly separable (Ace[hı]= 50%, Acc[/a] = 100%).
Figure 2: Forward vs. reverse probing. dataset X' embedded IR? by two different represen- tations, with meaningful clusters in representation space. Each data point has two binary attributes, color y1 (œ): red or blue and shape (=())): or O. Both representations separate these attributes; however, color is not linearly separable in (b), forward linear probing cannot recover this rela- tionship. Decision boundaries are shown for the forward linear probes. On the contrary, our reverse probing easily discovers that all combinations of shape and color map well separated clusters.
Relation to linear probing Linear probing is a common approach for assessing representations. A linear probe is simple predictor h2 f(x (x)) that maps the representation f() to specific concept yi (e.g., a binary attribute or a class label). The idea is that the representation captures concept yi if the probe performs well. As result, linear probing has become standard evaluation protocol in self-supervised representation to measure the (linear) classification accuracy of the learned representations against a labelled set, commonly ImageNet (Russakovsky et al., 2015). In interpretability studies, different formulations of linear probing (Alain and Bengio, 2017) are used to understand intermediate layers of neural networks. Similar to our approach, the simplicity of the probes (imposed via linearity and sometimes sparsity) is necessary bottleneck for interpretability: without such constraint, the best possible probe would take the raw image as input.
Our method is complementary to linear probes, with some advantages. To discuss this point further, we show an example in Fig. 2, where data points with color and shape attributes are encoded by two different representations, such that they form well-defined nameable clusters, i.e. blue square, red square, blue circle, and red circle. Forward linear probes (X' ()) can be trained as binary classifiers h1, h2 for each attribute, mapping feature vector (in this case, coordinates) to the corresponding attribute value for each data point. If attributes are linearly separable (Fig. 2a), then forward probes do well at separating the representation space. If an attribute is not linearly separable (color in Fig. 2b), the predictive accuracy of the corresponding probe h, reduces to chance, and consequently reduces the average score. As a result, linear probes cannot always assess the meaningfulness of clusters or, in other words, whether clusters in representation space respond to well-defined concepts. On the contrary, if all data points achieves a high score for, e.g., "red square" without requiring the concept "red" to be also encoded in the features. In this case, we can say that the model has discovered the concept of "red square"' without identifying, in isolation, the concept "red". Therefore, reverse probing handles combinations of attributes naturally and allows to better assess the semanticity of clusters.
cluster have consistent attributes, the reverse probe ()' + X))
EXPERIMENTS
We use our method to evaluate a wide range of recent self-supervised representation learning and clustering techniques. In the following, we compare these techniques according to the proposed criterion, which results a different ranking for state-of-the-art approaches. Further, we show qualitatively how our method discovers elementary concepts encoded in the raw representations.
4.1 IMPLEMENTATION DETAILS
Attributes and expert models As a first step in our approach, we collect set of attributes which expect find encoded in the self-supervised representations; these include semantic object cat-
5

---

Published as a conference paper at ICLR 2022
Table 1: Different types of concepts used in our evaluation and corresponding datasets.
Categories
Tasks/Datseets
OBJECTS
IN-Ik (Russokovsky et al,, 2015;; Open Images (Kuznetsova et al,, 2020); MSCOCO [things] (Lin al., 2014)
SCENE Places-366 (36 (Zhou et al,, 2016;, Scene attributes (Parterson and Hays, 2012)
MATERIAL
MINC (Bell al., 2015;; MSCOCO [stuff] (Caesar el al,, 2018)
TEXTURE
DTD (Cimpoi al., 2014;; Color (Bau al., 2017)
OTHER
Text detection; Sentiment; Photographic style
egories, scene types and attributes, materials and textures and possibly other information about the photographic style or overall sentiment of an image (Table 1). We thus look for relevant human- annotated datasets and expert models trained on these datasets, as proxy to human knowledge. This allows us to extract information related to such attributes on a different dataset or images in the wild; in our experiments we focus on ImageNet (IN-Ik) (Deng et al., 2009) the target data due to the large availability of pre-trained self-supervised models on this dataset. We concatenate all available attributes and form M -idimennionna binary vectors denoting the presence absence of each attribute in an image. We provide full details about the expert models in Appendix A. Since we primarily focus ImageNet, it is natural to also use the existing 1000 human-annotated labels. As IN-Ik already includes several fine-grained categories, we have not included further experts from common ine-grained recognition datasets, such as iNaturalist (Van Horn et al., 2018).
Linear model For each method, we train linear model mapping a set input attributes to clus- ters, which are obtained by K-means clustering on top of fixed representations and evaluate on held-out set. Further training details are provided in Appendix C.
4.2 COMPARISON OF SUPE-SUPER REPRESENTATIONS
We use reverse probing to evaluate recent self-supervised methods and rank them based on their se- manticity (Table 2). Following convention, we categorize these models as contrastive (00), positive- only, (0), clustering- bassed (4.), and handcrafted pretexts (++) details provided in Appendix B. For each method, we freeze the pre-trained model and extract feature vectors on the IN- Ik train data. We run 5x K = 1000-means to obtain different clusterings, each used to train linear model using all categories from Table 1. In addition to a normalized measure mutual information (NMJ), we also report the adjusted mutual information (AMI), classification accuracy (top-1) and mean aver- age precision (mAP). We also note the linear classification accuracy, as reported by the respective source for each method. For fairness of comparisons, we group methods based on the number of training epochs. For reference, we also evaluate the supervised counterparts in the same way as the self-supervised methods. Overall, we find a relatively strong correlation between our approach and the common linear evaluation protocol (Fig. 3). We also make the following observations:
Ranking Despite the strong correlation to linear classification accuracy, we obtain different ranking from the viewpoint of representational interpretability, using our approach. In particular, state-of-the-a art methods DINO, SwAV and BYOL fall slightly behind in our evaluation. The most recent version of MoCo (v3) performs the best across three metrics, with DeepCluster-v2 and OBoW ranking in the second place (across GesNet-50-bbsee models).
Number of pre-training epochs For some methods, learned weights are available various epochs. In terms of interpretability, we observe no benefit in longer pre-training in the case of DeepCluster-v2 and SwAV, despite the solid performance gains on standard benchmarks. However, there is noticeable gain for MoCo-v2, MoCHi and SimCLR between 200 and 800 epochs. Also notable is that one of the best performing methods, OBoW is only trained for 200 epochs.
Clustering vs contrastive approaches Methods such as SeLa, OBoW, PCL and DeepCluster-2,, which use clustering mechanism during training, have generally more interpretable representa- tions, i.e. they rank higher than methods with similar linear classification accuracy.
Architecture Interpretability increases with self-supervised methods that use a ViT (Touvron et al., 2021; Dosovitskiy et al., 2021) backbone, showing a significant boost in all metries, even more so than what linear classification accuracy suggests. However, a fair comparison between MoCo-v3 and DINO is not possible due to pre-training for a different number of epochs (300 vs. 800).

---

Published as a conference paper at ICLR 2022
Table 2: Evaluation of self-supervised learning methods on ImageNet- (Russakovsky et al., 2015). Lin. Acc. refers to the linear classification accuracy reported by the respective source for each model. The remaining metrics are computed for our approach and reported as mean (#o) over 5 clusterings.
Model
Lin. Acc.
NMI
AMI
Top-1
mAP
ResNet-50 (1x)
+. Jigsaw
(Goyal et 2019)
46.58
37.36 0.04
22.17
9.80
0.04
5,95
0.04
de
ClusterFit
(Yaner al., 2001g
53.63
51.10
: 0.07
38.78
4: 0.10
30.06
:: 0.09
25.41
0.11
00 CMC
(Tin al,, 2019)
58.60
51.96 = 0.06
39.33
= 0.07
29.01
0.15
25.77
0.23
00 MoCo-vl
(He atl 2011)
60.60
55.94 1 0.05
44,39
1: 0.07
34.36
1: 0.13
33.36
0.17
Epoch 200
à SeLa-vl
(Asano et 2020)
61.50
59.91 = 0.04
49.02
# 0.08
41.22
+ 0.13
39.94
0.30
00 SimCL.
(Chen al., 2000a)
66.61
63.73 + 0:07
54.87
:: 0.09
47.93
+ 0.09
53.95
0.08
00 MoCo-v2
(Chen et all 20006)
67.70
64.76 = 0:05
56.05
0.05
47.53
+ 0.10
51.97
0.19
00 InfoMin
(Tiu sl, 2020)
70.10
65.16 0.07
57.27
48.11
0.0.11
54.79
0.13
00 MoCHi
(Kalannidis 2020)
67.60
65.27 0.09
56.94
49.26
+ 0.22
56.15
0.40
G9 PCI.-v2
(laeral, aa, 2001)
67.60
68.89
0.06
62,
0.09
53.72
: 0.088
61.26
0.16
à
SwAV
(Caron al., 2020)
73.90
69.18
= 0.05
61.68 = 0,06
55,85
+ 0.09
62.77
0.27
de OBoW
(Gidtrie al. 2001)
73.80
71,50
1: 0.07
64.09
:: 0.09
57.25
J: 0.13
0.22 61.67
Epoch 400
00 SimCLR
(Chen et al, 2020u)
67.71
64.88
0.09
56,45
50.03
0.17
56,92
0.21
d SwAV
(Caron 2020)
74.60
69.09
0.12
61.66
0.18
56.06
+ 0.22
62.78
0.25
à SeLa-v2
(Asano 2020)
71.80
70.04
# 0.06
63,43
0.07
58.47
+ 0.15
68.33
0,34
d. DecpCluster-22
(Cronn et 2020)
74.32
70.85
0.07
64.01
+ 0.12
58.91
+: 0.15
0.27 67.60
00 SimCLR
(Chen et al., 2020u)
69.68
65.63
0.14
57.48
51.36
0.17
59.11
och- PIRL
and Masten, 2020)
69.90
65.92
0.05
58.79
0.09
51.95
+ 0.07
60.75
0.11
Epoch 800
0 DINO
(Caron 2021)
75.30
68.73
0.08
61.18
# 0.11
55,33
+ 0.21
59.87
0.21
d. SwAV
(Cron et 2000)
75.30
68.79 = 0.05
61.19 + 0.08
55.73 + 0.06
62. 17 0.28
00
MoCHi
(Kalantáis et 2020)
69.20
69.00 1: 0.07
61.77 +: 0.06
55.23
4: 0.07
63.81 1: 0.09
00
MoCo-v2 v2
(Chen et al., 20206)
71.10
69.02 = 0.07
61.55 = 0.11
54.17
+ 0.17
60 44 0.10
InfoMin
(Tax al., 2020)
73.00
69.20
= 0.02
62.54
+ 0.04
55.15
+ 0.10
63.8c 0.10
DeepCluster-v2 (Crron 2020)
78.18
69.44
0.04
62.12
0.05
57.01
+ 0.08
63.97
0.26
Epoch 1k
0
Barlow Twins
(Zootar et st., 2021)
73.50
69.37
1: 0.08
61.69
+: 0.13
56.84
J: 0.18
61.32
0.23
00
MoCHi
(Kalanidis et al. 2020)
70.60
70.16
= 0.06
63.37
= 0.09
57.08
: 0.16
66.
0.11
BYOL
(Grill 2120)
74.40
70.48
0.07
63.12
0.10
58.36
:: 0.09
63.26
0.25
MoCo-v3
(Chen et al., 2021)
74.60
71.45
0.06
64,49
0.09
59.58
0.11
64.95
0.43
Supervised
(He al., 2016)
83.20
0.06
78.82
0.07
76.16
J. 0.14
78.53
0.15
ViT--ase/16
00
MoCo-v3
(Chen al., 2021)
76.70
79.06
a 0.04
73.67
0.05
70.51
a: 0.09
74.39
0.28
O DINO
(Caron 2021)
78,20
81.46
0.08
76.70
0.111
72.95
+ 0.14
76.44
0.18
Supervised
(Tavvon 2011
94.36
0.11
93.13
92.02
0.199
80.82
0.15
4.3 EXPERT BREAKDOWN
Our next goal is to understand the effect of different concepts explaining the representation, iee answering the question: to which degree does the representation know certain concepts, e.g., material? As we focus on self-supervised representations, the answer this question is far from obvious. We measure this via the predictive ability of our probe, which we now train individually for each group of concepts. These vary in nature; from highly semantic object categories (e.g., dog, avocado) to lower-level features such as material (e.g., wood, brick), texture (e.g., bubbly, striped) and color (e.g., red).
In Fig. 4, we show how the different concept groups contribute to the overall performance for se- lected ResNeet-5-bbasee methods. We train and evaluate reverse probes for each shown combination, e.g., IN-IK+OBJECTS, then IN-IK-+BBJCTS+SSEEE using ground truth IN-1K categories as part of the input. We observe that for earlier methods, such as MoCo-v1, ImageNet categories alone are not sufficient for accurately predicting the cluster as- signments. In fact, it appears that using only semantic categories from MSCOCO and OpenImages (w/o ImageNet) provides about as much information about the clusters. This likely suggests that the discovered clusters do not reflect fine- mne-grained distinctions, since the most notable difference be- tween IN- 1K and the other semantic experts is the granularity of some categories (e.g.. dog breeds). On the other hand, the best performing methods do owe a big part of their performance to IN- 1K concepts, meaning that they are able recover clusters that are closer to the original label distribu-
etc. We compare variants with and without
7

---

Published as a conference paper at ICLR 2022
Meteol
Maseid
Rovere Ausinyy
Diha:
Clasoiti
PCL
35
68
Swe
Scland
DargCeddee 41
MuCen3
Linear Classfication Accunayy (9)
Figure 3: Linear classification accuracy on IN- 1k vs. classification accuracy of our probe (top- 1). A linear regression model is fit the data, suggesting strong correlation. Each point corre- sponds ResNet-50-basee model in Table 2.
Figure 4: Contribution of each expert group to the overall accuracy for selected methods. For each bar linear model is trained using in- put the current expert group and all previous ones (e.g., green bar: IN-IK+OBJECTS+ SCENES).
Table 3: Change in NMI when learning the reverse probe with individual concept groups on top of the mamman-nnnttaeed IN- 1K labels.
Method
IN-1K
OBJECTS
SCENE
MATERIAL
TEXTURE
OTHER
MoCo-vl
(He al.,, 2020)
47.70
+2.73
+2.92
+2.93
+1.04
+0.21
MoCo-v2
(Chen et al., 20206)
57.39
+2.43
+2.09
+2.03
+0.43
-0.24
SwAV
(Cuon et al,, 2000)
61.94
+2.42
+1.92
+2.01
+0.23
-0.20
SeLa-v2
(Asano et al., 2020)
63.29
+2.23
+1.69
+1.69
+0.03
-0.45
DeepCluster-v2
(Caon 2000)
64.29
+2.17
+1.54
+1.70
+0.12
-0.35
OBoW
(Gidaris 2021)
64.73
+2.77
+1.82
+2.13
+0.11
-0.15
MoCo-v3
2021)
67.69
+1.51
+0.32
+0.21
-0.36
-0.38
tion and that rely less on low-level concepts such as textures. This suggests that more performant methods learn features highly correlated with IN-1K labels, but less with other semantic categories.
Finally, in Table 3, we also show the (decorrelated) effect of each individual concept group in isola- tion but always in combination with the ground truth IN- 1K categories. We observe that in absence of concept combinations, the models we probed use as much (and sometimes more) information about scene and material properties as information about semantic object categories. Miscellancous concepts (OTHER) do not appear relevant for more recent methods. We again observe that the rep- resentations of the most recent, oate-off-the-ar MoCo-v3 share very little information with concepts other than IN-1K, despite the lack of label information during pre-training.
4.4 QUALITATIVE RESULTS
Since the clusters we examine are learned without supervision and expert models are not perfect predictors, it is crucial to verify that the clusters align with human interpretation, besides quantita- tive scores. By inspecting the probe's coefficients 0, one can qualitatively explore what concepts representative for specific clusters and how exposing the linear model to different sets of perts affects its predictive accuracy. In Fig. 5 we show pairs of clusters from MoCo-v2 for which we observed significant drop in pairwise confusion (estimated from the confusion matrix) with the inclusion of an additional concept group on top of the fixed ImageNet label set. For example, after including OBJECTS we notice that several pairs of clusters become more easily disambiguated thanks to the concept person, or related concepts (human hand, human face, etc.). In Fig. (top left), both clusters can be described by the ImageNet label French horn and it only possi- ble to explain their differences when more information becomes available. This finding supports our initial hypothesis that there could be interpretable properties the network that remain undetected since they not annotated a-priori in the benchmark data. Importantly, since ImageNet is only annotated with a single label per image, combined concepts, such as "person playing the french horn" cannot be discovered otherwise. Similarly, when adding SCENE concepts (top right), clusters
8

---

Published as a conference paper at ICLR 2022
Objects
Scenes
Material
Figure 5: Qualitative examples. We show pairs of clusters for which the estimated confusion was significantly reduced after training the linear model with the corresponding concepts. We show and discuss additional examples in Appendix E.
Table 4: Evaluation of selected IN- N--kk-pretriined methods the Places-365 dataset. K-means
(K = {500, 1000}) and the reverse probe are trained on features extracted on Places-365.
K
500
K
1000
Model
NMI
AMI
Top-1
mAP
NMI
AMI
Top-1
mAP
MoCo-v2
54.41
51.84
41.18
42.15
54.81
47.34
34.05
32.74
SeLa-v2
57.02
54.66
45.04
46.42
56.72
49.76
37.46
36.48
SwAV
57.30
54.65
44.84
46.48
57.96
50.44
39.08
38.34
DeepCluster-v2
58.09
55.62
46.23
48.06
58.30
50.94
39.54
39.35
are distinguished by scene type; one contains vol canos, while the other contain mounta iss and creeks (categories from Places-365). With MATERIAL, we can predict the shipwreck cluster more accurately, while TEXTURE can explain the difference the more stratified appearance of cliff's for the cluster on the right. Thus, it appears that, despite the object-centric nature of Ima- geNet, self-supervised representations rely significantly on context (e.g., SCENE and MATERIAL), and concept combinations. This likely also explains why in (Van Horn et al., 2021) self superviedd models perform equally or better than the supervised baseline at context, counting and gestalt tasks, and why structural downstream tasks benefit more from self-supervision Kotar al. (2021).
4.5 TRANSFER OTHER DATASETS
We finally evaluate the representations learned by selected methods by transferring them to a dif- ferent domain. For this experiment, the self-supervised methods are not fine-tuned or adapted on the new data. We perform the quantization on representations extracted on images from Places-365 and use all concepts from Table but IN- 1K (now using the available ground truth for Places-365 instead). We report the performance in Table 4 and observe that the same ranking generally holds.
5 CONCLUSION
We have introduced reverse linear probing, a measure of representation interpretability that allows to explain representations by combining multiple concepts. Driven by the need to better understand and characterize the representations learned by self-supervised methods, the proposed measure has a rigorous foundation in information theory and provides complementary information to existing benchmarks well as insights into the semanticity and the importance of different concepts for such models. Our approach is applicable to any representation and is considerably faster to train, as all concepts can be pre-extracted on any given image collection.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
I.L. is supported by the European Research Council (ERC) grant IDIU-638009 and EPSRC Vi- sualAI EP/T028572/1. Y.M.A. is thankful for funding from EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines & Systems (EP/L015897/1) and MLRA from AWS. A.V. is supported by the ERC grant IDIU-638009.
REFERENCES
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In Sth International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings, 2017. 1,3,5
Nikolaos Aletras and Mark Stevenson. Evaluating topic coherence using distributional semantics. In Proceed- ings of the 10th International Conference on Computational Semantics (IWCS 2013)-Long Pupers, pages 13-22, 2013. 26
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. theoret- ical analysis contrastive unsupervised representation learning. arXiv preprint 0rXiv:1902.09222 2019.
Yuki M. Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and rep- resentation learning. In Proceedings the International Conference Learning Representations (ICLR), 2020. 2, 7, 8,
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael Rabbat. Semi-supervised learning of visual features by non-parametrically predicting view as- signments with support samples. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 8443-8452, 2021.
Youngmin Back, Bado Lee. Dongyoon Han, Sangdoo Yun, and Hwalsuk Lee. Character region awareness for text detection. In Proc. CVPR, pages 9365-9374, 2019. 16
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Proc. NeurlPS, 32:9453-9463, 2019. 20
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proc. CVPR, 2017. 1, 3,4,4
Miguel A Bautista, Artsiom Sanakoyeu, Ekaterina Tikhoncheva, and Bjom Ommer. Cliqueenn: Deep unsuper- vised exemplar learning. In NeurlPS, pages 3846-3854, 2016. 2
Miguel A Bautista, Artsiom Sanakoyeu, and Bjorn Ommer: Deep unsupervised similarity learning using par- tially ordered sets. In CVPR, pages 7130-7139, 2017. 2
Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. Material recognition in the wild with the materials in context database. In Proc. CVPR, 2015. 6, 16
David M Blei, Andrew Y Ng, and Michael Jordan. Latent dirichlet allocation. the Journal of machine Learning research, 3:923-1022, 2003. 26
Holger Caesar, Jasper Uijlings, and Vittorio Ferran. Coco-stuff: Thing stuff classes in context. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 1209-122 1218, 2018. 6, 16
Mathildc Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual fcatures. In Proc. ECCV, 2018. 2
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Proc. NewrlPS, 2020. 2, 7,
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint 2rXvv:2004 /4294, 2021. 7
10

---

Published as a conference paper at ICLR 2022
Jonathan Chang, Sean Gerrish, Chong Wang, Jordan L Boyd-Graber. and David M Blei. Reading tea leaves: How humans interpret topic models. In Advances neural information processing systems, pages 288 296. 2009. 26
Liang-Chiee Chen, George Papandreou, Jasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Sc- mantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834-848 2017. 16
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework contrastive learning of visual representations. In International conference on machine learning, pages 1597-16077
PMLR, 2020a. 7
Xinlei Chen and Kaiming He. arXiv:201. 2020.
Exploring simple siamese representation learning. arXiv preprint
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003, 04297, 2020b. 2, 7,
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv: 104. 02057, 2021. 7,8
Yao Ching- Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debiased con- trastive learning. Advances in neural information processing systems, 13:8765-87755 2020. 2
Mircea Cimpoi, Subhransu Maji. lasonas Kokkinos. Sammy Mohamed, and Andrea Vedaldi. Describing tex- tures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014. URL http:ttttt/. http:://wwwwwwtwttte
wx.ac.u.. wwt..uuk/vvvg.g datp:///aa///////////// 16
Elijah Cole, Xuan Yang, Kimberly Wilber, Oisin Mac Aodha, and Serge Belongie. When does contrastive
visual representation learning work? arXiv preprint arXiv:2105.05837, 2021. 3
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Wiley, 2006. 3
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. ImageNet: large-scale hierarchical image database. In Proc. CVPR, 2009, 6
Carl Doersch, Abhinav Gupta, and Alexci Efros. Unsupervised visual representation learning by context prediction. In Proc. ICCV, 2015. 2
Alexey Dosovitskiy, Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Un- terthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition scale. Proc. ICLR, 2021.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: Nearest-neighhor contrastive learning of visual representations. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 9588-9597, 2021.
Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer? In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5414-5423, 2021. 3
Victor Escorcia, Juan Carlos Niebles, and Bernard Ghanem. On relationship between visual attributes and convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 1256-1264, 2015.
Ruth Fong and Andrea Vedaldi. Net2Vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Jonathan Frankle, David J Schwab, Ari S Morcos, et al. Are all negatives created equal in contrastive instance discrimination? arXiv preprint arXiv:2010.0666 2020. 2
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept- based explana- tions. In H. Wallach, H. Larochelle, A. Beygelzimer, d. Alché-Buc, E. Fox, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. Proc. ICI.R 2018 2
11

---

Published as a conference paper at ICLR 2022
Spyros Gidaris, Andrei Bursuc, Gilles Puy, Nikos Komodakis, Matthieu Cord, and Patrick Perez. Obow: Online vog-oo-visual-wors generation for self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6830-68800 6840, 2021. 7, 8, 19
Priya Goyal, Dhruv Mahajan, Abhinav Cupta, and Ishan Misra. Scaling and benchmarking self-supervised visual representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6391-6400, 2019. 3,7
Jean-Bastien Grill, Florian Struh, Florent Altché, Corentin Tallec, Piere H Richemond, Elena Buchatskaya, Carl Doersch, Beraardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. Proc. NeurIPS, 2020. 2,
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. CVPR, 2016. 7. 16
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proc. CVPR, pages 9729-9738, 2020. 2, 7, 8, 20
Olivier J. Hénaff, Ali Razavi, Carl Doersch, S. M. Ali Eslami, and Aäron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXivees, abs//995.09272, 2019.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, (13(576)):504 507, 2006.
Weihua Hu, Takeru Miyato, Sciya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. Learning discrete repre- sentations via information maximizing self-augmented training. In ICML, pages 1558-1567, 2017. 2
Xu Joao F. Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classi- fication and segmentation. In Pruceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019, 2
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. arXiv preprint
arXiv:100.008734, 2017. 18
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. Proc. NeurIPS, 2020. 2,7
Bingyi Kang, Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representa- tion learning. In International Conference on Learning Representations, 2020.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, al. Interpretabillity beyond feature attribution: Quantitative testing with concept activation vectors (tcav). International conference on machine learning, pages
PMLR, 2018. 3
Klemen Kotar, Gabriel Ilharco, Ludwig Schmidt, Kiana Ehsani, and Roozbeh Mottaghi. Contrasting contrastive self-supervised representation learning pipelines. In Proceedings the the IEEE/CVF International Conference
on Computer Vision, pages 9949-9959, 2021. 3,9
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. IJCV, pages 1-26, 2020. 6
Iro Laina, Ruth C. Fong, and Andrea Vedaldi. Quantifying Icarnaillity and describability of visual concepts emerging in representation learning. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020. 3
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic coloriza- tion. In Proc. ECCV, 2016. 2
Juho Lee, Yoonho Lee, and Yee Whye Teh. Deep amortized clustering. arXiv preprint arXiv:1909.13493. 2019. 2
Junnan Li, Pan Zhou, Caiming Xiong, and Steven C.H. Hoi. Prototypical contrastive learning of unsupervised representations. ICLR, 2021. 2,
Tsung-Y Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In Proc. ECCV, 2014. 6, 16
Stuart Lloyd. Least squares quantization in pem. IEEE transactions information theory, (8(2):1291137, 1982. 4
12

---

Published as a conference paper at ICLR 2022
Aravindh Mahendran and Andrea Vedaldi. Salient deconvolutional networks. In Proceedings of the European Conference on Computer Vision (ECCV), 201 3
Aravindh Mahendran and Andrea Vedaldi. Visualizing deep convolutional neural networks using natural pre- images. International Journal of Computer Vision (ICCV), 120()), 2016b. 3
David Mimno, Hanna Wallach, Edmund Talley, Miriam Leenders, and Andrew McCallum. Optimizing mantic coherence in topic models. In Proceedings of the 2011 conference on empirical methods in natural language processing, pages 262-272, 2011. 26
Ishan Misra and Laurens van der Maaten. Self-supervised learning pretext-invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6707-6717, 2020. 2,7
Jovana Mitrovic, Brian McWilliams, and Melanie Rey. Less can more contrastive learning. In Jessica Zosa Forde, Francisco Ruiz, Melanie F. Pradier, and Aaron Schein, editors, Proceedings on "I Can't 7 Believe It's Not Better!" at NewrlPS Workshops, volume 137 of Proceedings of Machine Learning Research, pages 70-75. PMLR, 12 Dec 2020. 2
Fred Morstatter and Huan Liu. In search of coherence and consensus: measuring the interpretability of statis- tical topics. Journal of Machine Learning Research, 18( (8(169):1-33 32, 2018. 26
David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. Automatic evaluation of topic coherence. In Human language technologies: The 2010 anmual conference the North American chapter of the asso- ciation for computational linguistics, pages 100-108, 2010. 26
Anh Nguyen. Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. Plug & play generative networks: Conditional iterative generation of images in latent space. In Proc. CVPR, 2017. 3
Anh Mai Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the pre- ferred inputs for neurons in neural networks via deep generator networks. In Proc. NeurIPS, 2016.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In Proc. ECCV, 2016. 2
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11). 2017.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 800::037870 2018. 2
Jose Oramas, Kaili Wang, and Tinne Tuytelaars. Visual explanation by interpretation: Improving visual feed- back capabilitics of deep neural networks. In International Conference on Learning Representations, 2019.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 2536-2544, 2016.
Genevieve Patterson and James Hays. SUN attribute database: Discovering, annotating, and recognizing scene attributes. Proc. CVPR, 2012. 6, 16
Senthil Purushwalkam and Abhinav Gupta. Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, edi- tors, Advances in Neural Information Processing Systems, volume 33, pages 3407-3418. Curran Associates, Inc., 2020. 2
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint 210x::200300022 2021. 17
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In Proc. NewrIPS, 2016. 16
Joshua Robinson, Ching- Yao Chuang, Suvrit Sra, and Stefamie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXi::001000592, 2020. 2
Michael Röder, Andreas Both, and Alexander Hinneburg. Exploring the space of topic coherence measures. In
Proceedings of the eighth ACM international conference Web search and data mining, pages 399-408, 2015. 26
13

---

Published as a conference paper at ICLR 2022
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bersstein, Alexander C. Berg, and Fei-Fei Li. ImageNet large scale visual recognition challenge. ICV, 115(3), 2015. 5, 6, 7, 16
Mert Bulent Sariyildiz, Yannis Kalantidis, Diane Larlus, and Karteek Alahari. Concept generalization in visual
representation learning. arXiv preprint FrXiiv2012.05644 2020. 3
Vikash Sehwag. Mung Chiang. and Prateek Mittal. On separability of supervise representations. ICML workshop on Uncertainty and Robustness Deep Learning (UDI), 2020. 3
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618 626, 2017. 3
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1rXiv:1409.1. 1556, 2014. 16
Karen Simonyan. Andrea Vedaldi. and Andrew Zisserman. Deep inside convolutional networks: Visualising
image classification models and saliency maps. In Proceedings of the International Conference on Learning Representations (ICLR), 2014. 3
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXivees, abs// htn//196.058449 2019. 7.7
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. Proc. NeurlPS, 2020. 7,7
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou.
Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, volume 139, pages 10347-10357, July 2021. 6,
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Demystifying self- supervised learning: An infornaaion-theortttiall framework. arXiv preprint arXiv2006.005776 2020. 2
Michael Tschannen, Josip Djolonga, Paul Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual informa- tion maximization for representation learning. Proc. ICLR, 2020. 2
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Mare Proesmans, and Lue Van Gool. Scan: Learning to classify images without labels. In Proceedings of the European Conference on Computer Vision, 2020. 2, 20
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pictro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 8769-8778, 2018. 6
Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmark- ing representation learning for natural world image collections. In Proceddiggs of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12884-12893, 2021. 3,
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Proc. ICML, pages 9929 9939. PMLR, 2020. 2
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non non- parametric instance discrimination. In Proc. CVPR, 2018.
Jia Xue, Hang Zhang, Ko Nishino, and Kristin Dana. Differential viewpoints for ground terrain material recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020. 16
Xueting Yan, Ishan Misra, Abhinav Gupta, Deepti Ghadiyaram, and Dhruv Mahajan. Clusterfit: Improving generalization of visual representations. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 66118, 2020. 2,
Chih-Kuan Yeh, Been Kim, Serean Arik, Chun-Liang Li, Tomas Pfister, and Pradeop Ravikumar, On completeness-awart concept-based explanations deep neural networks. In NewrlPS, 2020. 3
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint urXiv:2103.03: 033300 2021. 2.7
Matthew D. Zeiler and Roh Fergus. Visualizing and understanding convolutional networks. In Proc. ECCV, 2014. 3
14

---

Published as a conference paper at ICLR 2022
Xiaohua Zhai, Joan Puigeerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. large-scale study rep- resentation learning with the visual task adaptation benchmark. arXiv preprint arXiv: 1900::910.04486,, 2019. 3
Xiaohang Zhan, Jiahao Xie, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. Online deep clustering for unsupervised representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1688-6697, 2020. 2
Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. In Proc. ECCV, 2016. 2
Richard Zhang, Phillip Isola, and Alexci A. Efros. Split-brain autoencoders: Unsupervised learning by cross- channel prediction. In Proc. CVPR, 2017. 2
Nanxuan Zhao, Zhirong Wu, Rynson W.H. Lau, and Stephen Lin. What makes instance discrimination good for transfer learning? In Proc. ICLR, 2021.
Evgenii Zheltonozhskii, Chaim Baskin, Alex M Bronstein, and Avi Mendelson. Self-supe learning for large-scale unsupervised image clustering. arXiv preprint arXiv:2008, 10312, 2020. 3, 19
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene cans. arXiv preprint 141.:::412.6886, 2014. 3
Bolei Zhou, Aditya Khosla, Àgata Lapedriza, Antonio Torralba, and Aude Oliva. Places: An image database for deep scene understanding. PAMI, 2016.
Bolci Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 2017. 16
Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Interpretable hasis decomposition for visual ex- planation. In Proceedings of the European Conference Computer Vision (ECCV), pages 119-134, 2018.
15

---

Published as a conference paper at ICLR 2022
A EXPERTS: IMPLEMENTATION DETAILS
We provide all information about the concepts and experts used in our experiments in full detail. Expert models are trained on various tasks and image datasets extract information about the scene and its objects, material, texture and miscellancous (other). Then, the trained models are applied to the target dataset (e.g., ImageNet (Russakovsky et al., 2015)). While the tasks differ in nature- -e- e-g., segmentation (dense), detection, classification in all cases we convert the predictions to binary vectors. For example, in the case of segmentation and detection, we return only a (binary) label vector denoting whether class is present in given image, discarding dense/location information. We concatenate the experts' binary predictions forming the concept vector y(a)) for image z, ie..
y(z) = (x)) y (2)
(z)],
for expert models with ( denoting the i-th expert. In the following, we provide some expert- specific details and summarize them in Table
Detection on Open Images. We use the Tensorflow Object Detection API and publicly available Faster R- R-CNN model (Ren et al., 2016), trained on Open Images (600 object categories). We apply the model to our target datasets and keep only object categories predicted with a confidence higher than 0.5.
Segmentation on MS COCO. We use a DeepLab-v2 model (Chen et al., 2017) trained for seg- mentation on MS COCO 2017 (Lin al., 2014; Caesar et al.. 2018), which includes 91 'thing' and 91 'stuff" categories. 'Thing' categories include common, countable objects with well-defined shape, such as person, dog. bicycle, etc.; 'stuff" categories typically include background regions, such as sky, snow, wall, grass, etc. and are thus closely related to material properties. While we experimented with confidence and minimum area thresholds for predicted class to be considered in our scenario, we found that simply returning all predicted categories performed best.
Scene Classification and Scene Attributes. Although ImageNet is object-centric dataset, lot of images are related to specific scene types, e-g., outdoor scenery (mountains, seaside) or indoor scenery (houses, shops). In fact, some ImageNet categories can even further subdivided by scene type, for example dogs sitting indoors or playing in the park. It is thus crucial to consider scene classification in our investigations. We use pre-trained ResNet-50 model provided by the official repository of the Places-365 database (Zhou et al., 2017) and assign each image of our target dataset to scene category. Further, we use the provided unified model (wide ResNet- 18) to predict multiple scene attributes per image. The model is trained on the SUN Attribute Database (Patterson and Hays, 2012), which contains 102 categories describing scene properties, e.g., man-made vs. natural or enclosed vs. open area.
Texture and Material Classification. For further detecting material concepts, we train Deep Encoding Network (DEP) (Xue et al., 2020) on the Materials in Context (MINC) dataset (Bell et al., 2015), which consists of 23 categories (include a background class). We then do the same on the Describable Textures Dataset (DTD) (Cimpoi et al., 2014), which consists of close-up images for 47 texture categories. DEP uses ResNet-50 (He al., 2016) backbone and is trained for single-label classification. We apply the trained models to our target datasets, thus returning a single label per image in each case.
Next, we wish to identify the presence of text in the images to investigate whether this is deciding factor in clustering the image features (e.g., camera dates watermarks). As a text detection expert, we use CRAFT (Baek et al., 2019) (official implementation) without the refinement step. We only return one bit representing whether the image contains text or not, rather than character recognition. Overall, we found that text as a concept did not contribute significantly in our experiments.
Text Detection.
Sentiment Analysis. We use a VGG-19 classifier (Simonyan and Zisserman, 2014) trained on the Twitter for Sentiment Analysis Dataset (T4SA) to classify the sentiment for each image into one of three classes (positive/nettaalnegatvvee and use the sentiment prediction as an additional concept. As with text, we found that sentiment does not significantly affect the performance. We found the predictions of the expert model to be somewhat unreliable; this is also likely due to the fact that sentiment is rather subjective quality.
16

---

Published as a conference paper at ICLR 2022
Table Summarization of expert models and datasets used in our work for concept extraction.
Category
Datasets
#Classes
Task
Model
Source
Open Images
600
Segmentation
Faster R-CNN
OBJECTS
MS COCO (thing)
91
Detection
DeepLab-v2 (ResNet-101)
Places-365 365
365
Single-labe label Classification
ResNet-50
SCENES
SUN Attribute
Multi-label Classification
WideResNet-18
MS COCO (stuff)
91
Segmentation
DeepLab-v2 (ResNet-101)
MATERIAL
MINC
23
Single-label Classification
DEP (ResNet50)
DTD
47
Single-label Classification
DEP (ResNct-50)
TEXTURE
Color
11
Quantization
SynthText, ICI3, IC17
Text Detection
CRAFT
OTHER
T4SA
Sentiment Classification
VGG-19
OpenAI Data (unreleased)
Image/Text Similarity
CLIP
Photography. We have observed differences in the photographic style, quality and type of images in ImageNet, e-g., ranging from macro photography, usually for flowers insects, vector illus- trations. In order to probe the representations for style and image quality, we use the recent CLIP model (Radford et al., 2021), I), trained on 400 million image-text pairs collected from the web and create set of candidate sentences: "a high quality photo", "a noisy, grainy image", "a blurry im- age of low quality". "macro photography", "a photo with out of focus background, bokeh effect", "an animated picture, vector illustration", "a painting", "a portrait". We use the pre- pre-trained CLIP model to compute image and sentence features (normalized to unit norm) and use the dot product find the similarity between each image and each one of the sentences. We pick the sentence with the highest similarity as the predicted class for each image the score is at least 0.5, else we assign background class. Since CLIP operates on free-form text rather than fixed of categories, its use is not limited to this type of queries; it can be used as an expert where a labelled dataset might unavailable.
B SSL MODELS
We have evaluated a wide array of self-supervised models, which we divide into the following categories.
(00) Contrastive methods make use of positive and negative examples and learn representations by drawing positive samples together while pushing negative samples apart, Based on the idea of instance discrimination, positive examples are constructed from different views the same im- age, thus the representation learns invariance the choice of transformations, which are typically rather aggressive. We have evaluated MoCo, SimCLR, CMC, InfoMin and MoChi as contrastive approaches.
(o) Positive-only methods do not discriminate between instances, eliminating the need for negative examples altogether. However, simply removing negative examples can result in collapse (features can be a single constant vector). For this reason methods that use only positive examples avoid feature collapse through other techniques such as regularization or distillation. From this family of methods, we have evaluated BYOL, DINO and Barlow Twins.
(a)) Another family of methods can be categorized as musstrrng-basedd Clustering another way to enforce invariance and it relies on the assumption that meaningful groups exist in the data, such that intra-group similarities can be maximized and inter-group similarities should be low. Offline clustering methods typically alternate two steps, i.e. assign datapoints to clusters based their representations and optimize the model given the current cluster assignments. Online approaches, like SwAV, perform clustering in minibatch and only enforee consistent assignments for differ- ent views of the same image. We have evaluated ClusterFit, SeLa, DeepCluster-v2 and SwAV as clustering- based approaches, well PCL which combines clustering and contrastive objective.
(+x) Earlier methods on self-supervised visual representation learning devised handcrafted pretext tasks. As representative of this category, have evaluated Jigsaw, which trains the model to
17

---

Published as a conference paper at ICLR 2022
solve puzzles the pretext task. We also evaluated PIRL, which combines the Jigsaw task with NCE, to learn representations that are invariant to the input perturbations.
C TRAINING DETAILS
Linear model We evaluate representations from number of pre-trained self-supervised feature extractors (Table 2). For ResNet-50 models, we evaluate features at the output of the average pooling layer (2048-d). For ViT-Base models we evaluate the [CLS] ] token of the last self-attention layer (768-d). For each model, we pre-extract and store features for the entire training set of ImageNet and, prior to clustering, run K-means for 100 steps using faiss (Johnson et al., 2017) and choose the best out of 5 runs. We divide the data into train and test sets by splitting all cluster assignments with 80/20 ratio and stratified sampling; from the training split we also reserve 20% of the data for validation. Finally, we targets and concept vectors as inputs (where
standardize them to zero mean and unit standard deviation. We then
train the linear model with the cluster assignments the concepts are pre- pre-extracted on a given image collection, e.g., ImageNet). We train for up to 100 epochs with batch size 512 and optimize using SGD with a momentum of 0.9 and initial learning rate of 3.5 which is further reduced by factor of 10 at epochs 60 and 80. We also add L2-regularization with weight x 10-6
Computation Given a dataset (e-g., ImageNet) and set of experts (e.g., the ones listed in Table 5), all labels can be pre-computed for all images in the dataset. Feature vectors for given model can be also pre-extracted and stored for the whole dataset. Our method computes cluster assignments using the efficient K -means implementation of faiss (which takes less than 5min for 256k 2048-d vectors on 4 NVIDIA RTX A4000). Training of the linear model converges in less than 100 epochs in matter of minutes on single GPU (1 epoch takes 2sec). We should note that standard linear probing on ImageNet typically requires full images (including online augmentations) and multiple forward passes through the frozen feature extractor, which makes it significantly slower to train.
D FURTHER ANALYSIS
In addition the results and discussion in the main paper, we provide more investigations and insights.
D.1 MUTUAL INFORMATION BETWEEN CONCEPTS AND IMAGENET CATEGORIES
One interesting question that arises is to which extent the elementary concepts we are considering are predictive of ImageNet labels. We answer this question by training a linear model to predict the ground truth label (instead of pseudo-label) for each image from its concepts. This results in top-1 accuracy of 46.8% (NMI: 64.2, AMI: 54.2)-an and significantly less if we exclude 'object' concepts that overlap with ImageNet labels. This suggests that using additional concepts provides information which is complementary to the fixed label set of ImageNet, further justifying our approach.
D.2 VARYING K
As discussed in Section 3, the mutual information between a representation f(x) and a fixed set of concepts v(х) is maximized when f=x) = i.e. any processing of x cannot increase the amount of information. It is thus expected that with an increasing number of clusters (K), I I(xx (x), v(x)) will monotonically increase, as we approximate f(x) (i.e. when each sample is its own cluster). We verify this empirically in Fig. 6. For fair comparison across methods, we have fixed K = 1000 for all experiments on ImageNet experiments in the main paper. To better understand the effect of the number of clusters, in Fig. 7 we also show the performance of most methods for K E (500, 1000, 1500, 2000, 2500, 3000}, measuring NMI, AMI and top-1 accuracy, for the predictions of the reverse linear probe trained with all concepts. Importantly, we observe that for K > 1000 the relative ranking of methods remains mostly consistent regardless of the number of clusters. The top three methods, MoCo-v3, OBoW and DeepCluster-v2 perform similarly and converge for larger K, followed by SeLa-v2, SWAV, BarlowTwins and DINO. Further, we observe that BYOL scales gracefully for larger K, reaching the performance of the top methods; the opposite is observed for
18

---

Published as a conference paper at ICLR 2022
10
9
IUK()) v())) (bits)
4
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
x10*
Figure 6: Empirically estimated mutual information between fx (z) and y(x) for varying K.
SYUL
10p- Accitac (R)
NMT
AMI
20
KL-2
64
Z0NO
Setevivie
K
Ø
K
Figure 7: Performance of self-supervised methods (architecture: ResNet-50) for varying number of clusters K. The ranking remains mostly consistent.
PCL-v2 which does scale as well. Though it is important study the effect of varying K, should also note that our results do not appear sensitive to it, as similar trends are observed for all methods.
D.3 DISCUSSION ON CLUSTERING-BASEE METHODS
We have observed that methods that employ a clustering mechanism during training have gener- ally more interpretable representations, natural question that arises is whether clustering-base evaluation naturally favors such approaches.
However, clustering has already been shown work well even for contrastive ap- proaches (Zheltonozhskii et al., 2020), which is also further confirmed by our experiments. In our evaluations, the highest ranked models are MoCo-v3 (Ik epochs) and OBoW (200 epochs). MoCo- v3 uses a contrastive objective, while OBoW follows a different approach from previous methods with bag-of-words. prediction task that puts emphasis on contextual information and local feature statistics. Our results align with the intuition behind OBoW (Gidaris et al., 2021) that encoding local concepts via a bag-of-visual-wort. approach yields richer, context-aware representations. An- other example, SeLa, uses Sinkhorn- Knopp optimization to compute the cluster assignments, while D uses spherical K-means. All these clustering mechanisms differ from the one used specifically in our experiments. Moreover, SeLa and DeepCluster-v2 train with K 3000 proto- types, and there is no reason to believe that their representations are optimal for all values of K, and discussed above, the relative ranking of methods does not depend on the choice of K.
19

---

Published as a conference paper at ICLR 2022
Method
NMI
AMI
Top-1
mAP
SeLa-vl (K -mmeas))
(Asano et al., 2020)
64.66
37.37
33.05
29.29
SeLa-vl (self-label)
(Asano at 2020)
62.02
31.92
29.22
24.51
SCAN (K-means)
(Van Gunsboke et al., 2020)
69.02
61.56
54.17
60.44
SCAN (self-label)
(Vin Gansheke
73.76
73.76
62.10
63.63
Table 6: Evaluation of self-labelling methods. "Self-label" denotes clusters which are predicted directly from the model and which are used to train the reverse probe. We compare these to the clusters obtained with K-means on the representation vectors (as in the main paper). 3000 for SeLa and K 1000 for SCAN to match the number of pseudo-labels predicted by the respective models.
Method
Top-1
Top-5
NMI
AMI
40.30
75.57
60.14
50.38
MoCo(v2)
12.67
32.45
26.73
16.53
41.03
75.62
59.89
50.48
MoCHi
12.64
31.71
25.66
15.98
45.73
78.56
62.75
53.77
SwAV
17.71
43.64
26.39
18.72
46.92
78.73
64.82
55.75
OBoW
12.23
31.72
27.45
16.82
47.20
79.90
63.25
54.77
DeepCluster(v2)
19.73
46.81
26.84
19.43
47.62
80.87
62.88
54.91
SeLa(v2)
20.61
48.83
25.32
18.36
Table Reverse probes evaluated on ObjectNet comparison ImageNet (in black).
D.4 K-MEANS FOR SELF-LABELLING METHODS
We now look specifically self-labelling methods, SeLa (Asano et al., 2020) and SCAN (Van Gans- beke et al., 2020), to investigate the difference between (a) the pseudo-labels predicted directly by each method and (b) clustering the learned representations with K-means, i.e. same the experi- ments reported in the main paper. SeLa follows an optimal transport approach that yields clusters of approximately equal size. We found that this weakens the quality of its self-labels but results in stronger clusters after K-means and consequently in improved performance in our evaluations. Since SeLa is originally trained with 3000 classes', in Table we compare its pseudo-labels with K 3000-means clustering in Table 6. SCAN builds on top of representation learning method, i.e. MoCo (He al., 2020) and, with learnable clustering approach that removes adverse effects of low-level features, it produces more semantic pseudo-labels, improving upon K-means. Thus SCAN (self-label) results in improved performance Table 6.
D.5 TESTING ON OBJECTNET
Next we provide an evaluation of selected self- suff-supereieed methods using the reversed probes on the challenging test set ObjectNet (Barbu et al., 2019), evaluating the transferability of the quantized representations. Specifically, we use the reversed probes trained for each method on ImageNet to assign ObjectNet samples to clusters, measuring the success of the classification (accuracy, NMI, AMI) against the corresponding K-means assignments, í.e. we measure if we can successfully pre- dict the cluster assignments from predicted concepts, without any further training/adaptation. Here, we use the probes trained with all experts as input, except for the real ImageNet labels to account for
phttps:/gg thub. wwww.ccooyyceeeecccccccccc manoo/ sel f 1abel
20

---

Published as a conference paper at ICLR 2022
ObjectNet categories that do not appear in ImageNet. We notice significant drop in performance which suggests that a domain gap is indeed present, though the ranking stays again roughly the same (with the exception of OBoW).
E QUALITATIVE EXAMPLES
Finally, in Figs. 8, 9, 10 and 11 we present several additional qualitative examples that highlight the usefulness of reverse probing. Similar to Fig. 5 in the main paper, we begin by showing self- supervised clusters which are similar; they often contain images belonging to the same ImageNet categories. As a result, when training (reverse) linear probe from ImageNet labels to pseudo-labels (cluster assignments), pairs of clusters with overlapping concepts will typically have high confusion (computed from the confusion matrix). We then add concept group top of the ImageNet labels and train second linear probe, e-g., Fig. we include concepts from Places-365 and SUN At- tributes. We can then identify pairs of clusters for which the inclusion of these concepts significantly reduced the ambiguity of the mapping, therefore reducing the confusion. Finally, for each pair of clusters we show the difference in linear model's coefficients as word cloud, with larger differ- ences denoted by larger font size. In other words, we show the concepts that are most important to distinguish the two clusters (blue for (i) and red for (ii)).
In this way, we were able to discover and understand fine-grained differences between clusters and problematic cases when considering only ImageNet labels. For example, we find that several ImageNet categories appear combination with people (e.g., musical instruments). The quantized space of self-supervised representations (for all methods) appears separate the corresponding images based on whether they contain people or not (examples shown Fig. 8 (a), (c) and (d))- Noteworthy are the clusters in Fig. 8 (d), both of which correspond to the same ImageNet label (hockey puck), although they are visually very different. Including additional concepts, such as person or sports equipeent, justifies their distinction into separate clusters and aligns with human intuition perhaps even more so than assigning them to the same class. In Fig. 9 we show examples where clusters with images from the same ImageNet category differ by scene-related concepts such as flying vs. perched birds, wolves and orcas in different environments and even the inside vs. outside of a building. Material categories (Fig. 10) are helpful to understand environment or context and whether images contain hands (skin). Finally, texture (Fig. 11) can be used to tease apart details at a macro level, e.g., cut and whole fruits or peacocks with open or closed plumage. Overall, texture becomes less relevant models which are better semantic discrimination. We also found that other concepts, such as text, sentiment and image quality do not significantly affect the clustering.
To summarize, we have proposed reverse linear probing as a way to understand whether interpretable clusters form in the representation space of self-supervised methods. While provide quantifi- able measure for this - more interpretable clusters will be predicted with higher accuracy - we also show that we can qualitatively identify how concepts are encoded in the quantized representations through the linear probe's coefficients. We finally show that such concepts are often complemen- tary ImageNet labels and can in fact correctly push clusters which are seemingly semantically close -ooee-iie based on fixed label set- farther apart (e.g., Fig. (d)).
21

---

Published as a conference paper at ICLR 2022
(i)
(ii)
[IID] bicycle helmet
(coGo thingel motorcycle Koj hote
Icoco-things person
Coco-tingd pottedd COCD theest bes Po
[OID] person
(a)
GoOg and beat
[ID motorcycle
COOO--hiings bicycle
icoco-tingal Iran
[ID
man
[ID] helmet
aeteamber be [OID) wheel
MoCo(v2)
HARD
COCO- things
suitcase
COCO thiggal laptop poroppiteeeoo mingal
making meter
[COCO things car
[OID]
calculator
aton
(b)
[CoCO- GOco-things microwave
OID washing machine and bosten
coco-things cell phone
10O0 thingel
COCO things] book
OID car panga
RC vis
COCO things) bed
SwAV
OID man
COCO deggel
[OID] pester
CCOC things dining table
[OID] person
GoCO
Goco tangs pnocols
CoCd Hiopl treistare (cooo-thinga bed
(c)
human
ODD phoses
face
|coco-thimg
person
coco things) clock
(OID)
microphione
fart
COcO-things kite
OID clothing
2000 thingal lefe
DeepClustertv2)
(DD) poster OID equipment
Wassee eRAA
|COcO- things clock
1000
CCCC things) stop sign
(eaco
deegs donng babl
[OD footwear
CCCC- cocO-things skis
paac Hep
[OI)
(d)
(ID) wall clok COCO-things
fboand
OID person
(0c0
CAD
Ditan
[OI) ball OOD traffic sigh
REOT
-Coo-thiigs frisbee
COCO things) parking, meter
OD
SimCLR
Figure 8: Clusters found by unsupervised methods, where each pair (i)-(ii) contains the same (or similar) ImageNet label(s). Confusion is high when probing with ImageNet categories alone, but is significantly reduced after including Object concepts from experts trained COCO-thing and Open Images (OID) categories. The word clouds show the difference in the regressor coefficients for each pair (absolute value is denoted by increasing font size with blue: (i) > (ii) and red: (ii) > (i)).
22

---

Published as a conference paper at ICLR 2022
(i)
(i)
claaces airfield
Pleccal au, Jeps
ISUN
|Paacs florist shop/indoor
[SUN] no horizon
[Places]
aquarium
(a)
[Places] field
hutte
road
[Places] rainf (2.NN 2ON1 FIEE
waly All
Mo boturiiaal
garden
[Places sky
[Places] golf
course
Places veterinarians office
SeLo(v2)
Pllaes snowfield
B. [llacss Wa vattenng hole
[ igloo
Places land
[SN vegetation SUN Iatural light
(b)
[SUN]
cold
[Places] field
Pleas bert
road
Placess natural history
museum
Places
The meer
Ice
* Peathos
floe
(Plaess) rainforest [llacss] pasture
SWAV
[Places] water park
[SUN] keoo... [Places] shelf
[Places] ice. floe
(Flies baiin [Places] aquarium mani-oaae
Places) swimming hole [Places] lagoon
SUN ocean [Places] islet (un)
boating
[Paces]
(c)
[SUN] natural [SUN] open area patoe
feek meger exicatiio (Piaces) acegi
[Places] Cumiiner
pool /indoot
Yoca patta
Places underwator occan doep
[SUN] man- man--aade
[SUN] far-eway benicnn [SUN] diving
Places] Backe//nttraaSSSUU [SUN] swimming
DeepClustertv2)
(Places] pagoda [Places arch
[Places] temple /asa
[Places] church indoor
[Places] church outdoor
comparent Places) courthouse
[Places] atrium/ public
(d)
[Places] staircase
Places
(SUNE touring
[SUN] horizon
SUN enclosed area Aodio
Macen butal chanker
[Places] mosque outdoor
[Placss] elevator shaft
S.N0 open ARER
MoCHI
Figure 9: Clusters found by unsupervised methods, where each pair (i)-(ii) contains the same (or similar) ImageNet label(s). Confusion high when probing with ImageNet categories alone, but is significantly reduced after including Scene concepts from experts trained on Places-365 categories and SUN Autributtes. The word clouds show the difference the regressor coefficients for each pair (absolute value is denoted by increasing font size with blue: (i) (ii) and red: (ii) > (i)).
23

---

Published as a conference paper at ICLR 2022
(i)
(i)
MINC] Foliago Maliagellocoaalfi brunch
taicn (CO00 staff Food otrigt
CCCC stuff]
leaves
1000 calt
COCO-stuff plant-other
stuff) table 6o00 Loal reedd
dlor [MINC] food rerpoe
(a)
Sea nor dal (CCC stuff) wall other
[COCO-stuf]] bush 10000 muth window
Kooo yos CDCD-bufff ligttoee totfiii
1CCCO-stfff vegetable
COCD sulf saldi ea 201 stane
juke etter [COCO-stee -tuff)t sky-other
CCCD-stuff plastic [0000 staft Blor oie (CDCO staff cardboard.
SeLo(v2)
MINC| fabrie pavement
[MICC]
polishedstone Fastod 1OCO staff Fance
COCO-
IMINCO pointed
[MNC
metal
DOCo-stufi
metal
sooff table
stuff)
(b)
DOCo-stuffi Counter
COCO-stur stuff] desk stuff HuNN)
CoCo stuff] f light [MINC] other
surl
cage
[CocO-stuue stuff cabinet
MoCHi
COCO--tuff floor-tile 10000 staff haaee
(MNC) follage
(COcO-stur
metal
COCOO I
ground-other
MING|
skin
(c)
[MINC]
stone
wall-stone
[MINC] [MINC] other
IM plastic
COCO-sttuff
PuINC
dirt. Iasa
cocoo
coco taan been COCD begpebbblle
BYOL
[COCO-Stee stuff]
snow
MMA del Beo
Do00 andtia
(COCO-stual stuff]
my
[INCC
Cooo
coco
CCOC COCO-stuff
wood
stor
[CCC
stuff
branch
(d)
plant
[cOco-stuue
sky-other
fooco woseods (BocB
cOCO-stuff metal 10000 shell Jton
(bitt--nf fnod-athrr
(ANC carpet COCO-stuff] gravel
CCCO stuff dirt [C000-stuff] tree
COCO-stuff) ground-other
OBoW
Figure 10: Clusters found by unsupervised methods, where each pair (i)~(i) contains the same (or similar) ImageNet label(s). Confusion is high when probing with ImageNet categories alone, but is significantly reduced after including Material concepts from experts trained on COCO-stufi and Material Context (MINC). The word clouds show the difference each pair (absolute value is denoted by increasing font size with blue: (i) > (ii) and red: (ii) > (i)).
regressor coefficients for
24

---

Published as a conference paper at ICLR 2022
(i)
(ii
DTD erystalline
[TD freckled
[DDD]
spiralled
DDD
veined
[TTD] swirly
Color red
[Did] porous
[DTD]
cracked
(a)
[DTD]
bumpy
[Color]
green
Calori pink coaly
[DD]
frilly
DTD]
bubbly
[DTD]
cobwebbod
poantam
DDD
pitted
prilow
[DTD] sprinkled
DDTDIblochh
(DTD] blotchy
MoCHi
[Colo] red
DTD] scaly
DTD] paisley
[DTD] matted
(DTD) obbwerged
Jow (Calor] blue
DTD)
stratified
[DTD]
potholed
(b)
DTD]
bumpy
[DTD]
frilly
[DTD]
Eoo Notted
dotted
(DTD] bubbly
[DTD]
smeared
[DTI swirly
pirted
(DTD) veieed
[Color] green(oTD) wrinkeed
(DTD)
porous
McColv2)
[DT]]
DIDI zizzagged
cobwebbed
(DD
wrinkled
VIT
DTD sprinkled
SITO
purlec
CColoi
black
[DTD) perforated
(c)
[TD] studded
(TT)
meeher
oidl tno
[DTD]
frilly
[Color] orange
(oTD) braided
(DTD]
spiralled
(DTD)
potholed GGnly
[DTD] grid
Color
[DTD]
crosshatched
SWAV
DTD]
stratified DTD)
[DT1]
fibrous
interlaced
[DTD]
wrinkled
[DTD]
flecked
(d)
[TD
bubbly
(TD
smeared
Crler
bls
(OTD)
dottod
(DTD)
knitted
[TD]
porous
DTD crystalline
[DTD]
pitted
Mprch
MoCo(v2)
Figure 11: Clusters found by unsupervised methods, where each pair (i)-(ii) contains related Ima- geNet label(s). Using Texture concepts from experts on the Describable Textures Dataset (DTD) and 11 elementary colors helps reduce the confusion comparing to using only ImageNet categories. Clusters where texture helps are usually of lower quality and more recent methods depend less on textures.
25

---

Published as a conference paper at ICLR 2022
F RELATION TO TOPIC MODELS
Probabilistic topic models, such as LDA (Blei et al., 2003), often used in natural language process- ing, aim recover the latent semantic structure, i.e. of documents. The goal is to find the collection of the topics that best represent the corpus, with each document typically containing multiple topics in different proportions. However, the unsuper- vised nature of these models and the lack of gold standard makes their evaluation a long- stand-tnninng problem. For this reason, several automatic "topic coherence" measures have been proposed that evaluate the degree of semantic similarity of the top words in a topic (Mimno et al., 2011; Aletras and Stevenson, 2013; 3; Newman et al., 2010; Röder al., 2015), while human evaluations are also common (Chang et al., 2009; Morstatter and Liu, 2018).
of abstract "topics", from collection
There is a similarity between topic coherence and the way we evaluate models. However, this analogy breaks down rather quickly. To discuss this, can directly map the terms of topic models (topics, documents and words) to our scenario (clusters, images and concepts) in this order. The semantic coherence of topic (cluster) is often measured by the co-occurrence of words (concepts) across documents (images). One immediate difference that arises is that in topic modeling the original space (corpus) is authored by humans and be thus considered interpretable. Measures of coherence are introduced to understand whether the topics found by a model are also interpretable, i.e. evaluating the model producing the topics. our case, we do know the degree which the original space (representation) is interpretable and this is precisely what we aim to quantify with our method.
Another conceptual difference is that topic models operate on a higher level of abstraction: topic collection of words that describe higher-level idea defined by this collection of words. This is sensible as the goal in topic modelling is to discover which words describe higher-level ideas and can be likely grouped together vs. irrelevant words.
In our case, this is not necessary, as the expert annotations already define the relevant concepts. A further abstraction is unnecessary and fact may even be undesirable because we are interested in the minimum number of concepts that may explain a cluster. For example, a topic of "animals" may be coherent, yet a cluster of "animals" " lacks specificity because may contain a multitude of classes.
Finally, the main mechanism evaluating topic coherence is co-occurrence of words in the general corpus (documents). In our case, this translates visual concepts co-occurring in the same image and models relationships between concepts in the images/the real world, whereas we are interested in the relationship between clusters and concepts they have been learned by the model.
26


Reviews:
[{"review_id": "xBfxqhLgrDX", "reviewer": "Reviewer_phRe", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed reading the paper - it is well written and the proposed approach is simple. In general, interpretability or maybe rather explainability of representations is a subject worth studying, and I appreciate the effort that went into experimentation in this work. However, there are also issues that I will describe below that relate to 1) positioning with respect to related work, and 2) the ability for the reader to draw insights from the analysis, among more minor aspects, that would significantly strengthen the work. In the current state I believe the paper is not yet ready for publication at ICLR.\n\n**Related work: topic models**\n\nIn essence, the approach proposed here boils down to a linear model that predicts an output class for inputs in the shape of vectors that indicate image attributes (e.g. object presence, texture, etc). While this type of approach might be new in the assessment of semanticity of image representations, it is strikingly similar to classic topic models in NLP (e.g. a model identifying topics in documents represented as bag-of-words). The authors propose a measure to essentially measure the quality of those \"topics\" (i.e. clusters in this work), which is a well studied area in NLP where it is usually referred to as \"topic coherence\", e.g. [1]. All sorts of measures have been proposed to measure topic coherence, including those based on (pointwise) mutual information. I would therefore encourage the authors to take a deeper look into this relationship - it may strengthen the paper if the relationship of this approach to other methods can be understood more thoroughly.\n\n**Ability to draw insights**\n\nThe authors run plenty of experiments using recent methods to assess their semanticity, which I think should be the focus point of this paper. While I appreciate the effort that went into this, I think how these results are currently presented (is there a better way to present the findings than table 2?), and discussed falls short of what I expected given the focus of this work. While the authors provide some insights w.r.t. to different types of approaches (e.g. clustering vs contrastive) it would be useful to link this to more established taxonomies (see e.g. [2]). Grouping approaches based on some form of taxonomy may make it easier to identify differences. This is an area where this paper could shine and provide actionable insights for the practitioner but at the moment falls short of that expectation.\n\n**Choice of K**\n\nThe choice of the number of clusters seems a bit arbitrary to me, particularly as it impacts the (relative) performance of different methods. The authors outline in the appendix (B.2, in particular Fig 7) how BYOL's performance improves a lot with much larger K, essentially being en-par with the best performing methods. Yet in the main text the authors mention specifically that more recent methods (like BYOL) don't perform as well as others - Would the authors draw different conclusions if K was larger? Again, these differences may be alleviated with a more adequate grouping of approaches across experiments. Also in section 4.5 the authors drop K to 500, why?\n\n**Limitations of a linear probe**\n\nThe authors outline how the proposed approach allows more complex relationships between image attributes to be captured. Yet they only present results for a linear model (predicting the cluster in representation space). How well would a non-linear model do in this task, which would be able to capture more elaborate relationships between image attributes?\n\n[1] Roder et al. (2015) Exploring the Space of Topic Coherence Measures \n\n[2] Jing et al (2020) Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey", "summary_of_the_paper": "This paper presents an approach to investigate the semanticity of representations learned via self-supervision applied to images. The approach measures (via mutual information) how well a linear model can map a vector indicating image attributes (e.g. objects, texture, etc) to clusters within the representation space. The authors apply this method to recent methods from the literature and interpret the resulting ranking, aiming to learn more about which methods produce representations that map well to human judgements of similarity.", "summary_of_the_review": "The paper is well written and the proposed approach is simple. In general, interpretability of representations is a subject worth studying, and I appreciate the effort that went into experimentation in this work. However, there are also issues that relate to 1) positioning with respect to related work, and 2) the ability for the reader to draw insights from the analysis, among other aspects, that would significantly strengthen the work. In the current state I believe the paper is not yet ready for publication at ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "v2_Um2cjW-", "reviewer": "Reviewer_RSYr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I actually really enjoyed reading this paper and found the contribution to be compelling. My main suggestion is to surface your insights and findings in the abstract and introduction. These sections, currently, only describe your method and not your findings. The abstract and introduction made me ask “so what?”. The findings are buried; some in the appendix!\n\nI wish the choice of K-means as the clustering algorithm was questioned. For example, if you used hierarchical agglomerative clustering, would that lead to clusters where different levels of the hierarchy could represent compositions of children cluster concepts.\n\nWhat I think is missing:\n1) Which concepts are not captured by any of these models?\n\n2) Which models are capable of decoupling individual concepts (identifying each concept in isolation) versus only compositions of concepts?\n\n3) How does interpretability correlate with transfer to other downstream tasks like object detection, segmentation, etc. I know the paper includes a transfer experiment to Places but there could have been a more thorough exploration of transfer learning.\n", "summary_of_the_paper": "This paper studies how human-interpretable are the concepts learned within the representation space of self-supervised models. It argues that linear probing methods are unable to identify whether a representation space contains a specific concept because the input might contain multiple confounding concepts (“red apple”) and might encode them in a manner that renders “red” and “apple” individually linearly inseparable. Instead, it proposes a “reverse linear probe” moving from which maps combinations of binary concept labels to the representation space, which is clustered using k-means.\n\nThe paper uses this linear probing method to propose a normalized mutual information metric to measure how well human-interpretable concepts are encoded in the representation space. Even though they find that their metric is correlated with linear probe, there are some interesting insights. First, the paper finds that models trained on ImageNet images capture more than just semantic class labels. They capture information about textures, scenes, objects, etc. Second, they found that additional training epochs dont lead to more interpretable representations even though it increases end-task performance. They also identified that OBoW, does really well even with only 200 epochs (though the reasons for why is still left to be investigated). Clustering-based approaches generally produce more interpretable representations. Fourth, ViT is a better architecture. Finally, qualitative evaluation of clusters seem to justify their utility of the reverse probe since NMI increases as when accounting for multiple categories for objects that occur together (ex, french horn and people). \n", "summary_of_the_review": "Overall, I am still quite satisfied with where the paper currently stands. I listed things I would like to see improved or discussed. I also listed some potential questions for investigation. But these questions do not detract from my overall positive impression of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "d5oGu_Z4QJd", "reviewer": "Reviewer_UJ5M", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n[S1] The paper tries to answer a very important question: to what extent the self-supervised visual representations can be interpreted by humans. This question is important and interesting as these representations were learned without human interventions but showed to be very effective for many downstream recognition tasks. While there are many task-oriented evaluations, systematic analysis of the interpretability of the representations is missing.\n\n[S2] Formulating the “interpretability” as the mutual information between the representations and the semantic concepts is technically sound.\n\n[S3] The analysis covers a wide range of visual concepts and pretrained self-supervised models.\n\n[S4] The paper is well written.\n\n—------------------------------------------------------------------------------------------------------------------------\n\nWeaknesses\n\nThe proposed Quantized Reverse Probing is claimed to be complementary to linear probes with further advantages. At this point, I don’t think this claim is supported by the method or the experiments. To me, the proposed method is too similar to linear probes that they share the same drawbacks.\n\n[C1] “A limitation of this type of approaches is that one can only discover in the representation meanings that are represented in the available data annotations”. I’m afraid both linear probing and the proposed method have this problem, the proposed method relies on pseudo labels generated by the expert models that were trained with data annotations.\n\n[C2] “The representation might be predictive of combinations of attributes, i.e. it might understand the concept of ‘red apple’ without necessarily understanding the individual concepts, e.g. ‘red’ and ‘apple’.” This statement is correct. Unfortunately, neither linear probing nor the proposed method can resolve this problem, as they both are linear functions, so score(‘red apple’) == score(‘red’) + score(‘apple’) . In order to recognize “red apple”, both methods should have high scores on “red” and “apple”. In other words, the claim that the proposed method can better handle the compositions of the concepts is incorrect from the methodology perspective.\n\n[C3] “The proposed method provides a single principled figure characterizing the entire representation instead of a separate figure for each predicted attribute”. With the proposed linear mapping, “this principled figure” is obtained by simply averaging the edge weights between the concepts and the clusters. I don’t see why linear probing cannot obtain such a figure by averaging the per-category scores.\n\n[C4] “our measure can be computed much faster than linear probes”. Please elaborate more on this, as at first glance I thought the proposed method is slower as it involves clustering.\n\n[C5] From the experiments, it is still not obvious to me that the proposed method is better than linear probing. Some observations listed in the paper are as expected or already pointed out by previous literature: deepcluster-v2 performs better as the optimization objective of deepcluster-v2 is pretty similar to the optimization objective of the proposed metric; also the evaluated self-supervised methods have very similar performance so their rankings may be sensitive to the metrics; ViT + Self-supervised-learning performs better on KNN classifications (and clustering ) was already observed by DINO, etc.\n\n—------------------------------------------------------------------------------------------------------------------------\n\nMinor comments:\n\nPage 2, “either only cluster (Yan et al., 2020a) or cluster and further train and …”\n\n", "summary_of_the_paper": "The paper aims to measure the interpretability of the visual representations learned by the recent self-supervised models. In doing this, the paper formulates the “interpretability” as the mutual information between the feature clusters and a set of visual concepts that can be interpreted by humans. The mutual information is approximated by the proposed Quantized Reverse Probing, a linear function that maps the concept vectors to the feature clusters. The proposed method is claimed to be complementary to linear probes, with further advantages: (1) a single principled score rather than individual prediction scores for different labels in linear probes (2)  handle combinations of concepts better than linear probes. (3) Faster than linear probes.", "summary_of_the_review": "While the problem studied in the paper is very interesting, at this moment, I don't see there are any obvious advantages of the proposed method over a simple linear probing, as discussed in the \"weaknesses\". Therefore, I'm leaning toward a rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "SbKJpUMgpee", "reviewer": "Reviewer_AZhK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "On the stronger side, the proposed method aims at addressing a task, post-hoc interpretation of pre-trained models, that had received much reduced attention compared to its explanation counterpart. As such it could help push research in this direction further.\nIn addition, the proposed method is evaluated considering a good amount of self-supervised learning methods.\nFinally, code related to the paper will be released upon the publication of the manuscript which is always from the point of view of reproducibility.\n\nOn the weaker side,\n\nWhile the idea of having the proposed probing in reverse order is, to best of my knowledge novel to me, it is not clear to what extent the proposed method does provides insight in the representation learned by the model being probed. Is is by providing semantic means (via y(x)) to describe some modes (f_K(x)) in the representation f(x) ? Perhaps it should be made more explicit the way in whic the proposed method makes the interpretation, of the model being analyzed, possible.\n\nI find the way in which the paper approaches the idea of finding relationships between the internal representations learned by a model and a set of concepts very similar to [Escorcia et al., 2015], that did it at the attribute level, and to [Oramas et al., 2019], that further explore it towards interpretation of learned representations. \nUnfortunately, there is no descriptive nor quantitative comparison with respect to these methods. Given the similarity, I would suggest properly positioning against them.\n\nAt this point it is not clear how the quantized representation $f_K(x)$ is actually computed. For instance, is the internal representation f(x) taken at the layer level, fed to the clustering algorithm and the its result concatenated with that from other layers. Or perhaps you feed the concatenation of all the internal activations to the clustering components, or only use a specific [set] of layers, e.g. the last convolutional layers? Are there any normalization steps involved?\n\nIn Section 3, it is mentioned that \"we can further and cheaply increase the coverage of the label space by predicting the labels y(x)\nautomatically via a battery of expert classifiers learned using full manual supervision.\" While this indeed may seem as a positive and cheap means to further get annotations, as properly noted by the paper, it indirectly requires the existence of the expert predictors and the annotated data on which they were trained. While applicable in the more general imageNet like testing set considered in the experiments, this requirement might not in place to more specific/niche applications, which will limit the applicability of this practice.\n\nAt the end of Section 4.2, by using the proposed method, it is observed and stated that methods that use a clustering mechanism during training have generally more interpretable representations. Doesn't this observation originates from the fact that the proposed method relies on an intermediate clustering step and as such favors methods that rely in similar clustering steps as well. If this indeed the case, perhaps there is some bias indirectly injected by the method that favors some types of methods over others.\n\nIn Section 4.4, it is observed that confusion between different clusters decreases when additional concept groups are added. This observation is used to further stress the fact that there could be interpretable properties in the network that may remain undetected since they are not annotated a-priori in the benchmark data and that this further motivates the need for reverse linear probing.\nIn this regard, this is a known limitation of standard linear probing methods whose interpretation capabilities are bounded by the set of pre-defined annotated concepts. In my opinion, this is also a limitation of the proposed method which, as shown in Section 4.4. does require additional pre-defined concepts in order to decrease the confusion of the identified clusters. Therefore I do not agree with the statement that the proposed reverse linear probing is a solution to the problem.\n\nFinally, it might be good to provide some details on the linear models mentioned at the end of Section 4.1. Is this a single [fully-connected] layer neural network trained for several epochs or perhaps something more elaborated? ", "summary_of_the_paper": "The paper proposes a method for characterizing the \"meaning\" of the representation learned by a given model (interpretability). Towards this goal, it proposes reverse linear probing, a post-hoc method that aims at predicting a quantized version of the internal representation (as observed in specific examples) from semantic label.\n\nSpecial emphasis is given in the capability of the proposed method on enabling the interpretation of models learned in a self-supervised manner.\n\nPROs\n+ Code will be released upon acceptance.\n+ Evaluation covers a good variety of models.\n\nCONs \n- Missing related literature\n- Unclear how interpretability is gained.", "summary_of_the_review": "The proposed idea is sound and, to the best of my knowledge, novel. I believe the proposed method can perfectly complement the somewhat reduced amount of efforts along the line of interpretation of pre-trained models. However, I believe several aspects (please see my review) that should be polished/clarified before the paper is ready for publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for inspecting and interpreting the visual representations learned by self-supervised methods. 
The method is conceptually simple and intuititive, the authors assume that concept labels for the images are available, and then go on to learn a mapping between the learned image vectors and the human-provided descriptions of the images. The key insight is to learn a reverse mapping, i.e., to map label vectors to representation vectors. Specifically, feature vectors are quantized using k-means to obtain clusters;  images are labeled (automatically) with a diverse set of concepts from expert models trained with supervision on
external data sources, and  a linear model is trained  to map concepts to clusters, measuring the mutual information between the representation and human-interpretable concepts.

Reviewers raised some questions regarding the relation of the approach to topic models, the difference between reverse probing and linear probing, implementation details and computation. The authors addressed reviewers comments convincingly with additional experiments and/or explanations.
Decision: Accept (Poster)