Year: 2021

Abstract:
Self-supervised visual representation learning has attracted significant research interest. While a common way to evaluate self-supervised representations is through transfer to various downstream tasks, we instead investigate the problem of measuring their interpretability, i.e. understanding the semantics encoded in raw representations. We formulate the latter as estimating the mutual information between the representation and a space of manually labelled concepts. To quantify this we introduce a decoding bottleneck: information must be captured by simple predictors, mapping concepts to clusters in representation space. This approach, which we call reverse linear probing, provides a single number sensitive to the semanticity of the representation. This measure is also able to detect when the representation contains combinations of concepts (e.g. “red apple”) instead of just individual attributes (“red” and “apple” separately). Finally, we suggest that supervised classifiers can be used to automatically label large datasets to enrich the space of concepts used for probing. We use our method to evaluate a large number of self-supervised representations, ranking them by interpretability, highlight the differences that emerge compared to the standard evaluation with linear probes and discuss several qualitative insights.

heading: 1 INTRODUCTION
text: Relying on black-box models such as deep networks comes sometimes with significant methodological and ethical challenges. This is particularly true for unsupervised and self-supervised models which are learned without human supervision. While these models perform increasingly well in downstream applications, often outperforming supervised counterparts, there is very little understanding of what they learn, making their real-life deployment risky.
In this paper, we thus consider the problem of characterizing the meaning of data representations, with a particular focus on unsupervised and self-supervised representations of images. Given a representation f mapping images x to representation vectors f(x) ∈ RD, our goal is to find whether these vectors contain human-interpretable information. This is usually done by finding a relationship or correspondence between f(x) and human-provided descriptions y(x) of the images, essentially translating the information from the representation space to a concept space. Consider for example the popular linear probing method (Alain and Bengio, 2017). Given a large dataset X̂ of images with corresponding manual annotations y(x), one learns linear classifiers (probes) to map the feature vectors f(x) to labels y(x), measuring the resulting classification accuracy. If the predictor is accurate, then one can argue that the representation captures the corresponding concept y.
One possible issue with this type of approaches is that one can only discover in the representation concepts that are represented in the available data annotations. In order to maximize the semantic coverage of the analysis, it is thus customary to combine annotations for several types of attributes, such as object classes, textures, colour, etc. (Bau et al., 2017). In this case, y is a vector of image attributes, and one can compute several linear probes to predict each individual attribute yi. By doing so, however, attributes are treated independently, which may be unsatisfactory. In order to obtain a single number assessing the overall semanticity of a representation, one must combine the prediction accuracies of several independent classifiers, and there is no natural way of doing
so (e.g. a simple average would not account for the different complexity of the different prediction tasks). Furthermore, the representation might be predictive of combinations of attributes, i.e. it might understand the concept of “red apple” without necessarily understanding the individual concepts, “red” and “apple”. While it is in principle possible to test any combination of attributes via linear probing, most attribute combinations are too rare to generate significant statistics for this analysis.
In this paper, we thus propose a complementary assessment strategy to address these shortcomings. In contrast to linear probes, we start by considering the reverse prediction problem, mapping label vectors y(x) to representation vectors f(x). A key advantage is that the entire attribute vector is used for this mapping, which, as we show later, accounts for attribute combinations more effectively.
Next, we consider the challenge of deriving a quantity that allows to compare representations based on the performance of these reverse predictors. Obviously a simple metric such as the average L2 prediction error is meaningless as its magnitude would be affected by irrelevant factors such as the scale of the representation vectors. To solve this problem in a principled manner, we consider instead the mutual information between the concepts and quantized representation vectors. This approach, which we justify from the viewpoint of information theory, essentially measures whether the representation groups the data in a human-interpretable manner.
We use our approach to evaluate a large number of self-supervised and supervised image representations. Importantly, we show qualitatively that (a) all methods capture interpretable concepts that extend well beyond the underlying label distribution (e.g. ImageNet labels, which are not used for training), (b) while some clusters that form in the representation space are purely semantic, others carry information about scenery, material, textures, or combined concepts, and (c) context matters. We also show that more performant methods recover more of the original label distribution, i.e. learn better “ImageNet concepts”, and rely less on lower-level concepts. Quantitatively, we observe that our interpretability measure results in a similar but not identical ranking for state-of-the-art methods and that clustering-based approaches produce generally more interpretable representations.

heading: 2 RELATED WORK
text: Self-supervised representation learning (SSL) In this paper, we focus on self-supervised methods that learn representations from image data. Early self-supervised learning approaches devise numerous pretext tasks — such as colorization, predicting image rotations or solving image puzzles — to learn useful representations from unlabeled images (Gidaris et al., 2018; Pathak et al., 2016; Noroozi and Favaro, 2016; Doersch et al., 2015; Larsson et al., 2016; Zhang et al., 2017; 2016). More recent approaches follow the contrastive learning paradigm (Chen et al., 2020a; Frankle et al., 2020; Kalantidis et al., 2020; He et al., 2020; Chen et al., 2020b; Hénaff et al., 2019; Oord et al., 2018; Misra and Maaten, 2020; Tian et al., 2019; 2020; Wu et al., 2018), where the goal is to discriminate instances belonging to the same image from negative samples. Other methods (Zbontar et al., 2021; Chen and He, 2020; Grill et al., 2020; Caron et al., 2021) eliminate the need for negative samples. While there exists some theoretical approaches for explaining why the constrastive learning works (Arora et al., 2019; Wang and Isola, 2020; Tschannen et al., 2020; Tsai et al., 2020), in most cases the focus lies on showing empirical improvement in downstream tasks.
Clustering A related part of literature deals with unsupervised image clustering. Early approaches include using autoencoders (Hinton and Salakhutdinov, 2006), agglomerative clustering (Bautista et al., 2016) and partially ordered sets of hand-crafted features (Bautista et al., 2017). More recent methods combine representation learning with clustering, using mutual information (Ji et al., 2019; Hu et al., 2017), K-means (Caron et al., 2018; Zhan et al., 2020), prototypes (Li et al., 2021) or optimal transport (Asano et al., 2020; Caron et al., 2020). Other methods build on strong feature representations and, at a second stage, cluster and further refine the network (Yan et al., 2020a; Van Gansbeke et al., 2020; Lee et al., 2019). These methods produce pseudo-labels which can be used to evaluate representations by measuring the correlation to the ground-truth labels.
Evaluation and analysis of SSL There are two main directions in evaluating self-supervised learning methods. The first one uses pre-trained representations as an initialization for subsequent supervised tasks, e.g. for object detection. The second approach is to train a linear classifier on the embedding spaces which are kept frozen after pre-training. As a result, several benchmarking suites with various datasets and tasks have been proposed (Zhai et al., 2019; Goyal et al., 2019b; Van Horn
tion, extracting and quantizing feature vectors to obtain clusters ( ... ), (2) we label the image data with a diverse set of concepts ( ... ) from expert models trained with supervision on external data sources, and (3) we train a linear model hθ to map concepts to clusters, measuring the mutual information between the representation and human-interpretable concepts.
et al., 2021; Kotar et al., 2021). Complementary to this, clustering in the frozen feature space has also been proposed as an evaluation metric (Zheltonozhskii et al., 2020; Sariyildiz et al., 2020). In addition, downstream dataset (Ericsson et al., 2021) and pretraining dataset (Zhao et al., 2021; Cole et al., 2021) dependencies have been evaluated. Finally, recent investigations of self-supervised feature spaces aim to understand separability (Sehwag et al., 2020), concept generalization (Sariyildiz et al., 2020) and the effect of balanced vs. long-tailed training data (Kang et al., 2020).
Interpretability Although a large amount of work has studied feature representations in Convolutional Neural Networks (CNNs), it is heavily focused on models trained with full supervision. Zeiler and Fergus (2014) and Zhou et al. (2014) analyze networks by visualizing the most activating patches, while activation maximization methods (Mahendran and Vedaldi, 2016a;b; Nguyen et al., 2017; 2016; Olah et al., 2017; Simonyan et al., 2014) generate inputs to activate specific neurons.
Another line of work focuses on understanding what information is present in intermediate representations of CNNs, usually mapping activations to high-level concepts. Alain and Bengio (2017) introduce linear probes as a means to understanding the dynamics of intermediate layers, by predicting the target labels from these layers. Escorcia et al. (2015) also use a linear formulation to study the relationship of mid-level features and visual attributes, while Oramas et al. (2019) adopt this to predict task-specific classes from the features (similar to Alain and Bengio (2017)), select relevant features and generate a visual explanation . Similarly, Zhou et al. (2018) decompose feature vectors into a set of elementary and interpretable components and show decomposed Grad-CAM heat maps (Selvaraju et al., 2017) for concepts that contribute to the prediction. Furthermore, Kim et al. (2018) relate feature vectors and human-interpretable concepts using a set of user-provided examples for each concept and Bau et al. (2017) propose to quantify the interpretability of individual units by measuring the overlap between each unit and a set of densely annotated concepts, Finally, Ghorbani et al. (2019) propose a method to automatically assign concepts to image segments and Yeh et al. (2020) analyze the completeness of these concepts for explaining the CNN.
With the exception of (Laina et al., 2020; Bau et al., 2017; Fong and Vedaldi, 2018), relatively little work has been done to understand the emergence of interpretable visual concepts in self-supervised representations specifically. In particular, Laina et al. (2020) quantify the learnability and describability of unsupervised image groupings, by measuring how successful humans are at understanding the learned concept from a set of provided examples. Instead, our approach does not depend on human input and is thus easily scalable to a wide range of methods and number of classes.

heading: 3 METHOD
text: We are interested in measuring the semanticity of data representations. A representation f is a map f :X → RD that encodes data samples x∈X as vectors f(x)∈RD. In this paper, we assume that the data are images X = R3×H×W and f is a deep neural network, but other choices are possible. The “meaning” or semantic content of an image x can be characterized by obtaining a label or description y(x) of the image from a human annotator. If the representation f(x) captures the
meaning of the image too, then it should be predictive of the description y(x). The mutual information I(f(x), y(x)) provides a natural measure of predictivity, and it is thus tempting to use this quantity as a measure of the overall semanticity of the representation. However, information as such is not a useful measure in our context. To see why, note that, due to the data processing inequality (I(f(x), y(x)) ≤ I(x, y(x))), the information is maximized by observing the raw image, i.e. by the identity representation f(x) = x. Since data processing cannot increase information content, a useful representation must preserve information while also making it easier to decode and act on it.
There are many possible definitions of what constitutes ‘simple decoding’. Here we propose a definition that stems from the interpretation of differential entropy as the limit of discrete entropy for quantized versions of the corresponding variables (Cover and Thomas, 2006). In our case, the description y(x) is already discrete, but the representation vectors are continuous. We thus propose to vector-quantize the representation and to use the resulting discrete information as a measure of semanticity.
Intuitively, in the limit of infinitely-fine quantization, this quantity reduces (up to a normalizing term) to the mutual information above. For a finite number of clusters, however, a large value of information means that the representation groups images in a way that makes sense to a human observer. In this case, the ‘simple decoding’ amounts to grouping images based on the L2 distance between their representation vectors.
Formally, we use a vector quantization algorithm such asK-means (Lloyd, 1982) to learn a quantizer function µK : RD → {1, . . . ,K} for the representation vectors and estimate the mutual information I(µK(f(x)), y(x)) by rewriting it as:
I(fK(x), y(x)) = H(fK(x))−H(fK(x) | y(x)), fK(x) = µK(f(x)). (1)
In the above equation, the first term denotes the entropy of the cluster assignments. In practice, given a sample dataset X̂ of images, we first run K-means to compute the quantizer µK , and then compute the frequency of cluster assignments fK(x), x ∈ X to calculate the entropy. The second term in Eq. (1) is the conditional entropy:
H(fK(x) | y(x)) = Ep[− ln p(fK(x) | y(x))] ≤ Ep[− ln q(fK(x) | y(x))]
where p is the true joint distribution between variables fK(x) and y(x). While this is difficult to compute, we can upper-bound it by considering an auxiliary posterior distribution q interpreting the conditional entropy as the cross-entropy loss of a predictor hθ, parametrized by θ, that maps labels y(x) to clusters fK(x). We minimize the gap by learning this predictor in the usual way.
In practice, learning the predictor may result in overfitting, which would lead to an over-optimistic estimate of the mutual information. We address this issue by learning the predictor hθ on a subset X̂ ′ ⊂ X̂ of the data and evaluating its cross-entropy loss on the remaining subset X̂ − X̂ ′. Importantly, we consider a linear predictor (probe) for hθ, to further reduce the risk of overfitting. We refer to this as quantized reverse probing, as it maps semantic concepts to quantized data representations.
Obtaining labelled data via automatic experts Similar to prior work on interpretation, our method requires a dataset x ∈ X̂ of images equipped with manually-provided labels y(x) ∈ Y . Because we do not know a-priori which concepts may be capture by a given representation f(x) under analysis, these labels should provide a good coverage for a number of different concepts (e.g., textures, object types, parts, etc.). A good example of such a dataset is Broden (Bau et al., 2017).
Because it would be cumbersome, and maybe even infeasible, to annotate any target dataset with all possible visual concepts, we can further and cheaply increase the coverage of the label space by predicting the labels y(x) automatically via a battery of expert classifiers learned using full supervision. The noise of these automated predictions is small compared to the noise in the correlation between the concepts and the unsupervised/self-supervised visual representations under study.
Number of clusters The number of clusters K controls the size of the bottleneck. Mutual information increases monotonically as K increases, as we show in the Appendix (Fig. 6); when comparing representations, it is thus important to fix a value of K and use it consistently. For added convenience, in practice we report a normalised value of information (NMI).
Relation to linear probing Linear probing is a common approach for assessing representations, related to our method. A linear probe is a simple predictor hi ◦ f(x) ≈ yi(x) that maps the representation f(x) to a specific concept yi (e.g., a binary attribute or a class label). The idea is that the representation captures concept yi if the probe performs well. As a result, linear probing is a standard evaluation protocol in self-supervised representation to measure the (linear) classification accuracy of the learned representations against a labelled set, commonly ImageNet (Russakovsky et al., 2015). In interpretability studies, different formulations of linear probing (Alain and Bengio, 2017) are used to understand intermediate layers of neural networks. Similar to our approach, the simplicity of the probes (imposed via linearity and sometimes sparsity) is a necessary bottleneck for interpretability: without such a constrain, the best possible probe would take the raw image as input.
Our method is complementary to linear probes, with some advantages. To discuss this point further, we show a simplified example in Fig. 2, where data points with color and shape attributes are encoded by two different representations, such that they form well-defined nameable clusters, i.e. blue square, red square, blue circle, and red circle. Forward probes (X → Y) can be trained as binary classifiers h1, h2 for each attribute, mapping a feature vector (in this case, coordinates) to the corresponding attribute value for each data point. If attributes are linearly separable (Fig. 2a), then forward probes do well at separating the representation space. If an attribute is not linearly separable (color in Fig. 2b), the predictive accuracy of the corresponding probe h1 reduces to chance, and consequently reduces the average score. As a result, linear probes cannot always assess the meaningfulness of clusters or, in other words, whether a cluster in representation space responds to a well-defined concept. On the contrary, if all data points in a cluster have consistent attributes, reverse probes (Y → X ) can achieve a high score for, e.g. , “red square” without requiring the concept “red” to be also encoded in the features. In this case, we can say that the model has discovered the concept of a “red square” without identifying, in isolation, the concept “red”. Therefore, reverse probing handles combinations of attributes naturally and allows to better assess the semanticity of clusters.

heading: 4 EXPERIMENTS
text: We use our method to evaluate a wide range of recent self-supervised representation learning and clustering techniques. In the following, we compare these techniques according to the proposed criterion, which results in a new ranking for the state-of-the-art approaches. Further, we show qualitatively how our method discovers elementary concepts which are encoded in the raw representations.

heading: 4.1 IMPLEMENTATION DETAILS
text: Attributes and expert models As a first step in our approach, we collect a set of attributes which we expect to find encoded in the self-supervised representations; these include semantic object cat-
egories, scene types and attributes, materials and textures and possibly other information about the photographic style or overall sentiment of an image (Table 1). We thus look for relevant humanannotated datasets and expert models trained on these datasets, as a proxy to human knowledge. This allows us to extract information related to such attributes on a different dataset or images in the wild; in our experiments we focus on ImageNet (IN-1k) (Deng et al., 2009) as the target data due to the large availability of pre-trained self-supervised models on this dataset. We concatenate all available attributes and form M -dimensional binary vectors denoting the presence or absence of each attribute in an image. We provide full details about the expert models in Appendix A. Since we primarily focus on ImageNet, it is natural to also use the existing 1000 human-annotated labels. As IN-1k already includes several fine-grained categories, we have not included further experts from common fine-grained recognition datasets, such as iNaturalist (Van Horn et al., 2018).
Linear model For each method, we train a linear model mapping a set of input attributes to clusters, which are obtained by K-means clustering on top of fixed representations and evaluate on a held-out set. Further training details are provided in Appendix A.

heading: 4.2 COMPARISON OF SELF-SUPERVISED REPRESENTATIONS
text: We use reverse probing to evaluate recent self-supervised techniques in Table 2 and rank them based on their semanticity. We obtain Jigsaw, SimCLR, ClusterFit from the VISSL model zoo (Goyal et al., 2021) and for the rest we use the official implementations. For each method, we freeze the pre-trained model and extract feature vectors on the IN-1k data. Using all categories from Table 1, we run 5× K = 1000-means to obtain different clusterings, each used to train a linear model. In addition to a normalized measure of mutual information (NMI), we also report the adjusted mutual information (AMI), classification accuracy (top-1) and mean average precision (mAP). We also note the linear classification accuracy, as reported by the respective source for each method. For fairness of comparisons, we also group methods based on the number of training epochs. For reference, we also evaluate the supervised counterparts in the same way as for the self-supervised methods. Overall, we find that there is a relatively strong correlation between our approach and the common linear evaluation protocol (Fig. 3). However, we make the following additional observations:
Ranking Despite the strong correlation to linear classification accuracy, we obtain a different ranking from the viewpoint of representational interpretability, using our approach. In particular, state-of-the-art methods DINO, SwAV and BYOL fall slightly behind in our evaluation. The most recent version of MoCo (v3) performs the best across three metrics, with DeepCluster-v2 and OBoW ranking in the second place (across ResNet-50-based models).
Number of pre-training epochs For some methods, learned weights are available at various epochs. In terms of interpretability, we observe no benefit in longer pre-training in the case of DeepCluster-v2 and SwAV, despite the solid performance gains on standard benchmarks. However, there is a noticeable gain for MoCo-v2, MoCHi and SimCLR between 200 and 800 epochs. Notably, one of the best performing methods, OBoW is only trained for 200 epochs.
Clustering vs contrastive approaches Methods such as SeLa, OBoW, PCL and DeepCluster-v2, which use a clustering mechanism during training, have generally more interpretable representations, i.e. they rank higher than methods with similar linear classification accuracy.
Architecture Interpretability increases with self-supervised methods that use a ViT (Touvron et al., 2021; Dosovitskiy et al., 2021) backbone, showing a significant boost in all metrics, even more so than linear classification accuracy suggests. However, a fair comparison between MoCo-v3 and DINO is not possible due to pre-training for a different number of epochs (300 vs. 800).

heading: Model Lin. Acc. NMI AMI Top-1 mAP

heading: ViT-Base/16

heading: 4.3 EXPERT BREAKDOWN
text: Our next goal is to understand the effect of different concepts on explaining the representation, i.e. we want to answer the question: to which degree does the representation know certain concepts, e.g. material? As we are focusing on self-supervised representations here, the answer to this question is far from obvious. We again measure this via the predictive power of our probe, which we now train individually for each group of concepts. These vary in nature; from highly semantic object categories (e.g. dog, avocado) to lower-level features such as material (e.g. wood, brick), texture (e.g. bubbly, striped) and color (e.g. red).
In Fig. 4, we show how the different concept groups contribute to the overall performance for selected ResNet-50-based methods. We train and evaluate reverse probes for each shown combination, e.g. IN-1K+OBJECTS, then IN-1K+OBJECTS+SCENE, etc. We compare variants with and without using ground truth IN-1K categories as part of the input. We observe that for earlier methods, such as MoCo-v1, ImageNet categories alone are not sufficient for accurately predicting fK(x). In fact, it appears that using only semantic categories from MSCOCO and OpenImages (w/o ImageNet) provides about as much information about the clusters. This likely suggests that the discovered clusters do not reflect fine-grained distinctions, since the most notable difference between IN-1K and the other semantic experts are some fine-grained categories (e.g. dog breeds). On the other side of the spectrum, the best performing methods do indeed owe a big part of their performance to IN-1K concepts, meaning that they are able to recover clusters that are closer to the original label distribu-
tion and that rely less on low-level concepts such as textures. This suggests that more performant methods learn features highly correlated with IN-1K labels, but less with other semantic categories.
Finally, in Table 3, we also show the (decorrelated) effect of each individual attribute group in isolation but always in combination with the ground truth IN-1K categories. We observe that in absence of concept combinations, the models we probed use as much (and sometimes more) information about scene and material properties as information about semantic object categories. Miscellaneous concepts (OTHER) do not appear relevant for more recent methods. We again observe that the representations of the most recent, state-of-the-art MoCo-v3 share very little information with concepts other than IN-1K, despite the lack of label information during pre-training.

heading: 4.4 QUALITATIVE RESULTS
text: Since the clusters we examine are learned without supervision and expert models are not perfect predictors, it is crucial to verify whether the clusters align with human interpretation, besides quantitative scores. By inspecting the probe’s coefficients θ, one can qualitatively explore what concepts are representative for specific clusters and how exposing the linear model to different sets of experts affects its predictive accuracy. In Fig. 5 we show pairs of clusters from MoCo-v2 for which we observed a significant drop in the confusion (estimated from the confusion matrix) after including an additional concept group on top of the fixed ImageNet label set. For example, after including OBJECTS we notice that several pairs of clusters become more easily disambiguated thanks to the concept person, or related concepts (human hand, human face, etc.). In Fig. 5 (top left), both clusters contain the ImageNet label French horn and it is only possible to understand their differences when more information is available. This finding supports our initial hypothesis there could be interpretable properties in the network that may remain undetected since they are not annotated a-priori in the benchmark data. Importantly, since ImageNet is only annotated with a single label per image, combined concepts, such as “person playing the french horn” cannot be discovered otherwise. Similarly, when adding SCENE concepts (top right), clusters are distinguished by scene

heading: Model NMI AMI Top-1 mAP NMI AMI Top-1 mAP
text: type; one contains volcanos, while the other contain mountains and creeks (categories from Places-365). With MATERIAL, we can predict the shipwreck cluster more accurately, while TEXTURE can explain the difference in the more stratified appearance of cliffs for the cluster on the right. It therefore appears that, despite the object-centric nature of ImageNet, self-supervised representations rely significantly on context and concept combinations. This likely also explains why in (Van Horn et al., 2021) ImageNet self-supervised models do as well as or better than the supervised baseline at context, counting and gestalt tasks.

heading: 4.5 TRANSFER TO OTHER DATASETS
text: We finally evaluate the representations learned by selected methods by transferring them to a different domain. For this experiment, the self-supervised methods are not fine-tuned or adapted on the new data. We perform the quantization on representations extracted on images from Places-365 and use all concepts from Table 5 but IN-1K (now using the available ground truth for Places-365). We report the performance in Table 4 and observe that the same ranking generally holds.

heading: 5 CONCLUSION
text: We have introduced reverse linear probing, a measure of representation interpretability that allows to explain representations by combining multiple concepts. Driven by the need to better understand and characterize the representations learned by self-supervised methods, the proposed measure has a rigorous foundation in information theory and provides complementary information to existing benchmarks as well as insights into the semanticity and other interesting properties of such models. The proposed model is also considerably faster to train, as all concepts can be pre-extracted on any given image collection. Code and data will be released upon acceptance.

heading: Category Datasets #Classes Task Model Source
text: Photography. We have observed differences in the photographic style, quality and type of images in ImageNet, e.g. ranging from macro photography, usually for flowers or insects, to vector illustrations. In order to probe the representations for style and image quality, we use the recent CLIP model (Radford et al., 2021), trained on 400 million image-text pairs collected from the web and create a set of candidate sentences: “a high quality photo”, “a noisy, grainy image”, “a blurry image of low quality”, “macro photography”, “a photo with out of focus background, bokeh effect”, “an animated picture, a vector illustration”, “a painting”, “a portrait”. We use the pre-trained CLIP model to compute image and sentence features (normalized to unit norm) and use the dot product to find the similarity between each image and each one of the sentences. We pick the sentence with the highest similarity as the predicted class for each image if the score is at least 0.5, else we assign a background class. Since CLIP operates on free-form text rather than a fixed set of categories, its use is not limited to this type of queries; it can be used as an expert where a labelled dataset might unavailable.
Training details for linear model We evaluate representations from a number of pre-trained selfsupervised feature extractors (Table 2). For ResNet-50 models, we evaluate features at the output of the average pooling layer (2048-d). For ViT-Base models we evaluate the [CLS] token of the last self-attention layer (768-d). For each model, we pre-extract and store features for the entire training set of ImageNet and, prior to clustering, we standardize them to zero mean and unit standard deviation. We then train K-means for 100 steps using faiss (Johnson et al., 2017) and choose the best out of 5 runs. We divide the data into trainval and test sets by splitting all cluster assignments with a 80/20 ratio. Finally, we train the linear model with the cluster assignments as targets and concept vectors as inputs (where the concepts are pre-extracted on a given image collection, e.g. ImageNet). We train for up to 100 epochs with batch size 512 and optimize using SGD with a momentum of 0.9 and initial learning rate of 3.5 which is further reduced by a factor of 10 at epochs 60 and 80. We also add L2-regularization with weight 3× 10−6.
Computation Given a dataset (e.g. ImageNet) and a set of experts (e.g. the ones listed in Table 5), all labels can be pre-computed for all images in the dataset. Feature vectors for a given model can be also pre-extracted and stored for the whole dataset. Our method computes cluster assignments using the fastK-means implementation of faiss (which takes less than 5min for 256k 2048-d vectors on 4 NVIDIA RTX A1000). Training of the linear model converges in less than 100 epochs in a matter of minutes on a single GPU (1 epoch takes 2sec). We should note that standard linear probing on ImageNet typically requires full images (including online augmentations) and multiple forward passes through the frozen feature extractor, which makes it significantly slower to train.

heading: B RELATION TO TOPIC MODELS
text: Probabilistic topic models, such as LDA (Blei et al., 2003), often used in natural language processing, aim to recover the latent semantic structure, i.e. a set of abstract “topics”, from a collection of documents. The goal is to find the collection of the topics that best represent the corpus, with each document typically containing multiple topics in different proportions. However, the unsuper-
vised nature of these models and the lack of a gold standard makes their evaluation a long-standing problem. For this reason, several automatic “topic coherence” measures have been proposed that evaluate the degree of semantic similarity of the top words in a topic (Mimno et al., 2011; Aletras and Stevenson, 2013; Newman et al., 2010; Röder et al., 2015), while human evaluations are also common (Chang et al., 2009; Morstatter and Liu, 2018).
There is a similarity between topic coherence and the way we evaluate models. However, this analogy breaks down rather quickly. To discuss this, we can directly map the terms of topic models (topics, documents and words) to our scenario (clusters, images and concepts) in this order. The semantic coherence of a topic (cluster) is often measured by the co-occurrence of words (concepts) across documents (images). One immediate difference that arises is that in topic modeling the original space (corpus) is authored by humans and can be thus considered interpretable. Measures of coherence are introduced to understand whether the topics found by a model are also interpretable, i.e. evaluating the model producing the topics. In our case, we do not know the degree to which the original space (representation) is interpretable and this is precisely what we aim to quantify with our method.
Another conceptual difference is that topic models operate on a higher level of abstraction: a topic is a collection of words that describe a higher-level idea defined by this collection of words. This is sensible as the goal in topic modelling is to discover which words describe higher-level ideas and can be likely grouped together vs. irrelevant words.
In our case, this is not necessary, as the expert annotations already define the relevant concepts. A further abstraction is unnecessary and in fact it may even be undesirable because we are interested in the minimum number of concepts that may explain a cluster. For example, a topic of “animals” may be coherent, yet a cluster of “animals” lacks specificity because it may contain a multitude of classes.
Finally, the main mechanism in evaluating topic coherence is co-occurrence of words in the general corpus (documents). In our case, this translates to visual concepts co-occurring in the same image and models relationships between concepts in the images/the real world, whereas we are interested in the relationship between clusters and concepts as they have been learned by the model.

heading: C FURTHER ANALYSIS
text: In addition to the results and discussion in the main paper, we provide more investigations and insights.

heading: C.1 MUTUAL INFORMATION BETWEEN CONCEPTS AND IMAGENET CATEGORIES
text: One interesting question that arises is to which extent the elementary concepts we are considering are predictive of ImageNet labels. We answer this question by training a linear model to predict the ground truth label (instead of a pseudo-label) for each image from its concepts. This results in top-1 accuracy of 46.8% (NMI: 64.2, AMI: 54.2) — and significantly less if we exclude ‘object’ concepts that overlap with ImageNet labels. This suggests that using additional concepts provides information which is complementary to the fixed label set of ImageNet, further justifying our approach.
C.2 VARYING K
As discussed in Section 3, the mutual information between a representation f(x) and a fixed set of concepts y(x) is maximized when f(x) = x, i.e. any processing of x cannot increase the amount of information. It is thus expected that with an increasing number of clusters (K), I(fK(x), y(x)) will monotonically increase, as we approximate f(x) (i.e. when each sample is its own cluster). We verify this empirically in Fig. 6. For a fair comparison across methods, we have fixed K = 1000 for all experiments on ImageNet experiments in the main paper. To better understand the effect of the number of clusters, in Fig. 7 we also show the performance of most methods for K ∈ {500, 1000, 1500, 2000, 2500, 3000}, measuring NMI, AMI and top-1 accuracy, for the predictions of the reverse linear probe trained with all concepts. Importantly, we observe that for K ≥ 1000 the relative ranking of methods remains mostly consistent regardless of the number of clusters. The top three methods, MoCo-v3, OBoW and DeepCluster-v2 perform similarly and converge for larger
K, followed by SeLa-v2, SwAV, BarlowTwins and DINO. Further, we observe that BYOL scales gracefully for larger K, reaching the performance of the top methods; the opposite is observed for PCL-v2 which does not scale as well. Though it is important to study the effect of varying K, we should also note that our results do not appear sensitive to it, as similar trends are observed for all methods.

heading: C.3 DISCUSSION ON CLUSTERING-BASED METHODS
text: We have observed that methods that employ a clustering mechanism during training have generally more interpretable representations. A natural question that arises is whether clustering-based evaluation naturally favors such approaches.
However, clustering has already been shown to work well even for contrastive approaches (Zheltonozhskii et al., 2020), which is also further confirmed by our experiments. In our evaluations, the highest ranked models are MoCo-v3 (1k epochs) and OBoW (200 epochs). MoCov3 uses a contrastive objective, while OBoW follows a different approach from previous methods with a bag-of-words prediction task that puts emphasis on contextual information and local feature statistics. Our results align with the intuition behind OBoW (Gidaris et al., 2021) that encoding local concepts via a bag-of-visual-words approach yields richer, context-aware representations. Another example, SeLa, uses Sinkhorn-Knopp optimization to compute the cluster assignments, while DeepCluster-v2 uses spherical K-means. All these clustering mechanisms differ from the one used specifically in our experiments. Moreover, SeLa and DeepCluster-v2 train with K = 3000 prototypes, and there is no reason to believe that their representations are optimal for all values of K, and as we discussed above, the relative ranking of methods does not depend on the choice of K.
C.4 K-MEANS FOR SELF-LABELLING METHODS
We now look specifically at self-labelling methods, SeLa (Asano et al., 2020) and SCAN (Van Gansbeke et al., 2020), to investigate the difference between (a) the pseudo-labels predicted directly by each method and (b) clustering the learned representations with K-means, i.e. same as the experiments reported in the main paper. SeLa follows an optimal transport approach that yields clusters of approximately equal size. We found that this weakens the quality of its self-labels but results in stronger clusters after K-means and consequently in improved performance in our evaluations. Since SeLa is originally trained with 3000 classes1, in Table 6 we compare its pseudo-labels with K = 3000-means clustering in Table 6. SCAN builds on top of a representation learning method, i.e. MoCo (He et al., 2020) and, with a learnable clustering approach that removes adverse effects of low-level features, it produces more semantic pseudo-labels, improving upon K-means. Thus SCAN (self-label) results in improved performance in Table 6.

heading: C.5 TESTING ON OBJECTNET
text: Next we provide an evaluation of selected self-supervised methods using the reversed probes on the challenging test set ObjectNet (Barbu et al., 2019), evaluating the transferability of the quantized representations. Specifically, we use the the reversed probes trained for each method on ImageNet to assign ObjectNet samples to clusters, measuring the success of the classification (accuracy, NMI, AMI) against the corresponding K-means assignments, i.e. we measure if we can successfully predict the cluster assignments from predicted concepts, without any further training/adaptation. Here, we use the probes trained with all experts as input, except for the real ImageNet labels to account for ObjectNet categories that do not appear in ImageNet. We notice a significant drop in performance
1https://github.com/yukimasano/self-label
which suggests that a domain gap is indeed present, though the ranking stays again roughly the same (with the exception of OBoW).

heading: D QUALITATIVE EXAMPLES
text: Finally, in Figs. 8, 9, 10 and 11 we present several additional qualitative examples that highlight the usefulness of reverse probing. Similar to Fig. 5 in the main paper, we begin by showing selfsupervised clusters which are similar; they often contain images belonging to the same ImageNet categories. As a result, when training a (reverse) linear probe from ImageNet labels to pseudo-labels (cluster assignments), pairs of clusters with overlapping concepts will typically have high confusion (computed from the confusion matrix). We then add a concept group on top of the ImageNet labels and train a second linear probe, e.g. in Fig. 9 we include concepts from Places-365 and SUN Attributes. We can then identify pairs of clusters for which the inclusion of these concepts significantly reduced the ambiguity of the mapping, therefore reducing the confusion. Finally, for each pair of clusters we show the difference in linear model’s coefficients as a word cloud, with larger differences denoted by larger font size. In other words, we show the concepts that are most important to distinguish the two clusters (blue for (i) and red for (ii)).
In this way, we were able to discover and understand fine-grained differences between clusters and even problematic cases when considering only ImageNet labels. For example, we find that several ImageNet categories appear in combination with people (e.g. musical instruments). The quantized space of self-supervised representations (for all methods) appears to separate the corresponding images based on whether they contain people or not (examples shown in Fig. 8 (a), (c) and (d)). Noteworthy are the clusters in Fig. 8 (d), both of which correspond to the same ImageNet label (hockey puck), although they are visually very different. Including additional concepts, such as person or sports equipment, justifies their distinction into separate clusters and aligns with human intuition — perhaps even more so than assigning them to the same class. In Fig. 9 we show examples where clusters with images from the same ImageNet category differ by scene-related concepts such as flying vs. perched birds, wolves and orcas in different environments and even the inside vs. outside of a building. Material categories (Fig. 10) are helpful to understand environment or context and whether images contain hands (skin). Finally, texture (Fig. 11) can be used to tease apart details at a macro level, e.g. cut and whole fruits or peacocks with open or closed plumage. Overall, texture becomes less relevant models which are better at semantic discrimination. We also found that other concepts, such as text, sentiment and image quality do not significantly affect the clustering.
To summarize, we have proposed reverse linear probing as a way to understand whether interpretable clusters form in the representation space of self-supervised methods. While we provide a quantifiable measure for this — more interpretable clusters will be predicted with higher accuracy — we also show that we can qualitatively identify how concepts are encoded in the quantized representations through the linear probe’s coefficients. We finally show that such concepts are often complementary to ImageNet labels and can in fact correctly push clusters which are seemingly semantically close — based on a fixed label set — farther apart (e.g. Fig. 9(d)).

Review summaries:
The paper proposes a method for characterizing the "meaning" of the representation learned by a given model (interpretability). Towards this goal, it proposes reverse linear probing, a post-hoc method that aims at predicting a quantized version of the internal representation (as observed in specific examples) from semantic label.  Special emphasis is given in the capability of the proposed method on enabling the interpretation of models learned in a self-supervised manner.  PROs + Code will be released upon acceptance. + Evaluation covers a good variety of models.  CONs  - Missing related literature - Unclear how interpretability is gained.

This paper studies how human-interpretable are the concepts learned within the representation space of self-supervised models. It argues that linear probing methods are unable to identify whether a representation space contains a specific concept because the input might contain multiple confounding concepts (“red apple”) and might encode them in a manner that renders “red” and “apple” individually linearly inseparable. Instead, it proposes a “reverse linear probe” moving from which maps combinations of binary concept labels to the representation space, which is clustered using k-means.  The paper uses this linear probing method to propose a normalized mutual information metric to measure how well human-interpretable concepts are encoded in the representation space. Even though they find that their metric is correlated with linear probe, there are some interesting insights. First, the paper finds that models trained on ImageNet images capture more than just semantic class labels. They capture information about textures, scenes, objects, etc. Second, they found that additional training epochs dont lead to more interpretable representations even though it increases end-task performance. They also identified that OBoW, does really well even with only 200 epochs (though the reasons for why is still left to be investigated). Clustering-based approaches generally produce more interpretable representations. Fourth, ViT is a better architecture. Finally, qualitative evaluation of clusters seem to justify their utility of the reverse probe since NMI increases as when accounting for multiple categories for objects that occur together (ex, french horn and people).  

This paper presents an approach to investigate the semanticity of representations learned via self-supervision applied to images. The approach measures (via mutual information) how well a linear model can map a vector indicating image attributes (e.g. objects, texture, etc) to clusters within the representation space. The authors apply this method to recent methods from the literature and interpret the resulting ranking, aiming to learn more about which methods produce representations that map well to human judgements of similarity.

The paper aims to measure the interpretability of the visual representations learned by the recent self-supervised models. In doing this, the paper formulates the “interpretability” as the mutual information between the feature clusters and a set of visual concepts that can be interpreted by humans. The mutual information is approximated by the proposed Quantized Reverse Probing, a linear function that maps the concept vectors to the feature clusters. The proposed method is claimed to be complementary to linear probes, with further advantages: (1) a single principled score rather than individual prediction scores for different labels in linear probes (2)  handle combinations of concepts better than linear probes. (3) Faster than linear probes.