Measuring the Interpretability of Unsupervised Representations via Quantized Reversed Probing

1 INTRODUCTION . Relying on black-box models such as deep networks comes sometimes with significant methodological and ethical challenges . This is particularly true for unsupervised and self-supervised models which are learned without human supervision . While these models perform increasingly well in downstream applications , often outperforming supervised counterparts , there is very little understanding of what they learn , making their real-life deployment risky . In this paper , we thus consider the problem of characterizing the meaning of data representations , with a particular focus on unsupervised and self-supervised representations of images . Given a representation f mapping images x to representation vectors f ( x ) ∈ RD , our goal is to find whether these vectors contain human-interpretable information . This is usually done by finding a relationship or correspondence between f ( x ) and human-provided descriptions y ( x ) of the images , essentially translating the information from the representation space to a concept space . Consider for example the popular linear probing method ( Alain and Bengio , 2017 ) . Given a large dataset X̂ of images with corresponding manual annotations y ( x ) , one learns linear classifiers ( probes ) to map the feature vectors f ( x ) to labels y ( x ) , measuring the resulting classification accuracy . If the predictor is accurate , then one can argue that the representation captures the corresponding concept y . One possible issue with this type of approaches is that one can only discover in the representation concepts that are represented in the available data annotations . In order to maximize the semantic coverage of the analysis , it is thus customary to combine annotations for several types of attributes , such as object classes , textures , colour , etc . ( Bau et al. , 2017 ) . In this case , y is a vector of image attributes , and one can compute several linear probes to predict each individual attribute yi . By doing so , however , attributes are treated independently , which may be unsatisfactory . In order to obtain a single number assessing the overall semanticity of a representation , one must combine the prediction accuracies of several independent classifiers , and there is no natural way of doing so ( e.g . a simple average would not account for the different complexity of the different prediction tasks ) . Furthermore , the representation might be predictive of combinations of attributes , i.e . it might understand the concept of “ red apple ” without necessarily understanding the individual concepts , “ red ” and “ apple ” . While it is in principle possible to test any combination of attributes via linear probing , most attribute combinations are too rare to generate significant statistics for this analysis . In this paper , we thus propose a complementary assessment strategy to address these shortcomings . In contrast to linear probes , we start by considering the reverse prediction problem , mapping label vectors y ( x ) to representation vectors f ( x ) . A key advantage is that the entire attribute vector is used for this mapping , which , as we show later , accounts for attribute combinations more effectively . Next , we consider the challenge of deriving a quantity that allows to compare representations based on the performance of these reverse predictors . Obviously a simple metric such as the average L2 prediction error is meaningless as its magnitude would be affected by irrelevant factors such as the scale of the representation vectors . To solve this problem in a principled manner , we consider instead the mutual information between the concepts and quantized representation vectors . This approach , which we justify from the viewpoint of information theory , essentially measures whether the representation groups the data in a human-interpretable manner . We use our approach to evaluate a large number of self-supervised and supervised image representations . Importantly , we show qualitatively that ( a ) all methods capture interpretable concepts that extend well beyond the underlying label distribution ( e.g . ImageNet labels , which are not used for training ) , ( b ) while some clusters that form in the representation space are purely semantic , others carry information about scenery , material , textures , or combined concepts , and ( c ) context matters . We also show that more performant methods recover more of the original label distribution , i.e . learn better “ ImageNet concepts ” , and rely less on lower-level concepts . Quantitatively , we observe that our interpretability measure results in a similar but not identical ranking for state-of-the-art methods and that clustering-based approaches produce generally more interpretable representations . 2 RELATED WORK . Self-supervised representation learning ( SSL ) In this paper , we focus on self-supervised methods that learn representations from image data . Early self-supervised learning approaches devise numerous pretext tasks — such as colorization , predicting image rotations or solving image puzzles — to learn useful representations from unlabeled images ( Gidaris et al. , 2018 ; Pathak et al. , 2016 ; Noroozi and Favaro , 2016 ; Doersch et al. , 2015 ; Larsson et al. , 2016 ; Zhang et al. , 2017 ; 2016 ) . More recent approaches follow the contrastive learning paradigm ( Chen et al. , 2020a ; Frankle et al. , 2020 ; Kalantidis et al. , 2020 ; He et al. , 2020 ; Chen et al. , 2020b ; Hénaff et al. , 2019 ; Oord et al. , 2018 ; Misra and Maaten , 2020 ; Tian et al. , 2019 ; 2020 ; Wu et al. , 2018 ) , where the goal is to discriminate instances belonging to the same image from negative samples . Other methods ( Zbontar et al. , 2021 ; Chen and He , 2020 ; Grill et al. , 2020 ; Caron et al. , 2021 ) eliminate the need for negative samples . While there exists some theoretical approaches for explaining why the constrastive learning works ( Arora et al. , 2019 ; Wang and Isola , 2020 ; Tschannen et al. , 2020 ; Tsai et al. , 2020 ) , in most cases the focus lies on showing empirical improvement in downstream tasks . Clustering A related part of literature deals with unsupervised image clustering . Early approaches include using autoencoders ( Hinton and Salakhutdinov , 2006 ) , agglomerative clustering ( Bautista et al. , 2016 ) and partially ordered sets of hand-crafted features ( Bautista et al. , 2017 ) . More recent methods combine representation learning with clustering , using mutual information ( Ji et al. , 2019 ; Hu et al. , 2017 ) , K-means ( Caron et al. , 2018 ; Zhan et al. , 2020 ) , prototypes ( Li et al. , 2021 ) or optimal transport ( Asano et al. , 2020 ; Caron et al. , 2020 ) . Other methods build on strong feature representations and , at a second stage , cluster and further refine the network ( Yan et al. , 2020a ; Van Gansbeke et al. , 2020 ; Lee et al. , 2019 ) . These methods produce pseudo-labels which can be used to evaluate representations by measuring the correlation to the ground-truth labels . Evaluation and analysis of SSL There are two main directions in evaluating self-supervised learning methods . The first one uses pre-trained representations as an initialization for subsequent supervised tasks , e.g . for object detection . The second approach is to train a linear classifier on the embedding spaces which are kept frozen after pre-training . As a result , several benchmarking suites with various datasets and tasks have been proposed ( Zhai et al. , 2019 ; Goyal et al. , 2019b ; Van Horn tion , extracting and quantizing feature vectors to obtain clusters ( ... ) , ( 2 ) we label the image data with a diverse set of concepts ( ... ) from expert models trained with supervision on external data sources , and ( 3 ) we train a linear model hθ to map concepts to clusters , measuring the mutual information between the representation and human-interpretable concepts . et al. , 2021 ; Kotar et al. , 2021 ) . Complementary to this , clustering in the frozen feature space has also been proposed as an evaluation metric ( Zheltonozhskii et al. , 2020 ; Sariyildiz et al. , 2020 ) . In addition , downstream dataset ( Ericsson et al. , 2021 ) and pretraining dataset ( Zhao et al. , 2021 ; Cole et al. , 2021 ) dependencies have been evaluated . Finally , recent investigations of self-supervised feature spaces aim to understand separability ( Sehwag et al. , 2020 ) , concept generalization ( Sariyildiz et al. , 2020 ) and the effect of balanced vs. long-tailed training data ( Kang et al. , 2020 ) . Interpretability Although a large amount of work has studied feature representations in Convolutional Neural Networks ( CNNs ) , it is heavily focused on models trained with full supervision . Zeiler and Fergus ( 2014 ) and Zhou et al . ( 2014 ) analyze networks by visualizing the most activating patches , while activation maximization methods ( Mahendran and Vedaldi , 2016a ; b ; Nguyen et al. , 2017 ; 2016 ; Olah et al. , 2017 ; Simonyan et al. , 2014 ) generate inputs to activate specific neurons . Another line of work focuses on understanding what information is present in intermediate representations of CNNs , usually mapping activations to high-level concepts . Alain and Bengio ( 2017 ) introduce linear probes as a means to understanding the dynamics of intermediate layers , by predicting the target labels from these layers . Escorcia et al . ( 2015 ) also use a linear formulation to study the relationship of mid-level features and visual attributes , while Oramas et al . ( 2019 ) adopt this to predict task-specific classes from the features ( similar to Alain and Bengio ( 2017 ) ) , select relevant features and generate a visual explanation . Similarly , Zhou et al . ( 2018 ) decompose feature vectors into a set of elementary and interpretable components and show decomposed Grad-CAM heat maps ( Selvaraju et al. , 2017 ) for concepts that contribute to the prediction . Furthermore , Kim et al . ( 2018 ) relate feature vectors and human-interpretable concepts using a set of user-provided examples for each concept and Bau et al . ( 2017 ) propose to quantify the interpretability of individual units by measuring the overlap between each unit and a set of densely annotated concepts , Finally , Ghorbani et al . ( 2019 ) propose a method to automatically assign concepts to image segments and Yeh et al . ( 2020 ) analyze the completeness of these concepts for explaining the CNN . With the exception of ( Laina et al. , 2020 ; Bau et al. , 2017 ; Fong and Vedaldi , 2018 ) , relatively little work has been done to understand the emergence of interpretable visual concepts in self-supervised representations specifically . In particular , Laina et al . ( 2020 ) quantify the learnability and describability of unsupervised image groupings , by measuring how successful humans are at understanding the learned concept from a set of provided examples . Instead , our approach does not depend on human input and is thus easily scalable to a wide range of methods and number of classes . 3 METHOD . We are interested in measuring the semanticity of data representations . A representation f is a map f : X → RD that encodes data samples x∈X as vectors f ( x ) ∈RD . In this paper , we assume that the data are images X = R3×H×W and f is a deep neural network , but other choices are possible . The “ meaning ” or semantic content of an image x can be characterized by obtaining a label or description y ( x ) of the image from a human annotator . If the representation f ( x ) captures the meaning of the image too , then it should be predictive of the description y ( x ) . The mutual information I ( f ( x ) , y ( x ) ) provides a natural measure of predictivity , and it is thus tempting to use this quantity as a measure of the overall semanticity of the representation . However , information as such is not a useful measure in our context . To see why , note that , due to the data processing inequality ( I ( f ( x ) , y ( x ) ) ≤ I ( x , y ( x ) ) ) , the information is maximized by observing the raw image , i.e . by the identity representation f ( x ) = x . Since data processing can not increase information content , a useful representation must preserve information while also making it easier to decode and act on it . There are many possible definitions of what constitutes ‘ simple decoding ’ . Here we propose a definition that stems from the interpretation of differential entropy as the limit of discrete entropy for quantized versions of the corresponding variables ( Cover and Thomas , 2006 ) . In our case , the description y ( x ) is already discrete , but the representation vectors are continuous . We thus propose to vector-quantize the representation and to use the resulting discrete information as a measure of semanticity . Intuitively , in the limit of infinitely-fine quantization , this quantity reduces ( up to a normalizing term ) to the mutual information above . For a finite number of clusters , however , a large value of information means that the representation groups images in a way that makes sense to a human observer . In this case , the ‘ simple decoding ’ amounts to grouping images based on the L2 distance between their representation vectors . Formally , we use a vector quantization algorithm such asK-means ( Lloyd , 1982 ) to learn a quantizer function µK : RD → { 1 , . . . , K } for the representation vectors and estimate the mutual information I ( µK ( f ( x ) ) , y ( x ) ) by rewriting it as : I ( fK ( x ) , y ( x ) ) = H ( fK ( x ) ) −H ( fK ( x ) | y ( x ) ) , fK ( x ) = µK ( f ( x ) ) . ( 1 ) In the above equation , the first term denotes the entropy of the cluster assignments . In practice , given a sample dataset X̂ of images , we first run K-means to compute the quantizer µK , and then compute the frequency of cluster assignments fK ( x ) , x ∈ X to calculate the entropy . The second term in Eq . ( 1 ) is the conditional entropy : H ( fK ( x ) | y ( x ) ) = Ep [ − ln p ( fK ( x ) | y ( x ) ) ] ≤ Ep [ − ln q ( fK ( x ) | y ( x ) ) ] where p is the true joint distribution between variables fK ( x ) and y ( x ) . While this is difficult to compute , we can upper-bound it by considering an auxiliary posterior distribution q interpreting the conditional entropy as the cross-entropy loss of a predictor hθ , parametrized by θ , that maps labels y ( x ) to clusters fK ( x ) . We minimize the gap by learning this predictor in the usual way . In practice , learning the predictor may result in overfitting , which would lead to an over-optimistic estimate of the mutual information . We address this issue by learning the predictor hθ on a subset X̂ ′ ⊂ X̂ of the data and evaluating its cross-entropy loss on the remaining subset X̂ − X̂ ′ . Importantly , we consider a linear predictor ( probe ) for hθ , to further reduce the risk of overfitting . We refer to this as quantized reverse probing , as it maps semantic concepts to quantized data representations . Obtaining labelled data via automatic experts Similar to prior work on interpretation , our method requires a dataset x ∈ X̂ of images equipped with manually-provided labels y ( x ) ∈ Y . Because we do not know a-priori which concepts may be capture by a given representation f ( x ) under analysis , these labels should provide a good coverage for a number of different concepts ( e.g. , textures , object types , parts , etc. ) . A good example of such a dataset is Broden ( Bau et al. , 2017 ) . Because it would be cumbersome , and maybe even infeasible , to annotate any target dataset with all possible visual concepts , we can further and cheaply increase the coverage of the label space by predicting the labels y ( x ) automatically via a battery of expert classifiers learned using full supervision . The noise of these automated predictions is small compared to the noise in the correlation between the concepts and the unsupervised/self-supervised visual representations under study . Number of clusters The number of clusters K controls the size of the bottleneck . Mutual information increases monotonically as K increases , as we show in the Appendix ( Fig . 6 ) ; when comparing representations , it is thus important to fix a value of K and use it consistently . For added convenience , in practice we report a normalised value of information ( NMI ) . Relation to linear probing Linear probing is a common approach for assessing representations , related to our method . A linear probe is a simple predictor hi ◦ f ( x ) ≈ yi ( x ) that maps the representation f ( x ) to a specific concept yi ( e.g. , a binary attribute or a class label ) . The idea is that the representation captures concept yi if the probe performs well . As a result , linear probing is a standard evaluation protocol in self-supervised representation to measure the ( linear ) classification accuracy of the learned representations against a labelled set , commonly ImageNet ( Russakovsky et al. , 2015 ) . In interpretability studies , different formulations of linear probing ( Alain and Bengio , 2017 ) are used to understand intermediate layers of neural networks . Similar to our approach , the simplicity of the probes ( imposed via linearity and sometimes sparsity ) is a necessary bottleneck for interpretability : without such a constrain , the best possible probe would take the raw image as input . Our method is complementary to linear probes , with some advantages . To discuss this point further , we show a simplified example in Fig . 2 , where data points with color and shape attributes are encoded by two different representations , such that they form well-defined nameable clusters , i.e . blue square , red square , blue circle , and red circle . Forward probes ( X → Y ) can be trained as binary classifiers h1 , h2 for each attribute , mapping a feature vector ( in this case , coordinates ) to the corresponding attribute value for each data point . If attributes are linearly separable ( Fig . 2a ) , then forward probes do well at separating the representation space . If an attribute is not linearly separable ( color in Fig . 2b ) , the predictive accuracy of the corresponding probe h1 reduces to chance , and consequently reduces the average score . As a result , linear probes can not always assess the meaningfulness of clusters or , in other words , whether a cluster in representation space responds to a well-defined concept . On the contrary , if all data points in a cluster have consistent attributes , reverse probes ( Y → X ) can achieve a high score for , e.g . , “ red square ” without requiring the concept “ red ” to be also encoded in the features . In this case , we can say that the model has discovered the concept of a “ red square ” without identifying , in isolation , the concept “ red ” . Therefore , reverse probing handles combinations of attributes naturally and allows to better assess the semanticity of clusters .

Summary: The paper aims to measure the interpretability of the visual representations learned by the recent self-supervised models. In doing this, the paper formulates the “interpretability” as the mutual information between the feature clusters and a set of visual concepts that can be interpreted by humans. The mutual information is approximated by the proposed Quantized Reverse Probing, a linear function that maps the concept vectors to the feature clusters. The proposed method is claimed to be complementary to linear probes, with further advantages: (1) a single principled score rather than individual prediction scores for different labels in linear probes (2)  handle combinations of concepts better than linear probes. (3) Faster than linear probes.