Measuring the Interpretability of Unsupervised Representations via Quantized Reversed Probing

1 INTRODUCTION . Relying on black-box models such as deep networks comes sometimes with significant methodological and ethical challenges . This is particularly true for unsupervised and self-supervised models which are learned without human supervision . While these models perform increasingly well in downstream applications , often outperforming supervised counterparts , there is very little understanding of what they learn , making their real-life deployment risky . In this paper , we thus consider the problem of characterizing the meaning of data representations , with a particular focus on unsupervised and self-supervised representations of images . Given a representation f mapping images x to representation vectors f ( x ) ∈ RD , our goal is to find whether these vectors contain human-interpretable information . This is usually done by finding a relationship or correspondence between f ( x ) and human-provided descriptions y ( x ) of the images , essentially translating the information from the representation space to a concept space . Consider for example the popular linear probing method ( Alain and Bengio , 2017 ) . Given a large dataset X̂ of images with corresponding manual annotations y ( x ) , one learns linear classifiers ( probes ) to map the feature vectors f ( x ) to labels y ( x ) , measuring the resulting classification accuracy . If the predictor is accurate , then one can argue that the representation captures the corresponding concept y . One possible issue with this type of approaches is that one can only discover in the representation concepts that are represented in the available data annotations . In order to maximize the semantic coverage of the analysis , it is thus customary to combine annotations for several types of attributes , such as object classes , textures , colour , etc . ( Bau et al. , 2017 ) . In this case , y is a vector of image attributes , and one can compute several linear probes to predict each individual attribute yi . By doing so , however , attributes are treated independently , which may be unsatisfactory . In order to obtain a single number assessing the overall semanticity of a representation , one must combine the prediction accuracies of several independent classifiers , and there is no natural way of doing so ( e.g . a simple average would not account for the different complexity of the different prediction tasks ) . Furthermore , the representation might be predictive of combinations of attributes , i.e . it might understand the concept of “ red apple ” without necessarily understanding the individual concepts , “ red ” and “ apple ” . While it is in principle possible to test any combination of attributes via linear probing , most attribute combinations are too rare to generate significant statistics for this analysis . In this paper , we thus propose a complementary assessment strategy to address these shortcomings . In contrast to linear probes , we start by considering the reverse prediction problem , mapping label vectors y ( x ) to representation vectors f ( x ) . A key advantage is that the entire attribute vector is used for this mapping , which , as we show later , accounts for attribute combinations more effectively . Next , we consider the challenge of deriving a quantity that allows to compare representations based on the performance of these reverse predictors . Obviously a simple metric such as the average L2 prediction error is meaningless as its magnitude would be affected by irrelevant factors such as the scale of the representation vectors . To solve this problem in a principled manner , we consider instead the mutual information between the concepts and quantized representation vectors . This approach , which we justify from the viewpoint of information theory , essentially measures whether the representation groups the data in a human-interpretable manner . We use our approach to evaluate a large number of self-supervised and supervised image representations . Importantly , we show qualitatively that ( a ) all methods capture interpretable concepts that extend well beyond the underlying label distribution ( e.g . ImageNet labels , which are not used for training ) , ( b ) while some clusters that form in the representation space are purely semantic , others carry information about scenery , material , textures , or combined concepts , and ( c ) context matters . We also show that more performant methods recover more of the original label distribution , i.e . learn better “ ImageNet concepts ” , and rely less on lower-level concepts . Quantitatively , we observe that our interpretability measure results in a similar but not identical ranking for state-of-the-art methods and that clustering-based approaches produce generally more interpretable representations . 2 RELATED WORK . Self-supervised representation learning ( SSL ) In this paper , we focus on self-supervised methods that learn representations from image data . Early self-supervised learning approaches devise numerous pretext tasks — such as colorization , predicting image rotations or solving image puzzles — to learn useful representations from unlabeled images ( Gidaris et al. , 2018 ; Pathak et al. , 2016 ; Noroozi and Favaro , 2016 ; Doersch et al. , 2015 ; Larsson et al. , 2016 ; Zhang et al. , 2017 ; 2016 ) . More recent approaches follow the contrastive learning paradigm ( Chen et al. , 2020a ; Frankle et al. , 2020 ; Kalantidis et al. , 2020 ; He et al. , 2020 ; Chen et al. , 2020b ; Hénaff et al. , 2019 ; Oord et al. , 2018 ; Misra and Maaten , 2020 ; Tian et al. , 2019 ; 2020 ; Wu et al. , 2018 ) , where the goal is to discriminate instances belonging to the same image from negative samples . Other methods ( Zbontar et al. , 2021 ; Chen and He , 2020 ; Grill et al. , 2020 ; Caron et al. , 2021 ) eliminate the need for negative samples . While there exists some theoretical approaches for explaining why the constrastive learning works ( Arora et al. , 2019 ; Wang and Isola , 2020 ; Tschannen et al. , 2020 ; Tsai et al. , 2020 ) , in most cases the focus lies on showing empirical improvement in downstream tasks . Clustering A related part of literature deals with unsupervised image clustering . Early approaches include using autoencoders ( Hinton and Salakhutdinov , 2006 ) , agglomerative clustering ( Bautista et al. , 2016 ) and partially ordered sets of hand-crafted features ( Bautista et al. , 2017 ) . More recent methods combine representation learning with clustering , using mutual information ( Ji et al. , 2019 ; Hu et al. , 2017 ) , K-means ( Caron et al. , 2018 ; Zhan et al. , 2020 ) , prototypes ( Li et al. , 2021 ) or optimal transport ( Asano et al. , 2020 ; Caron et al. , 2020 ) . Other methods build on strong feature representations and , at a second stage , cluster and further refine the network ( Yan et al. , 2020a ; Van Gansbeke et al. , 2020 ; Lee et al. , 2019 ) . These methods produce pseudo-labels which can be used to evaluate representations by measuring the correlation to the ground-truth labels . Evaluation and analysis of SSL There are two main directions in evaluating self-supervised learning methods . The first one uses pre-trained representations as an initialization for subsequent supervised tasks , e.g . for object detection . The second approach is to train a linear classifier on the embedding spaces which are kept frozen after pre-training . As a result , several benchmarking suites with various datasets and tasks have been proposed ( Zhai et al. , 2019 ; Goyal et al. , 2019b ; Van Horn tion , extracting and quantizing feature vectors to obtain clusters ( ... ) , ( 2 ) we label the image data with a diverse set of concepts ( ... ) from expert models trained with supervision on external data sources , and ( 3 ) we train a linear model hθ to map concepts to clusters , measuring the mutual information between the representation and human-interpretable concepts . et al. , 2021 ; Kotar et al. , 2021 ) . Complementary to this , clustering in the frozen feature space has also been proposed as an evaluation metric ( Zheltonozhskii et al. , 2020 ; Sariyildiz et al. , 2020 ) . In addition , downstream dataset ( Ericsson et al. , 2021 ) and pretraining dataset ( Zhao et al. , 2021 ; Cole et al. , 2021 ) dependencies have been evaluated . Finally , recent investigations of self-supervised feature spaces aim to understand separability ( Sehwag et al. , 2020 ) , concept generalization ( Sariyildiz et al. , 2020 ) and the effect of balanced vs. long-tailed training data ( Kang et al. , 2020 ) . Interpretability Although a large amount of work has studied feature representations in Convolutional Neural Networks ( CNNs ) , it is heavily focused on models trained with full supervision . Zeiler and Fergus ( 2014 ) and Zhou et al . ( 2014 ) analyze networks by visualizing the most activating patches , while activation maximization methods ( Mahendran and Vedaldi , 2016a ; b ; Nguyen et al. , 2017 ; 2016 ; Olah et al. , 2017 ; Simonyan et al. , 2014 ) generate inputs to activate specific neurons . Another line of work focuses on understanding what information is present in intermediate representations of CNNs , usually mapping activations to high-level concepts . Alain and Bengio ( 2017 ) introduce linear probes as a means to understanding the dynamics of intermediate layers , by predicting the target labels from these layers . Escorcia et al . ( 2015 ) also use a linear formulation to study the relationship of mid-level features and visual attributes , while Oramas et al . ( 2019 ) adopt this to predict task-specific classes from the features ( similar to Alain and Bengio ( 2017 ) ) , select relevant features and generate a visual explanation . Similarly , Zhou et al . ( 2018 ) decompose feature vectors into a set of elementary and interpretable components and show decomposed Grad-CAM heat maps ( Selvaraju et al. , 2017 ) for concepts that contribute to the prediction . Furthermore , Kim et al . ( 2018 ) relate feature vectors and human-interpretable concepts using a set of user-provided examples for each concept and Bau et al . ( 2017 ) propose to quantify the interpretability of individual units by measuring the overlap between each unit and a set of densely annotated concepts , Finally , Ghorbani et al . ( 2019 ) propose a method to automatically assign concepts to image segments and Yeh et al . ( 2020 ) analyze the completeness of these concepts for explaining the CNN . With the exception of ( Laina et al. , 2020 ; Bau et al. , 2017 ; Fong and Vedaldi , 2018 ) , relatively little work has been done to understand the emergence of interpretable visual concepts in self-supervised representations specifically . In particular , Laina et al . ( 2020 ) quantify the learnability and describability of unsupervised image groupings , by measuring how successful humans are at understanding the learned concept from a set of provided examples . Instead , our approach does not depend on human input and is thus easily scalable to a wide range of methods and number of classes . 3 METHOD . We are interested in measuring the semanticity of data representations . A representation f is a map f : X → RD that encodes data samples x∈X as vectors f ( x ) ∈RD . In this paper , we assume that the data are images X = R3×H×W and f is a deep neural network , but other choices are possible . The “ meaning ” or semantic content of an image x can be characterized by obtaining a label or description y ( x ) of the image from a human annotator . If the representation f ( x ) captures the meaning of the image too , then it should be predictive of the description y ( x ) . The mutual information I ( f ( x ) , y ( x ) ) provides a natural measure of predictivity , and it is thus tempting to use this quantity as a measure of the overall semanticity of the representation . However , information as such is not a useful measure in our context . To see why , note that , due to the data processing inequality ( I ( f ( x ) , y ( x ) ) ≤ I ( x , y ( x ) ) ) , the information is maximized by observing the raw image , i.e . by the identity representation f ( x ) = x . Since data processing can not increase information content , a useful representation must preserve information while also making it easier to decode and act on it . There are many possible definitions of what constitutes ‘ simple decoding ’ . Here we propose a definition that stems from the interpretation of differential entropy as the limit of discrete entropy for quantized versions of the corresponding variables ( Cover and Thomas , 2006 ) . In our case , the description y ( x ) is already discrete , but the representation vectors are continuous . We thus propose to vector-quantize the representation and to use the resulting discrete information as a measure of semanticity . Intuitively , in the limit of infinitely-fine quantization , this quantity reduces ( up to a normalizing term ) to the mutual information above . For a finite number of clusters , however , a large value of information means that the representation groups images in a way that makes sense to a human observer . In this case , the ‘ simple decoding ’ amounts to grouping images based on the L2 distance between their representation vectors . Formally , we use a vector quantization algorithm such asK-means ( Lloyd , 1982 ) to learn a quantizer function µK : RD → { 1 , . . . , K } for the representation vectors and estimate the mutual information I ( µK ( f ( x ) ) , y ( x ) ) by rewriting it as : I ( fK ( x ) , y ( x ) ) = H ( fK ( x ) ) −H ( fK ( x ) | y ( x ) ) , fK ( x ) = µK ( f ( x ) ) . ( 1 ) In the above equation , the first term denotes the entropy of the cluster assignments . In practice , given a sample dataset X̂ of images , we first run K-means to compute the quantizer µK , and then compute the frequency of cluster assignments fK ( x ) , x ∈ X to calculate the entropy . The second term in Eq . ( 1 ) is the conditional entropy : H ( fK ( x ) | y ( x ) ) = Ep [ − ln p ( fK ( x ) | y ( x ) ) ] ≤ Ep [ − ln q ( fK ( x ) | y ( x ) ) ] where p is the true joint distribution between variables fK ( x ) and y ( x ) . While this is difficult to compute , we can upper-bound it by considering an auxiliary posterior distribution q interpreting the conditional entropy as the cross-entropy loss of a predictor hθ , parametrized by θ , that maps labels y ( x ) to clusters fK ( x ) . We minimize the gap by learning this predictor in the usual way . In practice , learning the predictor may result in overfitting , which would lead to an over-optimistic estimate of the mutual information . We address this issue by learning the predictor hθ on a subset X̂ ′ ⊂ X̂ of the data and evaluating its cross-entropy loss on the remaining subset X̂ − X̂ ′ . Importantly , we consider a linear predictor ( probe ) for hθ , to further reduce the risk of overfitting . We refer to this as quantized reverse probing , as it maps semantic concepts to quantized data representations . Obtaining labelled data via automatic experts Similar to prior work on interpretation , our method requires a dataset x ∈ X̂ of images equipped with manually-provided labels y ( x ) ∈ Y . Because we do not know a-priori which concepts may be capture by a given representation f ( x ) under analysis , these labels should provide a good coverage for a number of different concepts ( e.g. , textures , object types , parts , etc. ) . A good example of such a dataset is Broden ( Bau et al. , 2017 ) . Because it would be cumbersome , and maybe even infeasible , to annotate any target dataset with all possible visual concepts , we can further and cheaply increase the coverage of the label space by predicting the labels y ( x ) automatically via a battery of expert classifiers learned using full supervision . The noise of these automated predictions is small compared to the noise in the correlation between the concepts and the unsupervised/self-supervised visual representations under study . Number of clusters The number of clusters K controls the size of the bottleneck . Mutual information increases monotonically as K increases , as we show in the Appendix ( Fig . 6 ) ; when comparing representations , it is thus important to fix a value of K and use it consistently . For added convenience , in practice we report a normalised value of information ( NMI ) . Relation to linear probing Linear probing is a common approach for assessing representations , related to our method . A linear probe is a simple predictor hi ◦ f ( x ) ≈ yi ( x ) that maps the representation f ( x ) to a specific concept yi ( e.g. , a binary attribute or a class label ) . The idea is that the representation captures concept yi if the probe performs well . As a result , linear probing is a standard evaluation protocol in self-supervised representation to measure the ( linear ) classification accuracy of the learned representations against a labelled set , commonly ImageNet ( Russakovsky et al. , 2015 ) . In interpretability studies , different formulations of linear probing ( Alain and Bengio , 2017 ) are used to understand intermediate layers of neural networks . Similar to our approach , the simplicity of the probes ( imposed via linearity and sometimes sparsity ) is a necessary bottleneck for interpretability : without such a constrain , the best possible probe would take the raw image as input . Our method is complementary to linear probes , with some advantages . To discuss this point further , we show a simplified example in Fig . 2 , where data points with color and shape attributes are encoded by two different representations , such that they form well-defined nameable clusters , i.e . blue square , red square , blue circle , and red circle . Forward probes ( X → Y ) can be trained as binary classifiers h1 , h2 for each attribute , mapping a feature vector ( in this case , coordinates ) to the corresponding attribute value for each data point . If attributes are linearly separable ( Fig . 2a ) , then forward probes do well at separating the representation space . If an attribute is not linearly separable ( color in Fig . 2b ) , the predictive accuracy of the corresponding probe h1 reduces to chance , and consequently reduces the average score . As a result , linear probes can not always assess the meaningfulness of clusters or , in other words , whether a cluster in representation space responds to a well-defined concept . On the contrary , if all data points in a cluster have consistent attributes , reverse probes ( Y → X ) can achieve a high score for , e.g . , “ red square ” without requiring the concept “ red ” to be also encoded in the features . In this case , we can say that the model has discovered the concept of a “ red square ” without identifying , in isolation , the concept “ red ” . Therefore , reverse probing handles combinations of attributes naturally and allows to better assess the semanticity of clusters .

Summary: This paper studies how human-interpretable are the concepts learned within the representation space of self-supervised models. It argues that linear probing methods are unable to identify whether a representation space contains a specific concept because the input might contain multiple confounding concepts (“red apple”) and might encode them in a manner that renders “red” and “apple” individually linearly inseparable. Instead, it proposes a “reverse linear probe” moving from which maps combinations of binary concept labels to the representation space, which is clustered using k-means.  The paper uses this linear probing method to propose a normalized mutual information metric to measure how well human-interpretable concepts are encoded in the representation space. Even though they find that their metric is correlated with linear probe, there are some interesting insights. First, the paper finds that models trained on ImageNet images capture more than just semantic class labels. They capture information about textures, scenes, objects, etc. Second, they found that additional training epochs dont lead to more interpretable representations even though it increases end-task performance. They also identified that OBoW, does really well even with only 200 epochs (though the reasons for why is still left to be investigated). Clustering-based approaches generally produce more interpretable representations. Fourth, ViT is a better architecture. Finally, qualitative evaluation of clusters seem to justify their utility of the reverse probe since NMI increases as when accounting for multiple categories for objects that occur together (ex, french horn and people).