Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies how human-interpretable are the concepts learned within the representation space of self-supervised models. It argues that linear probing methods are unable to identify whether a representation space contains a specific concept because the input might contain multiple confounding concepts (“red apple”) and might encode them in a manner that renders “red” and “apple” individually linearly inseparable. Instead, it proposes a “reverse linear probe” moving from which maps combinations of binary concept labels to the representation space, which is clustered using k-means.

The paper uses this linear probing method to propose a normalized mutual information metric to measure how well human-interpretable concepts are encoded in the representation space. Even though they find that their metric is correlated with linear probe, there are some interesting insights. First, the paper finds that models trained on ImageNet images capture more than just semantic class labels. They capture information about textures, scenes, objects, etc. Second, they found that additional training epochs dont lead to more interpretable representations even though it increases end-task performance. They also identified that OBoW, does really well even with only 200 epochs (though the reasons for why is still left to be investigated). Clustering-based approaches generally produce more interpretable representations. Fourth, ViT is a better architecture. Finally, qualitative evaluation of clusters seem to justify their utility of the reverse probe since NMI increases as when accounting for multiple categories for objects that occur together (ex, french horn and people). 

main_review: I actually really enjoyed reading this paper and found the contribution to be compelling. My main suggestion is to surface your insights and findings in the abstract and introduction. These sections, currently, only describe your method and not your findings. The abstract and introduction made me ask “so what?”. The findings are buried; some in the appendix!

I wish the choice of K-means as the clustering algorithm was questioned. For example, if you used hierarchical agglomerative clustering, would that lead to clusters where different levels of the hierarchy could represent compositions of children cluster concepts.

What I think is missing:
1) Which concepts are not captured by any of these models?

2) Which models are capable of decoupling individual concepts (identifying each concept in isolation) versus only compositions of concepts?

3) How does interpretability correlate with transfer to other downstream tasks like object detection, segmentation, etc. I know the paper includes a transfer experiment to Places but there could have been a more thorough exploration of transfer learning.

summary_of_the_review: Overall, I am still quite satisfied with where the paper currently stands. I listed things I would like to see improved or discussed. I also listed some potential questions for investigation. But these questions do not detract from my overall positive impression of the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an approach to investigate the semanticity of representations learned via self-supervision applied to images. The approach measures (via mutual information) how well a linear model can map a vector indicating image attributes (e.g. objects, texture, etc) to clusters within the representation space. The authors apply this method to recent methods from the literature and interpret the resulting ranking, aiming to learn more about which methods produce representations that map well to human judgements of similarity.
main_review: I enjoyed reading the paper - it is well written and the proposed approach is simple. In general, interpretability or maybe rather explainability of representations is a subject worth studying, and I appreciate the effort that went into experimentation in this work. However, there are also issues that I will describe below that relate to 1) positioning with respect to related work, and 2) the ability for the reader to draw insights from the analysis, among more minor aspects, that would significantly strengthen the work. In the current state I believe the paper is not yet ready for publication at ICLR.

**Related work: topic models**

In essence, the approach proposed here boils down to a linear model that predicts an output class for inputs in the shape of vectors that indicate image attributes (e.g. object presence, texture, etc). While this type of approach might be new in the assessment of semanticity of image representations, it is strikingly similar to classic topic models in NLP (e.g. a model identifying topics in documents represented as bag-of-words). The authors propose a measure to essentially measure the quality of those "topics" (i.e. clusters in this work), which is a well studied area in NLP where it is usually referred to as "topic coherence", e.g. [1]. All sorts of measures have been proposed to measure topic coherence, including those based on (pointwise) mutual information. I would therefore encourage the authors to take a deeper look into this relationship - it may strengthen the paper if the relationship of this approach to other methods can be understood more thoroughly.

**Ability to draw insights**

The authors run plenty of experiments using recent methods to assess their semanticity, which I think should be the focus point of this paper. While I appreciate the effort that went into this, I think how these results are currently presented (is there a better way to present the findings than table 2?), and discussed falls short of what I expected given the focus of this work. While the authors provide some insights w.r.t. to different types of approaches (e.g. clustering vs contrastive) it would be useful to link this to more established taxonomies (see e.g. [2]). Grouping approaches based on some form of taxonomy may make it easier to identify differences. This is an area where this paper could shine and provide actionable insights for the practitioner but at the moment falls short of that expectation.

**Choice of K**

The choice of the number of clusters seems a bit arbitrary to me, particularly as it impacts the (relative) performance of different methods. The authors outline in the appendix (B.2, in particular Fig 7) how BYOL's performance improves a lot with much larger K, essentially being en-par with the best performing methods. Yet in the main text the authors mention specifically that more recent methods (like BYOL) don't perform as well as others - Would the authors draw different conclusions if K was larger? Again, these differences may be alleviated with a more adequate grouping of approaches across experiments. Also in section 4.5 the authors drop K to 500, why?

**Limitations of a linear probe**

The authors outline how the proposed approach allows more complex relationships between image attributes to be captured. Yet they only present results for a linear model (predicting the cluster in representation space). How well would a non-linear model do in this task, which would be able to capture more elaborate relationships between image attributes?

[1] Roder et al. (2015) Exploring the Space of Topic Coherence Measures 

[2] Jing et al (2020) Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey
summary_of_the_review: The paper is well written and the proposed approach is simple. In general, interpretability of representations is a subject worth studying, and I appreciate the effort that went into experimentation in this work. However, there are also issues that relate to 1) positioning with respect to related work, and 2) the ability for the reader to draw insights from the analysis, among other aspects, that would significantly strengthen the work. In the current state I believe the paper is not yet ready for publication at ICLR.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper aims to measure the interpretability of the visual representations learned by the recent self-supervised models. In doing this, the paper formulates the “interpretability” as the mutual information between the feature clusters and a set of visual concepts that can be interpreted by humans. The mutual information is approximated by the proposed Quantized Reverse Probing, a linear function that maps the concept vectors to the feature clusters. The proposed method is claimed to be complementary to linear probes, with further advantages: (1) a single principled score rather than individual prediction scores for different labels in linear probes (2)  handle combinations of concepts better than linear probes. (3) Faster than linear probes.
main_review: Strengths

[S1] The paper tries to answer a very important question: to what extent the self-supervised visual representations can be interpreted by humans. This question is important and interesting as these representations were learned without human interventions but showed to be very effective for many downstream recognition tasks. While there are many task-oriented evaluations, systematic analysis of the interpretability of the representations is missing.

[S2] Formulating the “interpretability” as the mutual information between the representations and the semantic concepts is technically sound.

[S3] The analysis covers a wide range of visual concepts and pretrained self-supervised models.

[S4] The paper is well written.

—------------------------------------------------------------------------------------------------------------------------

Weaknesses

The proposed Quantized Reverse Probing is claimed to be complementary to linear probes with further advantages. At this point, I don’t think this claim is supported by the method or the experiments. To me, the proposed method is too similar to linear probes that they share the same drawbacks.

[C1] “A limitation of this type of approaches is that one can only discover in the representation meanings that are represented in the available data annotations”. I’m afraid both linear probing and the proposed method have this problem, the proposed method relies on pseudo labels generated by the expert models that were trained with data annotations.

[C2] “The representation might be predictive of combinations of attributes, i.e. it might understand the concept of ‘red apple’ without necessarily understanding the individual concepts, e.g. ‘red’ and ‘apple’.” This statement is correct. Unfortunately, neither linear probing nor the proposed method can resolve this problem, as they both are linear functions, so score(‘red apple’) == score(‘red’) + score(‘apple’) . In order to recognize “red apple”, both methods should have high scores on “red” and “apple”. In other words, the claim that the proposed method can better handle the compositions of the concepts is incorrect from the methodology perspective.

[C3] “The proposed method provides a single principled figure characterizing the entire representation instead of a separate figure for each predicted attribute”. With the proposed linear mapping, “this principled figure” is obtained by simply averaging the edge weights between the concepts and the clusters. I don’t see why linear probing cannot obtain such a figure by averaging the per-category scores.

[C4] “our measure can be computed much faster than linear probes”. Please elaborate more on this, as at first glance I thought the proposed method is slower as it involves clustering.

[C5] From the experiments, it is still not obvious to me that the proposed method is better than linear probing. Some observations listed in the paper are as expected or already pointed out by previous literature: deepcluster-v2 performs better as the optimization objective of deepcluster-v2 is pretty similar to the optimization objective of the proposed metric; also the evaluated self-supervised methods have very similar performance so their rankings may be sensitive to the metrics; ViT + Self-supervised-learning performs better on KNN classifications (and clustering ) was already observed by DINO, etc.

—------------------------------------------------------------------------------------------------------------------------

Minor comments:

Page 2, “either only cluster (Yan et al., 2020a) or cluster and further train and …”


summary_of_the_review: While the problem studied in the paper is very interesting, at this moment, I don't see there are any obvious advantages of the proposed method over a simple linear probing, as discussed in the "weaknesses". Therefore, I'm leaning toward a rejection.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method for characterizing the "meaning" of the representation learned by a given model (interpretability). Towards this goal, it proposes reverse linear probing, a post-hoc method that aims at predicting a quantized version of the internal representation (as observed in specific examples) from semantic label.

Special emphasis is given in the capability of the proposed method on enabling the interpretation of models learned in a self-supervised manner.

PROs
+ Code will be released upon acceptance.
+ Evaluation covers a good variety of models.

CONs 
- Missing related literature
- Unclear how interpretability is gained.
main_review: On the stronger side, the proposed method aims at addressing a task, post-hoc interpretation of pre-trained models, that had received much reduced attention compared to its explanation counterpart. As such it could help push research in this direction further.
In addition, the proposed method is evaluated considering a good amount of self-supervised learning methods.
Finally, code related to the paper will be released upon the publication of the manuscript which is always from the point of view of reproducibility.

On the weaker side,

While the idea of having the proposed probing in reverse order is, to best of my knowledge novel to me, it is not clear to what extent the proposed method does provides insight in the representation learned by the model being probed. Is is by providing semantic means (via y(x)) to describe some modes (f_K(x)) in the representation f(x) ? Perhaps it should be made more explicit the way in whic the proposed method makes the interpretation, of the model being analyzed, possible.

I find the way in which the paper approaches the idea of finding relationships between the internal representations learned by a model and a set of concepts very similar to [Escorcia et al., 2015], that did it at the attribute level, and to [Oramas et al., 2019], that further explore it towards interpretation of learned representations. 
Unfortunately, there is no descriptive nor quantitative comparison with respect to these methods. Given the similarity, I would suggest properly positioning against them.

At this point it is not clear how the quantized representation $f_K(x)$ is actually computed. For instance, is the internal representation f(x) taken at the layer level, fed to the clustering algorithm and the its result concatenated with that from other layers. Or perhaps you feed the concatenation of all the internal activations to the clustering components, or only use a specific [set] of layers, e.g. the last convolutional layers? Are there any normalization steps involved?

In Section 3, it is mentioned that "we can further and cheaply increase the coverage of the label space by predicting the labels y(x)
automatically via a battery of expert classifiers learned using full manual supervision." While this indeed may seem as a positive and cheap means to further get annotations, as properly noted by the paper, it indirectly requires the existence of the expert predictors and the annotated data on which they were trained. While applicable in the more general imageNet like testing set considered in the experiments, this requirement might not in place to more specific/niche applications, which will limit the applicability of this practice.

At the end of Section 4.2, by using the proposed method, it is observed and stated that methods that use a clustering mechanism during training have generally more interpretable representations. Doesn't this observation originates from the fact that the proposed method relies on an intermediate clustering step and as such favors methods that rely in similar clustering steps as well. If this indeed the case, perhaps there is some bias indirectly injected by the method that favors some types of methods over others.

In Section 4.4, it is observed that confusion between different clusters decreases when additional concept groups are added. This observation is used to further stress the fact that there could be interpretable properties in the network that may remain undetected since they are not annotated a-priori in the benchmark data and that this further motivates the need for reverse linear probing.
In this regard, this is a known limitation of standard linear probing methods whose interpretation capabilities are bounded by the set of pre-defined annotated concepts. In my opinion, this is also a limitation of the proposed method which, as shown in Section 4.4. does require additional pre-defined concepts in order to decrease the confusion of the identified clusters. Therefore I do not agree with the statement that the proposed reverse linear probing is a solution to the problem.

Finally, it might be good to provide some details on the linear models mentioned at the end of Section 4.1. Is this a single [fully-connected] layer neural network trained for several epochs or perhaps something more elaborated? 
summary_of_the_review: The proposed idea is sound and, to the best of my knowledge, novel. I believe the proposed method can perfectly complement the somewhat reduced amount of efforts along the line of interpretation of pre-trained models. However, I believe several aspects (please see my review) that should be polished/clarified before the paper is ready for publication.
review_title: null

Meta-review: This paper proposes a method for inspecting and interpreting the visual representations learned by self-supervised methods. 
The method is conceptually simple and intuititive, the authors assume that concept labels for the images are available, and then go on to learn a mapping between the learned image vectors and the human-provided descriptions of the images. The key insight is to learn a reverse mapping, i.e., to map label vectors to representation vectors. Specifically, feature vectors are quantized using k-means to obtain clusters;  images are labeled (automatically) with a diverse set of concepts from expert models trained with supervision on
external data sources, and  a linear model is trained  to map concepts to clusters, measuring the mutual information between the representation and human-interpretable concepts.

Reviewers raised some questions regarding the relation of the approach to topic models, the difference between reverse probing and linear probing, implementation details and computation. The authors addressed reviewers comments convincingly with additional experiments and/or explanations.
Decision: Accept (Poster)