Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a representation analysis technique to particularly understand what information is contained in self-supervised models. The authors propose a visualization technique based on conditional diffusion models that seeks to synthesize realistic-looking inputs whose representation matches that of a target image. Finding commonalities among these synthesized images likely provides clues about the target representation. 
main_review: Strengths: 

1. The question is meaningful, and the proposed technique is interesting. 
2. Visualizations generated using the proposed technique look promising in addressing the question laid out in the study.
3. The paper is very well-written, and the analyses are well-motivated.

Critiques: 
1. Some conclusions are stated as though they are specific to SSL but it might be good to know whether or not they are applicable for supervised training as well. For instance, the authors claim that `mapping to the same SSL representation as a natural image is not sufficient for producing a similar realistic-looking image’. My intuition is that that should be true for the supervised case as well due to the very nature of gradient-based matching on a deep layer representation when starting from random noise images. This result is not very surprising and has been described before. 

2. Imposing the 'naturalness’ constraint always arises out of a need for human understandability but it is not the most faithful account of what information is contained in the representation. I am a little concerned by the use of the phrase 'high fidelity’ in this setting to describe the visualization/model inversion process.  Perhaps it would be useful to briefly discuss the effect of different constraints on the input space in terms of fidelity – one with fewer control knobs (regularizations, naturalness prior etc.) is going to rank better on the fidelity scale..?  And the perceptual quality of image generation is not likely a good reflection of its faithfulness to the representation. 


3. Given that the goal was to characterize (through visualization) what is contained in a self-supervised representation, I think the results are not particularly intriguing and seem inadequate in their present form. Additional work and more rigorous analyses, including exploring more models, producing stronger results by analyzing more images etc., seem needed to strengthen the paper. 

4. The proposed method for analyzing representations is also highly subjective (not unlike other interpretability approaches I concede) but still there are no recommendations or thorough analysis about how to find the key factors driving a representation. e.g., how many images should be sampled from the RCDM to find the commonalities? Further, the conclusions that are drawn using this conditional image syntheses approach could just as well have been drawn by analyzing the representations directly (for instance, by seeing how well the scale of images can be decoded from the representation). 

summary_of_the_review: I enjoyed reading the paper. However, I'm uncertain about how much value this paper adds in efforts towards understanding the nature of self-supervised representations and the results are not very interesting. As a result, I think the paper falls slightly below the acceptance threshold.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a technique for visualizing the representations of fixed pretrained self-supervised neural networks. More specifically, by using a conditional denoising diffusion probabilistic model (DDPM, coined _RCDM_ here), a learned neural network representation $h$ can be mapped (back) to image space (i.e., the space of "natural images") and, by drawing multiple samples conditioned on the same $h$, the invariances learned by the (self-supervised) network can be visualized. Experiments with different pre-trained models show that the method is indeed able to synthesize high quality natural images corresponding to the learned representations.
main_review: The main idea of the paper, i.e. explaining the representations (and their invariances) learned by deep neural networks via visualizations through natural images is intuitive and very reasonable. By demonstrating that gradient-based methods do not suffice (Sec. 2) to reconstruct _natural_ images, the authors show the need for a probabilistic approach and, by opting for a conditional diffusion model, provide a method capable of generating high-quality images corresponding to the representations, as demonstrated in the experimental section. The main drawback of this paper, however, is that this idea is not novel and the paper fails to discuss prior work which uses different implementations for the conditional generative model, e.g. GANs [1], autoregressive models [2] or normalizing flows [3]. Authors should clearly situate their work within this existing literature and discuss what advantages/differences an approach based on DDPMs offers. For example, is it possible to use a deterministic approximation of the diffusion process (such as DDIM, see [4]) to extract, analogously to [3], an explicit representation of the learned invariances? Further, I do not quite understand the focus on SSL representations, as the proposed techniques can easily be applied to any type of representation, independent of the training paradigm. 

Moreover, a quantitative assessment of the correspondences between representations and synthesized images is missing. For example, can the synthesized representations be fed back into the (SSL) model and does this then yield the same representation $h$?

_Mixed comments:_

- I think the paper would benefit from focusing on __explaining__ (SSL) representations and thus dropping the part on synthesis (mainly Appendix B), as it mainly relies on methods that already have been developed and discussed in the context of pure synthesis (ADM).
- On the contrary, Appendix E and especially Fig. 20 should be moved to the main paper, because they provide interesting analysis of different SSL models/representations.
- 1st paragraph of 4.2.: I do not quite understand the motivation here: In the case the density of $p(x|h)$ approaches a delta-distribution, the task becomes more easy and in the limit, could be implemented through a regression model.
- Following on from this: the paper should demonstrate the need for a probabilistic approach by comparing, for example, with a simple regression baseline $x = f_{\theta}(h)$. A more advanced baseline would be the IC-GAN discussed in Sec. 4.2 (without neighborhood sampling).
- _"Diffusion models do not usually explicitly express an energy function"_ (Footnote 4) The usual training mechanism is to train the reweighted epsilon-parameterization from [5] which corresponds to a denoised score matching objective. This score corresponds to the derivative of an energy which can be plugged into MCMC sampling, see [6].
- The figure on the first page should show several examples of each model to emphasize the visualization capability of the invariances.
- The interpolation in Fig. 4c should show several "interpolation-paths", demonstrating the stochasticity of the proposed approach (DDPM)
- Does the proposed conditioning mechanism via conditional batch norm provide significant advantages over other methods (e.g. addition to the timestep-embedding, AdaIN, cross-attention, ...)?

__References:__
- [1]: Shocher, A., Gandelsman, Y., Mosseri, I., Yarom, M., Irani, M., Freeman, W.T., Dekel, T.: Semantic Pyramid for Image Generation. 
- [2]: Nash, C., Kushman, N., Williams, C.K.: Inverting Supervised Representations with Autoregressive Neural Density Models.
- [3]: Rombach, R., Esser, P., Ommer, B.: Making Sense of CNNs: Interpreting Deep Representations & Their Invariances with INNs
- [4]: Song, J., Meng, C., Ermon, S.: Denoising Diffusion Implicit Models
- [5]: Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models
- [6]: Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations

summary_of_the_review: I think that the present paper deals with a very important problem, namely the "understanding" of learned neural representations. The proposed approach to do this by synthesizing "natural" images generated by a generative model conditioned on these learned representations (here a diffusion model) is intuitive and reasonable. The generated samples are of high quality and seem to be in agreement with the representations. 

Unfortunately, this approach is not novel (only the use of a diffusion model instead of {GANS, ARMs, flows} is new here) and the present work would, in my opinion, benefit greatly from being placed in the context of this existing work. In addition, there are some specific questions (see above) that I think are not yet answered satisfactorily. 

++++++++++++++++++++++++++

Score raised to 6 after the rebuttal
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work introduces develops a method for sampling different natural images that a pretrained encoder maps to the same or similar representation. To do this, the authors propose a modification to the diffusion model of Dhariwal & Nichol for sampling conditioned on a given encoded representation $h=f(x)$ that is well suited to conditioning on high dimensional vectors (i.e., representations). This new method is used to qualitatively compare encoders trained via a number of methods (supervised, DINO, SimCLR, Barlow Twins, VICReg etc.).


main_review: 
This paper considers a very worthy problem, that of analyzing encoders via visualizations, and clearly develops a high quality generative modeling approach with which to do this. I think this paper may be on its way to being a vey nice paper, however I have concerns with it’s current state. 

Current strengths include:

- high quality of generative models used (the generated samples are high fidelity). 
- plenty of qualitative evaluation - the appendix contains many pages of additional visualizations. 
- the method is able to visually discern some differences between supervised and self-supervised trained encoders. The differences between  self-supervised methods seems a little harder to extract, besides noting that some (e.g., DINO) exhibit better invariance than others (e.g., SimCLR).


My two broad concerns (i.e., "weaknesses"), which I discuss in more detail momentarily, are:

- A lack of *quantitative results*. All results presented are qualitative and visual in nature (with the exception of a sanity check that the generative models are producing photorealistic samples). 
- A lack of demonstration of useful applications for the proposed method. 

**Quantitative results:**

While a large part of generative modeling is focused on qualitative production of photorealistic samples, in order to properly compare you method and analyze pertained encoders I think it is critical to have more quantitative evaluation. 

I am sure the authors can come up with many better ideas for additional quantitative evaluations than I can, but I will also attempt to offer some actionable feedback:

- The main premise of the work is that the samples $x’$ from your diffusion model are such that $f(x’)$ are very close to the original representation h. It seems almost a must that this is quantitatively confirmed. 
- You argue verbally that for a given representation $h$, IC-GAN generates samples which map to a neighborhood of $h$, whereas your method generates samples that map very closely to $h$. While this may very well be true, it would be much more convincing if you quantitatively showed this. 

**Applications of method:**

At present, the work mainly attempts to visualize some simple properties that one would expect a self-supervised encoder to have - e.g., a more instance-level invariance then supervised encoders, and invariance to the data augmentations used training training. Unfortunately I don't find this to be sufficiently insightful right now. In order to truly demonstrate the value of this method, it would be very valuable to take more steps towards devising practically useful tools that are enabled by this method. Some ideas of the kind of thing I am imagining are:

- Could your visualization method be used to determine a priori which out of a group of encoders will perform best at a given downstream task. If possible, this would be very cool since the trial and error evaluation of pretrained encoders is annoyingly heuristic. E.g. specifically, could you develop some metric based on your visualization method that correlates well with (e.g.,) segmentation performance. 
- Suppose you have two encoders $f$ and $f’$, and you know that f performs much better on some downstream task $T$. Could you use the diffusion model for $f$ to *generate new training samples* with which to finetune the encoder f’ itself (not just the linear probe) so that the performance of $f’$ on task $T$ improves. 


To be clear, I am not asking you to try any of these ideas specifically. Instead I am trying to illustrate the type of thing I envisage when I ask for extensions that analyze encoders in more detail. This is a promise that was made in the abstract — “new avenues to analyze and improve self-supervised models” — but which was underdelivered on in my eyes. 

---
**Miscellaneous comments and questions:**

- The difference in FID and IS for your method compared to ADM seems very large considering your method is a modification of theirs. Do you know what accounts for this? Is it better model training? Is it the fact that your method is instance conditioned, & therefore requires less variation in samples? Something else?
- Some experiments seem orthogonal to the idea of analyzing pretrained encoders. E.g. the value of Fig 4c) is unclear to me (4 a) and b) are good) and seem to be more of the form of a “sanity check” that the generative model is working sensibly. 
- Please fix your bibliography. Many citations are missing their publication venues (e.g. Barlow Twins and at least three of Yang Song’s papers), and some citation simply list: 1) authors names, 2) paper’s name, 3) year — i.e. they don’t even have an arXiv number, let alone a publication venue (e.g. SimCLR and VICReg papers). 


summary_of_the_review: 
The reasons I am concerned about these two main objections are: 
- quantitative results are almost a must in order to properly evaluate methodological progress, and  check that the method is behaving as it should, and to properly compare models/methods etc.,
- The methodological contribution of the representation conditioned diffusion model is relatively small (it sufficed to simply describe the changes one needs to make to ADM in words). This is not a problem in itself. But I do think it is therefore important to demonstrate the method's usefulness in analyzing and understanding pretrained models. 

Consequently, I am currently not in favor of acceptance. However, to recognize the generally high quality of the work thus far I am opting for a weak reject. I am unlikely to raise the score to an accept without significant updates to the work, or strong arguments in favor from other reviewers.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a conditional diffusion model that can be used to visualize representations learned by SSL or supervised models.

The proposed Representation-conditioned Diffusion Model (RCDM) can generate images that are both close in the representation space to a given image, and looking realistic.

The paper also provides empirical results comparing conditionally generated samples from models trained on features from the backbone or projector of a SSL model, as well as supervised features. These experiments provide insight to understanding SSL, such as showing that projector is the key of SSL being invariance to transformations.

main_review: **Strength**: This work provides a visualization tool for understanding NN representations. The proposed RCDM is not technically challenging (it is adapted from ADM in the prior work), but it works well for probing the information in NN representations without sacrificing generation quality.

The experiments cover different SSL methods and the comparison with supervised learning, showing results on 1) in/out of distribution generation, 2) interpolation, 3) super-resolution, 4) unconditioned generation, 5) algebraic manipulation. 

Other aspects:
  - The writing is clear and easy to follow.
  - The relation and comparison with prior work is discussed adequately.
  - The code will be released.

**Questions**:
  - Figure 1: it would be better to provide more context for the relative distance: e.g. is the difference between 0.4% and 3.3% significant?  Should a 3% distance considered small?
  - Figure 6: projector head vs backbone: are features from the projector able to get as close to the original feature as features from the backbone? If not, how much further are the projector features compared to the backbone features?

Minor comments:
  - The writing for Figure 7 is repetitive, i.e. the last paragraph on page 8 and the caption for Figure 7.
  - last sentence in the first paragraph: "...or discriminating instances."
  - the paragraph below equation (2): should the dimension be $K$ rather than $S$?
  - Sec 4.1, second line in the second paragraph: model "dependent"
  - Last sentence of the caption of Figure 4: should be column 2 to 6 (rather than 7)
  - Appendix section C: it should link to Figure 15 rather than Figure 21.


summary_of_the_review: The method in this paper is unsurprising but gives good results.
I find the experiments thorough and insightful, and would recommend an accept.
review_title: null

Meta-review: This paper proposes a method for visualizing representations of neural networks trained with self-supervised learning with conditional denoising diffusion probabilistic models. By generating multiple images conditioned on a representation, one can identify what aspects the representation is and is not sensitive to. The proposed method allows for high fidelity generated images that can be used to compare different self-supervised methods and layers.

Reviewers agreed that the paper proposed reasonable methodology, targeted an interesting problem of understanding what is learned by self-supervised methods, and presented interesting qualitative evaluations. However, there remained concerns on the novelty of results in comparison to other methods for probing representations (e.g. classification based), subjectiveness of interpretation of the qualitative results, and limited quantifications of the intuition gained from the visualizations. While the authors have argued that the point of the paper is to showcase the merits of qualitative visual analysis method, reviewers found that the presented results were insufficient to demonstrate the value of the proposed approach. A number of ideas were discussed with reviewers on how to highlight the value of visualization which could strengthen the paper in the future. Given the lack of novelty on the conditional generation side, and limited insight gained from the qualitative results, I cannot recommend this paper for acceptance in its current form.
Decision: Reject