High Fidelity Visualization of What Your Self-Supervised Representation Knows About
Authors: Florian Bordes
Randall Balestriero
Pascal Vincent
Venue: ICLR 2022 Submitted (2022)
Keywords: self-supervised learning
visualization
diffusion model
conditional generative model
representation

Abstract: Discovering what is learned by neural networks remains a challenge. In self-supervised learning, classification is the most common task used to evaluate how good a representation is. However, relying only on such downstream task can limit our understanding of how much information is contained in the representation of a given input. In this work, we study how to visualize representations learned with self-supervised models. We investigate a simple gradient descend based method to match a target representation and show the limitations of such techniques. We overcome these limitations by developing a representation-conditioned diffusion model (RCDM) that is able to generate high-quality inputs that share commonalities with a given representation. We further demonstrate how our model's generation quality is on par with state-of-the-art generative models and how the representation conditioning brings new avenues to analyze and improve self-supervised models.

Under review as a conference paper at ICLR 2022
HIGH FIDELITY VISUALIZATION OF WHAT YOUR SELF- SUPERVISED REPRESENTATION KNOWS ABOUT
Anonymous authors
Paper under double-blind review
ABSTRACT
Discovering what is learned by neural networks remains a challenge. In self- supervised learning, classification is the most common task used to evaluate how good a representation is. However, relying only on such downstream task can limit understanding of how much information is retained in the representation of a given input. In this work, we showcase the use of a conditional diffusion based generative model (RCDM) visualize representations learned with self-supervised models. We further demonstrate how this model's 's generation quality on par with tate-of-the-ar generative models while being faithful the representation used as conditioning. By using this new tool to analyze self-supervised models, we can show visually that i) SSL (backbone) representation are not really invariant many data augmentation they were trained on. ii) SSL projector embedding appear too invariant for tasks like classifications. iii) SSL representations are more robust to small adversarial perturbation of their inputs iv) there is an inherent structure learned with SSL model that can used for image manipulation.
Earth from spacc
an untrained representation supervised representation
SSL representation
INTRODUCTION AND MOTIVATION
Approaches aimed at learning useful representations, from unlabeled data, have long tradition in ma- chine learning. These include probabilistic latent variable models and variants of auto-encoders (Ack
ley et al. 1985] Hinton et al., 2006] Salakhutdinov et al. [007]] Vincent et al. 2008 Kingma & Welling 2014 Rezende al. (2014) that traditionally put under the broad umbrella term of unsu- pervised learning (Bengio et al. 2013). More recent approaches, under the term of self-supervised learning (SSL) have used various kinds of "pretext-tasks" to guide the learning of useful repre- sentations. Filling-ing Filling-in-the- taa-the-blaleee tasks, proposed earlier in (Vincent et al.) 2008 2010). later proved remarkably successful learning potent representations for natural language processing (Vaswani et al. 2017 Devlin et al. 2019). Pretext tasks for the image domain include solving higsaw-puzzle
(Noroozi & Favaro 2016), predicting rotations or affine transformations (Gidaris et al., 2018 Zhang
al.
2019b) discriminating instances (Wu et al. 2018 van den Oord et al.) 2018). The latest,
most
successful, modern family of SSL approaches for images (Misra Maaten 2020 Chen et al.
2020
Chen & He 2020 He et al.) 2020 Grill et al. 2020: Caron et al. 2020 2021 Zbontar et al.
2021
Bardes et al. 2021 have two noteworthy characteristics that markedly distinguish them from
We use representations of the real picture of Earth on the left (source: NASA) as conditioning for RCDM. We show samples (resolution 256 x 256) in cases where the representations 0048-dimensions) were obtained respectively with a random initialized ResNet50, supervised-trained one, and a SSL -trained one. More samples Fig.

---

Under review as a conference paper at ICLR 2022
traditional ceapperrvied-leerningg models such autoencoder variants GANs (Goodfellow et al. 2014): a) their training criteria are not based on any input-space reconstruction or generation, but instead depend only on the obtained distribution in the representation embedding space b) they encourage invariance to explicitly provided input transformations a.k.a. data-augmentations, thus injecting important additional domain knowledge.
Despite their remarkable success in learning representations that perform well on downstream classification tasks, rivaling with supervised-trainet models (Chen all. 2020), much remains be understood about SSL algorithms and the representations they learn. How do the particularities of different algorithms affect the representation learned and its usefulness? What information does the learned representation contain? Empirical analyses have so far attempted to analyse SSL algorithms almost exclusively through the limited lens of the numerical performance they achieve on downstream tasks such as classification. Contrary to their older unsupervised learning cousins, due to characteristic a) highlighted above, modern SSL, methods do not provide any direct way of mapping back the representation in image space, to allow visualizing it. The main goal of our work is thus to enable the visualization of representations learned by SSL methods, as a tool to improve our understanding.
More precisely, we suppose that we are given mapping function ʃ a (part of) SSL or otherwise trained neural network- that takes an input image I E X and maps it to a representation h E H as in h f(x). The input space X will typically be RGB pixel space represented as X 1_,, and the representation space H will be the output space of a deeper layer. We denote the representation space's 's dimension by K as in HH = RRK Now want, when given specific representation h HHI, to visualize what inputs æ yield this representation. As f is typically not bijective, e.g. if it computes higher level representation of reduced dimension, there may be many inputs that yield that same representation, most of which will not resemble natural images. Our approach (Section (3) thus aims at finding inputs that not only map the target h but are also visually recognizable images. For this we build a conditional generative model that (implicitly) models p(z (h) and allows to sample diverse æ' ~ (((r((). For reasons that we will explain later, we opted for a conditional diffusion model, inspired by Dhariwal & Nichol (202T), for our conditional generative model.
This paper's main contributions are:
To devise conditional diffusion model architecture (RCDM) suitable for conditioning on large vector representations s.a. SSL representations. Our model provides high-quality images, measured in term of FID, par with otate-of-the-art models (Tab. 2a), and is suited for out-of-distte disiiiiibnttonnn samples (see Fig. []. The conditionally generated images are also highly representation-fane faithfuu i... they closely match the representations of the images used for the conditioning (Tab. 2b Fig. 24).
To showcase its usefulness for qualitatively analyzing SSL representations and embeddings (also in contrast with supervised representations), by shedding light on what information about the input image or isn't 't retained in them.
Specifically, by repeatedly sampling from same conditioning representation, one can observe which aspects are common all samples, thus identifying what encoded in the representation, while the aspects that vary greatly show what was not retained in the representation. We make the following observations: (i) SSL projector embeddings appear most invariant, followed by supervised- trained representation and last SSL representations (Fig. 3). (ii) SSL-trained representations retain more detailed information on the content of the background and object style while supervised- trained representations appear oblivious to these (Fig. 4 (iii) despite their invariant training criteria, SSL representations appear to retain information on object scale, grayscale vs color, and color palette of the background, much like supervised representation (Fig. (4). (iv) Supervised representations appear more susceptible to adversarial attacks than SSL ones (Fig. (0 (v) We can explore and exploit structure inside SSL representations leading to meaningful manipulation of image content (s.a. splitting representation in foreground/backgroum components to allow background substitution)
(Fig. 31 32).
2 RELATED WORK
Deterministic visualization methods: Many early works (Erhan et al., 2009; Zeiler & Fergus
2014 Simonyan et al.) 2013 Selvaraju et al. 2016 Smilkov et al. 2017 used gradient based
Thhe representation that is produced by Resnet50 backbone, before the projector.
2

---

Under review as a conference paper at ICLR 2022
techniques to visualize what is learned by neural networks. This led to successful interpretability of DNs internal features, especially when applied on a unit belonging to the first few layers of a DN (Cadena et al. 2018). More recently, Caron et al. (202T) used the attention mask of transformers to perform unsupervised object segmentation. By contrast, our method is not model dependent, we can plug any type of representation as conditioning for the diffusion model. Another possibility, explored in Zhao et al. (2021): Appalaraju et al. (2020); [Eicsson et al. (2021). is to learn to invert the DN features through Deep Image Prior (DIP) go as in ming d(ge( (f fz) ())), F). In fact, as we experimented in Appendix A performing unconstrained gradient based optimization of sample to match target representation leads to unrealistic generation. The use of DIP however requires to retrain the DIP network for each feature-image pairs and only quantify how much information about æ is retained in (a) while we are interested in finding all the æs that are seen to have the same information content.
Several families of techniques have been developed as generative models, that can be trained on unlabeled data and then employed to generate images. These include auto regr-seeressvve models (Van Den Oord et al. 2016), variational auto-encoders (Kingma & Welling 2014 Rezende al. 2014), GANs (Goodfellow a 2014), autoregressive flow models (Kingma et al.. 2016).
Generative models:
and diffusion models (Sohl-Dickstein et al.) 2013). Conditional versions are typically developed shortly after their unconditional versions (Mirza & Osindero 2014] van den Oord et al., 2016). In principle one could envision training a conditional model with any of these techniques, to condition on an SSL. or other representation for visualization purpose, as we are doing in this paper with a diffusion model. One fundamental challenge when conditioning on rich representation such as the one produced by a SSL model, is that for a given conditioning h we will usually have available only a single corresponding input instance æ, precious few to learn distribution. This can lead model training astray. By contrast a particularly successful model such as the conditional version of BigGAN (Brock al., 2019) conditions on categorical variable, the class label, that for each value of the conditioning has a large number of associated x data.
One closely related work to ours is the recent work on Iistance-Conditionell GANs (IC-GAN) of Casanova et al. (202T). Similar to us it also uses SSL or supervised representations as conditioning when training conditional generative model, here GAN (Goodfellow et al., 2014), specifically variant of BigGAN (Brock et al. 2019)) or AtyleGAN2 (Karras et al. 2020). However, the model is trained such that, from specific representation h. it learns to generate not only images that should map to this representation, but a much broader neighborhood of the training data. Specifically up to 50 training points that are nearest neighbors in representation space to h. It remains be seen whether such GAN architecture could be trained successfully without resorting to nearest neighbor set. IC-GAN to be understood as a conditional generative model of an image's ' broad neighborhood, and the primary focus of this work was on developing superior quality controllable generative model. By contrast we want to sample images that map as closely as possible to the original image in the representation space, as our focus is to build tool to analyse SSL representations, to enable visualising what images correspond precisely to a representation. (See Fig. 24 a comparison.)
As previously stated, our choice of a diffusion-based model rather than GAN was motivated by the simple stable training of such models, by the high quality of generated images demonstrated with the model we build on Dhariwal & Nichol (2021) that rivals that of GAN, and by the similarity of the input-space gradient-based sampling procedure with the simple approach we explored in Appendix AA] While conditional versions of their diffusion model were already developed in Dhariwal & Nichol (202I), these were unsuitable for conditioning on high dimensional distributed representation s.a. those obtained with SSL models, as we discussed in details in section 3 This prompted us develop the architecture variant of this paper. Despite their qualities. diffusion models also have drawbacks, in particular they are resource- resource-hungry and slow for generation. It is thus very likely that alternative approaches for representation conditioond generative models will be developed and employed for analysis and visualisation purposes the future.
Lastly, few approaches have focused on conditional generation to unravel the information encoded in representations of supervised models. Shocher et al. (2020), a hierarchical LSGAN generator is trained with class-conditional discriminator (Zhang et al. 2019a). While the main applications focused on inpainting and ttyle-rrn,sfer, this allowed to visually quantify the increasing invariance of representations associated to deeper and deeper layers. This method however requires labels to train the generator. On the other hand, Nash et al. (2019) proposed to use an autoregressive model, in particular PixeICNN+ (Salimans al. 2017), to specifically study the invariances that each layer
3

---

Under review as a conference paper at ICLR 2022
of DN inherits. In that case, the conditioning was incorporated by regressing context vector to the generator biases. As far as we are aware, PixeICNN+ generator falls short on high-resolution images e.g. most papers focus on 32 x 32 Imagenet. Lastly, RRobach et al. (2020) proposes to learn a Variational AutoEncoder that is combined with an invertible neural network (INN) whose role is model the relation between the VAE latent space and the given representations. To allow for interpretable manipulation, a second invertible network (Esser et al., 2020) is trained using labels disentangle the factors of variations present in the representation. By contrast we train end-to-end single decoder to model the entire diversity of inputs that correspond to the conditioning representation, without imposing constraints of a structured prior or requiring labels for image manipulation.
3 CONDITIONING A DIFFUSION MODEL ON REPRESENTATION h
We propose build novel conditional diffusion process whose goal is to directly generate realistic images that match a given target representation. Given a representation h the failure of the method in Appendix A suggests we need way to further constrain the type of samples mere constraint of belonging to S S(h). More precisely, we want to be able to sample, among S(h), inputs that are more like the training data (here natural images), i.e. that are likely under the same distribution. That is we would like not merely to find x' E S(h) but rather to sample x' ~ p( h). Informally we might picture the set of likely natural images (points whose density p(x) is above some threshold) as a subset of X that we will loosely refer to as the "data manifold" MM. Where our first approach attempted to sample points more or less uniformly within S(h), modeling and sampling form p(z) h) will more likely produce points from M n S(h). We propose to train a conditional diffusion model to implicitly model p(> [z|p /)) and allow sampling from it.
generate, beyond the
While we could have considered other conditional generative approaches (we discussed some of the alternatives in section 2). the choice of the reverse diffusion approach (Sohl-Dickstein et al. 2015
Ho et al., 2020 Song & Ermon 2019 2020 Song al.; 2021a b Nichol & Dhariwal 2021
not entirely y arbitrary. It is motivated by the remarkable quality of image generation they recently proved capable of (Dhariwal & Nichol 2021), well the closeness of their Langevin-MCMC-
like generation process to the input-gradient-dieedd Appendix A Indeed sampling from reverse diffusion model similarly starts from random noise image, and takes multiple steps in input space that can be thought of as (noisy) gradient steps on an (implicit) energy function (Ho et al., 2020, Song al. 202 b) Informally, one can think of these steps as progressively moving this initial random point closer to the "data manifold" M. A reverse diffusion conditioned on h will move it towards M n S(h). The three sampling approaches are depicted and contrasted in Fig. Tla
optimization we used to obtain samples in
We base our work on the Ablated Diffusion Model (ADM) developed by Dhariwal Nichol (2021) which uses a UNet architecture (Ronneberger et al. 2015) to learn the reverse diffusion process. Our conditional variant - called CepreseetaaioonCondiiee Conditionned Diffusioon Diffisusi Model (RCDM) - is illustrated in To suitably condition on representation h = f(æ), we replaced the Group Normalization
Fig. 11b layers of ADM by conditional batch normalization layers (Dumoulin et al. 2017) that take h as conditioning More precisely we apply a fully connected layer to h that reduces dimension to vector of size 512. This vector is then given as input to multiple conditional batch normalization layers that are placed in each residual block of the diffusion model.
In contrast with Dhariwal & Nichol (021)) we don't 't use the input gradient of a classifier to bias the reversed diffusion process towards more probable images, nor do we use any label information for training our model - recall that our goal is building a visualization tool for SSL models that train on unlabeled data. Our batch normalization based conditioning also different from the approach that was used by Dhariwal& & Nichol (202)) when conditioning their super- resouutiso model on a low-resolution image. Their technique of upscaling and appending the conditioning image as extra channels to the input would not work for our application. Our representation h typically has 2048 "channels" with no spatial extent: upscaling it to the size of the input image would blow up memory constraints.
4

---

Under review as a conference paper at ICLR 2022
(a)
(b)
(c)
Figure 1: a) In distribution conditional image generation. An image from ImageNet validation set (first column) is used to compute the representation output by trained SSL model (Dino). The representation is used as conditioning for the diffusion model. Resulting samples are shown in the subsequent columns (see Fig. 12 We observe that our conditional diffusion model produces samples that are very close to the original image. b) Out of distribution (OOD) conditional. How well does RCDM generalize when conditioned on representations given by images from different distribution? (here a WikiMedia Commons image, see Fig. 13 for more). Even with OOD an conditionning, the images produced by RCDM are very close visually to the original image. c) Interpolation between two images from ImageNet validation data. We apply a linear interpolation between the SSL representation of the images on the first column and the representation of the images on the last column. We use the interpolated vector as conditioning for our model that produce the samples that showed in column 2 to column 6. Fig. T6 in appendix shows more sampled interpolation paths.
(b) For each encoder, compute the rank and mean reciprocal rank (MRR) of the image used condi- tioning within the closest set of neighbor in the rep- resentation space of the samples generated from the valid set (50K samples). A rank of one means that all of the generated samples for given model have their representations matching the representation used as conditioning.
(a) We report results for ImageNet to show that our approach reliable for generating images which look realistic. Since the focus of our work is not generative modelling but to showcase and encourage the use of such model for representation analysis, only show results for one conditional generative models. For each method, we computed FID and IS with the same evaluation setup in Pytorch.
Model
[Man rank
"MRR
Method
Res.
JFID
118
ADM Dhariwal & Nichol 2021
256
26.8
34.5 + 1.8
Dino (Caron et al.
2021
1.00
0.99
IC-GAN
IC-GAN
Casanova ef al.
Casanova
T
(KDE*)
256
256
21.6
20.8
51.3 :: 2.2
38.6
- 1.1
SimCTR (C hen et
Swav (Caron at
Barlow Zbontar
2027)
A720
2021
1.00
1.16
0.99
0.99
0.97
RCDM (ours)
256
19.0
$1.9 + 2.6
Supervised
5.65
0.69
Figure 2: a) Table of results ImageNet. We compute the FID (Heusel al. 2017) and IS (Salimans)
et al. 2016) on 10 000 samples generated by each models with 10 000 images from the validation set of ImageNet as reference. KDE* means that used our unconditional representation sampling scheme based on KDE (Kernel Density Estimation) for conditioning IC-GAN instead of the method based on K-means introduces by Casanova al. (202T). b) Table of ranks and mean reciprocal ranks for different encoders. This table show that RCDM is faithful to the conditioning by generating images which have their representations close the original one.
EXPERIMENTS USING RCDM TO MAP BACK REPRESENTATIONS TO IMAGES
Our first experiments aim at evaluating the abilities of our model to generate realistic-looking images whose representations are close the conditioning. To do so, we trained our Representation- Conditionned Diffusion Model (RCDM), conditioned on the 2048 dimensional representation given by Resnet50 (He et al. (0166) trained with Dino (Caron et al.) 2021) on ImageNet (Russakovsky et al. 2015). Then we compute the representations of a set of images from ImageNet validation data to condition the sampling from the trained RCDM. Fig. Ia shows is able to sample images that are very close visually from the one that is used to get the conditioning. We also evaluated the generation abilities of our model on out of distribution data. Fig. Tb shows that our model is able sample new
A similar technique was used by Casanova et al. 2021 for IC-GAN, discussed in the next section.
5

---

Under review as a conference paper at ICLR 2022
(2348)
1280
Dina Hoad (250)
8mC (2040)
3mCLF (126
Figure 3: On the first and second row, RCDM samples conditioned on the usual backbone representa- tion (size 2048) and projector representation (size 256) with Dino. Same the third and forth row with SimCLR representations (2048 and 128). For comparison, we also added samples from RCDM trained representation given by supervised trained model. We clearly observe that the projector only keeps global information and not its context, as opposed to the backbone. This indicates that invariances SSL models are mostly achieved in the projector representation, not the backbone. Additional comparisons provided in Fig. 26
views of an OOD image. We also quantitatively validate that the generated images' representations are close to the original image representation in Tab. 2b
Fig.
18
Fig. 19 and Fig. 20
This implies that there much information kept inside the SSL representation so that the conditional generative model is able to reconstruct many characteristics of the original image. We also perform interpolations between two SSL representations in Fig. Ic This shows that our model able to produce interpretable images even for SSL representations that correspond to unlikely mix of factors. Both the interpolation and OOD generation clearly show that the RCDM model is not merely outputting training set images that it could have memorized. This is also confirmed by Fig. 17 in the appendix that shows nearest neighbors of generated points.
The conditional diffusion model might also unconditional generative model. Any technique suitable for modeling and sampling the distribution of (lower dimensional) representations could be used. As this not our primary goal in the present study, we experimented only with simple kernel density estimation (see appendix for details). This allow us quantify the quality of our generative process in an unconditional manner fairly compare against state-of of the-trt-rt generative models such as ADM. We provide some generative model metrics in Tab. 2a along some samples in Fig. 12 to show that our method competitive with the current literature.
building block to hierarchically build
VISUAL ANALYSIS OF REPRESENTATIONS LEARNED WITH SELF-SUPERVISE MODEL
Having generated samples that are close in the representation space conditioning image can gives us an insight on what's hidden in the representations learned with self-supervised models. As demonstrated in the previous section, the samples that are generated with RCDM are really close visually to the image used as conditioning. This give an important proof of how much is kept inside a SSL representation. However, it's also important to consider how much this amount of "hidden" information varied depending on the SSL representation that is used. Therefore, we train several RCDM on SSL representations given by VicReg (Bardes et al. 2021), Dino (Caron et al. 2021), Barlow Twins (Zbontar et al.) 2021) and SimCLR (Chen et al. 2020). In many applications that used self-supervised models, the representation that used is the one that corresponds to the backbone of the ResNet50. Usually, the representation given by the projector of the SSL--model (on which the SSL

---

Under review as a conference paper at ICLR 2022
Conditioning
Supervised
Vertioal
oNh
Zoom out
Zoom
Grayscale
Color
Figure 4: We use our conditional generative model to get insight on the invariance (or covariance) of representations with respect several data augmentations. On original image (top left) apply specific transformations (visible in the first column). For each transformed image, we compute the SSL representation with Dino, SimCLR and a supervised network and condition the corresponding RCDM with that representation to sample 3 images. We see that despite their invariant training criteria, the 2048 dimensional SSL representations appear retain information on object scale, grayscale vs color, and color palette of the background, much like the supervised representation. They appear insensitive to vertical shifts. We also see that supervised representation constrain the appearance much less. Refer to Fig. 27 in Appendix for comparison with using the lower dimensional projector head embedding as the representation.
criterion is applied) is discarded because the results on many downstream tasks like classification as good as the backbone. However, since our work is visualize and better understand the differences between SSL representations, we also trained RCDM on the representation given by the projector of Dino, Barlow Twins and SimCLR. Fig. 3 and Fig. 25 we condition all the RCDM with the image labelled conditioning and sample 9 images for each model. We observe that Dino representation does not allow much variance meaning that even information about the pose of the animal is kept inside the representation. In contrast, the SimCLR representation seems to be more invariant to the pose of the kangaroo. We also observe class-crossing, the kangaroo becomes a rabbit. VicReg seems to be more robust in the sense that the animal doesn't I cross the class boundary despite changes in the background.
5.1 WHAT ARE REPRESENTATIONS REALLY INVARIANT TO?
In Fig. we apply specific transformations (augmentations) to a test image and we check whether the samples generated by the diffusion model change accordingly. We also compare with the behavior of a supervised model. We note that despite their invariant training criteria, the 2048 dimensional SSL represeenationns do retain information on object scale, grayscale status, and color palette of the background, much like the supervised representation. They do appear invariant to vertical shifts. In the Appendix, Fig. [27 applies the same transformations, but additionally compares using the 2048 representation with using the lower dimensional projector head embedding as the representation. There, we observe that the projector representation seems to encode object scale, but contrary to the 2048 representation, it appears to have gotten rid of grayscale- status and background color information. Currently, researchers need to use custom datasets (in which the factors variation of a specific image annotated) to verify how well the representations learned are invariant those
7

---

Under review as a conference paper at ICLR 2022
Eps
0.0
0.01
0.25
0.1
Adversarial RCDM
Supervised
SIMCLR
RCDM
Swar
Figure 5: Visualization of adversarial attacks with RCDM. We use Fast Gradient Sign attack a given image (top-left corner) with various values for the attack coefficient epsilon. On the first row, we only show the adversarial images obtained from a supervised encoder: refer to Fig. 30 in the Appendix to see the (similar looking) adversarial examples for each model. On the following rows, we show the reconstruction of the attacked images with RCDM for various models. For a supervised representation, RCDM reconstructs an animal that belong to another class, a lion in this case. However, we observe that we use SimCLR or Swav as encoder (third and forth row), the images generated by RCDM are still dogs even with higher values for epsilon.
factors. We hope that RCDM will help researchers in self-supervised learning to alleviate this concern since our method "plug and play" and can be use on any dataset with any type of representation.
5.2 VISUALIZATION OF ADVERSARIAL EXAMPLES
Since our model is able to project any representation to the manifold of real images, we follow the same experiment protocol as Rombach et al. (2020) to visualize how adversarial examples seen by RCDM. We apply Fast Gradient Sign attacks (FGSM) (Goodfellow et al.) 2013)) over a given image and compute the representation associated to the attacked image. When using RCDM conditioned on the representation of the adversarial examples, we can visualize if the generated images still belong the class of the attacked image or not. In Fig. 5 and 30 the adversarial attacks change the dog in the samples to lion in the supervised setting whereas SSL methods doesn't 't seem to be impacted by the adversarial perturbations i.e the samples are still dogs until the adversarial attack became visible to the human eye.
5.3 MANIPULATION OF REPRESENTATIONS
Experimental manipulation of representations can be needed to analyze how much specific dimensions of the representation can be associated to specific factors of variation of the data. In a self-supervised setting in which we don't 't have access labelled data, can be difficult gain insight on how the information about the data is encoded in the representation. We showcase a very simple and heuristic setup to remove the most common information in the representations within set of the nearest neighbors of specific example. We experimentally saw that the nearest neighbors of given representation share often similar factors of variation. Having this information in mind, we investigate how many dimensions are shared in between this set of neighbors. Then, we remove the most common non-zero dimensions by setting them to zero and use RCDM to decode this truncated representation. In Fig. 6, such simple setup induces the removing of all information about the background and the dog to only keep the information about clothing (Only one dog had clothes the set of neighbors used to find the most common dimensions). Since the information about the dog and the background are removed, RCDM produces images of different clothes. On the third and forth row, instead of setting the most common dimensions to zeros, set them to the value of other images
8

---

Under review as a conference paper at ICLR 2022
zero mask most common indioes where dim of representation is non zero
where dim representation non zero Figure 6: Visualization of direct manipulations
Least common
the backbone representation space. In this experiment, we find the most common non zero dimension across the neighborhood of the image used conditioning (top-left dog). On the second row, set these dimensions RCDM to decode the truncated representation. We observe that RCDM produces examples with variety of clothes meaning that all information about the background and the dog is removed. In the third and forth row, instead of setting the most common dimensions to zero, we set them to the value of the corresponding dimension in the representation associated the image on the left. As we can see, the corresponding dog get various clothes which were not present in the original image.
and
at the exact same dimensions. By using these new representations, RCDM is able to generate the corresponding dog with clothes. This setup works better with SSL methods since supervised models learned to put away most of the information that is not needed to predict class labels. We have similar experiment with background removal Figure 31
6 CONCLUSION
Most of the Seff-Supervise Learning literature uses downstream tasks that require labeled data to measure how good the learned representation is and to quantify its invariance to specific data- augmentations. However one cannot in this way see the entirety of what is retained in representation, beyond testing for specific invariances known beforehand, or predicting specific labeled factors, for limited (and costly to acquire) set of labels. Yet, through conditional generation, all the stable information can be revealed and discerned from visual inspection of the samples. We showcased how simple conditional generative model (RCDM) to visualize representations, enabling the visual analysis of what information is contained in self-supervised representation, without the need of any labelled data. After verifying that our conditional generative model produces high-quality samples (attested qualitatively and by FID scores) and eepresentation-faithii samples, we turned to exploring representations obtained under different frameworks. Our findings clearly separate supervised from SSL models along variety of aspects: their respective invariances or lack thereof to specific image transformations. the discovery of exploitable structure in the representation' 's dimensions, and their differing sensitivity to adversarial noise.
REPRODUCIBILITY STATEMENT
The data and images in this paper only used for the sole purpose of exchanging reproducible research results with the academic community.
Our results should be easily reproducible

---

Under review as a conference paper at ICLR 2022
RCDM, is based on the same code as Dhariwal & Nichol (2021)
com/ owweoo/pp//////ooo htt.dd/dddddffiiiii..... f
and uses the same hyper-paramenn (See Ap-
pendix of Dhariwal & Nichol (2021) for details about the hyper- parametee..m.............
To obtain our conditional RCDM, one just needs replace the GroupNormalization layers in that architecture by conditional batch normalization layer of Brock et al. (2019) (using the code from ht //gi thub com/ www.aooooiiiggie.c.c /ttpa//cccciiiiiaaaPPPoooo.
The self-supervised pretrained models we used to extract the conditioning representations
were obtained from the model-zoo of VISSL (Goyal al. 2021 (code from https
The unconditonal sampling process is straightforward, as explained in Appendix C
We are working on cleaning and preparing to release any remaining code glue to easily reproduce the results in this paper.
REFERENCES
David H. Ackley, Geoffrey E. Hinton, and Terrence J. Sejnowski. A learning algorithm for boltzmann machines. Cognitive Science, 9:147-169, 1985.
Srikar Appalaraju, Yi Zhu, Yusheng Xie, and István Fehérvári. Towards good practices in self- supervised representation learning, arXiv preprint arXiv:2012.00866 2020.
Randall Balestriero and Richard Baraniuk. spline theory of deep learning. In Jennifer Dy and
Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning. volume 80 Proceedings of Machine Learning Research, pp. 374-383. PMLR, 10-15 Jul 2018. URL https::/ppoceecinggg mlr wttp..ppeeeeeeee/aasstttteeaaaea
html
Randall Balestriero, Romain Cosentino, Behnaam Aazhang, and Richard Baraniuk. The geometry of deep networks: Power diagram subdivision. Advances in Neural Information Processing Systems, 32: 32:15832-1588 15841, 2019.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variane-iivarrane--vvvaaaeee heg.areeeccvrrannnnneeggaaaaiiia for self-supervised learning. arXiv preprint arXiv:2105.044906 2021.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 1798)11998,,888, 2013. URL http:://aax/....o//
/200 5538 cite arxiv: arxiv:1206. 1206.5538.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL
httts://opeereei//
net. httee//oommmddddxeeeeeee
Santiago A Cadena, Marissa A Weis, Leon A Gatys, Matthias Bethge, and Alexander Ecker. Diverse feature visualizations reveal invariances in early layers of deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 217-232, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 9912-9924. Curran Asso- ciates, Inc., 2020. URL httttps:/pproceediine wwwpiiinnnnn.euuurrre cc httpscccpppppeeeee cc/pppper/022//E1e.....
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference Computer Vision (ICCV), 2021.
Arantxa Casanova, Marlène Careil, Jakob Verbeek, Michal Drozdzal, and Adriana Romero. Instance- conditioned gan. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
10

---

Under review as a conference paper at ICLR 2022
Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2020.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
Ingrid Daubechies, Ronald De Vore, Simon Foucart, Boris Hanin, and Guergana Petrova. Nonlinear approximation and (deep) relu networks. Constructive Approximation, pp. 1 1-46, 2021.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT
2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics. 2019. doi: (0.18653/v1/n199 1423. URL www.oo.g.....o....ooooooo......o 1...0..8653//11///
https://doi. wttp://///...../......1...1
1423
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis.
In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. URL https
/oopenreieew.net net.eeewweeetteeeee wwt.:/euttoteeee A AAWuCvzaVE
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https::/ openreview net hett:/forumidddBBB BuTlg
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1, 2009.
Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5413-5423, 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. A disentangling invertible interpretation network for explaining latent representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9223-9232, 2020.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by pre- dicting image rotations. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net. 2018. URL
net forum? htt.:/o.m.....-111142222222 Sat.mmm21-SS111211----
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015. URL http://
wwt.oogg//bbb/aa/oo 6572
Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski. Armand Joulin, and Ishan Misra. Vissl. https: thub. com http://///ccccooooeeeerrtrtao
sssl 2021.
Cean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach self-supervised learning. In NewrIPS, 2020.
11

---

Under review as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729- 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to local Nash equilibrium. In NeurIPS, 2017.
Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18:1527-1554, 2006.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances Neural Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020, URL htt.tttss///ccccep//itttt neurips hctps/aaee//22////aaaaaaaa 4c5bcfec8
Abstaa44-bbbttrtee html
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In CVPR, 2020.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations (ICLR 2014), 2014.
Durk Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In D. Lee. M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL
neur www.rpps... cc/ pdf
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp. 5542-5550, 2017.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint,
(urXiv:1411 1.77444) 2014.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of seapeextiinaariattteeeeeenee In CVPR, 2020.
Guido F. Montúfar, Razvan Pascanu, KyungHyun Cho, and Yoshua Bengio. On the number of linear
regions of deep neural networks. In Zoubin Ghahramani. Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger (eds.), (eds.).AAvvanll Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Mon- treal, Quebec, Canada, pp. 2924 2932, 2014. URL httttstttst/// httptttppppppcpeeseitttttee nww..i.eerrpiii ips. cc/ paper Ttter/2014/aaa wwwp//aahh1199222113333333333332 http:///6weee7777111111122211 Abst .htmll
Charlie Nash, Nate Kushman, and Christopher KI Williams. Inverting supervised representations with autoregressive neural density models. In The 22nd International Conference on Artificial
Intelligence and Statistics, pp. 1620-1629. PMLR, 2019.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8162-8171. PMLR, 2021. URL http::tt///oeeeeee// iggs
mlr.
html
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In Proceedings of the European Conference on Computer Vision (ECCV), 2016.
12

---

Under review as a conference paper at ICLR 2022
D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning (ICML 2014), 2014.
Robin Rombach, Patrick Esser, and Björn Ommer. Making sense of cnns: Interpreting deep rep- resentations and their invariances with inns. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVII 16, pp. 647-664. Springer, 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. volume 9351, pp. 234-241, 10 2015. ISBN 978-3-319-24573-7. doi: 10.1007/978-33319-24554444
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 1 11((3): 211-252, 2015.
doi: 10.1007/s112633 103-007/11263--015-01166
Ruslan Salakhutdinov, Andriy Mnih, and Geoffrey Hinton. Restricted boltzmann machines for collaborative filtering. In Proceedings of the 24th International Conference on Machine Learning. ICML '07, pp. 791-798, New York, NY, USA, 2007. Association for Computing Machinery. URL/https://doi..... 0tt.///0....../1..11111 1145/
ISBN 9781595937933.. doi: 10.1145/1273496.1275333 1273496.1273557
Salimans, I. Goodfellow, W. Zaremba, Cheung, A. Radford, and X. Chen. Improved techniques for training GANs. In NeurlPS, 2016.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. Pixelenn++: A pixelenn implementation with discretized logistic mixture likelihood and other modifications. ICLR, 2017.
Ramprasaath R Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? arXiv preprint 1rXiv:1611.074455 2016.
Assaf Shocher, Yossi Gandelsman, Inbar Mosseri, Michal Yarom, Michal Irani, William T Freeman, and Tali Dekel. Semantic pyramid for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 7457-7466, 2020.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv: 13.660034 2013.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint 1xXxv:1706.088// 2017.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 2256-2265. JMLR.org 2015. URL httl:ttpppppcceednngg. mmr..rrssr.lrrr.. mlr httpa//ppppe/////11//aaaaaaaaae. html
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In 9th Interna- tional Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Buc Emily B. Fox, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 32: Annual Confer- Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Van- couver, BC, pp. 11895 11907, URL |ttp://sioooooooooooooaaaaaaaaap
www.siinnnnninnnnnnn.........m.. /c/ .tml
httll

---

Under review as a conference paper at ICLR 2022
Yang Song and Stefano Ermon. Improved techniques for training score-based generative mod- els. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Flore Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Con- ference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL 92cc30151166517732007633333383
neurips..........meem..m.....s. htcplp...ccpppppp hccc/aaper/2222/haaaae Abstseeaaaaa---ttccaatttee Abst .html
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In 9th Interna- tional Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, OpenReview.net, 2021b. BRLLhttts://ppenreeeee
nnt/ forum idd P
Aaron van den Oord, Nal Kalchbrenner, Lasse Espeholt, koray kavukcuoglu, Oriol Vinyals, and Alex Graves. Conditional image generation with pixelcnn decoders. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances Newral Information Processing Systems, volume 29. Curran Associates, Inc., 2016. htt.tttss:/ /twp://eeee/epoooooeeeeooooeeeeo ing neur
neur wesurrpp. ps. cc/
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML' 16, pp. 1747-1756. JMLR.org, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprini
2018
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention all you need. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL httpp::/ 3f5ee24354 91 (eardd53ccc4488444 Saa---aa-PPprrr..
.wwrips...... wwtppcpppppppeeeeeeeeeeeeeeeeeep 20tp////1////fi/e
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th International Conference on Machine Learning, ICML '08, pp. 1096-1103 103, New York, NY, USA, 2008. Associ- ation for Computing Machinery. ISBN 9781605582054, doi: 10.1145//390156 19.19999911111199944 URL attpss:/ddos www.oo.......o.o.oooooooo......o 1.w..1115111. 3ttp:/199155660 33919
1390294
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations deep network with local denoising criterion. Journal of Machine Learning Research, 1 11(110):3371 330004488,,888 2010. URL
/ettp://jmlrr www.llr..rgpppppaaaee wwwl/viiineetie Ca. html
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018.
Jure Zhontar, Li Jing, Ishan Misra, Yann Lecun, and Stephane Deny. Barlow twins: supersuper learning via redundancy reduction. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 12310-12320. PMLR, 18-24 Jul 2021. URL htt.httss//prcceedmm.g
mlr. pre.spres..ooo / 139/ zbont html
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference computer vision, pp. 818 033.833.355 Springer, 2014.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks, In International conference on machine learning, pp. 7354-7363, PMLR,
2019a.
Liheng Zhang, Guo-Jun Qi, Liqiang Wang. and Jiebo Luo. Aet aed: Unsupervised representation learning by auto-encoding transformations rather than data. arXiv preprint, MarXiv:1901.045000 2019b.
14

---

Under review as a conference paper at ICLR 2022
Nanxuan Zhao, Zhirong Wu, Rynson W. H. Lau, and Stephen Lin. What makes instance dis- crimination good for transfer learning? In 9th International Conference on Learning Repre- sentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL htttps://openevvier net ftrem?/rrmmmmmm :e:::id===iiiie W20Ub.f
15

---

Under review as a conference paper at ICLR 2022
A MATCHING A REPRESENTATION BY FOLLOWING INPUT GRADIENTS
We want to visualize what kinds of input images would be mapped to the same representation h as that of an input image by a given network function f. This section proposes to sample inputs that fall into that equivalence class by solving an explicit optimization problem, while the next section leverages a conditional diffusion model.
A.1 PEEKING INTO THE REPRESENTATION
INPUT SET
We call the set of inputs that trained network function f maps to given representation h the pipresentationmmatcine input set, defined formally as
(h)) _ (ar E X d(h, f 0),
where d could be any desired distance We would like to see what kinds "images" S(h) contains (besides the that may have used to obtain representation h to begin with). We tackle this problem through simple gradient-based optimization. We start from a random input xx X, sampled from basic distribution (s.a. uniform). We then performing T gradient steps in input-space towards minimizing objective d( (zx), ht, i.. to "match the representation" yielding final sample aF)) Note that the minimizer is usually not unique, so that we can obtain quite different ((") depending on the random x (0) we started from.
A.2 DO SAMPLES FROM S(f())) LOOK LIKE x??
We performed the above-described procedure to sample examples from S()())), using for I' the same ResNet50 backbone (He al., 2016) trained with the DINO SSL criterion (Caron al.] 2021) on ImageNet (Russakovsky et al. 2015)). We took from the validation set of ImageNet. In addition to standard DINO training, we also trained second SSL network (termed DINO+n) that uses independent additive noise as extra augmentation 15 which led to 71% top-1 Imagenet accuracy. Examples of obtained images are provided in Figure 7 Appendix A.3 has more details and more results for these experiments. We see that even though gradient- based input optimization manages to produce samples that match target embedding representation, this technique fails to produce realistic images. Mapping to the same SSL representation natural image is not sufficient for being a similar realistic- realisticllooking image.
The gradient directions not enough. The updates producing the sequence x!)), all follow a trajectory that only involves the Jacobian matrix of deep network ff at each step. This is due to the chain rule of caleulus and reads
x(() x
(f+1)
(t)
(2)
linear combination of Jacobian matrix
where (x))))) the linear combination coefficients, is given by (f As a result, is clear that ((+1) is constrained to be within the affine space spanned by J, (a=())""((((()) u(x(()) and shifted by (*). Given that f is a mapping from R to R.K with in general K < D, the dimension of that affine space is at most K.
We see that the representation and mapping function f obtained through SSL training are by them- selves not sufficient to recover corresponding natural-image-like inputs.
A.3 MORE ON S(f()) SAMPLING
In this section, propose in Fig. 9 additional gradient based matching that employ the projector head of DINO. We also provide in Tab. 32 the distances that those gradient based matched input
'In practice, we may be content with finding elements of a relaxed representation representation-constaaaddd set S, (h) {xe % d(h, f(=') S <} allowing for small tolerance
'The motivation was to learn a smoother map f for gradient- mudient-bbssdd based representation matching to be easier. Although the optimization proved no more difficult in the non pon-noised case, DINO+n yields qualitativeyy markedly different samples, where one can more casily distinguish natural-image like edges. The reason is unclear; one hypothesis that reliably discriminate noised instances the representation must focus more edges.
16

---

Under review as a conference paper at ICLR 2022
target
Dino, Adam, L2
Dino, LBFGS, cos
Dino+n, Adam. L2
Dinotn, LBFGS, cos
Figure Gradient based samples from (h). Leftmost image z used obtain target SSL representation h (2048 dimensions) with either a standard LINO-ResNett (Dino) one trained with additive noise as extra augmentation (Dino+n). random initialized input moved so that its representation will match h, by minimizing either 1.2 or cosine distance using Adam or L-BFGS respectively (indicated in column headers). We display the samples (T)) obtained after 7 = 10. 000 iterations obtained from the respective optimizers and distances, in all cases starting from a random Gaussian image (0) These w(22) have the same SSL representation as a very close (relative distance 0.4%, 0.1%, 3.6%. 3.3% respectively. see details in appendix Tab, 1 but do not resemble natural images. Samples obtained from Dinotn look slightly more natural: distinguish faint edges similarly shaped to the original. Similar experiments but applied on the lower-dimensional projection head embeding are reported Figure in appendix.
Figure 8: Depiction four rows the Jacobian matrix fz(x) for the input given in Figure 7 with f being a Resnet50 trained with standard DINO (left) and additive noise (DINO+n) (right). The Jacobian matrix in the noisy case shows more structures looking somewhat more natural-image like. Recalling ( this observation justifies the more natural images observed in Figure 7 Addi- tional rows provided in the appendix in Figure 10)
distance
82
f
cosine
relative & (%)
relative & (%)
relative cosine (%)
Adam plateau
0.8
30.0
0.0
0.1
5.5
0.1
Adam cosine
0.4
11.0
0.0
0.1
2.1
0.1
GD plateau
2.8
48.0
0.1
0.4
8.8
5.7
GD cosine
2.2
17.0
0.1
0.3
3.1
7.8
L-BFGS plateau
0.1
23.0
0.0
0.0
4.3
0.0
L-BEGS cosine
0.1
26.0
0.0
0.0
4.9
0.0
Table 1: We depict here the final value of the input optimization step (x!T)) We experiment with different distances (each column) and we provide the actual value of the distance along with relative distance which is obtained by 100 100 x [d([(=)), [() (z) - d(f(xx)))))))))))))(()))) (z= ((=))) That is, the relative distance gives a proportion of how close to the target is the obtained representation as ratio with respect the distance using the initial (random) image, value between 0 and 100. In
this table, we are looking at the DINO model. We provide the noise models in the below tables.
can read term of representation from a target one. the next section we also provide additional theoretical arguments supporting the challenge of following gradient directions to obtain realistic samples from S.
A.4 DEEP NETWORKS AND CPAS
In this section we propose to further characterize what S(h) looks like by using a specific form for the DN input output mapping.
17

---

Under review as a conference paper at ICLR 2022
Figure 9: Reprise of Fig. 8 but now when considering the mapping be the resnet50 backbone and the projection head of DINO. Top row is when using DINO+noise and the bottom row is when using standard DINO. As was the case when using the resnet50 backbone only, we do obtain realistic inputs from ((x) when following gradient directions.
distance
L2
f
cosine
relative le (%)
relative & (%)
relative cosine (%)
Adam plateau
3.1
54.0
0.0
0.9
13.5
0.9
Adam cosine
3.6
41.0
0.0
1.0
10.2
0.8
GD plateau
12.2
107.0
1.3
3.5
26.6
91.8
GD cosine
14.0
90.0
1.3
4.1
22.5
95.1
L-BFGS plateau
2.2
50.3
0.0
0.6
12.4
0.7
L-BFGS cosine
3.3
560.0
0.0
0.9
61.6
0.5
Table 2: Reprise of Tab. 1 but with DINO noise
distance
82
cosine
relative & (%)
relative & (%)
relative cosine (%
Adam plateau
1.8
52.4
0.0
0.3
15.6
0.3
Adam cosine
2.8
39.3
0.0
0.5
11.7
0.5
GD plateau
27.2
131.0
0.7
4.5
39.3
85.4
GD cosine
50.1
170.0
0.7
8.4
50.9
89.9
L-BFGS plateau
3.6
53.7
0.0
0.6
16.0
0.5
L-BFGS cosine
3.4
52.3
0.0
0.6
15.6
0.5
Table 3: Reprise of Tab. but with DINO noise
Without loss of generality we consider a mapping ff that continuous piecewise affine (CPA), as is the case for most DNs(Balttrriero & Baraniuk] 2018). The DN input-output is then given by
f(x)
Σ (A + bw) (wew))
with Ω a partition of the DN input space. In the case of f being smooth, a simple approximation argument will allow fall back the above setting (Daubechies et al.) [202T). Using this formulation, we can now characterize more precisely the form of (x)) as follows
S(f())) U (x (w(x')) n {a' u. u E ker (Awte'))
(4)
linear subspace ker (Awcer)) shifted by
where w(x') is the region from Ω in which x' lives in, and X(xx) is a finite set of inputs that depend on x such that each point lives a separate region from the others as Vu, v E ((f(x)) w(u)
w(v) =
18

---

Under review as a conference paper at ICLR 2022
With noise during training
Without noise (baseline) during training
Figure 10: Depiction of 36 rows of the Jacobian matrix of trained DINO model that either employed Gaussian noise on the images during training (top) or did not (bottom). Clearly the use of noise during training produce Jacobian matrices with more natural patterns.
19

---

Under review as a conference paper at ICLR 2022
Input space X
Auperao S(
Samoling RCOM
UIx hJ deep rebeat
fix)
gesdlens copastnstiitoo
Irgunt for
cond representation
Ubx. U-Netix.
x090
x7
Sampling diffudiee process)
Utar R)
Training
fis)
data
mandfold
ingul
Trantngsample
h
xfff (
M
slols h. JE-
(berwiding)
U(s
(a)
(b)
Figure 11: (a) Illustration of considered image generation methods. A real input x yields representa- tion h. All methods start from a random noise image (0) Gradient- representation matching (light blue arrows) will move it towards S(h) i.e. until its representation matches h. but won't land on the data-manifold M. Unconditional reverse diffusion (ADM model, green arrows) will move it towards the data manifold. Our representationnccnniiion will move it towards M n S(h), yielding a different natural-looking image with the same given representation. (b) Representation- RepresetaatoonCCnndttnnneed Diffusion Model (RCDM). From diffusion process that progressively corrupts an image, the model learns the reverse process by predicting the noise that it should remove each step. We also add as conditioning a vector h, which is the representation given by SSL. or supervised model for a given image c. Thus, the network is trained explicitly to denoise towards a specific example given the corresponding conditioning. The diffusion model used is the same the one presented by Dhariwal & Nichol (202I) with the exception of the conditioning on the representations.
diffusion model (RCDM, red arrows)
The optimization problem needs to be constrained. The first challenge of our method comes from the fact that the set (=)) consists of a union of affine subspace that are highly localized in the input space (recall
) In fact, the regions w E f2 are often extremely localized in the input space, especially as the architecture involves many layers (Montáfar et al., 20144] |Balestriero et al. (2019). In addition to that optimization difficulty, we have that the equivalence class does not constrain the inputs to lie within the data manifold. In fact, each affine subspace that form ((f is very high dimension as we emphasize below.
Proposition 1. Given a model f : X + H. the set S(f(x)) is a union of Card(X((x)) affine subspaces, each with dimension at least D -K-K. K.
In other words, regardless of the chosen distance d, as soon as the dimensions of the affine subspaces forming ())))) are greater than the dimension of the data manifold M sufficient condition being D - S > dim(M) we obtain that ((f()) contains samples that do not belong to M. That is, performing gradient descent from randomly initialized samples (0) will almost surely produce samples !²²) E M. This is particularly true for case such as Imagenet in which D = 150528 and
S 2048
B CONDITIONAL AND
SAMPLING WITH RCDM
As presented in the main text, we introduce RCDM to generate samples that preserved well the semantics of the images used for the conditioning. As showed in Figure Ia RCDM is constraint to map back the representation to the manifold of real images which answers the concerns raised in Appendix A, The training of the model is very simple and presented in Figure 1Tb] We show in Figure 12 additional samples of RCDM when conditioning on the SSL representation of ImageNet validation set images (which were never used for training). We observe that the information hidden in the SSL representation is so rich that RCDM is almost able to reconstruct entirely the image used for conditioning, To further evaluate the abilities of this model, we present in Figure 13 similar experiment except that we use out of distribution images as conditioning. We used cell images from microscope and a photo of status (Both from Wikimedia Commons), sketch and cartoons from
20

---

Under review as a conference paper at ICLR 2022
PACS (Li et al. 2017). image segmentation from Cityscape (Cordts al. 2016) and an image of the Earth by NASA. Even in the OOD scenario, RCDM is able to generate images that are very close to the one used as conditioning because of the richness of ssl representations.
We also use the super-resolution model presented by Dhariwal & Nichol (202T) to generate images of higher resolutions. In Figure [ we use the small images the top of the bigger images as conditioning for a RCDM trained on images of size 128x 128. Then, we feed the 128x128 samples into the super-resolution model of Dhariwal Nichol (2021) p to get images of size 512x512. Since the model of Dhariwal & Nichol (2021) is conditional and need labels, we used a random label when upsampling from RCDM. Despite using the "wrong" label, the high resolution samples are still very close to the conditioning. This show that RCDM can be used jointly with super- reperrresolution model to sample high fidelity images in the close neighhorhood of the conditioning.
To verify how well our model can produces realistic samples from different combinations of repre- sentations, we take two images from which we compute their representations and perform linear interpolation between those. This give us new vectors of representation that can be used as condition- ing for RCDM. We can see on Figure and Figure 16 that RCDM is able to generate samples that contains the semantic characteristics of both images.
we search the nearest neighbors of series of samples in the ImageNet training set. As demonstrated by Figure 17 RCDM samples images that are new and far enough from images belonging to the training set of ImageNet.
Finally, in Figure 17
C A HIERARCHICAL DIFFUSION MODEL FOR UNCONDITIONAL. GENERATION
We provided a novel and conditional generative model based a given latent representation e.g. from a SSL embedding. and a diffusion model. This provided tremendous insights into interpreting what is encoded in those representations. We can go one step further and augment this conditional model with an unconditional one that can generate those representations. This will provide us with the ability generate new samples without the need to condition will allow us to quantify the quality of our generative process in an unconditional manner to fairly compare against state-of-the-
given input. As by-product,
generative models.
We shall recall that our goal is to employ the conditional generative model to provide understanding into learned (SSL) representations. The unconditional model is only developed compare our generative model and ensure that its quality is reliable for any further down analysis. As such, we propose learn the representation distribution in a very simple manner via the usual Kernel Density Estimation (KDE). That is, the distribution is modeled as
p(h) Σ
N(: I(xn)
with a 0.01. By using the above distribution, we are able to sample representations to then sample images x conditionally to that h using our diffusion model. We provide some samples in Figure[1] to show that even with our very simple conditioning, our method is still able to generate realistic images.
D ON THE CLOSENESS OF THE SAMPLES IN THE REPRESENTATION SPACE
Even if we show that RCDM is able to generate images that seems visually close to the image used for the conditioning, it's still unclear how close those images are in the representation space. We can compute euclidean distances but to know how close the generated samples are to the conditioning, we need to have references that can be used to compare this distance with. As references, we compute the euclidean distance between a conditioning image and random images in the validation set of ImageNet, random images belonging to the same class as the conditioning, the closest images in the training set, the conditioning image on which we applied single data augmentations and the conditoning image
which applied the data augmentation performed by Swav and Dino (Caron) jet al. 2020 2021). ). The results can be seen in Figure 22 for a RCDM trained with Dino representations and in Figure 23 for RCDM trained with SimCLR representations. On both Figure, observe that
21

---

Under review as a conference paper at ICLR 2022
Figure 12: Generated samples from RCDM on 256x256 images trained with representations pro- duced by Dino. We put on the first column the images that are used to compute the representation conditioning. On the following column, we can see the samples generated by RCDM. It is worth to denote our generated samples are qualitatively close to the original image.
22

---

Under review as a conference paper at ICLR 2022
Figure 13: Generated samples from RCDM model on 256x256 images trained with representations produced by Dino on Out of Distribution data. We put on the first column the images that are used to compute the representation. On the following column, we can see the samples generated by RCDM. It is worth to denote our generated sample are close to the original image. The images used for the conditioning are from Wikimedia Commons, Cityscapes (Cordts et al. 20166. PACS (Li al, 2017) and the image of earth from NASA.
23

---

Under review as a conference paper at ICLR 2022
Figure 14: High resolution samples from our conditional diffusion generative model using the super resolution model of Dhariwal & Nichol (022)). We use the small images on the top of each bigger image conditioning for diffusion model trained with Dino representation on 128x 128 images. Then, we feed the samples generated to the super resolution model of Dhariwal & Nichol (2021) which produces images size 512x512. Since the super resolution model is conditional, we sample random label. We note that the high resolution samples are still very close to the conditioning.
24

---

Under review as a conference paper at ICLR 2022
Conditioning
Conditioning
Conditioning 2
Conditioning 2
2ondiooning
(a) I. Linear interpolation between the image of the golden retriever in conditioning 1 with various other images helonging the same class conditioning
Conditioning
Conditoninn 2
Conditoning2 Conditioning
Conditioning
(b) Linear interpolation between the image of the pug in conditioning with various other images belonging to the espresso class conditioning 2.
Figure 15: Each vectors that result from the linear interpolation is feed to a RCDM trained with Barlow Twins representation.
the generated images with RCDM are closer to the conditioning than the closest neighbors in the entire training set of ImageNet. We also computed the and reciprocal mean rank in the main paper (Table which show that for most SSL models the closest examples in the representation space of the generated images is the image used conditioning. We also added Figure 18 to show
25

---

Under review as a conference paper at ICLR 2022
Figure 16: Diversity of the samples generated by RCDM on interpolated representations. Each row corresponds to different random noise for the same conditioning. On the first and last column are the real images used for the interpolation. All of the images in-between those rows are samples from RCDM.
26

---

Under review as a conference paper at ICLR 2022
which rank is associated to samples generated by RCDM. For SimCLR, the rank is mostly always 1 whereas we got more diversity for the supervised case. This difficulty of RCDM to generated samples which have their representation that map back to the one used for the conditioning can be explain by the nature of a supervised training. In such scenario, the encoder is trained to map a big set of images (often specific class) to specific type of representation whereas SSL models are explicitly train to push each examples farther away from each others. Thus, it seems more likely that a little perturbation on the supervised representation induces change of nearest neighbor. This hypothesis is supported by Figure 30 which show that small adversarial attack are enough to induces change of class in the representation which is not the case for SSL encoders.
E ANALYSIS OF REPRESENTATIONS LEARNED WITH MODEL
Having generated samples that are close in the representation space to a conditioning image can give us an insight on what's hidden in the representations learned with self-supervised models. As demonstrated in the previous section, the samples that are generated with RCDM are really close visually the image used as conditioning. This give an important proof of how much kept inside a SSL representation. However, I's also important to consider how much this amount of "hidden" information varied depending on the SSL representation that is used. Therefore, we train several RCDM on SSL representations given by VicReg (Bardes et al.] 2021), Dino (Caron et al., 202 I). Barlow Twins (Zbontar et al.) 202T) and SimCLR (Chen et al. 2020p. In many applications that used self-supervised models, the representation that is used is the corresponding the backbone of the ResNet50. Usually, the representation given by the projector of the SSL-model (on which the SSL criterion is applied) is discarded because the results many downstream tasks like classification is not as good as the backbone. However, since our work is to visualize and better understand the differences between SSL representations, also trained RCDM the representation given by the projector of Dino, Barlow Twins and SimCLR. In Figure 25 and 26 we condition all the RCDM with the image labelled conditioning and sample 9 images for each model. We observe that Dino representation does not allow much variance meaning that even information about the pose of the animal is kept inside the representation. In contrast, the SimCLR representation seems to be more invariant to the pose of the kangaroo, maybe too much because on many images the kangaroo become a rabbit. VicReg seems to be more robust in the sense that the animal doesn't cross the class boundary despite changes in the background. If we look at the projector of the SSL models, the generated samples have a higher variance except for Barlow Twins. This can be explained by the fact that the dimension of the representation given by the projector has size of 8192 which is much bigger than the one used by other methods.
To further compare and analyse the different SSL models, we visualize how much SSL representations can be invariant with respect to a transformation that is applied on the conditioning image. In Figure we apply several Data Augmentation: Vertical shift, Zoom out, Zoom In, Grayscale and Collor Jitter on given conditioning image. Then we compute the SSL. representation of the transformed image with different SSL models and use our corresponding RCDM see how much the samples have changed with respect to the samples generated on the vanilla conditioning image. We observe that the representation (the 2048 backbone one) of all SSL methods are not invariant to scale and change of colors. Whereas the representation of the projector doesn't 't seem to take into account any small transformation in the original conditioning outside the scale for Dino. For SimCLR, there is still some information about the background that is kept in the representation however the samples are not as close visually with respect to the 2048 representation. Barlow Twins interesting because there isn't 't much differences between the backbonn representation (2048) one and the representation of the projector (Size 8192). With the exception that this last representation seems to be more invariant to color shift than the backbone one.
27
E. VISUALIZATION OF ADVERSRIAL EXAMPLES
We use RCDM to visualize adversarial examples for different models. For each model, we trained a linear classifier top of their representations to predict class labels for the ImageNet dataset. Then, we use FGSM attacks over the trained model using a NLL loss to generate adversarial examples. In Figure 30 we show the adversarial examples that are created for each model, the samples generated
27

---

Under review as a conference paper at ICLR 2022
by RCDM with respect to the representation of the adversarial perturbed example and the class label predicted by the linear classifier over the adversarial examples. The supervised model is very sensitive to the attack whereas SSL models seems more robust.
E.2 MANIPULATION OF SSL REPRESENTATIONS
It is also possible to manipulate SSL representations to generate new images. We try to apply addition and subtractions over SSL representations (similarly what has been done in NLP). From two different images, we compute the difference between the two corresponding representations and add the difference vector to third image. F3 shows that it possible apply such transformations meaningfully in the SSL space. We also used another setup where choose specific dimensions in the representation based on how many times these dimensions are non zero in the representation space of set of neighbors. Then set this dimension to which surprisingly induces the removing of the background in the generated images. We also replace them by the same corresponding dimension of another images which induces change of background toward the one of the new image. Results are shown in Figure 31
28

---

Under review as a conference paper at ICLR 2022
Semples
Dicoaent traningg Imagee
Samples
Crosest maaning Iraege
Sumphha
Closeest traning Image
(a) Closest real images (ImageNet training set) from images sampled with RCDM trained Dino (backbone) representation (2048).
Semples
Choceet trainigg Imagge
Samples
Clcoeet traiiin Iragge
Sumphis
Otovest traming
Image
(b) Closest real images (ImageNet training set) from images sampled with RCDM trained on Dino projector representation (256).
Figure 17: We find the nearest neighbors in the representation space of samples generated by RCDM. The images in the red squared are the ones used for conditioning.
29

---

Under review as a conference paper at ICLR 2022
SimCLR
Supervised
RCOM
Rank
1
1
1
4
1
2
2
Rank
2
8
1
1
5
2
RCDM
NN
Rank
1 2 8
1
RCOM
1
Figure 18: After generating samples with respect to a specific conditioning, we compute back the representation of the generated samples and find which are the closest neighbors in the validation set. Then, we compute the rank of the original image that was used as conditioning within the set of neighbors. When the rank is one, it implies that the nearest neighbors of the generated samples is the conditioning itself, meaning that the generated samples have their representation that is very close the representation space to the one used as conditioning. the generated samples are much closer in the representation space to their conditioning than the supervised representation. This casily explain by the fact that supervised model learn to map images from same class toward similar representation whereas SSL models try to push further away different examples.
can see that for SimCLR,
30

---

Under review as a conference paper at ICLR 2022
Figure 19: Visual analysis of the variance of the generated samples for a specific image when using a supervised encoder. The first image (in red) in the one used as conditioning.
31

---

Under review as a conference paper at ICLR 2022
Figure 20: Visual analysis of the variance of the generated samples for a specific image when using SimCLR encoder. The first image (in red) in the one used as conditioning.
32

---

Under review as a conference paper at ICLR 2022
Unconditional generation following the protocol section C Our simple generative model of representations consists applying small Gaussian noise over representation computed from random training images of ImageNet. We use these noisy vector as conditioning for our 256x256 RCDM trained with Dino representations. We note that the generated images looks realistic despite some generative artefact like a two-headed dog and an elephant-horse.
Figure 21:
33

---

Under review as a conference paper at ICLR 2022
Random
0.0
636.5
868.7
1042.9
712.5
1443.8
Same class
1833.4
1040.7
673.9
650.4
745.7
Clasest Images
340.1
344.8
356.5
364.2
D.A.
23.8
478.3
64.5
173.9
147.3
DA.2
417.7
510.0
444.5
445.6
379.5
Generated
132.5
93.8
91.4
122.0
124.5
Figure 22: Squared Euclidean distances in the Dino representation space. We show the squared euclidean distance between the conditioning image the leftmost column first and different images to get an insight about how close the samples generated by the diffusion model stay close to the representation used conditioning. The distances with the conditioning printed below each images. On the first row, we show random images from the ImageNet validation data. On the second row, we take random validation examples belonging to the same class as the conditioning. third row, we find the closest training neighbors of the conditioning in the representation space. On forth row, D.A. means Data Augmentation which consist in horizontal flip, CenterCrop, ColorJitter, Scale and solarization. On fifth row AD.A. 2), use the random data Augmentation used in the paper of (Caron et al. 2020 2021). On the last row, we show the generated samples from our conditional diffusion model that Dino representation. The samples produces by model much closer to the conditioning than other images.
the
34

---

Under review as a conference paper at ICLR 2022
UNLEADED
Random
0.0
8.8
7.1
4.8
26.4
8.1
Same class
35.4
13.7
6.0
9.0
13.3
Closest images
FAMILY PORTRAIT
CARBAGE TRUCH FAIL
2.7
D.A
0.3
2.6
0.4
0.8
1.6
DA.2
5.2
3.9
5.1
5.6
2.7
Generated
1.3
2.4
1.5
2.4
2.2
Figure 23: Squared Euclidean distances in the SimCLR projector head representation space. We show the squared euclidean distance between the conditioning image on the leftmost column on first row and different images get insight about how close the samples generated by the diffusion model stay close to the representation used as conditioning. The distances with the conditioning is printed below each images. On the first row, we show random images from the ImageNet validation data. On the second row, we take random validation example belonging to the same class as the conditioning. On third row, we find the closest training neigbords of the conditioning in the representation space. On forth row, D.A. means Data Augmentation which consist in horizontal flip, CenterCrop. ColorJitter, GrayScale and solarization. On fifth row (D.A. 2), we use the random data Augmentation used in the paper of (Caron al. 2020 2021). On the last row, show the generated samples from our conditional diffusion model that use SimCLR projector head representation. The samples produces by our model are much closer to the conditioning than other images.
35

---

Under review as a conference paper at ICLR 2022
RCNK
M
Distance: 0
17.9
21.
19.7
24.2
23.6
21.2
N
Distance:
35.5
42.4
37.2
R
D
M
Distance:
31.5
34,9
35.2
34.6
33.7
31.9
Distance:
43.2
38.5
45.4
42,4
43.2
44,5 .4
R
C D
M
Distance:
31
20.5
28.8
26.2
26.8
25.6
C
G
Distance:
50.6
43.8
43.2
40.7
43.1
Figure 24: Comparison of the euclidean distance between IC-GAN and RCDM. We use the same self-supervised representation as conditioning (Swav encoder) for RCDM and IC-GAN. We compute the cuclidean distance between the representation of the generated images versus the representation used as conditioning. We observe that samples of RCDM are much closer in the representation space (and also visually) the conditioning. Samples of IC-GAN show higher variability, thus farther away in the representation space.
36

---

Under review as a conference paper at ICLR 2022
Cardllantno
SimCLA
Head
SimOLR Head Superised
Figure 25: Generated samples from RCDM trained with representation from various self-supervised models. The image used for conditioning is baby kangaroo the top row, left-most column. Then, we generate 9 samples for each model with different random seed. We observe that the representation given by dino isn't t very invariant while the one given by SimCLR invariance. We also show the samples of RCDM trained on the representation given by the projector (The embedding which is usually applied the SSL criterion). There is much higher variability in the generated samples. Maybe too much to be used for classification task since we can observe class crossing.
VicReg show much better
37

---

Under review as a conference paper at ICLR 2022
Bofoow
Dins Head
SimC CLt
Figure 26: Generated samples from RCDM trained with representation from various self-supervised models. We generate 9 samples for each model with different random seeds. We observe that the representation given dino isn't 't very invariant while the one given by SimCLR or VicReg show much better invariance. We also show the samples of RCDM trained on the representation given by the projector (The embedding which is usually applied the SSL criterion). There is much higher variability in the generated samples. Maybe too much to be used for a classification task since we can observe class crossing.
38

---

Under review as a conference paper at ICLR 2022
Conditioning
Dino
SimCLR
Barlow
Vartical
givit
Zoom
Zoom
Greyscale
Coler Jitter
Conditioning
Dino head
SimCLR head
Barlow head
Vertical
Zoom
Zoom
Grayscals
Color Jitter
Figure 27: We compare how much the samples generated by RCDM change depending on different transformations of a given image and the model and layer used to produces the representation. Top half uses 2048 representation. Bottom half uses the lower dimensional projector head embedding. We observe that using the projector head representation leads to much larger variance in the generated samples whereas using the traditional backbone (2048) representation leads to samples that are very close to the original image. We also observe that the projector representation seems to encode object scale, but contrary to the 2048 representation, it seems to have gotten rid grayscalc-status and background color information.
39

---

Under review as a conference paper at ICLR 2022
Condiioning
Barlow T.
SmCLR
Supervised
Vertical
Zoom
Zoom
Grayscale
Celor Jitter
Conditioning
Berlow T.
SimCLR
Supervised
Vertical
Zoom
Zocm
Grsacale
Color litter
Figure 28: Same setup as Figure 27 except with other images as conditioning.
40

---

Under review as a conference paper at ICLR 2022
200
Batoo T.
SInCLA
Dino freadd
Barlow T. Head
SinCLA head
Superrtee
Figure 29: Generated samples from RCDM using the mean representation for a specific class (golden retriever) in ImageNet for various SSL models.
41

---

Under review as a conference paper at ICLR 2022
Supandsed
0.0
0.01
0:06
0.1
0.5
1.0
20
1000
Golden Retrlever
Learberg
Golder Retriover Golcien Retriever
Coyoite
Chain meil
Cheik meill
0.3
0.01
0.05
0.1
0.5
1.0
2.0
Golttin Ratrieeer Goldenn Ratriever
Lion
Chow chow
Biain cara
Brain conal
Namatode
0.0
0.01
0.06
0.1
0.5
2.0
Ejs Advussitial
Barlow T.
ROOM
Sma Chad
Ceere
Goiden Retrievor Golden Retrivvrr Golden Retreavrr
Chow chow
Chsinink fance
Brain coral
Brsin cOIN
Figure 30: Visualization of adversarial examples We RCDM visualize adversarial examples for different models. For each model, we trained linear classifier on top of their representations to predict class labels for the ImageNet dataset. Then, NLL loss to generate adversarial examples towards the class lion. For each model, we visualize adversarial examples for different values of which is the coefficient used in front of the gradient sign. In the supervised scenario, even for small values of epsilon which doesn' 't seem to change the original image, the decoded image as well as the predicted label by the linear classifier becomes lion. However it's not the case in the self sepfrsuper setting where the dog still get the same class or get another breed of dog as label until the adversarial attack becomes more visible to the human eye (For value superior to 0.5).
FGSM attack the trained model using
42

---

Under review as a conference paper at ICLR 2022
mask most common indices where dim representation is non Least common dim where dim of representation is non zero
Figure 31: Background suppression and addition Visualization of direct manipulations over the representation space. On the first row, we used the full representation of the dog's 's image on the top-left conditioning for RCDM. Then, we find the most common the neighborhood of the image used as conditioning. On the second row, we set these dimensions zero and use RCDM to decode the truncated representation. We observe that RCDM produces examples of the dog with high variety of unnatural background meaning that all information about the background is removed. In the third and forth row, instead of setting the most common non zero dimension to zero, we set them to the value of corresponding dimension of the representation associated to the image on the left. As we can see, the original dog get a new background and new pose.
zero dimension across
zero common indices representation non
Least common dim of where dim of representation is non zero
except that instead using the most common non zero- dimensions mask, we used the least common non-zero dimensions mask. On the second row, we observe that some information about the original dog removed such that each column, we get slightly different breed of dog while the background stay fixed. On the third and forth that the information about the background (grass) propagated through the samples (which was not the case in Figure 31
Figure 32: Same setup as Figure 31
saw
43

---

Under review as a conference paper at ICLR 2022
Figure 33: Algebraic manipulation of representations from real images hleft-hand side of =) allows RCDM to generate new images with novel combination factors. Here we use this technique with ImageNet images, to attempt background substitutions.
44

---

Under review as a conference paper at ICLR 2022
Condowee
(a) Earth from an untrained representation (Random initialized Resnet 50),
Ged
(b) Earth from supervised representation (Pretrained resnet50 on ImageNet)
(c) Earth from a SSL representation (Dino Resnet50 backhone).
Figure 34: Different samples of RCDM conditioned
satellite image of the earth (source: NASA).
We show the samples we obtained in a) when using random initialized network to get representations, b) when using pretrained resnet50, c) when using self supervised model (Dino).
45


Reviews:
[{"review_id": "qAfb1xqone6", "reviewer": "Reviewer_rQaS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main idea of the paper, i.e. explaining the representations (and their invariances) learned by deep neural networks via visualizations through natural images is intuitive and very reasonable. By demonstrating that gradient-based methods do not suffice (Sec. 2) to reconstruct _natural_ images, the authors show the need for a probabilistic approach and, by opting for a conditional diffusion model, provide a method capable of generating high-quality images corresponding to the representations, as demonstrated in the experimental section. The main drawback of this paper, however, is that this idea is not novel and the paper fails to discuss prior work which uses different implementations for the conditional generative model, e.g. GANs [1], autoregressive models [2] or normalizing flows [3]. Authors should clearly situate their work within this existing literature and discuss what advantages/differences an approach based on DDPMs offers. For example, is it possible to use a deterministic approximation of the diffusion process (such as DDIM, see [4]) to extract, analogously to [3], an explicit representation of the learned invariances? Further, I do not quite understand the focus on SSL representations, as the proposed techniques can easily be applied to any type of representation, independent of the training paradigm. \n\nMoreover, a quantitative assessment of the correspondences between representations and synthesized images is missing. For example, can the synthesized representations be fed back into the (SSL) model and does this then yield the same representation $h$?\n\n_Mixed comments:_\n\n- I think the paper would benefit from focusing on __explaining__ (SSL) representations and thus dropping the part on synthesis (mainly Appendix B), as it mainly relies on methods that already have been developed and discussed in the context of pure synthesis (ADM).\n- On the contrary, Appendix E and especially Fig. 20 should be moved to the main paper, because they provide interesting analysis of different SSL models/representations.\n- 1st paragraph of 4.2.: I do not quite understand the motivation here: In the case the density of $p(x|h)$ approaches a delta-distribution, the task becomes more easy and in the limit, could be implemented through a regression model.\n- Following on from this: the paper should demonstrate the need for a probabilistic approach by comparing, for example, with a simple regression baseline $x = f_{\\theta}(h)$. A more advanced baseline would be the IC-GAN discussed in Sec. 4.2 (without neighborhood sampling).\n- _\"Diffusion models do not usually explicitly express an energy function\"_ (Footnote 4) The usual training mechanism is to train the reweighted epsilon-parameterization from [5] which corresponds to a denoised score matching objective. This score corresponds to the derivative of an energy which can be plugged into MCMC sampling, see [6].\n- The figure on the first page should show several examples of each model to emphasize the visualization capability of the invariances.\n- The interpolation in Fig. 4c should show several \"interpolation-paths\", demonstrating the stochasticity of the proposed approach (DDPM)\n- Does the proposed conditioning mechanism via conditional batch norm provide significant advantages over other methods (e.g. addition to the timestep-embedding, AdaIN, cross-attention, ...)?\n\n__References:__\n- [1]: Shocher, A., Gandelsman, Y., Mosseri, I., Yarom, M., Irani, M., Freeman, W.T., Dekel, T.: Semantic Pyramid for Image Generation. \n- [2]: Nash, C., Kushman, N., Williams, C.K.: Inverting Supervised Representations with Autoregressive Neural Density Models.\n- [3]: Rombach, R., Esser, P., Ommer, B.: Making Sense of CNNs: Interpreting Deep Representations & Their Invariances with INNs\n- [4]: Song, J., Meng, C., Ermon, S.: Denoising Diffusion Implicit Models\n- [5]: Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models\n- [6]: Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations\n", "summary_of_the_paper": "This paper proposes a technique for visualizing the representations of fixed pretrained self-supervised neural networks. More specifically, by using a conditional denoising diffusion probabilistic model (DDPM, coined _RCDM_ here), a learned neural network representation $h$ can be mapped (back) to image space (i.e., the space of \"natural images\") and, by drawing multiple samples conditioned on the same $h$, the invariances learned by the (self-supervised) network can be visualized. Experiments with different pre-trained models show that the method is indeed able to synthesize high quality natural images corresponding to the learned representations.", "summary_of_the_review": "I think that the present paper deals with a very important problem, namely the \"understanding\" of learned neural representations. The proposed approach to do this by synthesizing \"natural\" images generated by a generative model conditioned on these learned representations (here a diffusion model) is intuitive and reasonable. The generated samples are of high quality and seem to be in agreement with the representations. \n\nUnfortunately, this approach is not novel (only the use of a diffusion model instead of {GANS, ARMs, flows} is new here) and the present work would, in my opinion, benefit greatly from being placed in the context of this existing work. In addition, there are some specific questions (see above) that I think are not yet answered satisfactorily. \n\n++++++++++++++++++++++++++\n\nScore raised to 6 after the rebuttal\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "h8KZkBLY1vQ", "reviewer": "Reviewer_MCTj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n\n1. The question is meaningful, and the proposed technique is interesting. \n2. Visualizations generated using the proposed technique look promising in addressing the question laid out in the study.\n3. The paper is very well-written, and the analyses are well-motivated.\n\nCritiques: \n1. Some conclusions are stated as though they are specific to SSL but it might be good to know whether or not they are applicable for supervised training as well. For instance, the authors claim that `mapping to the same SSL representation as a natural image is not sufficient for producing a similar realistic-looking image’. My intuition is that that should be true for the supervised case as well due to the very nature of gradient-based matching on a deep layer representation when starting from random noise images. This result is not very surprising and has been described before. \n\n2. Imposing the 'naturalness’ constraint always arises out of a need for human understandability but it is not the most faithful account of what information is contained in the representation. I am a little concerned by the use of the phrase 'high fidelity’ in this setting to describe the visualization/model inversion process.  Perhaps it would be useful to briefly discuss the effect of different constraints on the input space in terms of fidelity – one with fewer control knobs (regularizations, naturalness prior etc.) is going to rank better on the fidelity scale..?  And the perceptual quality of image generation is not likely a good reflection of its faithfulness to the representation. \n\n\n3. Given that the goal was to characterize (through visualization) what is contained in a self-supervised representation, I think the results are not particularly intriguing and seem inadequate in their present form. Additional work and more rigorous analyses, including exploring more models, producing stronger results by analyzing more images etc., seem needed to strengthen the paper. \n\n4. The proposed method for analyzing representations is also highly subjective (not unlike other interpretability approaches I concede) but still there are no recommendations or thorough analysis about how to find the key factors driving a representation. e.g., how many images should be sampled from the RCDM to find the commonalities? Further, the conclusions that are drawn using this conditional image syntheses approach could just as well have been drawn by analyzing the representations directly (for instance, by seeing how well the scale of images can be decoded from the representation). \n", "summary_of_the_paper": "This paper proposes a representation analysis technique to particularly understand what information is contained in self-supervised models. The authors propose a visualization technique based on conditional diffusion models that seeks to synthesize realistic-looking inputs whose representation matches that of a target image. Finding commonalities among these synthesized images likely provides clues about the target representation. ", "summary_of_the_review": "I enjoyed reading the paper. However, I'm uncertain about how much value this paper adds in efforts towards understanding the nature of self-supervised representations and the results are not very interesting. As a result, I think the paper falls slightly below the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Wx7CGOt496", "reviewer": "Reviewer_mFTx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strength**: This work provides a visualization tool for understanding NN representations. The proposed RCDM is not technically challenging (it is adapted from ADM in the prior work), but it works well for probing the information in NN representations without sacrificing generation quality.\n\nThe experiments cover different SSL methods and the comparison with supervised learning, showing results on 1) in/out of distribution generation, 2) interpolation, 3) super-resolution, 4) unconditioned generation, 5) algebraic manipulation. \n\nOther aspects:\n  - The writing is clear and easy to follow.\n  - The relation and comparison with prior work is discussed adequately.\n  - The code will be released.\n\n**Questions**:\n  - Figure 1: it would be better to provide more context for the relative distance: e.g. is the difference between 0.4% and 3.3% significant?  Should a 3% distance considered small?\n  - Figure 6: projector head vs backbone: are features from the projector able to get as close to the original feature as features from the backbone? If not, how much further are the projector features compared to the backbone features?\n\nMinor comments:\n  - The writing for Figure 7 is repetitive, i.e. the last paragraph on page 8 and the caption for Figure 7.\n  - last sentence in the first paragraph: \"...or discriminating instances.\"\n  - the paragraph below equation (2): should the dimension be $K$ rather than $S$?\n  - Sec 4.1, second line in the second paragraph: model \"dependent\"\n  - Last sentence of the caption of Figure 4: should be column 2 to 6 (rather than 7)\n  - Appendix section C: it should link to Figure 15 rather than Figure 21.\n\n", "summary_of_the_paper": "This paper proposes a conditional diffusion model that can be used to visualize representations learned by SSL or supervised models.\n\nThe proposed Representation-conditioned Diffusion Model (RCDM) can generate images that are both close in the representation space to a given image, and looking realistic.\n\nThe paper also provides empirical results comparing conditionally generated samples from models trained on features from the backbone or projector of a SSL model, as well as supervised features. These experiments provide insight to understanding SSL, such as showing that projector is the key of SSL being invariance to transformations.\n", "summary_of_the_review": "The method in this paper is unsurprising but gives good results.\nI find the experiments thorough and insightful, and would recommend an accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "BQhP1rmp6dK", "reviewer": "Reviewer_YLR1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThis paper considers a very worthy problem, that of analyzing encoders via visualizations, and clearly develops a high quality generative modeling approach with which to do this. I think this paper may be on its way to being a vey nice paper, however I have concerns with it’s current state. \n\nCurrent strengths include:\n\n- high quality of generative models used (the generated samples are high fidelity). \n- plenty of qualitative evaluation - the appendix contains many pages of additional visualizations. \n- the method is able to visually discern some differences between supervised and self-supervised trained encoders. The differences between  self-supervised methods seems a little harder to extract, besides noting that some (e.g., DINO) exhibit better invariance than others (e.g., SimCLR).\n\n\nMy two broad concerns (i.e., \"weaknesses\"), which I discuss in more detail momentarily, are:\n\n- A lack of *quantitative results*. All results presented are qualitative and visual in nature (with the exception of a sanity check that the generative models are producing photorealistic samples). \n- A lack of demonstration of useful applications for the proposed method. \n\n**Quantitative results:**\n\nWhile a large part of generative modeling is focused on qualitative production of photorealistic samples, in order to properly compare you method and analyze pertained encoders I think it is critical to have more quantitative evaluation. \n\nI am sure the authors can come up with many better ideas for additional quantitative evaluations than I can, but I will also attempt to offer some actionable feedback:\n\n- The main premise of the work is that the samples $x’$ from your diffusion model are such that $f(x’)$ are very close to the original representation h. It seems almost a must that this is quantitatively confirmed. \n- You argue verbally that for a given representation $h$, IC-GAN generates samples which map to a neighborhood of $h$, whereas your method generates samples that map very closely to $h$. While this may very well be true, it would be much more convincing if you quantitatively showed this. \n\n**Applications of method:**\n\nAt present, the work mainly attempts to visualize some simple properties that one would expect a self-supervised encoder to have - e.g., a more instance-level invariance then supervised encoders, and invariance to the data augmentations used training training. Unfortunately I don't find this to be sufficiently insightful right now. In order to truly demonstrate the value of this method, it would be very valuable to take more steps towards devising practically useful tools that are enabled by this method. Some ideas of the kind of thing I am imagining are:\n\n- Could your visualization method be used to determine a priori which out of a group of encoders will perform best at a given downstream task. If possible, this would be very cool since the trial and error evaluation of pretrained encoders is annoyingly heuristic. E.g. specifically, could you develop some metric based on your visualization method that correlates well with (e.g.,) segmentation performance. \n- Suppose you have two encoders $f$ and $f’$, and you know that f performs much better on some downstream task $T$. Could you use the diffusion model for $f$ to *generate new training samples* with which to finetune the encoder f’ itself (not just the linear probe) so that the performance of $f’$ on task $T$ improves. \n\n\nTo be clear, I am not asking you to try any of these ideas specifically. Instead I am trying to illustrate the type of thing I envisage when I ask for extensions that analyze encoders in more detail. This is a promise that was made in the abstract — “new avenues to analyze and improve self-supervised models” — but which was underdelivered on in my eyes. \n\n---\n**Miscellaneous comments and questions:**\n\n- The difference in FID and IS for your method compared to ADM seems very large considering your method is a modification of theirs. Do you know what accounts for this? Is it better model training? Is it the fact that your method is instance conditioned, & therefore requires less variation in samples? Something else?\n- Some experiments seem orthogonal to the idea of analyzing pretrained encoders. E.g. the value of Fig 4c) is unclear to me (4 a) and b) are good) and seem to be more of the form of a “sanity check” that the generative model is working sensibly. \n- Please fix your bibliography. Many citations are missing their publication venues (e.g. Barlow Twins and at least three of Yang Song’s papers), and some citation simply list: 1) authors names, 2) paper’s name, 3) year — i.e. they don’t even have an arXiv number, let alone a publication venue (e.g. SimCLR and VICReg papers). \n\n", "summary_of_the_paper": "This work introduces develops a method for sampling different natural images that a pretrained encoder maps to the same or similar representation. To do this, the authors propose a modification to the diffusion model of Dhariwal & Nichol for sampling conditioned on a given encoded representation $h=f(x)$ that is well suited to conditioning on high dimensional vectors (i.e., representations). This new method is used to qualitatively compare encoders trained via a number of methods (supervised, DINO, SimCLR, Barlow Twins, VICReg etc.).\n\n", "summary_of_the_review": "\nThe reasons I am concerned about these two main objections are: \n- quantitative results are almost a must in order to properly evaluate methodological progress, and  check that the method is behaving as it should, and to properly compare models/methods etc.,\n- The methodological contribution of the representation conditioned diffusion model is relatively small (it sufficed to simply describe the changes one needs to make to ADM in words). This is not a problem in itself. But I do think it is therefore important to demonstrate the method's usefulness in analyzing and understanding pretrained models. \n\nConsequently, I am currently not in favor of acceptance. However, to recognize the generally high quality of the work thus far I am opting for a weak reject. I am unlikely to raise the score to an accept without significant updates to the work, or strong arguments in favor from other reviewers. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for visualizing representations of neural networks trained with self-supervised learning with conditional denoising diffusion probabilistic models. By generating multiple images conditioned on a representation, one can identify what aspects the representation is and is not sensitive to. The proposed method allows for high fidelity generated images that can be used to compare different self-supervised methods and layers.

Reviewers agreed that the paper proposed reasonable methodology, targeted an interesting problem of understanding what is learned by self-supervised methods, and presented interesting qualitative evaluations. However, there remained concerns on the novelty of results in comparison to other methods for probing representations (e.g. classification based), subjectiveness of interpretation of the qualitative results, and limited quantifications of the intuition gained from the visualizations. While the authors have argued that the point of the paper is to showcase the merits of qualitative visual analysis method, reviewers found that the presented results were insufficient to demonstrate the value of the proposed approach. A number of ideas were discussed with reviewers on how to highlight the value of visualization which could strengthen the paper in the future. Given the lack of novelty on the conditional generation side, and limited insight gained from the qualitative results, I cannot recommend this paper for acceptance in its current form.
Decision: Reject