Generative Models as a Data Source for Multiview Representation Learning
Authors: Ali Jahanian
Xavier Puig
Yonglong Tian
Phillip Isola
Venue: ICLR 2022 Poster (2022)
Keywords: Generative models
GANs
Contrastive Learning
Representation Learning

Abstract: Generative models are now capable of producing highly realistic images that look nearly indistinguishable from the data on which they are trained. This raises the question: if we have good enough generative models, do we still need datasets? We investigate this question in the setting of learning general-purpose visual representations from a black-box generative model rather than directly from data. Given an off-the-shelf image generator without any access to its training data, we train representations from the samples output by this generator. We compare several representation learning methods that can be applied to this setting, using the latent space of the generator to generate multiple "views" of the same semantic content. We show that for contrastive methods, this multiview data can naturally be used to identify positive pairs (nearby in latent space) and negative pairs (far apart in latent space). We find that the resulting representations rival or even outperform those learned directly from real data, but that good performance requires care in the sampling strategy applied and the training method. Generative models can be viewed as a compressed and organized copy of a dataset, and we envision a future where more and more "model zoos" proliferate while datasets become increasingly unwieldy, missing, or private. This paper suggests several techniques for dealing with visual representation learning in such a future. Code is available on our project page https://ali-design.github.io/GenRep/.

Published as a conference paper at ICLR 2022
GENERATIVE MODELS AS A DATA SOURCE FOR MULTIVIEW REPRESENTATION LEARNING
Ali Jahanian, Xavier Puig, Yonglong Tian, Phillip Isola Massachusetts Institute of Technology Cambridge, MA 02139, USA (jahanian, xpuig, yong ong, chillippl phillipijemii eemptiimmmm...mmmmmmim (emem..e....... edu
ABSTRACT
Generative models now capable of producing highly realistic images that look nearly indistinguishable from the data on which they are trained. This raises the question: if we have good enough generative models, do still need datasets? We investigate this question in the setting of learning general-purpose visual rep- resentations from a black-box generative model rather than directly from data. Given an off-the-shelf image generator without any access to its training data, we train representations from the samples output by this generator. We compare sev- eral representation learning methods that can be applied to this setting, using the latent space of the generator to generate multiple "views" of the same semantic content. We show that for contrastive methods, this multiview data can naturally used to identify positive pairs (nearby in latent space) and negative pairs (far apart in latent space). We find that the resulting representations rival or even out- perform those learned directly from real data, but that good performance requires care in the sampling strategy applied and the training method. Generative models can be viewed as a compressed and organized copy of a dataset, and we envision future where more and more "model zoos" proliferate while datasets become increasingly unwieldy, missing, or private. This paper suggests several techniques for dealing with visual representation learning in such future. Code is available on our project page https: / //aii-ai-e/ design.nnnnnnnn github. io /ennRep/.
Learring Pou POOIAN
Generative Model
Data
H
Fix-te
F:x Embedding function
Learning
Embedding function
N.M
G(T.(K)
Figure I: Visual representation learning typically consists of training image embedding function, F x -> e, given a dataset of real images {x.) (left panel). our work (right panel), we study how to learn representations given instead a black-box generative model G. Generative models allow us to sample continuous streams of synthetic data. By applying transformations Ta on the latent vectors z of the model, we create multiple data "views" that can serve effective training data for representation learners.
INTRODUCTION
The last few years have seen great progress the diversity and quality of generative models. For almost every popular data domain there is now generative model that produces realistic samples from that domain, be it images (Radford al., 2015; Brock et al., 2019; Ramesh al., 2021), music (Dhariwal what we used to do with real data, can we now do instead with synthetic data, sampled from generative model?
al.. 2020), text (Brown et al., 2020). This raises intriguing possibility:
If so, there would be immediate advantages. Models are highly compressed compared to the datasets they represent and therefore easier share and store. Synthetic data also circumvents some of the concerns around privacy and usage rights that limit the distribution of real datasets (Tucker et al., 2020; DuMont Schütte al., 2021), and models can be edited censor sensitive attributes (Liao

---

Published as a conference paper at ICLR 2022
et al., 2019), remove biases that exist in real datasets (Tan et al., 2020; Ramaswamy et al., 2020), or steer toward other task-specific needs (Jahanian et al., 2020; Goetschalckx et al., 2019; Shen et al., 2020). Perhaps because of these advantages, it is becoming increasingly common for pre- pre-trained generative models to be shared online without their original training data being made easily acces- sible. This approach has been taken by individuals who may not have the resources or intellectual property rights to release the original data' and in the case of large-scale models such as GPT- 3 (Brown al., 2020), where the training data has been kept private but model samples are available through an API.
Our work therefore targets problem setting that has received little prior attention: given access only to a trained generative model, and no access to the dataset that trained it, can we learn effective visual representations?
To this end, we provide an exploratory study of representation learning in the setting of synthetic data sampled from pre-trained generative models: we analyze which representation learning algo- rithms are applicable, how well they work, and how they can be modified to make use of the special structure provided by deep generative networks.
Figure. lays out the framework we study: we compare learning visual embedding functions F from real data x ~ {x, vs. from generated data x G controlled via latent transformations. We study generation and representation learning both with and without class labels, and test repre- sentation learners based on several objectives. We evaluate representations via transfer performance
on held out datasets and tasks.
For representation learners, we focus primarily on contrastive methods that learn to associate mul- tiple "views" of the same scene. These views may be co-occurring sensory signals such as imagery and sounds (e.g., (de Sa, 1994)), or may be different augmented or transformed versions of the same image (e.g.. (Becker & Hinton, 1992; Chen et al., 2020c)). Interestingly, generative models can also create multiple views of an image: by steering in their latent spaces they can achieve camera and color transformations (Jahanian al., 2020) and more (Härkönen et al., 2020; Wu al., 2020). Figure 2 diagrams the currently popular setting, where views are generated as pixel-space transfor- mations, versus the setting we focus on, where views are generated via latent-space transformations.
(n) SimCLR views
(b) Latent views
Objective: these are two views the same thing
Objective: these two the same thing
2
Figure 2: Different ways of creating multiple views of the same "content". (a) SimCLR (Chen et al., 2020c) creates views by transforming an input image with standard pixel-space (2') data augmentations (example images taken from (Chen al., 2020c)). (b) With generative model, we instead create views by sampling nearby points in latent space Z, exploiting the fact that nearby points in latent space tend generate imagery of the same semantic object. Note that these examples are illustrative, the actual transformations that achieve the best results are shown in Fig. 7.
We study the properties of these latent views contrastive learning and ask under what conditions they can lead to enhanced learners. We then compare these learners with non-contrastive encoders, also trained on samples from generative models. Further, we study the long-standing promise of generate models in being capable of generating more samples than the number of datapoints that trained them. Knowing current models still have finite diversity, we ask: how many samples are necessary to get good performance
task? Our main findings are:
1. Contrastive learning algorithms can be naturally extended to learning from generative samples, where different "views" of the data are created via transformations in the model's 's latent space.
'F.g, see https: : / / igitti thub. com/ just npi nkney/ http://wwwwoeeeeeeeeeeeeeeeeeeee indd stllstyyees egan, host- ing many pretrained models, without datasets.
2

---

Published as a conference paper at ICLR 2022
2. These latent-space transformations can be combined with standard data transformations ("aug- mentations") in pixel-space to achieve better performance than either method alone. 3. To generate positive training pairs in latent space, simple Gaussian transformations work well as more complicated latent-space transformations, with the optimal standard deviation being not Livers-U-sshapr curve like that observed for contrastive learning from real data (Tian et al., 2020).
too large and not too small, obeying
Generative models can potentially produce an unbounded number of samples train on; we find that performance improves by training on more samples, but uub-logarithmiclly.
Learning representations from generative models can outperform learning directly from the data that trained those models, provided that the generative model is of sufficiently quality (we observe this result on StyleGAN2 (Karras et al., 2020) fit to cars but not on BigBiGAN (Donahue & Simonyan, 2019) or BigGAN (Brock et al., 2019) fit to ImageNet (Deng et al., 2009)).
2 RELATED WORK
Learning from Synthetic Data. Using synthetic data has been prominent method for learning in different domains of science and engineering, with different goals including and alternative sample generation (Sakshaug & Raghunathan, 2010; Tucker et al., 2020; Nußberger et al., 2020; Khan et al., 2019; Dan et al., 2020). In computer vision, synthetic data has been exten- sively used source for training models, for example in semantic segmentation (Chen et al., 2019; Ros et al., 2016), human pose estimation (Varol et al., 2017; Ionescu et al., 2013; Shakhnarovich al., 2003), optical flow (Mayer et al., 2016; Fischer et al., 2015), self-supervised multitask learning (Ren & Lee, 2018). In most prior work, the synthetic data comes from a traditional sim- ulation pipeline, c.g., via rendering 3D world with a graphies engine, or recently noise genera- tor (Baradad et al., 2021). We instead study the setting where the synthetic data is sampled from a deep generative model.
Recent works have used generative networks, such as GANs (Goodfellow et al., 2014), to improve images generated by graphics engines, closing the domain gap with real images (Shrivastava al., 2017; Hoffman et al., 2018). Ravuri & Vinyals (2019) show that even though there is a gap between GAN-generated images and real ones, mixing the two sources can lead to improvements over models trained purely on real data. GAN-generated images are also used to learn inverse graphics. Zhang et al. (2021a) use StyleGAN to generate multiple views of a given object, allowing to extract 3D knowledge that can be mapped back into the latent space.
Highly related to our paper is the work Besnier et al. (2020), which uses GAN MAN-generated images to train classifier. While they focus using a class-conditional generator to train classifier of those same classes, our work targets the more general setting of visual representation learning, and mainly focus the "unsupervised" setting where unconditional generative model produces unlabeled samples to learn from.
Recent works (Yang et al., 2021; Jeong & Shin, 2021) have studied combining contrastive and adversarial losses in GAN training, showing benefits in both image synthesis and representation learning. In contrast our work, these approaches require access to real data during training.
Recent work has also explored using StyleGAN (Karras et al., 2019) to generate training data for semantic segmentation (Zhang al., 2021b; Tritrong al., 2021). These approaches differ from ours in that they use intermediate layers of the generator network as their representation of images, whereas our method does not require internal access the generator; instead we simply treat the generator as black-box source of training data for a downstream representation learner.
GAN- based data augmentation has also been used, in conjunction with real data, to improve data efficiency for GAN training (Yang et al., 2021) as well as robustness at training time (Mao et al., 2020) and test-time (Chai et al., 2021). In contrast to these methods, we explore representation learning without access to the real data at all, relying only on samples from black-box generative model to train our systems.
Contrastive Representation Learning. Contrastive learning methods (van den Oord et al., 2019; Wu et al., 2018; Hénaff et al., 2019; Tian et al., 2019; He et al., 2020; Chen et al., 2020c) have greatly advanced the state of the art of self-super representation learning. The idea of con- trastive learning is to contrast positive pairs with negative pairs (Hadsell et al., 2006). Such pairs can be easily constructed with various data formats, and examples include different augmentations
3

---

Published as a conference paper at ICLR 2022
or transformations of images (Chen et al., 2020c), cross-modality alignment (Tian et al., 2019; Mor- gado et al., 2020; Patrick et al., 2020), and graph structured data (Hassani & Khasahmadi, 2020). One key ingredient for the success of contrastive learning is well-chosen data transformations (Chen et al., 2020c; Tian et al., 2020; Xiao et al., 2020). While all these approaches conduct data trans- formations, "augmentations", in the raw pixel space, in this paper we explore the possibility of transforming training points in the latent space of GAN.
Generative Representation Learning. Generative models learn representations by modeling the data distribution pxx). In VAEs (Kingma & Welling, 2013), each data point is encoded into a latent distribution, from which codes are sampled to reconstruct the input by maximizing the data likeli- hood. GANs (Goodfellow et al., 2014) model data generation through a minimax game, after which the discriminator can serve as a good representation extractor (Radford et al., 2015). In ALI (Du- moulin et al., 2016), BiGAN (Donahue et al., 2017), and BigBiGAN (Donahue & Simonyan, 2019) both image encoding and latent decoding are modeled simultaneously, and the encoder turns out to be a representation learner. Similar to the trend in natural language processing (Devlin et al., 2018; Radford et al.), autoregressive models (Van den Oord et al., 2016; Chen et al., 2020b) have been adopted to learn representations from raw pixels. These prior works show ways jointly train representation alongside a generative model, using a training set of real data for learning. Our ex- ploration qualitatively differs in that we assume we given black-box generative model, and no real training data, and the goal is to learn an effective representation by sampling from the model.
3 METHOD
Standard representation learning algorithms take a dataset {x, i=1 as input, and produce an encoder vector representation of x. Our method, in contrast, takes a generative model G' as input, in order to produce F' as output. We restrict our attention to implicit generative models (IGMs) (Mohamed & Lakshminarayanan, 2016), which map from latent variables z to sampled images i.e. G : z + x. Many currently popular generative models have this form, including GANs (Goodfellow et al., 2014), VAEs (Kingma & Welling, 2013), and normalizing flows (Rezende & Mohamed, 2015). We also consider cass-conditional variants, which we denote as Gy z. y + x, where y is a discrete class label. In our experiments, we only investigate GANs, but note that the method is general to any IGM with latent variables.
F : x + as output, where
To learn F. from either real data or model samples, we can pick any of a large variety of repre- sentation learners: autoencoders (Ballard, 1987), rotation prediction (Gidaris et al., 2018), coloriza- tion (Zhang et al., 2016), etc. We focus on contrastive methods due to their strong performance and natural extension to using latent transformations to define positive pairs. These methods are illus- trated in Fig. 3 in the first and the second rows. We also examine the effectiveness of non-contrastive methods, namely an inverter (similar to the encoder in the autoencoders) as illustrated in the third row of Fig. 3. For conditional IGMs, which generate data from class labels, we also consider using a labsl-classifier as the representation learner.
In the following sections, we first define the contrastive framework with different sampling strategies for creating views via pixel transformations and latent transformations. We then define the non- contrastive frameworks.
3.1 CONTRASTIVE LEARNING FRAMEWORK
Modern contrastive methods learn embeddings that are invariant to certain nuisance transformations of the data, or different "viewing" conditions. Two "views" of the same scene are pulled together in embedding space while views of different scenes are pushed apart. A common objective is the InfoNCE (van den Oord al., 2019). We use the following variant of this objective:
e F(x.)) F(xx)
Lucx -E
log e (F((x))T Ex (((x)) (
Here Xx is anchor image, (Xa, Xp) is a positive pair of images, i.e. pair want to bring together in embedding space. (x,, is negative pair of images, which we want to push apart.
Different contrastive learning algorithms differ in how the positive and negative pairs are defined. Commonly, x, and x, are two different transformations (i.e. data augmentations in pixel space) of
4

---

Published as a conference paper at ICLR 2022
Contrastive Augmentation
(SimCLR, etc)
Data space
Representation space
Given:
Dalasel
Learn: F
Eneoder
Contrastive Latent Views + Data Augmentation
Latent space
Data space
Representation space
Given:
Generator
F2
E
e
F
Generator
Encader
Inverter
Latent space
Data space
Latent space
Given:
Generator
Learn: F
Generator
Eneoder
Figure 3: Three different methods for learning representations. The first illustrates standard contrastive learning framework (e.g., SimCLR (Chen al., 2020c)) which positive pairs sampled as transformations training images x. The second and third rows show the new setting we consider: we are given generator, rather than datasct, and can use the latent space (input) of the generator to control the production of effective training data. Tx refers to transformations applied in pixel-space and T2 denotes transformations in latent- space. The second row illustrates contrastive learning approach in this setting and the third shows approach that simply inverts the generative model. For contrastive learning, negatives are omitted clarity.
same underlying image, and x is transformation of a different randomly selected image (Chen et al., 2020c; He et al., 2020; Gidaris et al., 201 8).
Our analysis focuses both on how the datapoints are sampled in the first place, and on how they are transformed to create negative and positive pairs:
What happens when x is fake data sampled from an IGM rather than real data from a dataset?
2. How can we use transformations in pixel space and in latent space to define positives/negatives?
Different answers to these two questions yield the specific methods we compare, as described the next sections.
3.2
SAMPLING POSITIVE AND NEGATIVE PAIRS
We first describe several schemes for sampling from both real and generated datapoints, with trans- formations applied in either latent space or pixel space.
3.2.1
CONTRASTIVE PIXEL TRANSFORMATIONS (L.E. SIMCLR)
If are given dataset, and of pixel-space transformations, Tx, that we wish to be invariant to, a standard approach is to use the following sampling strategy:
Sa & D Na Xp ~ (xta)
4 sample anchor and negatives
(2)
generate positive pair
(3)
= ~ (x))
4 generate negatives
(4)
where D refers to drawing random image uniformly from the dataset D )x This setting depicted in the top row Fig. 3. We use SimCLR (Chen et al., 2020c) as an instantiation this approach.
5

---

Published as a conference paper at ICLR 2022
3.2.2 CONTRASTIVE LATENT VIEWS + PIXEL TRANSFORMATIONS
If are given an unconditional IGM G, may also define set of latent transformations, T2 that we wish our representation to be invariant to. We can use this method with or without pixel-space transformations Tx applied as well. With both T₂ and Tx applied, we have the following fake data generating process:
Za 2n ~ Pz
sample latents
(5)
Xa (G((Z))
generate anchor sample
(6)
xp (x(((T(()))
generate positive sample
(7)
~ (x(G(T2(2..))
generate negatives
(8)
In practice we set ps to the truncated normal distribution, A (1, a, t), with truncation / (Brock et al., 2019). This scenario is depicted in the middle row of Fig. 3. Refer to App. C.I.1 for the class-conditional formulation.
3.3 CREATING VIEWS WITH Tx AND T₂
We refer to transformations as "views" of the data (either real fake). Figure shows different views generated according to all the following methods. For pixel-space transformations, 7x. many options have been previously proposed, and in our experiments we choose the transformations from SimCLR (Chen et al., 2020c) as they are currently standard and effective. Our framework could be updated with better transformations they are developed in future work.
Our work is the first, to our knowledge, to explore latent-space transformations, T2. for contrastive representation learning. Therefore, we focus our analysis on studying the properties of Te and explore the following options:
3.3.1 GAUSSIAN LATENT VIEWS
Many IGMs have the remarkable property that nearby points in latent space map to semantically related generated images (Jahanian et al., 2020). This suggests that nearby latent vectors (refer to Sec. 4.1.2 for an empirical proof). simple way to do so is to define the latent transformation just be small offset applied the sampled vector. We truncated Gaussian offsets W WGavas as a simple instantiation of this idea:
define positive views as
Wesuss "A'
(9)
Tz(z) = 2 + WGauss
(10)
where (1, o, 1) is the truncated Normal distribution with truncation / (Brock al., 2019).
(a) Original
Gaussian Views
(c) Steered Views
(d) SimCCR Views
(c) Grazzian SinCLLR Views
(1 Steer a SinCILR Views
Figure 4: Examples of different transformation methods for unconditional IGM data. Top row shows samples of BigBiGAN trained Im0geNet1000, and the bottom row shows samples from the StyleGAN2 LSUN CAR.
3.3.2 STEERED LATENT VIEWS
Can we find latent transformations that are more directly related to semantics? We leverage the recent method of (Jahanian et al., 2020), which finds latent vectors Wateer that achieve target effects in image space, such as shifting an image up and down or changing it brightness. That is. (z) transformation T according to the objective Wstees = arg min Ez [[[z+a)) (a-tww--re T(G((z) (G(()) I]] For example, T could be "shift left", parameterized by the strength of the transformation a, e.g. the number of pixels to shift. In App. B we describe the exact steering transformations in more detail.
+ Wsteer. where W Wsteer is a latent vector trained to match target pixel-space

---

Published as a conference paper at ICLR 2022
3.4 NON-CONTRASTIVE ALTERNATIVES
We can also learn image representations without contrastive loss. Here we describe one possible approach, an inverter. Appendix C.2.3 describes another non-contrastive approach, a classifier, that only applies in the class-conditional IGM setting.
Inverter The z vector that generates an image can itself be considered a vector representation of that image. This suggests that simply inverting the generator may result in useful embedding function. For a given IGM G, we learn an encoder F that inverts G by minimizing [[F())))) z l
EXPERIMENTS
To study the methods in Sec. 3, we analyze the behaviour and effectiveness of the representations learned from unconditional IGMs here and, in the interest of space, report the results on class- conditional IGMs in App. C. We experiment on two unconditional IGMs: the generator from Big- BiGAN (Donahue & Simonyan, 2019) and the StyleGAN2 L.UN CAR generator (Karras et al., 2020). Note, we do not use the encoder or training scheme from BigBiGAN, we simply use its learned generator G as an off-the-shelf unconditional ImageNet generator.
For ImageNet representation learning, we investigate two settings in our experiments: 1) training and evaluating representations on data at the scale of ImageNet1000 (Deng et al., 2009), and 2) lighter weight protocol where we train and evaluate representations only on data at the scale of ImageNet100 (Tian et al., 2019) (a 100-class subset of ImageNet1000), In the ImageNet1000 set- ting, the real data encoders are trained the ImageNet1000 dataset, the IGM encoders are trained on 1.3M anchor images (which roughly matches the size of ImageNet 1000). We use SGD with learning rate of 0.03, batch size of 256, and 20 epochs. In the ImagetNet100 setting, the real data encoders are trained on ImageNet1000, the unconditional IGM encoders are trained on 130K anchor images sampled unconditionally (note that the unconditional model implicitly is still sampling from all 1000 classes since the generative model itself was fit to ImageNet1 ing rate of 0.03, batch size of 256, for 200 epochs. In all settings, we use 128 x 128 images, leading to lower than state-of-the-arr performance on the real datasets, more details in App. D.
We use SGD with learn-
We evaluate the ImageNet learned representations by training linear classifier on top of the learned embeddings, for either ImageNet 1000 or ImageNet100. matching the setting in which the encoder was trained, and report Top-1 class accuracy. For both ImageNet1 100 and ImageNet100 settings, we use SGD with batch size of 256 over 60 epochs, and learning rates of 0.3 for real and 2 for IGMs, using a cosine decay schedule. We also evaluate on the Pascal VOC2007 dataset (Everingham et al., 2010) held out data setting, selecting blith et al., 2019) and report the mean average precision (AP) measures. See App. E for results on the object detection task.
via cross- cross-validatee validation, following (Korn-
For the StyleGAN2 LSUN CAR experiments, we use the pretrained model on 893K images from the LSUN CAR dataset. See App. A for obtaining the real data. For the contrastive model trained on this data, we use SGD with learning rate of 0.03, batch size of 256, and 20 epochs. We evaluate the learned representations by training a linear classifier on top of the learned embeddings, for either ImageNet 1000 or Stanford Car classification task (Krause et al., 2013) (196 car models with roughly 8K train and 8K val), and report Top-1 class accuracy. For both experiments, we use SGD with batch size of 256 over 100 epochs, and learning rates of 2, using cosine decay schedule.
4.1 CONTRASTIVE LEARNING METHODS
To generate datasets containing multiple views produced by latent transformations, we first sample 1.3M anchor view images in the ImageNet1000 setting (described above). All the anchors are generated with z (0, 1, 2). We then generate one neighbor view for each anchor view by following different strategies for view creation, Gaussian and steer, described in Sec. 3.3. For the Gaussian view, we tune the standard deviation of Weauss to be 0.2 for our unconditional IGMs (i.e. Weauss ~ (O 0.2, 2)), and we use this setting in all of our experiments. For steering, we learn the latent walk W Wateer as the summation of individual walks with randomly sampled steps for different camera transformations, i.e. horizontal and vertical shifts, zoom, 2D and 3D rotations as well as color transformations. The details for training Wazeer follows (Jahanian et al., 2020), and we report them in App. B. We use similar view creation strategies for Stylegan2 LSUN CAR, i.e., each anchor is generated with z ~ A/ 1, 0.9), and per anchor image, a neighbor is generated with the same steering or Gaussian strategy with std 0.25 apart from its anchor in the case of the Gaussian
7

---

Published as a conference paper at ICLR 2022
views. The resulting images of our latent view creations shown in Fig. 4(b,c). When needed, we further combine the latent views with pixel transformations, as shown in Fig. 4(e.f). See App. A for details how we handle cropping. In the following sections, we learn visual representations of the generated images by training a ResNet-50 via SimCLR (Chen et al., 2020c) for the datasets generated from unconditional IGMs.
4.1.1 EFFECT OF THE LATENT AND IMAGE TRANSFORMATIONS
First, we experiment with the effect of using pixel transformations on images sampled by the IGMs, Next, we enable both latent and pixel transformations
without latent transformations, i.e. T₂ together, where the latent transformation is either Gaussian or steer, i.e. z Waauss or + Wateez respectively. We report in Table and Table the transfer results on unconditional IGMs, for the described transformations. From this table we observe similar trends across the unconditional IGMs: while pixel transformations Tx lead to strong representations, these are significantly improved by adding latent transformations T5. Comparing different types of latent transformations, we find that
Training Method
Transfer Task
ImageNer1000
VOCU7 Classification
Data distribution
T₂
Tx
Objective
Top-1 Accuracy
AP
Real
SimCLR Augs. RimCLR Augs.
Contrastive
43.90
0.67
Generated
Contrastive
35.69
0.57
Generated
2- + Woases
Contrastive
28 88
0.52
Generated
2
Weasss
SimCLR Augs.
Contrastive
42.58
0.64
Generated
Contrastive
26.52
0.49
Generated
SimCLR Augs.
Contrustive
41.78
0.63
Generated
Inverter
26.43
0.49
Table 1: Results on unconditional IGMs where real data is sampled from ImageNet1000) and is distributed as (D), and generated data is sampled from BigBiGAN and distributed as (z)((())))) T2
indicates that no transformation applied. For the Contrastive objective, positive and negative views are defined as described in Sec. 3.2.1 (without using class labels).
Training Method
Transfer Task (Top-1 Accuracy)
Data distribution
T2
Tx
Objective
ImageNet1000
Stanford Car Classification
Real
SimCLR Augs.
Contrastive
30.30
39.68
Generated
SimCLK Augs.
Contrastive
28.33
40.39
Generated
Wacusa
Contrastive
24.50
24.50
Generated
+ Wasuss
SimCLR Augs.
Contrastive
30.06
49.79
Table 2: Results on unconditional IGMs where generated data is sampled from StyleGAN2 LSUN CAR and distributed T. (z)T,)))),, Tr indicates that transformation is applied. For the Contrastive objective, positive and negative views are defined as described Sec. 3.2.1 (without using class labels).
Gaussian transformations work best. This is a somewhat surprising result. Despite designing views through steering methods, transforming randomly in all latent directions provides similar or better performance.
Another important observation that training on StyleGAN2 fit LSUN CAR outperforms training directly on LSUN CAR (Table 2). This result demonstrates that generative data can, in some cases, be more useful for representation learning than training directly on real photographs. The success here may be due to the fact that the StyleGAN2 samples are nearly identical to real photos in their realism and diversity (FID = 2.32). Therefore we might expect StyleGAN2 samples, trained only with Tx, to more less match the performance of training what we find (Table 2; compare first two rows). It is then unsurprising that adding Tz, which can only be done for generative data, boost performance further.
real data with Tx, and indeed this is
4.1.2 LIMITS OF THE LATENT TRANSFORMATION
Prior work on contrastive learning has found that there is an optimal point in how much two views share information (Tian al., 2020). Similarly, in our Gaussian latent view creation, we may expect there to be a sweet spot for how far we can go when sampling neighbor view relative to an anchor view. To study that sweet spot for the BigBiGAN generator, we vary the standard deviation of the Gaussian transformation. For this experiment we use the ImageNet setting. Fig. 5 illustrates the results: linear transfer performance exhibits an inverse- U-shaped curve with respect Gaussian standard deviation, peaking at std = 0.2.
4.2 NON-CONTRASTIVE METHODS
Inverter We experiment with an inverter, as illustrated in Fig. 3. We learn an image encoder by minimizing the mean squared error between the original and predicted latent code Similar the
8

---

Published as a conference paper at ICLR 2022
contrastive setting, we train a ResNet-50 encoder and replace the last layer with fully connected layer to predict the latent vector z of a 128 x 128 image. We train the encoder with a dataset of 1.3M images and their latent vectors, Lec pairs of (G(z),z) z). Note we don't 't apply image transfor- mations on ((2) given that directions in z-Space encode hasic transformations like shifting and color change (Jahanian et al., 2020) so z is not invariant to these transformations. The results of using the unconditional IGM representations in Table suggest this non-contrastive approach performs poorly in comparison with the contrastive methods.
Comparison to BigBiGAN's encoder We finally compare the pretrained BigBiGAN en- coder (Donahue & Simonyan, 2019) (a non-contrastive approach) with our proposed methods. Note the BigBiGAN encoder is jointly trained with the generator, thus requiring access to the original real dataset during training. This is not the setting we are targeting but we provide this compari- son nonetheless to see where our methods stand compared to prior work that involved generative model in learning representation. To match the setup with our experiments, we take the BigBiGAN ResNet50 encoder after average pooling (without the head) and use it in our linear classification of ImageNet100 test. Further, note that this encoder trained on ImageNet1000 and accepts as input images of size 256 x 256, whereas for this experiment we compare against our encoders trained on the ImageNet100 training sets and use images of size 128 x 128. The top-1 I accuracy from the Big- BiGAN encoder is 70% which is better than inverter but still worse than all the contrastive methods trained and tested on ImageNet100 (see the numbers in in Table 4 in App. E).
4.3 EFFECT OF THE NUMBER OF SAMPLES
In the synthetic data setting, we can create infinite data (albeit with finite diversity). A natural question arises: how many samples do we need for good coverage and good visual representations? To answer this question, we run experiments in the ImageNet 10 setting and evaluate the linear classification performance for Gaussian views combined with pixel transformations, as vary the number of unique samples. We show the results in Fig. 6. Note that we train all the encoders for the same number of iterations, equivalent to 200 epochs on ImageNet100 (matching the experiments shown in table 4 in App. E). This means, as the number of images (x-axis) increases, the number of times the model revisits a seen image decreases. As Fig. 6 illustrates, the performance increases with more samples, but sub-logarithmically These findings are consistent with recent work that found a small gap in generalization performance between online learning (infinite data) and a sufficiently large offline regime (Nakkiran et al., 2020).
Imsgennctio Accurace Unconditioal Data
Imagmetton Arerayy Unronditional 1GM Data
60
44-
Tre-
30-
Cecsuint CessiinUnseeeiiee Uncediteee KIME 17 Cuesiien Uwwediteed KIM only
Tragelioun
Seatroyte Clansiin Sowdbard Diciation
Skmbeer of towriigh
Figure 5: Effect of the distance between latent views on contrastive learning. We vary the stan- dard deviation of Gaussian T2 = z + Woouss and measure linear transfer to ImageNet100.
Figure 6: Effect of the number of samples for training the representation learner, evaluated us- ing linear transfer to ImageNet100. "Gaussian" refers to the Gaussian views (T2- + Wawnss).
5 CONCLUSION
We investigated how to learn visual representations from IGMs (implicit generative models), when they are given as a black-box and without any access to their training data. IGMs make it possible to generate multiple views of similar image content by sampling nearby points in the latent space of the IGM. These views can be used for contrastive learning or as input to other representation learning algorithms that associate multiple views of the same scene. Our results demonstrate that leveraging latent-space views can improve performance beyond learning from pixel-space transfor- mations alone. Representations learned from large quantities of generative data rival and sometimes outperform the transfer performance of representations learned from the real datasets. As gener- ative models improve, we expect that training vision systems on generative data may become an increasingly important tool in our toolkit.
9

---

Published as a conference paper at ICLR 2022
6 ETHICS STATEMENT
This work studies how to learn useful image representations given data generated from IGMs as opposed to real data. This framework can provide several societal advantages currently faced real datasets, including protecting the privacy and usage rights of real images (Tucker et al., 2020; Du- Mont Schüttee et al., 2021), removing sensitive attributes (Liao et al., 2019), or reducing biases (Tan al., 2020; Ramaswamy et al., 2020) and therefore learning fairer representations.
At the same time, learning representations from IGMs brings several challenges and risks, that need be addressed. Generative models can in some cases reveal the data they are trained on Hayes al. (2018); Chen et al. (2020a), posing risks for privacy preservation. They can also amplify biases in datasets (Jain et al., 2020; Menon al., 2020), which can lead negative societal impacts if they are not audited properly (Raji et al., 2020a;b; Mitchell et al., 2019), or they are used in the wrong contexts Buolamwini & Gebru (2018). Making good use of the learned representations will require addressing the above issues by studying mitigation strategies, as well as methods to audit implicit generative models.
ACKNOWLEDGMENTS
Author A.J. thanks Kamal Youcef-Toumi, Boris Katz, and Antonio Torralba for their support. We thank Antonio Torralba, Janne Hellsten, David Bau, and Tongzhou Wang for helpful discussions.
This research was supported in part by IBM through the MIT-IBM Watson AI Lab. The research was also partly sponsored by the United States Air Foree Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number 1A-750-19-2-110 1000. The views and conclusions contained in this document those of the authors and should not be interpreted as representing the official policies, either expressed or im- plied, of the United States Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
REFERENCES
Dana H Ballard. Modular learning in neural networks. In AAAI, 1987.
Manel Baradad, Jonas Wulff, Tongzhou Wang, Phillip Isola, and Antonio Torralba. Learning to see by looking noise. Advances in Neural Information Processing Systems, 2021.
Suzanna Becker and Geoffrey E. Hinton. Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 1992.
Victor Besnier, Himalaya Jain, Andrei Bursue, Matthieu Cord, and Patrick Pérez. This dataset does not exist: training models from generated images. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, IEEE, 2020.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019.
Tom Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14 165, 2020.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commer- cial gender classification. Conference on fairness, accountability and transparency, pp. 77-91. PMLR, 2018
Lucy Chai. Jun- Yan Zhu, Eli Shechtman, Phillip Isola, and Richard Zhang. Ensembling with deep generative views. arXiv preprint arXiv:2104.14553 2021.
Dingfan Chen, Ning Yu. Yang Zhang, and Mario Fritz. Gan-leaks: A taxonomy of membership inference attacks against generative models. Proceedings of the 2020 ACM SIGSAC Conference Computer and Communications Security, 2020a.
10

---

Published as a conference paper at ICLR 2022
Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, and Ilya Sutskever. Generative pretraining from pixels. ICML, 2020b.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffey Hinton. A simple framework for contrastive learning of visual representations, 2020c.
Yuhua Chen, Wen Li, Xiaoran Chen, and Luc Van Gool. Learning semantic segmentation from synthetic data: A geometrically guided input-output adaptation approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1841-1850, 2019.
Yabo Dan, Yong Zhao, Xiang Li, Shaobo Li, Ming Hu, and Jianjun Hu. Generative adversarial networks (gan) based efficient sampling of chemical composition space for inverse design of inorganic materials. npj Computational Materials, 6(1):1-7, 2020.
Virginia R de Sa. Learning classification with unlabeled data. In Advances neural information processing systems, pp. 112-119. Citeseer, 1994.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale
hierarchical image database. In CVPR, 2009.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1rXiv:/810.04805 2018.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and llya Sutskever. Jukebox: generative model for music. arXiv preprint arXiv:2005.0034, 2020.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In NeurIPS, 2019.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. ICLR, 2017.
August DuMont Schütte, Jürgen Hetzel, Sergios Gatidis, Tobias Hepp, Benedikt Dietz, Stefan Bauer, and Patrick Schwab. Overcoming barriers to data sharing with medical image generation: a comprehensive evaluation. npj Digital Medicine, 4(1):1-14, 2021.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Ar- jovsky, and Aaron Courville. Adversarially learned inference. arXiv preprint arXiv: 160iv:1666.1 (6066000044 2016.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2); 303-338, 2010.
Philipp Fischer, Alexey Dosovitskiy, Eddy Ilg, Philip Häusser, Caner Hazıraa, Vladimir Golkov, Patrick Van der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks, arXiv preprint orXiv::504 (0l.:1504.0852,, 2015.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 1803.0778, 2018.
Lore Goetschalekx, Alex Andonian, Aude Oliva, and Phillip Isola. Ganalyze: Toward visual defi- nitions of cognitive image properties. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5744-5753, 2019.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR. IEEE, 2006.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable GAN controls. NeurlPS, 2020.
Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive milti-view representation learning on graphs. arXiv preprint arXiv:2006.05582 2020.

---

Published as a conference paper at ICLR 2022
Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro, Logan: Membership infer- ence attacks against generative models, 2018.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Olivier J Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv:1905.09272 2019.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun- Jun Zhu, Phillip Isola, Kate Saenko, Alexci A Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, 2018.
Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence, (6(7):1325-1332 2013.
Ali Jahanian, Lucy Chai, and Phillip Isola. On the "steerability" of generative adversarial networks, 2020.
Niharika Jain, Alberto Olmo, Sailik Sengupta, Lydia Manikonda, and Subbarao Kambhampati. Im- perfect imaganation: Implications of gans exacerbating biases on facial data augmentation and snapchat selfie lenses. arXiv preprint arXiv:2001. 009528, 2020.
Jongheon Jeong and Jinwoo Shin. Training GANs with stronger augmentations via contrastive discriminator. In International Conference Learning Representations, 2021. URL https:
/ //openreview.... fett://orrmmeeeeoottttAora
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of StyleGAN. In Proc. CVPR, 2020.
Shahroz Khan, Erkan Gunpinar, Masaki Moriguchi, and Hiromasa Suzuki. Evolving a psycho- physical distance metric for generative design exploration of diverse shapes. Journal of Mechan- ical Design, (41( 2019.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning, 2020.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint
1 2013.
Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2661-2671, 2019.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554-561, 2013.
Jiachun Liao, Chong Huang, Peter Kairouz, and Lalitha Sankar. Learning generative adversarial representations (gap) under faimess and censoring constraints. arXiv preprint (rXiv:1910.00411 2019.
Chengzhi Mao, Amogh Gupta, Augustine Cha, Hao Wang, Junfeng Yang, and Carl Vondrick. Gen- erative interventions for causal learning. arXiv e-prints, pp. arXiv-2012, 2020.
Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4040-4048, 2016.
12

---

Published as a conference paper at ICLR 2022
Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. Pulse: Self- supervised photo upsampling via latent space exploration of generative models. In Proceedings of the ieee/cvf conference computer vision and pattern recognition, pp. 2437-2445, 2020.
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Luey Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pp. 220-229, 2019.
Shakir Mohamed and Balaji Lakshminarayanan. Learning in implicit generative models. arXiv
preprint arXiv:1610.03483, 2016.
Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. arXiv:2004.12943, 2020.
Preetum Nakkiran, Behnam Neyshabur, and Hanie Sedghi. The deep bootstrap: Good online learn- are good offline generalizers. arXiv preprint arXiv:2010.08122 2020.
Jens Nuβberger, Frederic Boesel, Stefan Maria Lenz, Harald Binder, and Moritz Hess. Synthetic ob- servations from deep generative models and binary omics data with limited sample size. bioRxiv, 2020.
Mandela Patrick, Yuki M Asano, Ruth Fong, João Henriques, Geoffrey Zweig, and Andrea Vedaldi. Multi-modal self-supervision from generalized data transformations. 042292 2020.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 2015.
Inioluwa Deborah Raji, Timnit Gebru, Margaret Mitchell, Joy Buolamwini, Joonseok Lee, and Emily Denton. Saving face: Investigating the ethical concerns of facial recognition auditing. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pp. 145-151, 2020a.
Inioluwa Deborah Raji, Andrew Smart, Rebecca N White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. Closing the ai accountability gap: Defining end-to-end framework for internal algorithmic auditing. In Proceedings of the 2020 conference on fairness, accountability, and transparency, pp. 33-44, 2020b.
Vikram V Ramaswamy, Sunnis SY Kim, and Olga Russakovsky. Fair attribute classification through latent space de-biasing. arXiv preprint arXiv:2012.01469, 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. sero-shot text-to-image generation. arXiv preprint arXiv:2102.12092, 2021.
Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models. In NeurIPS, 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91.9,19,,0000 2015.
Zhongzheng Ren and Yong Jae Lee. Cross-domain self-supervised multi-task feature learning us- ing synthetic imagery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 762-771, 2018.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv: /505 055700 2015.
Genman Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3234- 3243, 2016.
13

---

Published as a conference paper at ICLR 2022
Joseph W Sakshaug and Trivellore E Raghunathan. Synthetic data for small estimation. In International Conference on Privacy in Statistical Databases, pp. 162-173. Springer, 2010.
Gregory Shakhnarovich, Paul Viola, and Trevor Darrell. Fast pose estimation with parameter- sensitive hashing. In null. pp. 750. IEEE, 2003.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9243-9252, 2020.
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2107-2116, 2017.
Shuhan Tan, Yujun Shen, and Bolei Zhou. Improving the fairness of deep generative models without retraining, 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint 19XX::1906005849 2019.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning, arXiv preprint arXiv:2005. 00020005.02245 10243, 2020.
Nontawat Tritrong, Pitchaporn Rewatbowornwong, and Supasorn Suwajanakom. Repurposing gans for one-shot semantic part segmentation. arXiv preprint arXiv:2103.04379) 2021.
Allan Tucker, Zhenchen Wang. Ylenia Rotalinti, and Puja Myles. Generating high-fidelity synthetic patient data for assessing machine learning healthcare software. NPJ digital medicine, 3(1):1-13, 2020.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Con- ditional image generation with pixelenn decoders. In Advances neural information processing systems, pp.
2016.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, 2019.
Gul Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael Black, Ivan Laptev, and Cordelia Schmid. Learning from synthetic humans. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 109-117, 2017.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. CVPR, 2018.
Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Disentangled controls for stylegan image generation. CoRR, abs/201 2020. URL https: wrgp/iiooooraaaaeoooo abs/ 2011.122 12799.
Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. arXiv preprint arXiv:2008.05659, 2020.
Ceyuan Yang, Yujun Shen, Yinghao Xu, and Bolei Zhou. Data-efficient instance generation from instance discrimination. arXiv preprint arXiv: 20:: 04566. 2021.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint 1rXiv:1904.00962, 2019.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European confer- ence on computer vision, pp. 646-666. Springer, 2016.
Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba, and Sanja Fidler. Image gans meet differentiable rendering for inverse graphics and interpretable 3d neural rendering. In International Conference Learning Representations, 2021a.
14

---

Published as a conference paper at ICLR 2022
Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Tean-Francoii Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. arXiv preprint arXiv:2104.06490, 2021b.
15

---

Published as a conference paper at ICLR 2022
APPENDIX
A CROPPING DETAILS
For all methods that use only 2, we generate images the native resolution of the generator output (128 x 128 for BigBiGAN, 256 256 for BigGAN, and 512 x 512 for StyleGAN) then take a center crop to produce the 128 x 128 pixel inputs to our models. preliminary experiments we found that this strategy worked better than downsampling the generated images to 128 128. Note that in our figures for T₂ we visualize the generated images before this crop is applicd.
For StyleGAN LSUN CAR, the dataset consists of rectangular images padded with black borders. For all methods we remove the padding before training on these images. This results in non-square images. Methods that use Tx take random square crop (one of the SimCLR augmentations) and for methods without T2 we take a 128 x 128 pixel center square crop.
Furthermore, for StyleGAN LSUN CAR, compare against training on real data, we need these 893K images; as they are not publicly released, we recreate this dataset by using the LSUN CAR dataset sampling code from the StyleGAN2 repository".
B CREATING STEERING VIEWS
For creating steering views, we apply the walk Wateer in Sec. 3.3.2. Here we expand this vector as the summation of individual walks with randomly sampled steps for different camera transforma- tions, i.e. horizontal and vertical shifts, zoom, 2D and 3D rotations as well as color transformations. In fact, the final walk is:
Wteeer= Σ Raw'tteer
(11)
for E {shi ftx, shi fty, zoom, rot20, rot or),, each representing a pixel-space transfor- mation. That is:
w attee
arg min Eg, El.aa - ())
(12)
where Ti can be "shift left/right", parameterized by the strength of the transformation a, e.g. the number of pixels to shift.
Following (Jahanian et al., 2020), coefficient a determines what magnitude we want for each trans- formation. For example, for color, we add to each RGB channel of an image (and have three w vectors in the latent space). Similarly, for each E {shiftx, shi fty, rot rot rot3D), a deter- mines how many pixels we shift rotate the image. For zoom, scale we should apply to the image. For that reason, we use log(a) in the latent space but a in the pixel space (to define the edits).
to determine how much
In order to train W Waeear. we create a dataset of target images and optimize for the best w steer walks that in linear summation yield images close the target images. Here, each target image a composition of all the pixel-space transformations applied to the G(z) image with random magnitudes (a for each T.).
After learning the w Wateer walks together, we can randomly draw a vector and add them it to (with random steps az), and this results in transformed image. This transformed image will serve positive view for the image of the given z vector.
C CLASS-CONDITIONAL IGMs
C.1 METHOD
Given a class-conditional IGM, we can do the same formulations as in Sec. 3.2.2 except conditioned on class labels, i.e. we simply replace G(-) with G(: y). using y Dp Cat(M) for the anchor and positive sample, and an independent draw y = In ~ Cat(M) for the negative sample. Additionally,
httpps: :// thub. com. wwto////waas///y/aaa/aaae
httpa///ggggatbooooo/aaaaaaagag/ pytorch
16

---

Published as a conference paper at ICLR 2022
taking inspiration from SupCon, in each batch, one positive is set with the same the anchor (but different pixel space transformation) while the other positives in the batch are generated from independently drawn z Px-
C.1.1 SUPERVISED CONTRASTIVE PIXEL TRANSFORMATIONS (L.E. SUPCON)
Given a labeled dataset D (x;, MiJi-1 with M classes, can leverage the labels to define positives as images that share the same labels and negatives as randomly sampled images from other classes:
Up ~ (at(MM
< sample positive class
(13)
xa; Xp ~ Dop pyr 8th Diur
sample pairs
(14)
Xa ~ Tx (8aa) Xy ~ (xp)
generate positive
(15)
xxx Tx (x
generate negatives
(16)
where Cat is the categorical distribution and Dy, {ax Dly, yph, D/u, }p Following SupCon (Khosla et al., 2020), in each batch one of the positives is specially set as
{xz E Dly *
Xp ~ (8a)).
C.1.2 INDEPENDENT LATENT VIEWS
The simplest method is:
T2(-) Pz
(17)
That is, the transformation simply produces a new random draw from Px. In the unconditional setting, this can be considered a naive baseline where the two views share no information about the image. However, in the class- cluss-oonditionll case, this strategy is actually quite sensible, and in a sense optimal if the goal to extract class semantics (Tian et al., 2020): the two positive views are two different images that are independent except that they share the same class 1/p-
C.2 EXPERIMENTS
For the class-conditional IGM, we use BigGAN (Brock et al., 2019) trained on (ma (Deng et al., 2009) with the "deep-256" and truncation= 2.0. We use image size of 128 x 128 for all the settings (i.e. scaling down BigGAN images).
Similar to unconditional experiments, for ImageNet representation learning, we investigate two set- tings in our experiments: 1) training and evaluating representations data the scale of Im- ageNet1000 (Deng et al., 2009), and 2) a lighter weight protocol where we train and evaluate representations only data at the scale of ImageNer100 (Tian et al., 2019) (a 100 class subset of ImageNer1000). In the ImageNet 000 setting, the real data encoders are trained on the Ima- geNet1000 dataset, the class-conditional IGM encoders images conditioned on each of the 1000 ImageNet1000 classes, which roughly matches the size of 1000), We use SGD with learning rate of 0.03, batch size of 256, and 20 epochs. In the ImagetNet 100 setting, the real data encoders are trained on ImageNet100 the class-conditional IGM encoders are trained on 130K anchor images (1300 images conditioned on each of the 100 classes in ImageNer100). We use SGD with learning rate of 0.03, batch size of 256, for 200 epochs.
0000
trained on 1.3. anchor images (1300
We evaluate the ImageNet learned representations by training linear classifier on top of the learned embeddings, for either ImageNet1 1000 or ImageNet 00, matching the setting in which the encoder was trained, and report Top-1 class accuracy. For both ImageNet1000 and ImageNet100 settings, use SGD with batch size of 256 over 60 epochs, and learning rates of 0.3 for real and 2 for IGMs, using a cosine decay schedule. We also evaluate on the Pascal VOC2007 dataset (Everingham et al., 2010) held out data setting, selecting syppe-parammtert via cross voos-valii,,,nn, following (Kor-- blith et al., 2019) and report the mean average precision (AP) measures. See App. E for evaluating on the object detection task.
17

---

Published as a conference paper at ICLR 2022
C.2.1 CONTRASTIVE LEARNING METHODS
IGMs can create multiple views of images via their latent transformations (Jahanian et al., 2020), making them useful for contrastive multi-view learning. In this section, we study the effectiveness of contrastive methods for learning representations from IGMs.
To generate datasets containing multiple views produced by latent transformations, we first sample 1.3. anchor view images in the ImageNet1000 setting (described above). All the anchors are generated with z ~ (0, 1,2). We then generate one neighbor view for each anchor view by following different strategies for view creation, Gaussian and steer, described in Sec. 3.3. For the Gaussian view, we tune the standard deviation of WGauss to be 1.0 for our class-conditional IGMs (i.e. Waauss (0, 1.0, 2)), and we use this setting in all of our experiments. For steering, we learn the latent walk Wteer as the summation of individual walks with randomly sampled steps for different camera transformations, i.c. horizontal and vertical shifts, zoom, 2D and 3D rotations as well as color transformations. See the details in App. B.
The resulting images of our latent view creation strategies are shown in Fig. 7(b,c). In the class- conditional case, test independent views, i.e. Тn Ps and qualitatively show examples in Fig. 7(d). When needed, we further combine the latent views with pixel transformations, as shown in Fig. 7(f).
In the following sections, we learn visual representations of the generated images by training a ResNet-50 via SupCon (Khosla et al., 2020) for the class-conditional IGMs.
Original
Vieus
(c) Views
Independent
SupCon
SupCon
Examples of different transformation methods for class-conditional IGMs data. The images are sampled from BigGAN.
Figure 7:
C.2.2 EFFECT OF THE LATENT AND IMAGE TRANSFORMATIONS
First, we experiment with the effect of using pixel transformations on images sampled by the IGMs, without latent transformations, i.e. Tg together, where the latent transformation either Gaussian or steer, i.e. z + Waauss or z + Wateers respectively. There is third latent transformation independent view, i.e. Tz Pu, only applicable to the class-conditional IGM because of conditioning on the class semantics. We report Table the transfer results class-conditional IGMs, respectively, for the described transformations. From this table we observe similar trends across both unconditional and class-conditional IGMs (compare with Table 1. The view generation works well along with the contrastive methods: while pixel transformations Tx lead to strong representations, these are significantly improved by adding latent transformations Tzz the unconditional case. In the class-conditional case, on the other hand, there is only marginal improvement using Tz, indicating that the class-label supervision may be sufficient already lead strong semantic representations.
Next, we enable both latent and pixel transformations
C.2.3 NON-CONTRASTIVE METHODS
We further study representations learned from non-contrastive methods.
Inverter We first experiment with an inverter, as illustrated Fig. 3. We learn an image encoder by minimizing the mean squared emror between the original and predicted latent code For the olass-conditional IGMs, we include an auxiliary cross-entropy loss, to predict the category of the encoded image. For this experiment, use the ImageNet1000 setting.
Similar to the contrastive setting, we train a ResNet-50 encoder and replace the last layer with a fully connected layer to predict the latent vector and the labels y of 128 x 128 image. We
18

---

Published as a conference paper at ICLR 2022
Training Method
Transfer Task
ImageNet1000
VO007 Classification
Data distribution
I
Tx
Objective
Top-1 Accuracy
AP
Real
SimCLR Augs.
Sup. Contrastive
50.84
0.76
Generated
SimCLR Augs.
Sup. Contrastive
0.75
Generated
Sup. Contrastive
35.74 75
0.66
Generated
W Wannee
SimCLR Augs.
Sup. Contrastive
46.43
0.74
Generated
Sup. Contrastive
36.74
0.68
Generated
Watous
SimCLR Augs.
Sup. Contrastive
49.25
0.75
Generated
Sup. Contrastive
36.21
0.65
Generated
SimCLR Augs.
Sup. Contrastive
48.97
0.76
Generated
0.60
Table 3: Results on class-conditional IGMs. Real data is sampled from ImageNet1000 and distributed as
(D). Generated data is sampled from BigGAN and distributed ~ Tx (G(T, (x). Tx indicates that transformation is applied. T, ys, indicates that the transformation draws new sample Pa. independent
original z. the Sup. Contrastive objective, positives are defined following App. C.I.I, where two views are treated as positive and only if they share the same label y.
train the encoder with a dataset of 1.3M images and their latent vectors, i.e. pairs of (G(z, y), z). Note don't 't apply image transformations G(z, y) given that directions in z-space encode basic transformations like shifting and color change (Jahanian et al., 2020) so z is not invariant to these transformations.
The results of using the the class-conditional IGMs in Table suggest this non-contrastive approach performs poorly in comparison with the contrastive methods.
Classifier When learning representations from class-conditional IGMs, we can leverage the labels available and learn representations through a classification objective (softmax cross-entropy) We study the performance of this learning objective in the ImageNet100 setting. We train classifier with ResNet-50 backbone to classify images into one of the 100 classes of ImageNer100, applying Tx as data augmentation but not Tz. After embedding has been learned in this way, we evaluate the Top-1 Accuracy ImageNet100 of linear classifier on top of the learned representations. This approach achieves 65. 65.2%. performing similarly to the Supervised Contrastive objective, which achieves 66.8% accuracy in the same setting (only Tx, evaluation on ImageNet100), Overall, the classifier is subset of the inverter (which reconstructs both and class label y), but allows to use data pixel augmentations which also helps for higher performance. We provide additional results App. E on the classifier experiments in Table 6. Note these experiments are in the ImageNet100 setting and can be compared with Table 5.
C.2.4 EFFECT OF THE NUMBER OF SAMPLES
As discussed See. 4.3, in the synthetic data setting, we can create infinite data (albeit with finite diversity). A natural question arises: how many samples do need for good coverage and good visual representations?
To answer this question for conditional IGMs, we run experiments in the ImageNet100 setting and evaluate the linear classification performance for Gaussian views combined with pixel transforma- tions, vary the number of unique samples. We show the results in Fig. 8. Note that we train all the encoders for the same number iterations, equivalent to 200 epochs on ImageNet100 (matching the experiments shown in table 5 in App. E). This means, as the number of images (x-axis) increases, the number of times the model revisits seen image decreases.
As Fig. 8 illustrates, the performance increases with more samples, but sub-logarithmically These findings are consistent with recent work that found a small gap in generalization performance be- tween online learning (infinite data) and sufficiently large offline regime (Nakkiran al., 2020).
D FURTHER TRAINING DETAILS AND COMPARISON TO SIMCLRV1
We report in this section more training details for the unconditional baselines real data, specifi- cally SimCLR Chen et al. (2020c) training on ImageNet1000, to help clarify the reasons for the gap with Chen et al. (2020c).
19

---

Published as a conference paper at ICLR 2022
Imagenetion Acturacy Conditional
The
40
Laueias Conditiona HGM 1x
Causnia Conditional IGM cnl ImngeNietioo
UP
Numberru in for Leaming the modder
Figure 8: Effect of the number of samples for training the representation learner, evaluated using linear transfer ImageNer100. "Gaussian" refers to the Gaussian views (T, Wisuss)
In our work, we train the image representations via contrastive learning using a batch size of 256 and evaluate them via linear evaluation using the same batch size. As such, the most comparable result Chen et al. (2020c) would be 57.5% 57.5%/62.88 /6..8 from Table B.I, which is still higher than our reported 43.9% in Table 1. We list below the main differences between results and theirs.
1. Lower image resolution: Given that the IGMs study in work generate images, 128 128, we train our representations on real data using the same resolution, both when training the encoder as well as when performing linear evaluation. This has a great effect performance; in fact, when performing a linear evaluation using the same pre-trained encoder but with 256 x 256 images, the performance increases 50.3% for ImageNet1000,, and 49.7% for BigBiGAN with SimCLR pixel augmentations and gaussian latent augmentations.
Optimizer in linear evaluation: We use an SGD optimizer, as opposed to LARS (You et al., 2019), as proposed in Chen et al. (2020c). According to their experiments, linear performance drops from 62.8% to 57.5% based on the optimizer, with a lower effect when the batch size increases.
3. Training time: We train encoder for 20 epochs, opposed to 200 epochs in Chen et al. (2020c). To test the effect of training time, we train SimCLR on real data at 5, 10 and 15 epochs separately, measure the effect of performance in transfer learning ImageNet classifi- cation with the same setup we described in Sec. 4 and we obtained 15.98%, 23.66%, 32.48% respectively.
E FURTHER TRANSFER LEARNING TASKS
In this section, we further investigate the lighter weight protocol: train and evaluate representations only on data at the scale of ImageNet100 (Tian et al., 2019). This setting is described in the main text in Sec 4.
Moreover, we test on object detection task. For object detection, following (He et al., 2020), we use a Faster-RCNN (Ren et al., 2015) with the R50-C4 architecture. We fine-tune layers for 24000 iterations on PASCAL VOC r i nnva 07-12, 2, with batch size of 8. We report the standard COCO metrics, including AP (averaged across multiple thresholds), APso. AP7s.
We report the results in Table 4 for unconditional IGM and Table 5 for the conditional IGM.
F MIXING REAL AND SYNTHETIC IGM DATA
Following (Ravuri & Vinyals, 2019), we test whether unsupervised models trained using real data can benefit from using data generated by an unconditional IGM. For these experiments, we use the ImageNet100 setting. Starting from ImageNet100, samples from BigBiGAN (Donahue Simonyan, 2019), and a ResNet-50 using the SimCLR (Chen al., 2020c) framework. For each model, train linear classifier ImageNet100, using the
replace given percentage of the images by
20

---

Published as a conference paper at ICLR 2022
Training Method
Transfer Task
Data distribution
Ts
Objective
ImageNet100 Top-1 Accuracy
VOCO7 Classification
VOC07 Detection
AP
AP
APso
APrs
Real
SimCLR Augs.
Contrastive
67.36
59.45
45.93
73.49
48.72
Generated
SimCI.R Augs.
Contrastive
57.10
46.11
Generated
SimCLR Augs.
Contrastive
61.52
55.07
46.84
74.88
49.68
Gencrated
2
Wateer
SimCLR Augs.
Contrastive
60.74
55.03
46.55
74.51
49.37
Generated
Inverter
28.36
22.95
39.56
66.38
40.56
Table 4: Results on unconditional IGMs. Real data is sampled from ImageNct1000 and distributed as x Tx (D).. Generated data is sampled from BigBiGAN and distributed as x ~ T (G (x)))))()))) TN = indicates that no transformation is applied. For the Contrustive objective, positive and negative views are defined as described in Sec. 3.2.1 (without using class labels).
Training Method
Trunsfer Task
ImageNet100 Top-1 Accuracy
VOCO7 Classilication
VOCI7 Detection
distribution
Objective
AP
AP
APtn
APrs
Real
SimCCR Augs.
Contrustive
81.18
65.79
45.76
74.72
48.58
Generated
SImCERR Augs.
Sup. Contrastive
66.82
56.41
43.87
72.88
45.50
Generated
Wasuss
SimCLR Augs.
Sup. Contrustive
70.16
58.26
44.23
73.51
45.73
Generated
2 Watous
SimCI.R Augs.
Sup. Contrastive
67.86
57.47
44.49
73.31
45.85
Generated
SimCLR Augs.
Sup. Contrustive
68
57.94
44.08
73.06
45.49
Generated
Inverter
38.84
24.93
42.02
69.29
43.62
Table 5: Results class-conditional IGMs. Real data is sampled from ImageNet1000 and distributed as (D). Generated data sampled from BigGAN and distributed the transformation draws a new sample ps.
T (GL, ()). T.,Ta
indicates that no transformation is applied. T, ps indicates independent of the original z. For the Sup. Contrastive objective, positives are defined following Sec. C.I.I, where two views are treated as positive and only if they share the same label y.
Training Method
Transfer Task VOCO7 Classificution
ImageNet100 Top-1 Accuracy
VOCU7 Detection
Data distribution
Tx
Objective
AP
AP
AP50
APrs
Real
SimCLR Augs.
Classilier
80.64
67.99
45.90
75.08
48.24
Generated
SimCLR Augs.
Classifier
65.18
62.41
43.45
72.88
45.40
Table 6: Results on class-conditional IGMs. Real data is sampled from ImageNet1000 and distributed as x ~ (D). Generated data sampled from BigGAN and distributed as x ~ [G y)). We report the performance on real data and synthetic data with no latent transforms, for the Supervised Contrastive and Classifier objectives.
Imagenetro0 Accuracy of Unconditional IGM Data
68.00
Top-1 Accuneyy
$4.00
62.00
$8.00
100
Percontagnol of the Synthotic IGM data
Figure 9:
Mixing real data and synthetic IGM data as way augmenting real data. The plot shows the percentage of real data replaced the fake one versus the top-1 percent accuracy.
learned features, and report Top-1 I accuracy. Similar to (Ravuri & Vinyals, 2019), we see a decreas- ing trend in the performance the number of real images decreases, but find a sweet spot in using a small percentage (5%) of synthetic images.
G VISUALIZATION OF LATENT TRANSFORMATIONS
Here we present qualitative examples of our methods in latent view creation for both unconditional and class-conditional IGMs as illustrated in Fig. 10 and Fig. 11, respectively.
21

---

Published as a conference paper at ICLR 2022
Original
Gaussian
Steering
Figure 10: Examples of different latent transformation methods for unconditional IGMs data using BigBiGAN and StyleGAN LSUN CaR (the last three rows). Columns from left: anchor, Gaussian neighbors, and steering neighbors.
Original
Gaussian
Steering
Examples of different latent transformation methods for conditional IGMs data using BigGAN. Columns from left: anchor, Gaussian neighbors, and steering neighbors.
Figure 11:
22


Reviews:
[{"review_id": "VZknh46Q7F9", "reviewer": "Reviewer_pxGz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an exploratory study of the effects of performing representation learning based on a generative model trained on data, as opposed to the data itself. It is very well written and coherently laid out, and was a pleasure to read. There are no theoretical advances, but the paper provides a useful first foray into an interesting topic, based on well executed empirical studies. \n\nStrengths: the paper provides a well reasoned empirical intro into a fairly novel area. The experiments appear to be well executed and thought through. The results are not earth shattering, but solid work like this should be published and discussed. The concept of using latent space perturbations rather than pixel level perturbations is an interesting (in a good way!) one. One area of further research might be into how one chooses these perturbations in a disciplined way, and how these choices influence the mapping from one latent space to another (the learned latent space). In extremis, would this result in latent space 'cloning'?\n\nWeaknesses: The elephant in the room is that the results depend on the quality of the IGM (implicit generative model) itself - by quality i mean how well has the IGM learnt the data-generating distribution. This of course is a function of both the model's expressivity as well as the complexity of the data-generating distribution (and how well / evenly the training data covers the distribution). The paper makes the claim that the IGM can be considered a compact representation of the data itself - were this true, the performance of models trained on the IGM would be near to indistinguishable from the performance of models trained directly on the data, which isn't the case.\nI was less surprised than the authors that Gaussian perturbations in the latent space performed as well as more reasoned perturbations using steering methods - I think because I assume that how we think about data may not be as useful as we think (!); indeed much of modern machine learning bears this out.", "summary_of_the_paper": "This is an exploratory study of the effects of performing representation learning based on a generative model trained on data, as opposed to the data itself. It is very well written and coherently laid out, and was a pleasure to read. There are no theoretical advances, but the paper provides a useful first foray into an interesting topic, based on well executed empirical studies. ", "summary_of_the_review": "No theoretical advances, but some clever thinking and solid results. Happy to recommend acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "JIFeve-4dc7", "reviewer": "Reviewer_ZWWA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- The paper is well-written and easy to follow. The proposed method is described and illustrated clearly and appears technically sound.\n- Overall, the experiments and ablations systematically support the effectiveness of the proposed method. Both, unconditional and class-conditional representation learning is considered.\n- Using generative models as a data source is an interesting idea that is gaining attention recently as it might be beneficial to compress excessively large datasets, address privacy concerns, and enable more diverse data augmentation for training.\n\n### Weaknesses\n- A highly similar idea to the proposed latent-space transformations for contrastive learning was already proposed by Yang et al. [1] as “Noise perturbation”. This work is not referred to in the paper. While this work focuses on more stable GAN training and not representation learning, it takes away from the novelty of the proposed method.\n\n> [1] Ceyuan Yang, Yujun Shen, Yinghao Xu, Bolei Zhou. Data-Efficient Instance Generation from Instance Discrimination. arXiv preprint arXiv:2106.04566, 2021.\n\n### Additional questions / comments:\n- As the performance of the proposed method probably largely depends on the quality of the generated images, an ablation considering the truncation value would be interesting. I.e., what is more important for representation learning: image diversity or image fidelity? Also, please report how the truncation parameter was determined for the existing experiments.\n- What are the details for evaluating the BigBiGAN encoder? In the paper, you mention that it is trained on images of size 256x256 while you use resolution 128x128. Does this mean you resize the images before showing them to the BigBiGAN encoder? Further, please add the result of this experiment to the corresponding table (Table 4) rather than in the text.\n- For the ablation in Figure 6: Why does it make sense to keep the number of iterations constant here? Would you not be rather interested in the best possible performance, i.e. training each model for the best validation error with the same number of sufficient epochs for convergence?\n- What is your recommendation for tuning the standard deviation? Did you perform a grid search for each dataset, similar to the ablation on ImageNet1000 in Figure 5?  \n- Why does steering perform better than Gaussian perturbation in the class-conditional setting (Table 3)?\n- ContraD (https://arxiv.org/pdf/2103.09742.pdf) combines contrastive learning with GAN training and finds that training a joint discriminator can be beneficial for both tasks, image synthesis, and representation learning. This should be added and put into context in your related work.\n- Another interesting related work where generative models are used as a data source, to generate multiview images and allow for efficient data labeling: https://nv-tlabs.github.io/GANverse3D/. Should be added to related work.\n\n### Misc:\n- Table 3: bold number for ImageNet1000 missing", "summary_of_the_paper": "The work investigates **generative models as a data source for self-supervised representation learning**. In particular, the authors propose to form contrastive pairs in the latent space of the generative model and combine it with standard contrastive learning where pairs are formed in image space. \nWhile performance is **slightly inferior** to training with the full dataset, the proposed approach **only requires storing the model weights** of the generator instead of the full dataset.", "summary_of_the_review": "The considered task, i.e. generative models as a data source for representation learning, is **very interesting for the community** and has recently received increasing attention. The experiments in the paper appear sound and provide **interesting insights**. My major concern is about **the contribution regarding the latent space transformations**, as a very similar strategy was already proposed by Yang et al. Nonetheless, the findings of this work are interesting, so currently I am leaning towards accepting the paper.\n\n**Post-rebuttal**:\nThe authors successfully addressed my main concern about the paper by Yang et al. so I update my rating from 6 to 8.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6Q2hevMzBLq", "reviewer": "Reviewer_ZFjV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The problem of learning with only generated data is very relevant, and the authors achieve reasonably good results.\n- The methods are very clearly exposed, the results are comprehensive and the experimental protocol is very detailed.\n- It should be appreciated that while the paper is about benchmarking, it makes an effort to play to generative models' strength by also using latent space augmentations for contrastive learning. This approach is novel and the empirical results support its importance.\n\nWeaknesses:\n- It might be a little bit misleading that much of the paper is about implicit generative models but only GANs are tested. Especially since while the general idea is universally applicable, the specifics, and in particular latent augmentation, are not necessarily trivial to transferred to other generative models.\n", "summary_of_the_paper": "The paper investigates if synthetic datasets obtained from implicit generative models can be used for representation learning in place of the original dataset.\nTo do so, they compare the performance of multiple popular contrastive learning methods trained on real data against the same methods trained on synthetic data generated by GANs.\nNotably, for the generated data, in addition to standard pixel transformation, they also evaluate latent space augmentations methods for contrastive learning.", "summary_of_the_review": "The paper is a step forward towards training only using synthetic data, a long-standing goal that has been of interest since the introduction of deep generative models. \nMoreover, it introduces and evaluates a novel way of combining generative data and contrastive learning that is shown to be very promising.\nWith the rapid advances in representation learning, this submission providing a strong and comprehensive empirical study is likely to be valuable to the community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "0UmXMuUiggz", "reviewer": "Reviewer_P9Ys", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n* The paper is well motivated. The authors tackle an important and relevant problem, how to use samples from a generative model in a meaningful way for downstream tasks.\n* To the best of my knowledge, this is the first paper that attempts to utilize black-box generative models for unsupervised representation learning.\n* The entire paper is easy to read and self-contained. The literature review is also very solid.\n\n**Weaknesses**\n\nThe major weaknesses of the papers are the empirical results. Here are my concerns:\n\n* It seems that the SimCLR baseline is very weak. The lowest reported number in Chen et al 2020 is 62.8% ImageNet top 1 accuracy using a batch-size of 256 and a number of epochs equal to 100 while the paper reports 43.9%. What are the differences in this paper as compared to the SimCLR baseline? Even though achieving state-of-the-art performance is not the goal of this paper, the authors should strive for a reasonable baseline.\n* The authors could try to provide more insight on their negative result which is that the contrastive algorithm trained on generated views underperforms the SimCLR baseline by 1.3%. Is it that the generated samples have poor quality as compared to the ground truth dataset, or do they lack in diversity (for eg, the typical mode collapse issue with GANS)?\n* **Steered Latent Views**: It seems that $w_{steer}$ in $z + w_{steer}$ is optimized to match pixel-level transformations targets $T_z(z)$. Then why not use the pixel-level transformations directly? Can the authors clarify the setting where it is preferred to use $T_z(z + \\alpha w)$ instead of $T(G(z), \\alpha)$?\n* The baseline for the real algorithm in Table 2 is missing.\n* The ImageNet100 benchmark is not apples-to-apples since both the “real” and “generated” algorithms have access to samples with a varying number of classes. Could the authors clarify the motivation behind using ImageNet100?\n* The authors should also plot the \"real baseline\" bound in Figure 6, to see if the real baseline can be approached simply by increasing the number of samples.\n\nTypos:\n* Legends in Figure 6 have the same color.\n* Eq 6) should also have T_z?\n", "summary_of_the_paper": "The paper explores applying a black-box generative model (instead of a fixed training dataset) to learn unsupervised visual representations, mainly using contrastive learning. Standard contrastive learning algorithms generate multiple views of a datapoint using transformations in pixel space. The authors explore using transformations in latent space, either in isolation or in combination with pixel-level transformation methods.\n\nResults are reported for two generative models.\n* BigBiGan trained on ImageNet and finetuned on ImageNet100 and ImageNet1000\n* StyleGan2 trained on LSUN Car and finetuned on ImageNet\n", "summary_of_the_review": "The direction that the authors pursue in this paper is challenging and novel. However, the results are mainly negative.\n\nFor negative results, to be useful to the community, imho either the results should be suprising or should offer some insight. I have given a reject rating mainly because of the empirical results.\n\nPlease see the weaknesses section above. If atleast the first two bullets points can be addressed convinvingly by the authors, I can reconsider my rating.\n\n**Update after Rebuttal**:\n\nI thank the authors for their additional experiments and discussion. I've upgraded my rating from 3 (Reject) to 6 (Lean Accept).\nThe rationale for my rating change:\n\n* The authors confirmed that the SimCLR baseline is lower than the reported results, due to the low resolution (128x128) of the generated model and reduced training time.\n* The models (baseline and generative) are trained for 20 epochs using cyclic learning rates so I think the results might be transferrable to full training.\n* The authors have shown more promsing results on pretrain LSUN + transfer on Stanford Car classification.\n* ImageNet100 results as far as I see are provided only in Section 4.1.2 and 4.3 as a diagnostic tool, so the comparisons are fair.\n\nThe authors have taken a first-step towards usage of generative models for unsupervised representation learning. Even though, the results are somewhat underwhelming on ImageNet itself, this work can serve as a promising and strong baseline for future work in this topic.\n\n**For the final version**\nIf the paper gets accepted, I encourage the authors:\n* To report results for full training (100 Epochs), at least with the \"Real baseline\"  and their best setting for generated data\n* To open-source their code.\n* To explicitly state that the latent transform is applied only once, as compated to pixel-level transformations, which are applied twice. This might be a trivial detail but can be extremely important to reproduce the authors results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All of the reviewers appreciate the clarity of exposition and the importance of the problem studied. That said, I agree with Reviewer P9Ys that the results are somewhat underwhelming. The baselines appear weak and are likely not well tuned on the Stanford car dataset. Key question that remains unanswered in my opinion is whether this is the most effective approach to using synthetic data to improve classification accuracy (e.g., in contrast to [Ravuri & Vinyals, 2019](https://arxiv.org/abs/1905.10887) and follow-up work). Nevertheless, I believe the community will benefit from this paper's contributions and this line of work.
Decision: Accept (Poster)