Year: 2022

Abstract:
Generative models are now capable of producing highly realistic images that look nearly indistinguishable from the data on which they are trained. This raises the question: if we have good enough generative models, do we still need datasets? We investigate this question in the setting of learning general-purpose visual representations from a black-box generative model rather than directly from data. Given an off-the-shelf image generator without any access to its training data, we train representations from the samples output by this generator. We compare several representation learning methods that can be applied to this setting, using the latent space of the generator to generate multiple “views” of the same semantic content. We show that for contrastive methods, this multiview data can naturally be used to identify positive pairs (nearby in latent space) and negative pairs (far apart in latent space). We find that the resulting representations rival or even outperform those learned directly from real data, but that good performance requires care in the sampling strategy applied and the training method. Generative models can be viewed as a compressed and organized copy of a dataset, and we envision a future where more and more “model zoos” proliferate while datasets become increasingly unwieldy, missing, or private. This paper suggests several techniques for dealing with visual representation learning in such a future. Code is available on our project page https://ali-design.github.io/GenRep/ Learner Generative Model Le ar ni ng fr om m od el s z<latexit F : x ! e Tz(z) x ⇠ G(Tz(z)) Embedding function Data Le ar ni ng fr om d at a

heading: 1 INTRODUCTION
text: The last few years have seen great progress in the diversity and quality of generative models. For almost every popular data domain there is now a generative model that produces realistic samples from that domain, be it images (Radford et al., 2015; Brock et al., 2019; Ramesh et al., 2021), music (Dhariwal et al., 2020), or text (Brown et al., 2020). This raises an intriguing possibility: what we used to do with real data, can we now do instead with synthetic data, sampled from a generative model?
If so, there would be immediate advantages. Models are highly compressed compared to the datasets they represent and therefore easier to share and store. Synthetic data also circumvents some of the concerns around privacy and usage rights that limit the distribution of real datasets (Tucker et al.,
2020), and models can be edited to censor sensitive attributes (Liao et al., 2019), remove biases that exist in real datasets (Tan et al., 2020; Ramaswamy et al., 2020), or steer toward other taskspecific needs (Jahanian et al., 2020; Goetschalckx et al., 2019; Shen et al., 2020). Perhaps because of these advantages, it is becoming increasingly common for pre-trained generative models to be shared online without their original training data being made easily accessible. This approach has been taken by individuals who may not have the resources or intellectual property rights to release the original data1 and in the case of large-scale models such as GPT-3 (Brown et al., 2020), where the training data has been kept private but model samples are available through an API.
Our work therefore targets a problem setting that has received little prior attention: given access only to a trained generative model, and no access to the dataset that trained it, can we learn effective visual representations?
To this end, we provide an exploratory study of representation learning in the setting of synthetic data sampled from pre-trained generative models: we analyze which representation learning algorithms are applicable, how well they work, and how they can be modified to make use of the special structure provided by deep generative networks.
Figure. 1 lays out the framework we study: we compare learning visual embedding functions F from real data x ∼ {xi}Ni=1 vs. from generated data x ∼ G controlled via latent transformations. We study generation and representation learning both with and without class labels, and test representation learners based on several objectives. We evaluate representations via transfer performance on held out datasets and tasks.
For representation learners, we focus primarily on contrastive methods that learn to associate multiple “views” of the same scene. These views may be co-occurring sensory signals such as imagery and sounds (e.g., (de Sa, 1994)), or may be different augmented or transformed versions of the same image (e.g., (Becker & Hinton, 1992; Chen et al., 2020c)). Interestingly, generative models can also create multiple views of an image: by steering in their latent spaces they can achieve camera and color transformations (Jahanian et al., 2020) and more (Härkönen et al., 2020; Wu et al., 2020). Figure 2 diagrams the currently popular setting, where views are generated as pixel-space transformations, versus the setting we focus on, where views are generated via latent-space transformations.
We study the properties of these latent views in contrastive learning and ask under what conditions they can lead to enhanced learners. We then compare these learners with non-contrastive encoders, also trained on samples from generative models. Further, we study the long-standing promise of generate models in being capable of generating more samples than the number of datapoints that trained them. Knowing current models still have finite diversity, we ask: how many samples are necessary to get good performance on a task? Our main findings are:
1. Contrastive learning algorithms can be naturally extended to learning from generative samples, where different “views” of the data are created via transformations in the model’s latent space.
1E.g., see https://github.com/justinpinkney/awesome-pretrained-stylegan, hosting many pretrained models, without datasets.
2. These latent-space transformations can be combined with standard data transformations (“augmentations”) in pixel-space to achieve better performance than either method alone. 3. To generate positive training pairs in latent space, simple Gaussian transformations work as well as more complicated latent-space transformations, with the optimal standard deviation being not too large and not too small, obeying an inverse-U-shaped curve like that observed for contrastive learning from real data (Tian et al., 2020). 4. Generative models can potentially produce an unbounded number of samples to train on; we find that performance improves by training on more samples, but sub-logarithmically. 5. Learning representations from generative models can outperform learning directly from the data that trained those models, provided that the generative model is of sufficiently quality (we observe this result on StyleGAN2 (Karras et al., 2020) fit to cars but not on BigBiGAN (Donahue & Simonyan, 2019) or BigGAN (Brock et al., 2019) fit to ImageNet1000 (Deng et al., 2009)).

heading: 2 RELATED WORK
text: Learning from Synthetic Data. Using synthetic data has been a prominent method for learning in different domains of science and engineering, with different goals including privacy-preservation and alternative sample generation (Sakshaug & Raghunathan, 2010; Tucker et al., 2020; Nußberger et al., 2020; Khan et al., 2019; Dan et al., 2020). In computer vision, synthetic data has been extensively used as a source for training models, for example in semantic segmentation (Chen et al., 2019; Ros et al., 2016), human pose estimation (Varol et al., 2017; Ionescu et al., 2013; Shakhnarovich et al., 2003), optical flow (Mayer et al., 2016; Fischer et al., 2015), or self-supervised multitask learning (Ren & Lee, 2018). In most prior work, the synthetic data comes from a traditional simulation pipeline, e.g., via rendering a 3D world with a graphics engine. We instead study the setting where the synthetic data is sampled from a deep generative model.
Recent works have used generative networks, such as GANs (Goodfellow et al., 2014), to improve images generated by graphics engines, closing the domain gap with real images (Shrivastava et al., 2017; Hoffman et al., 2018). Ravuri & Vinyals (2019) show that even though there is a gap between GAN-generated images and real ones, mixing the two sources can lead to improvements over models trained purely on real data. GAN-generated images are also used to learn inverse graphics. Zhang et al. (2021a) use StyleGAN to generate multiple views of a given object, allowing to extract 3D knowledge that can be mapped back into the latent space.
Highly related to our paper is the work of Besnier et al. (2020), which uses GAN-generated images to train a classifier. While they focus on using a class-conditional generator to train a classifier of those same classes, our work targets the more general setting of visual representation learning, and we mainly focus on the “unsupervised” setting where an unconditional generative model produces unlabeled samples to learn from.
Recent works (Yang et al., 2021; Jeong & Shin, 2021) have studied combining contrastive and adversarial losses in GAN training, showing benefits in both image synthesis and representation learning. In contrast to our work, these approaches require access to real data during training.
Recent work has also explored using StyleGAN (Karras et al., 2019) to generate training data for semantic segmentation (Zhang et al., 2021b; Tritrong et al., 2021). These approaches differ from ours in that they use intermediate layers of the generator network as their representation of images, whereas our method does not require internal access to the generator; instead we simply treat the generator as a black-box source of training data for a downstream representation learner.
GAN-based data augmentation has also been used, in conjunction with real data, to improve data efficiency for GAN training (Yang et al., 2021) as well as robustness at training time (Mao et al., 2020) and test-time (Chai et al., 2021). In contrast to these methods, we explore representation learning without access to the real data at all, relying only on samples from a black-box generative model to train our systems.
Contrastive Representation Learning. Contrastive learning methods (van den Oord et al., 2019; Wu et al., 2018; Hénaff et al., 2019; Tian et al., 2019; He et al., 2020; Chen et al., 2020c) have greatly advanced the state of the art of self-supervised representation learning. The idea of contrastive learning is to contrast positive pairs with negative pairs (Hadsell et al., 2006). Such pairs can be easily constructed with various data formats, and examples include different augmentations or transformations of images (Chen et al., 2020c), cross-modality alignment (Tian et al., 2019; Mor-
gado et al., 2020; Patrick et al., 2020), and graph structured data (Hassani & Khasahmadi, 2020). One key ingredient for the success of contrastive learning is well-chosen data transformations (Chen et al., 2020c; Tian et al., 2020; Xiao et al., 2020). While all these approaches conduct data transformations, or “augmentations”, in the raw pixel space, in this paper we explore the possibility of transforming training points in the latent space of a GAN.
Generative Representation Learning. Generative models learn representations by modeling the data distribution p(x). In VAEs (Kingma & Welling, 2013), each data point is encoded into a latent distribution, from which codes are sampled to reconstruct the input by maximizing the data likelihood. GANs (Goodfellow et al., 2014) model data generation through a minimax game, after which the discriminator can serve as a good representation extractor (Radford et al., 2015). In ALI (Dumoulin et al., 2016), BiGAN (Donahue et al., 2017), and BigBiGAN (Donahue & Simonyan, 2019) both image encoding and latent decoding are modeled simultaneously, and the encoder turns out to be a representation learner. Similar to the trend in natural language processing (Devlin et al., 2018; Radford et al.), autoregressive models (Van den Oord et al., 2016; Chen et al., 2020b) have been adopted to learn representations from raw pixels. These prior works show ways to jointly train a representation alongside a generative model, using a training set of real data for learning. Our exploration qualitatively differs in that we assume we are given a black-box generative model, and no real training data, and the goal is to learn an effective representation by sampling from the model.

heading: 3 METHOD
text: Standard representation learning algorithms take a dataset {xi}Ni=1 as input, and produce an encoder F : x → e as output, where e is a vector representation of x. Our method, in contrast, takes a generative model G as input, in order to produce F as output. We restrict our attention to implicit generative models (IGMs) (Mohamed & Lakshminarayanan, 2016), which map from latent variables z to sampled images x, i.e. G : z → x. Many currently popular generative models have this form, including GANs (Goodfellow et al., 2014), VAEs (Kingma & Welling, 2013), and normalizing flows (Rezende & Mohamed, 2015). We also consider class-conditional variants, which we denote as Gy : z, y → x, where y is a discrete class label. In our experiments, we only investigate GANs, but note that the method is general to any IGM with latent variables.
To learn F , from either real data or model samples, we can pick any of a large variety of representation learners: autoencoders (Ballard, 1987), rotation prediction (Gidaris et al., 2018), colorization (Zhang et al., 2016), etc. We focus on contrastive methods due to their strong performance and natural extension to using latent transformations to define positive pairs. These methods are illustrated in Fig. 3 in the first and the second rows. We also examine the effectiveness of non-contrastive methods, namely an inverter (similar to the encoder in the autoencoders) as illustrated in the third row of Fig. 3. For conditional IGMs, which generate data from class labels, we also consider using a label-classifier as the representation learner.
In the following sections, we first define the contrastive framework with different sampling strategies for creating views via pixel transformations and latent transformations. We then define the noncontrastive frameworks.

heading: 3.1 CONTRASTIVE LEARNING FRAMEWORK
text: Modern contrastive methods learn embeddings that are invariant to certain nuisance transformations of the data, or different “viewing” conditions. Two “views” of the same scene are pulled together in embedding space while views of different scenes are pushed apart. A common objective is the InfoNCE (van den Oord et al., 2019). We use the following variant of this objective:
LNCE = −E [ log
eτF (xa) TF (xp)
eτF (xa) TF (xp) + ∑K k=1 e τF (xa)TF (xkn)
] (1)
Here xa is an anchor image, {xa,xp} is a positive pair of images, i.e. a pair we want to bring together in embedding space. {xa,xkn} is a negative pair of images, which we want to push apart. Different contrastive learning algorithms differ in how the positive and negative pairs are defined. Commonly, xa and xp are two different transformations (i.e. data augmentations in pixel space) of
the same underlying image, and xkn is a transformation of a different randomly selected image (Chen et al., 2020c; He et al., 2020; Gidaris et al., 2018).
Our analysis focuses both on how the datapoints are sampled in the first place, and on how they are transformed to create negative and positive pairs:
1. What happens when x is fake data sampled from an IGM rather than real data from a dataset? 2. How can we use transformations in pixel space and in latent space to define positives/negatives?
Different answers to these two questions yield the specific methods we compare, as described in the next sections.

heading: 3.2 SAMPLING POSITIVE AND NEGATIVE PAIRS
text: We first describe several schemes for sampling from both real and generated datapoints, with transformations applied in either latent space or pixel space.

heading: 3.2.1 CONTRASTIVE PIXEL TRANSFORMATIONS (I.E. SIMCLR)
text: If we are given a dataset, and a set of pixel-space transformations, Tx, that we wish to be invariant to, a standard approach is to use the following sampling strategy:
x̂a, x̂ k n ∼ D / sample anchor and negatives (2) xa,xp ∼ Tx(x̂a) / generate positive pair (3) xkn ∼ Tx(x̂kn) / generate negatives (4)
where∼ D refers to drawing a random image uniformly from the datasetD = {xi}Ni=1. This setting is depicted in the top row of Fig. 3. We use SimCLR (Chen et al., 2020c) as an instantiation of this approach.

heading: 3.2.2 CONTRASTIVE LATENT VIEWS + PIXEL TRANSFORMATIONS
text: If we are given an unconditional IGM G, we may also define a set of latent transformations, Tz, that we wish our representation to be invariant to. We can use this method with or without pixel-space transformations Tx applied as well. With both Tz and Tx applied, we have the following fake data generating process:
za, z k n ∼ pz / sample latents (5) xa ∼ Tx(G(za)) / generate anchor sample (6) xp ∼ Tx(G(Tz(za))) / generate positive sample (7) xkn ∼ Tx(G(Tz(zkn))) / generate negatives (8)
In practice we set pz to be the truncated normal distribution, N t(µ, σ, t), with truncation t (Brock et al., 2019). This scenario is depicted in the middle row of Fig. 3. Refer to Appendix B.1.1 for the class-conditional formulation.
3.3 CREATING VIEWS WITH Tx AND Tz
We refer to transformations as “views” of the data (either real or fake). Figure 4 shows different views generated according to all the following methods. For pixel-space transformations, Tx, many options have been previously proposed, and in our experiments we choose the transformations from SimCLR (Chen et al., 2020c) as they are currently standard and effective. Our framework could be updated with better transformations as they are developed in future work.
Our work is the first, to our knowledge, to explore latent-space transformations, Tz, for contrastive representation learning. Therefore, we focus our analysis on studying the properties of Tz and explore the following options:

heading: 3.3.1 GAUSSIAN LATENT VIEWS
text: Many IGMs have the remarkable property that nearby points in latent space map to semantically related generated images (Jahanian et al., 2020). This suggests that we can define positive views as nearby latent vectors (refer to Sec. 4.1.2 for an empirical proof). A simple way to do so is to define the latent transformation to just be a small offset applied to the sampled z vector. We use truncated Gaussian offsets wGauss as a simple instantiation of this idea:
wGauss ∼ N t(µ, σ, t) (9) Tz(z) = z+wGauss (10)
where N t(µ, σ, t) is the truncated Normal distribution with truncation t (Brock et al., 2019).

heading: 3.3.2 STEERED LATENT VIEWS
text: Can we find latent transformations that are more directly related to semantics? We leverage the recent method of (Jahanian et al., 2020), which finds latent vectors wsteer that achieve target effects in image space, such as shifting an image up and down or changing it is brightness. That is, Tz(z) = z + wsteer, where wsteer is a latent vector trained to match a target pixel-space transformation T according to the objective wsteer = argminw Ez,α[||G(z+αw)−T (G(z), α)||]. For example, T could be “shift left”, parameterized by the strength of the transformation α, e.g. the number of pixels to shift. App A describes the exact steering transformations we use in more detail.

heading: 3.4 NON-CONTRASTIVE ALTERNATIVES
text: We can also learn image representations without a contrastive loss. Here we describe one possible approach, an inverter. App B.2.3 describes another non-contrastive approach, a classifier, that only applies in the class-conditional IGM setting. Inverter The z vector that generates an image can itself be considered a vector representation of that image. This suggests that simply inverting the generator may result in a useful embedding function. For a given IGMG, we learn an encoder F that inverts G by minimizing ‖F (G(z))− z‖22.

heading: 4 EXPERIMENTS
text: To study the methods in Sec. 3, we analyze the behaviour and effectiveness of the representations learned from unconditional IGMs here and, in the interest of space, report the results on classconditional IGMs in Appendix B. We experiment on two unconditional IGMs: the generator from BigBiGAN (Donahue & Simonyan, 2019) and the StyleGAN2 LSUN CAR generator (Karras et al., 2020). Note, we do not use the encoder or training scheme from BigBiGAN, we simply use its learned generator G as an off-the-shelf unconditional ImageNet generator.
For ImageNet representation learning, we investigate two settings in our experiments: 1) training and evaluating representations on data at the scale of ImageNet1000 (Deng et al., 2009), and 2) a lighter weight protocol where we train and evaluate representations only on data at the scale of ImageNet100 (Tian et al., 2019) (a 100-class subset of ImageNet1000). In the ImageNet1000 setting, the real data encoders are trained on the ImageNet1000 dataset, the IGM encoders are trained on 1.3M anchor images (which roughly matches the size of ImageNet1000). We use SGD with learning rate of 0.03, batch size of 256, and 20 epochs. In the ImagetNet100 setting, the real data encoders are trained on ImageNet100, the unconditional IGM encoders are trained on 130K anchor images sampled unconditionally (note that the unconditional model implicitly is still sampling from all 1000 classes since the generative model itself was fit to ImageNet1000). We use SGD with learning rate of 0.03, batch size of 256, for 200 epochs. In all settings, we use 128× 128 images, leading to lower than state-of-the-art performance on the real datasets, more details in Appendix C.
We evaluate the ImageNet learned representations by training a linear classifier on top of the learned embeddings, for either ImageNet1000 or ImageNet100, matching the setting in which the encoder was trained, and report Top-1 class accuracy. For both ImageNet1000 and ImageNet100 settings, we use SGD with batch size of 256 over 60 epochs, and learning rates of 0.3 for real and 2 for IGMs, using a cosine decay schedule. We also evaluate on the Pascal VOC2007 dataset (Everingham et al., 2010) as a held out data setting, selecting hyper-parameters via cross-validation, following (Kornblith et al., 2019) and report the mean average precision (AP ) measures. See Appendix D for results on the object detection task.
For StyleGAN2 LSUN CAR, the IGM encoders are trained on 893K anchor images. We use SGD with learning rate of 0.03, batch size of 256, and 20 epochs. We evaluate the learned representations by training a linear classifier on top of the learned embeddings, for either ImageNet1000 or Stanford Car classification task (Krause et al., 2013) (196 car models with roughly 8K train and 8K val), and report Top-1 class accuracy. For both experiments, we use SGD with batch size of 256 over 100 epochs, and learning rates of 2, using a cosine decay schedule.

heading: 4.1 CONTRASTIVE LEARNING METHODS
text: To generate datasets containing multiple views produced by latent transformations, we first sample 1.3M anchor view images in the ImageNet1000 setting (described above). All the anchors are generated with z ∼ N (0, 1, 2). We then generate one neighbor view for each anchor view by following different strategies for view creation, Gaussian and steer, described in Sec. 3.3. For the Gaussian view, we tune the standard deviation of wGauss to be 0.2 for our unconditional IGMs (i.e. wGauss ∼ N (0, 0.2, 2)), and we use this setting in all of our experiments. For steering, we learn the latent walk wsteer as the summation of individual walks with randomly sampled steps for different camera transformations, i.e. horizontal and vertical shifts, zoom, 2D and 3D rotations as well as color transformations. The details for training wsteer follows (Jahanian et al., 2020), and we report them in Appendix A. We use similar view creation strategies for Stylegan2 LSUN CAR, i.e., each anchor is generated with z ∼ N (0, 1, 0.9), and per anchor image, a neighbor is generated with the
same steering or Gaussian strategy with std = 0.25 apart from its anchor in the case of the Gaussian views. The resulting images of our latent view creations are shown in Fig. 4(b,c). When needed, we further combine the latent views with pixel transformations, as shown in Fig. 4(e,f). In the following sections, we learn visual representations of the generated images by training a ResNet-50 via SimCLR (Chen et al., 2020c) for the datasets generated from unconditional IGMs.

heading: 4.1.1 EFFECT OF THE LATENT AND IMAGE TRANSFORMATIONS
text: First, we experiment with the effect of using pixel transformations on images sampled by the IGMs, without latent transformations, i.e. Tz = −. Next, we enable both latent and pixel transformations together, where the latent transformation is either Gaussian or steer, i.e. z+wGauss or z+wsteer, respectively. We report in Table 1 and Table 2 the transfer results on unconditional IGMs, for the described transformations. From this table we observe similar trends across the unconditional IGMs: while pixel transformations Tx lead to strong representations, these are significantly improved by adding latent transformations Tz. Comparing different types of latent transformations, we find that
Gaussian transformations work best. This is a somewhat surprising result. Despite designing views through steering methods, transforming randomly in all latent directions provides similar or better performance.
Another important observation is that training on StyleGAN2 fit to LSUN CAR outperforms training directly on LSUN CAR (Table 2). This result demonstrates that generative data can, in some cases, be more useful for representation learning than training directly on real photographs. The success here may be due to the fact that the StyleGAN2 samples are nearly identical to real photos in their realism and diversity (FID = 2.32). Therefore we might expect StyleGAN2 samples, trained only with Tx, to more or less match the performance of training on real data with Tx, and indeed this is what we find (Table 2; compare first two rows). It is then unsurprising that adding Tz, which can only be done for generative data, can boost performance further.

heading: 4.1.2 LIMITS OF THE LATENT TRANSFORMATIONS
text: Prior work on contrastive learning has found that there is an optimal point in how much two views share information (Tian et al., 2020). Similarly, in our Gaussian latent view creation, we may expect there to be a sweet spot for how far we can go when sampling a neighbor view relative to an anchor view. To study that sweet spot for the BigBiGAN generator, we vary the standard deviation of the Gaussian transformation. For this experiment we use the ImageNet100 setting. Fig. 5 illustrates the results: linear transfer performance exhibits an inverse-U-shaped curve with respect to Gaussian standard deviation, peaking at std = 0.2.

heading: 4.2 NON-CONTRASTIVE METHODS
text: Inverter We experiment with an inverter, as illustrated in Fig. 3. We learn an image encoder by minimizing the mean squared error between the original and predicted latent code z. Similar to the
contrastive setting, we train a ResNet-50 encoder and replace the last layer with a fully connected layer to predict the latent vector z of a 128 × 128 image. We train the encoder with a dataset of 1.3M images and their latent vectors, i.e. pairs of 〈G(z), z〉. Note we don’t apply image transformations onG(z) given that directions in z-space encode basic transformations like shifting and color change (Jahanian et al., 2020) so z is not invariant to these transformations. The results of using the unconditional IGM representations in Table 1 suggest this non-contrastive approach performs poorly in comparison with the contrastive methods. Comparison to BigBiGAN’s encoder We finally compare the pretrained BigBiGAN encoder (Donahue & Simonyan, 2019) (a non-contrastive approach) with our proposed methods. Note the BigBiGAN encoder is jointly trained with the generator, thus requiring access to the original real dataset during training. This is not the setting we are targeting but we provide this comparison nonetheless to see where our methods stand compared to prior work that involved a generative model in learning a representation. To match the setup with our experiments, we take the BigBiGAN ResNet50 encoder after average pooling (without the head) and use it in our linear classification of ImageNet100 test. Further, note that this encoder is trained on ImageNet1000 and accepts as input images of size 256× 256, whereas for this experiment we compare against our encoders trained on the ImageNet100 training sets and use images of size 128× 128. The top-1 accuracy from the BigBiGAN encoder is 55.70% which is better than our inverter but still worse than all the contrastive methods trained and tested on ImageNet100 (see the numbers in in Table 4 in Appendix D).

heading: 4.3 EFFECT OF THE NUMBER OF SAMPLES
text: In the synthetic data setting, we can create infinite data (albeit with finite diversity). A natural question arises: how many samples do we need for good coverage and good visual representations? To answer this question, we run experiments in the ImageNet100 setting and evaluate the linear classification performance for Gaussian views combined with pixel transformations, as we vary the number of unique samples. We show the results in Fig. 6. Note that we train all the encoders for the same number of iterations, equivalent to 200 epochs on ImageNet100 (matching the experiments shown in table 4 in Appendix D). This means, as the number of images (x-axis) increases, the number of times the model revisits a seen image decreases. As Fig. 6 illustrates, the performance increases with more samples, but sub-logarithmically. These findings are consistent with recent work that found a small gap in generalization performance between online learning (infinite data) and a sufficiently large offline regime (Nakkiran et al., 2020).

heading: 5 CONCLUSION
text: We investigated how to learn visual representations from IGMs (implicit generative models), when they are given as a black-box and without any access to their training data. IGMs make it possible to generate multiple views of similar image content by sampling nearby points in the latent space of the IGM. These views can be used for contrastive learning or as input to other representation learning algorithms that associate multiple views of the same scene. Our results demonstrate that leveraging latent-space views can improve performance beyond learning from pixel-space transformations alone. Representations learned from large quantities of generative data rival and sometimes outperform the transfer performance of representations learned from the real datasets. As generative models improve, we expect that training vision systems on generative data may become an increasingly important tool in our toolkit.

heading: 6 ETHICS STATEMENT
text: This work studies how to learn useful image representations given data generated from IGMs as opposed to real data. This framework can provide several societal advantages currently faced in real datasets, including protecting the privacy and usage rights of real images (Tucker et al., 2020), removing sensitive attributes (Liao et al., 2019), or reducing biases (Tan et al., 2020; Ramaswamy et al., 2020) and therefore learning fairer representations.
At the same time, learning representations from IGMs brings several challenges and risks, that need to be addressed. Generative models can in some cases reveal the data they are trained on Hayes et al. (2018); Chen et al. (2020a), posing risks for privacy preservation. They can also amplify biases in datasets (Jain et al., 2020; Menon et al., 2020), which can lead to negative societal impacts if they are not audited properly (Raji et al., 2020a;b; Mitchell et al., 2019), or they are used in the wrong contexts Buolamwini & Gebru (2018). Making good use of the learned representations will require addressing the above issues by studying mitigation strategies, as well as methods to audit implicit generative models.

heading: ACKNOWLEDGMENTS
text: Author A.J. thanks Kamal Youcef-Toumi, Boris Katz, and Antonio Torralba for their support. We thank Antonio Torralba, Janne Hellsten, David Bau, and Tongzhou Wang for helpful discussions.
This research was supported in part by IBM through the MIT-IBM Watson AI Lab. The research was also partly sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the United States Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.

heading: A CREATING STEERING VIEWS
text: For creating steering views, we apply the walk wsteer in Sec. 3.3.2. Here we expand this vector as the summation of individual walks with randomly sampled steps for different camera transformations, i.e. horizontal and vertical shifts, zoom, 2D and 3D rotations as well as color transformations. In fact, the final walk is:
wsteer = ∑
i
αiw i steer, (11)
for i ∈ {shiftx,shifty,zoom,rot2D,rot3D,color}, each representing a pixel-space transformation. That is:
wisteer = argmin w Ez,α[||G(z+αw)− Ti(G(z), α)||], (12)
where Ti can be “shift left/right”, parameterized by the strength of the transformation α, e.g. the number of pixels to shift.
Following (Jahanian et al., 2020), coefficient α determines what magnitude we want for each transformation. For example, for color, we add α to each RGB channel of an image (and have three w vectors in the latent space). Similarly, for each i ∈ {shiftx,shifty,rot2D,rot3D}, α determines how many pixels we shift or rotate the image. For zoom, we use α to determine how much scale we should apply to the image. For that reason, we use log(α) in the latent space but α in the pixel space (to define the edits).
In order to train wsteer, we create a dataset of target images and optimize for the best wisteer walks that in a linear summation yield images close to the target images. Here, each target image is a composition of all the pixel-space transformations applied to the G(z) image with random magnitudes (α for each Ti).
After learning the wisteer walks together, we can randomly draw a z vector and add them it to (with random steps αi), and this results in a transformed image. This transformed image will serve as a positive view for the image of the given z vector.

heading: B CLASS-CONDITIONAL IGMS
text: B.1 METHOD
Given a class-conditional IGM, we can do the same formulations as in Sec. 3.2.2 except conditioned on class labels, i.e. we simply replaceG(·) withG(·, y), using y = yp ∼ Cat(M) for the anchor and positive sample, and an independent draw y = yn ∼ Cat(M) for the negative sample. Additionally, taking inspiration from SupCon, in each batch, one positive is set with the same z as the anchor (but different pixel space transformation) while the other positives in the batch are generated from independently drawn z ∼ pz.
B.1.1 SUPERVISED CONTRASTIVE PIXEL TRANSFORMATIONS (I.E. SUPCON)
Given a labeled dataset D = {xi, yi}Ni=1, with M classes, we can leverage the labels to define positives as images that share the same labels and negatives as randomly sampled images from other classes:
yp ∼ Cat(M) / sample positive class (13) x̂a, x̂p ∼ Dyp , x̂kn ∼ D/yp / sample pairs (14) xa ∼ Tx(x̂a),xp ∼ Tx(x̂p) / generate positive (15) xkn ∼ Tx(x̂kn) / generate negatives (16)
where Cat is the categorical distribution and Dyp = {xi ∈ D|yi = yp}, D/yp = {xi ∈ D|yi 6= yp}. Following SupCon (Khosla et al., 2020), in each batch one of the positives is specially set as xp ∼ Tx(x̂a).
B.1.2 INDEPENDENT LATENT VIEWS
The simplest method is: Tz(·) = pz (17)
That is, the transformation simply produces a new random draw from pz. In the unconditional setting, this can be considered a naive baseline where the two views share no information about the image. However, in the class-conditional case, this strategy is actually quite sensible, and in a sense optimal if the goal is to extract class semantics (Tian et al., 2020): the two positive views are two different images that are independent except that they share the same class yp.
B.2 EXPERIMENTS
For the class-conditional IGM, we use BigGAN (Brock et al., 2019) trained on ImageNet1000 (Deng et al., 2009) with the “deep-256” and truncation= 2.0. We use image size of 128 × 128 for all the settings (i.e. scaling down BigGAN images).
Similar to unconditional experiments, for ImageNet representation learning, we investigate two settings in our experiments: 1) training and evaluating representations on data at the scale of ImageNet1000 (Deng et al., 2009), and 2) a lighter weight protocol where we train and evaluate representations only on data at the scale of ImageNet100 (Tian et al., 2019) (a 100 class subset of ImageNet1000). In the ImageNet1000 setting, the real data encoders are trained on the ImageNet1000 dataset, the class-conditional IGM encoders are trained on 1.3M anchor images (1300 images conditioned on each of the 1000 ImageNet1000 classes, which roughly matches the size of ImageNet1000). We use SGD with learning rate of 0.03, batch size of 256, and 20 epochs. In the ImagetNet100 setting, the real data encoders are trained on ImageNet100, the class-conditional IGM encoders are trained on 130K anchor images (1300 images conditioned on each of the 100 classes in ImageNet100). We use SGD with learning rate of 0.03, batch size of 256, for 200 epochs.
We evaluate the ImageNet learned representations by training a linear classifier on top of the learned embeddings, for either ImageNet1000 or ImageNet100, matching the setting in which the encoder was trained, and report Top-1 class accuracy. For both ImageNet1000 and ImageNet100 settings, we use SGD with batch size of 256 over 60 epochs, and learning rates of 0.3 for real and 2 for IGMs, using a cosine decay schedule. We also evaluate on the Pascal VOC2007 dataset (Everingham et al., 2010) as a held out data setting, selecting hyper-parameters via cross-validation, following (Kornblith et al., 2019) and report the mean average precision (AP ) measures. See Appendix D for evaluating on the object detection task.
B.2.1 CONTRASTIVE LEARNING METHODS
IGMs can create multiple views of images via their latent transformations (Jahanian et al., 2020), making them useful for contrastive multi-view learning. In this section, we study the effectiveness of contrastive methods for learning representations from IGMs.
To generate datasets containing multiple views produced by latent transformations, we first sample 1.3M anchor view images in the ImageNet1000 setting (described above). All the anchors are generated with z ∼ N (0, 1, 2). We then generate one neighbor view for each anchor view by following different strategies for view creation, Gaussian and steer, described in Sec. 3.3. For the Gaussian view, we tune the standard deviation of wGauss to be 1.0 for our class-conditional IGMs (i.e. wGauss ∼ N (0, 1.0, 2)), and we use this setting in all of our experiments. For steering, we learn the latent walk wsteer as the summation of individual walks with randomly sampled steps for different camera transformations, i.e. horizontal and vertical shifts, zoom, 2D and 3D rotations as well as color transformations. See the details in Appendix A.
The resulting images of our latent view creation strategies are shown in Fig. 7(b,c). In the classconditional case, we test independent views, i.e. Tz = pz and qualitatively show examples in Fig. 7(d). When needed, we further combine the latent views with pixel transformations, as shown in Fig. 7(f).
In the following sections, we learn visual representations of the generated images by training a ResNet-50 via SupCon (Khosla et al., 2020) for the class-conditional IGMs.
B.2.2 EFFECT OF THE LATENT AND IMAGE TRANSFORMATIONS
First, we experiment with the effect of using pixel transformations on images sampled by the IGMs, without latent transformations, i.e. Tz = −. Next, we enable both latent and pixel transformations together, where the latent transformation is either Gaussian or steer, i.e. z+wGauss or z+wsteer, respectively. There is a third latent transformation, independent view, i.e. Tz = pz, only applicable to the class-conditional IGM because of conditioning on the class semantics. We report in Table 3 the transfer results on class-conditional IGMs, respectively, for the described transformations. From this table we observe similar trends across both unconditional and class-conditional IGMs (compare with Table 1. The view generation works well along with the contrastive methods: while pixel transformations Tx lead to strong representations, these are significantly improved by adding latent transformations Tz, in the unconditional case. In the class-conditional case, on the other hand, there is only marginal improvement using Tz, indicating that the class-label supervision may be sufficient to already lead to strong semantic representations.
B.2.3 NON-CONTRASTIVE METHODS
We further study representations learned from non-contrastive methods.
Inverter We first experiment with an inverter, as illustrated in Fig. 3. We learn an image encoder by minimizing the mean squared error between the original and predicted latent code z. For the class-conditional IGMs, we include an auxiliary cross-entropy loss, to predict the category of the encoded image. For this experiment, we use the ImageNet1000 setting.
Similar to the contrastive setting, we train a ResNet-50 encoder and replace the last layer with a fully connected layer to predict the latent vector z and the labels y of a 128 × 128 image. We train the encoder with a dataset of 1.3M images and their latent vectors, i.e. pairs of 〈G(z, y), z〉. Note we don’t apply image transformations on G(z, y) given that directions in z-space encode basic transformations like shifting and color change (Jahanian et al., 2020) so z is not invariant to these transformations.
The results of using the the class-conditional IGMs in Table 3 suggest this non-contrastive approach performs poorly in comparison with the contrastive methods.
Classifier When learning representations from class-conditional IGMs, we can leverage the labels available and learn representations through a classification objective (softmax cross-entropy). We study the performance of this learning objective in the ImageNet100 setting. We train a classifier with a ResNet-50 backbone to classify images into one of the 100 classes of ImageNet100, applying Tx as data augmentation but not Tz. After an embedding has been learned in this way, we evaluate the Top-1 Accuracy on ImageNet100 of a linear classifier on top of the learned representations. This approach achieves 65.2%, performing similarly to the Supervised Contrastive objective, which achieves 66.8% accuracy in the same setting (only Tx, evaluation on ImageNet100). Overall, the classifier is a subset of the inverter (which reconstructs both z and class label y), but it allows us to use data pixel augmentations which also helps for higher performance. We provide additional results in Appendix D on the classifier experiments in Table 6. Note these experiments are in the ImageNet100 setting and can be compared with Table 5.
B.2.4 EFFECT OF THE NUMBER OF SAMPLES
As discussed in Sec. 4.3, in the synthetic data setting, we can create infinite data (albeit with finite diversity). A natural question arises: how many samples do we need for good coverage and good visual representations?
To answer this question for conditional IGMs, we run experiments in the ImageNet100 setting and evaluate the linear classification performance for Gaussian views combined with pixel transformations, as we vary the number of unique samples. We show the results in Fig. 8. Note that we train all the encoders for the same number of iterations, equivalent to 200 epochs on ImageNet100 (matching the experiments shown in table 5 in Appendix D). This means, as the number of images (x-axis) increases, the number of times the model revisits a seen image decreases.
As Fig. 8 illustrates, the performance increases with more samples, but sub-logarithmically. These findings are consistent with recent work that found a small gap in generalization performance between online learning (infinite data) and a sufficiently large offline regime (Nakkiran et al., 2020).

heading: C FURTHER TRAINING DETAILS AND COMPARISON TO SIMCLRV1
text: We report in this section more training details for the unconditional baselines on real data, specifically SimCLR Chen et al. (2020c) training on ImageNet1000, to help clarify the reasons for the gap with Chen et al. (2020c).
In our work, we train the image representations via contrastive learning using a batch size of 256 and evaluate them via linear evaluation using the same batch size. As such, the most comparable result in Chen et al. (2020c) would be 57.5%/62.8% from Table B.1, which is still higher than our reported 43.9% in Table 1. We list below the main differences between our results and theirs.
1. Lower image resolution: Given that the IGMs we study in our work generate images, at 128× 128, we train our representations on real data using the same resolution, both when training the
encoder as well as when performing linear evaluation. This has a great effect on performance; in fact, when performing a linear evaluation using the same pre-trained encoder but with 256× 256 images, the performance increases to 50.3% for ImageNet1000, and 49.7% for BigBiGAN with SimCLR pixel augmentations and gaussian latent augmentations.
2. Optimizer in linear evaluation: We use an SGD optimizer, as opposed to LARS (You et al., 2019), as proposed in Chen et al. (2020c). According to their experiments, linear performance drops from 62.8% to 57.5% based on the optimizer, with a lower effect when the batch size increases.
3. Training time: We train our encoder for 20 epochs, as opposed to 200 epochs in Chen et al. (2020c). To test the effect of training time, we train SimCLR on real data at 5, 10 and 15 epochs separately, to measure the effect of performance in transfer learning on ImageNet1000 classification with the same setup we described in Sec. 4 and we obtained 15.98%, 23.66%, 32.48%, respectively.

heading: D FURTHER TRANSFER LEARNING TASKS
text: In this section, we further investigate the lighter weight protocol: train and evaluate representations only on data at the scale of ImageNet100 (Tian et al., 2019). This setting is described in the main text in Sec 4.
Moreover, we test on object detection task. For object detection, following (He et al., 2020), we use a Faster-RCNN (Ren et al., 2015) with the R50-C4 architecture. We fine-tune all layers for 24000 iterations on PASCAL VOC trainval07+12, with a batch size of 8. We report the standard COCO metrics, including AP (averaged across multiple thresholds), AP50, AP75.
We report the results in Table 4 for unconditional IGM and Table 5 for the conditional IGM.

heading: E MIXING REAL AND SYNTHETIC IGM DATA
text: Following (Ravuri & Vinyals, 2019), we test whether unsupervised models trained using real data can benefit from using data generated by an unconditional IGM. For these experiments, we use the ImageNet100 setting. Starting from ImageNet100, we replace a given percentage of the images by
samples from BigBiGAN (Donahue & Simonyan, 2019), and a ResNet-50 using the SimCLR (Chen et al., 2020c) framework. For each model, we train a linear classifier on ImageNet100, using the learned features, and report Top-1 accuracy. Similar to (Ravuri & Vinyals, 2019), we see a decreasing trend in the performance as the number of real images decreases, but find a sweet spot in using a small percentage (5%) of synthetic images.
F VISUALIZATION OF LATENT TRANSFORMATIONS
Here we present qualitative examples of our methods in latent view creation for both unconditional and class-conditional IGMs as illustrated in Fig. 10 and Fig. 11, respectively.
Anchor Gaussian Steering

Review summaries:
The work investigates **generative models as a data source for self-supervised representation learning**. In particular, the authors propose to form contrastive pairs in the latent space of the generative model and combine it with standard contrastive learning where pairs are formed in image space.  While performance is **slightly inferior** to training with the full dataset, the proposed approach **only requires storing the model weights** of the generator instead of the full dataset.

The paper explores applying a black-box generative model (instead of a fixed training dataset) to learn unsupervised visual representations, mainly using contrastive learning. Standard contrastive learning algorithms generate multiple views of a datapoint using transformations in pixel space. The authors explore using transformations in latent space, either in isolation or in combination with pixel-level transformation methods.  Results are reported for two generative models. * BigBiGan trained on ImageNet and finetuned on ImageNet100 and ImageNet1000 * StyleGan2 trained on LSUN Car and finetuned on ImageNet 

This is an exploratory study of the effects of performing representation learning based on a generative model trained on data, as opposed to the data itself. It is very well written and coherently laid out, and was a pleasure to read. There are no theoretical advances, but the paper provides a useful first foray into an interesting topic, based on well executed empirical studies. 

The paper investigates if synthetic datasets obtained from implicit generative models can be used for representation learning in place of the original dataset. To do so, they compare the performance of multiple popular contrastive learning methods trained on real data against the same methods trained on synthetic data generated by GANs. Notably, for the generated data, in addition to standard pixel transformation, they also evaluate latent space augmentations methods for contrastive learning.