Generative Models as a Data Source for Multiview Representation Learning

1 INTRODUCTION . The last few years have seen great progress in the diversity and quality of generative models . For almost every popular data domain there is now a generative model that produces realistic samples from that domain , be it images ( Radford et al. , 2015 ; Brock et al. , 2019 ; Ramesh et al. , 2021 ) , music ( Dhariwal et al. , 2020 ) , or text ( Brown et al. , 2020 ) . This raises an intriguing possibility : what we used to do with real data , can we now do instead with synthetic data , sampled from a generative model ? If so , there would be immediate advantages . Models are highly compressed compared to the datasets they represent and therefore easier to share and store . Synthetic data also circumvents some of the concerns around privacy and usage rights that limit the distribution of real datasets ( Tucker et al. , 2020 ) , and models can be edited to censor sensitive attributes ( Liao et al. , 2019 ) , remove biases that exist in real datasets ( Tan et al. , 2020 ; Ramaswamy et al. , 2020 ) , or steer toward other taskspecific needs ( Jahanian et al. , 2020 ; Goetschalckx et al. , 2019 ; Shen et al. , 2020 ) . Perhaps because of these advantages , it is becoming increasingly common for pre-trained generative models to be shared online without their original training data being made easily accessible . This approach has been taken by individuals who may not have the resources or intellectual property rights to release the original data1 and in the case of large-scale models such as GPT-3 ( Brown et al. , 2020 ) , where the training data has been kept private but model samples are available through an API . Our work therefore targets a problem setting that has received little prior attention : given access only to a trained generative model , and no access to the dataset that trained it , can we learn effective visual representations ? To this end , we provide an exploratory study of representation learning in the setting of synthetic data sampled from pre-trained generative models : we analyze which representation learning algorithms are applicable , how well they work , and how they can be modified to make use of the special structure provided by deep generative networks . Figure . 1 lays out the framework we study : we compare learning visual embedding functions F from real data x ∼ { xi } Ni=1 vs. from generated data x ∼ G controlled via latent transformations . We study generation and representation learning both with and without class labels , and test representation learners based on several objectives . We evaluate representations via transfer performance on held out datasets and tasks . For representation learners , we focus primarily on contrastive methods that learn to associate multiple “ views ” of the same scene . These views may be co-occurring sensory signals such as imagery and sounds ( e.g. , ( de Sa , 1994 ) ) , or may be different augmented or transformed versions of the same image ( e.g. , ( Becker & Hinton , 1992 ; Chen et al. , 2020c ) ) . Interestingly , generative models can also create multiple views of an image : by steering in their latent spaces they can achieve camera and color transformations ( Jahanian et al. , 2020 ) and more ( Härkönen et al. , 2020 ; Wu et al. , 2020 ) . Figure 2 diagrams the currently popular setting , where views are generated as pixel-space transformations , versus the setting we focus on , where views are generated via latent-space transformations . We study the properties of these latent views in contrastive learning and ask under what conditions they can lead to enhanced learners . We then compare these learners with non-contrastive encoders , also trained on samples from generative models . Further , we study the long-standing promise of generate models in being capable of generating more samples than the number of datapoints that trained them . Knowing current models still have finite diversity , we ask : how many samples are necessary to get good performance on a task ? Our main findings are : 1 . Contrastive learning algorithms can be naturally extended to learning from generative samples , where different “ views ” of the data are created via transformations in the model ’ s latent space . 1E.g. , see https : //github.com/justinpinkney/awesome-pretrained-stylegan , hosting many pretrained models , without datasets . 2 . These latent-space transformations can be combined with standard data transformations ( “ augmentations ” ) in pixel-space to achieve better performance than either method alone . 3 . To generate positive training pairs in latent space , simple Gaussian transformations work as well as more complicated latent-space transformations , with the optimal standard deviation being not too large and not too small , obeying an inverse-U-shaped curve like that observed for contrastive learning from real data ( Tian et al. , 2020 ) . 4 . Generative models can potentially produce an unbounded number of samples to train on ; we find that performance improves by training on more samples , but sub-logarithmically . 5 . Learning representations from generative models can outperform learning directly from the data that trained those models , provided that the generative model is of sufficiently quality ( we observe this result on StyleGAN2 ( Karras et al. , 2020 ) fit to cars but not on BigBiGAN ( Donahue & Simonyan , 2019 ) or BigGAN ( Brock et al. , 2019 ) fit to ImageNet1000 ( Deng et al. , 2009 ) ) . 2 RELATED WORK . Learning from Synthetic Data . Using synthetic data has been a prominent method for learning in different domains of science and engineering , with different goals including privacy-preservation and alternative sample generation ( Sakshaug & Raghunathan , 2010 ; Tucker et al. , 2020 ; Nußberger et al. , 2020 ; Khan et al. , 2019 ; Dan et al. , 2020 ) . In computer vision , synthetic data has been extensively used as a source for training models , for example in semantic segmentation ( Chen et al. , 2019 ; Ros et al. , 2016 ) , human pose estimation ( Varol et al. , 2017 ; Ionescu et al. , 2013 ; Shakhnarovich et al. , 2003 ) , optical flow ( Mayer et al. , 2016 ; Fischer et al. , 2015 ) , or self-supervised multitask learning ( Ren & Lee , 2018 ) . In most prior work , the synthetic data comes from a traditional simulation pipeline , e.g. , via rendering a 3D world with a graphics engine . We instead study the setting where the synthetic data is sampled from a deep generative model . Recent works have used generative networks , such as GANs ( Goodfellow et al. , 2014 ) , to improve images generated by graphics engines , closing the domain gap with real images ( Shrivastava et al. , 2017 ; Hoffman et al. , 2018 ) . Ravuri & Vinyals ( 2019 ) show that even though there is a gap between GAN-generated images and real ones , mixing the two sources can lead to improvements over models trained purely on real data . GAN-generated images are also used to learn inverse graphics . Zhang et al . ( 2021a ) use StyleGAN to generate multiple views of a given object , allowing to extract 3D knowledge that can be mapped back into the latent space . Highly related to our paper is the work of Besnier et al . ( 2020 ) , which uses GAN-generated images to train a classifier . While they focus on using a class-conditional generator to train a classifier of those same classes , our work targets the more general setting of visual representation learning , and we mainly focus on the “ unsupervised ” setting where an unconditional generative model produces unlabeled samples to learn from . Recent works ( Yang et al. , 2021 ; Jeong & Shin , 2021 ) have studied combining contrastive and adversarial losses in GAN training , showing benefits in both image synthesis and representation learning . In contrast to our work , these approaches require access to real data during training . Recent work has also explored using StyleGAN ( Karras et al. , 2019 ) to generate training data for semantic segmentation ( Zhang et al. , 2021b ; Tritrong et al. , 2021 ) . These approaches differ from ours in that they use intermediate layers of the generator network as their representation of images , whereas our method does not require internal access to the generator ; instead we simply treat the generator as a black-box source of training data for a downstream representation learner . GAN-based data augmentation has also been used , in conjunction with real data , to improve data efficiency for GAN training ( Yang et al. , 2021 ) as well as robustness at training time ( Mao et al. , 2020 ) and test-time ( Chai et al. , 2021 ) . In contrast to these methods , we explore representation learning without access to the real data at all , relying only on samples from a black-box generative model to train our systems . Contrastive Representation Learning . Contrastive learning methods ( van den Oord et al. , 2019 ; Wu et al. , 2018 ; Hénaff et al. , 2019 ; Tian et al. , 2019 ; He et al. , 2020 ; Chen et al. , 2020c ) have greatly advanced the state of the art of self-supervised representation learning . The idea of contrastive learning is to contrast positive pairs with negative pairs ( Hadsell et al. , 2006 ) . Such pairs can be easily constructed with various data formats , and examples include different augmentations or transformations of images ( Chen et al. , 2020c ) , cross-modality alignment ( Tian et al. , 2019 ; Mor- gado et al. , 2020 ; Patrick et al. , 2020 ) , and graph structured data ( Hassani & Khasahmadi , 2020 ) . One key ingredient for the success of contrastive learning is well-chosen data transformations ( Chen et al. , 2020c ; Tian et al. , 2020 ; Xiao et al. , 2020 ) . While all these approaches conduct data transformations , or “ augmentations ” , in the raw pixel space , in this paper we explore the possibility of transforming training points in the latent space of a GAN . Generative Representation Learning . Generative models learn representations by modeling the data distribution p ( x ) . In VAEs ( Kingma & Welling , 2013 ) , each data point is encoded into a latent distribution , from which codes are sampled to reconstruct the input by maximizing the data likelihood . GANs ( Goodfellow et al. , 2014 ) model data generation through a minimax game , after which the discriminator can serve as a good representation extractor ( Radford et al. , 2015 ) . In ALI ( Dumoulin et al. , 2016 ) , BiGAN ( Donahue et al. , 2017 ) , and BigBiGAN ( Donahue & Simonyan , 2019 ) both image encoding and latent decoding are modeled simultaneously , and the encoder turns out to be a representation learner . Similar to the trend in natural language processing ( Devlin et al. , 2018 ; Radford et al . ) , autoregressive models ( Van den Oord et al. , 2016 ; Chen et al. , 2020b ) have been adopted to learn representations from raw pixels . These prior works show ways to jointly train a representation alongside a generative model , using a training set of real data for learning . Our exploration qualitatively differs in that we assume we are given a black-box generative model , and no real training data , and the goal is to learn an effective representation by sampling from the model .

Summary: The work investigates **generative models as a data source for self-supervised representation learning**. In particular, the authors propose to form contrastive pairs in the latent space of the generative model and combine it with standard contrastive learning where pairs are formed in image space.  While performance is **slightly inferior** to training with the full dataset, the proposed approach **only requires storing the model weights** of the generator instead of the full dataset.