Generative Models as a Data Source for Multiview Representation Learning

1 INTRODUCTION . The last few years have seen great progress in the diversity and quality of generative models . For almost every popular data domain there is now a generative model that produces realistic samples from that domain , be it images ( Radford et al. , 2015 ; Brock et al. , 2019 ; Ramesh et al. , 2021 ) , music ( Dhariwal et al. , 2020 ) , or text ( Brown et al. , 2020 ) . This raises an intriguing possibility : what we used to do with real data , can we now do instead with synthetic data , sampled from a generative model ? If so , there would be immediate advantages . Models are highly compressed compared to the datasets they represent and therefore easier to share and store . Synthetic data also circumvents some of the concerns around privacy and usage rights that limit the distribution of real datasets ( Tucker et al. , 2020 ) , and models can be edited to censor sensitive attributes ( Liao et al. , 2019 ) , remove biases that exist in real datasets ( Tan et al. , 2020 ; Ramaswamy et al. , 2020 ) , or steer toward other taskspecific needs ( Jahanian et al. , 2020 ; Goetschalckx et al. , 2019 ; Shen et al. , 2020 ) . Perhaps because of these advantages , it is becoming increasingly common for pre-trained generative models to be shared online without their original training data being made easily accessible . This approach has been taken by individuals who may not have the resources or intellectual property rights to release the original data1 and in the case of large-scale models such as GPT-3 ( Brown et al. , 2020 ) , where the training data has been kept private but model samples are available through an API . Our work therefore targets a problem setting that has received little prior attention : given access only to a trained generative model , and no access to the dataset that trained it , can we learn effective visual representations ? To this end , we provide an exploratory study of representation learning in the setting of synthetic data sampled from pre-trained generative models : we analyze which representation learning algorithms are applicable , how well they work , and how they can be modified to make use of the special structure provided by deep generative networks . Figure . 1 lays out the framework we study : we compare learning visual embedding functions F from real data x ∼ { xi } Ni=1 vs. from generated data x ∼ G controlled via latent transformations . We study generation and representation learning both with and without class labels , and test representation learners based on several objectives . We evaluate representations via transfer performance on held out datasets and tasks . For representation learners , we focus primarily on contrastive methods that learn to associate multiple “ views ” of the same scene . These views may be co-occurring sensory signals such as imagery and sounds ( e.g. , ( de Sa , 1994 ) ) , or may be different augmented or transformed versions of the same image ( e.g. , ( Becker & Hinton , 1992 ; Chen et al. , 2020c ) ) . Interestingly , generative models can also create multiple views of an image : by steering in their latent spaces they can achieve camera and color transformations ( Jahanian et al. , 2020 ) and more ( Härkönen et al. , 2020 ; Wu et al. , 2020 ) . Figure 2 diagrams the currently popular setting , where views are generated as pixel-space transformations , versus the setting we focus on , where views are generated via latent-space transformations . We study the properties of these latent views in contrastive learning and ask under what conditions they can lead to enhanced learners . We then compare these learners with non-contrastive encoders , also trained on samples from generative models . Further , we study the long-standing promise of generate models in being capable of generating more samples than the number of datapoints that trained them . Knowing current models still have finite diversity , we ask : how many samples are necessary to get good performance on a task ? Our main findings are : 1 . Contrastive learning algorithms can be naturally extended to learning from generative samples , where different “ views ” of the data are created via transformations in the model ’ s latent space . 1E.g. , see https : //github.com/justinpinkney/awesome-pretrained-stylegan , hosting many pretrained models , without datasets . 2 . These latent-space transformations can be combined with standard data transformations ( “ augmentations ” ) in pixel-space to achieve better performance than either method alone . 3 . To generate positive training pairs in latent space , simple Gaussian transformations work as well as more complicated latent-space transformations , with the optimal standard deviation being not too large and not too small , obeying an inverse-U-shaped curve like that observed for contrastive learning from real data ( Tian et al. , 2020 ) . 4 . Generative models can potentially produce an unbounded number of samples to train on ; we find that performance improves by training on more samples , but sub-logarithmically . 5 . Learning representations from generative models can outperform learning directly from the data that trained those models , provided that the generative model is of sufficiently quality ( we observe this result on StyleGAN2 ( Karras et al. , 2020 ) fit to cars but not on BigBiGAN ( Donahue & Simonyan , 2019 ) or BigGAN ( Brock et al. , 2019 ) fit to ImageNet1000 ( Deng et al. , 2009 ) ) . 2 RELATED WORK . Learning from Synthetic Data . Using synthetic data has been a prominent method for learning in different domains of science and engineering , with different goals including privacy-preservation and alternative sample generation ( Sakshaug & Raghunathan , 2010 ; Tucker et al. , 2020 ; Nußberger et al. , 2020 ; Khan et al. , 2019 ; Dan et al. , 2020 ) . In computer vision , synthetic data has been extensively used as a source for training models , for example in semantic segmentation ( Chen et al. , 2019 ; Ros et al. , 2016 ) , human pose estimation ( Varol et al. , 2017 ; Ionescu et al. , 2013 ; Shakhnarovich et al. , 2003 ) , optical flow ( Mayer et al. , 2016 ; Fischer et al. , 2015 ) , or self-supervised multitask learning ( Ren & Lee , 2018 ) . In most prior work , the synthetic data comes from a traditional simulation pipeline , e.g. , via rendering a 3D world with a graphics engine . We instead study the setting where the synthetic data is sampled from a deep generative model . Recent works have used generative networks , such as GANs ( Goodfellow et al. , 2014 ) , to improve images generated by graphics engines , closing the domain gap with real images ( Shrivastava et al. , 2017 ; Hoffman et al. , 2018 ) . Ravuri & Vinyals ( 2019 ) show that even though there is a gap between GAN-generated images and real ones , mixing the two sources can lead to improvements over models trained purely on real data . GAN-generated images are also used to learn inverse graphics . Zhang et al . ( 2021a ) use StyleGAN to generate multiple views of a given object , allowing to extract 3D knowledge that can be mapped back into the latent space . Highly related to our paper is the work of Besnier et al . ( 2020 ) , which uses GAN-generated images to train a classifier . While they focus on using a class-conditional generator to train a classifier of those same classes , our work targets the more general setting of visual representation learning , and we mainly focus on the “ unsupervised ” setting where an unconditional generative model produces unlabeled samples to learn from . Recent works ( Yang et al. , 2021 ; Jeong & Shin , 2021 ) have studied combining contrastive and adversarial losses in GAN training , showing benefits in both image synthesis and representation learning . In contrast to our work , these approaches require access to real data during training . Recent work has also explored using StyleGAN ( Karras et al. , 2019 ) to generate training data for semantic segmentation ( Zhang et al. , 2021b ; Tritrong et al. , 2021 ) . These approaches differ from ours in that they use intermediate layers of the generator network as their representation of images , whereas our method does not require internal access to the generator ; instead we simply treat the generator as a black-box source of training data for a downstream representation learner . GAN-based data augmentation has also been used , in conjunction with real data , to improve data efficiency for GAN training ( Yang et al. , 2021 ) as well as robustness at training time ( Mao et al. , 2020 ) and test-time ( Chai et al. , 2021 ) . In contrast to these methods , we explore representation learning without access to the real data at all , relying only on samples from a black-box generative model to train our systems . Contrastive Representation Learning . Contrastive learning methods ( van den Oord et al. , 2019 ; Wu et al. , 2018 ; Hénaff et al. , 2019 ; Tian et al. , 2019 ; He et al. , 2020 ; Chen et al. , 2020c ) have greatly advanced the state of the art of self-supervised representation learning . The idea of contrastive learning is to contrast positive pairs with negative pairs ( Hadsell et al. , 2006 ) . Such pairs can be easily constructed with various data formats , and examples include different augmentations or transformations of images ( Chen et al. , 2020c ) , cross-modality alignment ( Tian et al. , 2019 ; Mor- gado et al. , 2020 ; Patrick et al. , 2020 ) , and graph structured data ( Hassani & Khasahmadi , 2020 ) . One key ingredient for the success of contrastive learning is well-chosen data transformations ( Chen et al. , 2020c ; Tian et al. , 2020 ; Xiao et al. , 2020 ) . While all these approaches conduct data transformations , or “ augmentations ” , in the raw pixel space , in this paper we explore the possibility of transforming training points in the latent space of a GAN . Generative Representation Learning . Generative models learn representations by modeling the data distribution p ( x ) . In VAEs ( Kingma & Welling , 2013 ) , each data point is encoded into a latent distribution , from which codes are sampled to reconstruct the input by maximizing the data likelihood . GANs ( Goodfellow et al. , 2014 ) model data generation through a minimax game , after which the discriminator can serve as a good representation extractor ( Radford et al. , 2015 ) . In ALI ( Dumoulin et al. , 2016 ) , BiGAN ( Donahue et al. , 2017 ) , and BigBiGAN ( Donahue & Simonyan , 2019 ) both image encoding and latent decoding are modeled simultaneously , and the encoder turns out to be a representation learner . Similar to the trend in natural language processing ( Devlin et al. , 2018 ; Radford et al . ) , autoregressive models ( Van den Oord et al. , 2016 ; Chen et al. , 2020b ) have been adopted to learn representations from raw pixels . These prior works show ways to jointly train a representation alongside a generative model , using a training set of real data for learning . Our exploration qualitatively differs in that we assume we are given a black-box generative model , and no real training data , and the goal is to learn an effective representation by sampling from the model .

Summary: This is an exploratory study of the effects of performing representation learning based on a generative model trained on data, as opposed to the data itself. It is very well written and coherently laid out, and was a pleasure to read. There are no theoretical advances, but the paper provides a useful first foray into an interesting topic, based on well executed empirical studies.