RPGAN: random paths as a latent space for GAN interpretability

In this paper , we introduce Random Path Generative Adversarial Network ( RPGAN ) — an alternative design of GANs that can serve as a tool for generative model analysis . While the latent space of a typical GAN consists of input vectors , randomly sampled from the standard Gaussian distribution , the latent space of RPGAN consists of random paths in a generator network . As we show , this design allows to understand factors of variation , captured by different generator layers , providing their natural interpretability . With experiments on standard benchmarks , we demonstrate that RPGAN reveals several interesting insights about the roles that different layers play in the image generation process . Aside from interpretability , the RPGAN model also provides competitive generation quality and allows efficient incremental learning on new data . 1 INTRODUCTION . Nowadays , deep generative models are an active research direction in the machine learning community . The dominant methods for generative modeling , such as Generative Adversarial Networks ( GANs ) , are currently able to produce diverse photorealistic images ( Brock et al. , 2019 ; Karras et al. , 2019 ) . These methods are not only popular among academicians , but are also a crucial component in a wide range of applications , including image editing ( Isola et al. , 2017 ; Zhu et al. , 2017 ) , super-resolution ( Ledig et al. , 2017 ) , video generation ( Wang et al. , 2018 ) and many others . Along with practical importance , a key benefit of accurate generative models is a more complete understanding of the internal structure of the data . Insights about the data generation process can result both in the development of new machine learning techniques as well as advances in industrial applications . However , most state-of-the-art generative models employ deep multi-layer architectures , which are difficult to interpret or explain . While many works investigate interpretability of discriminative models ( Zeiler & Fergus , 2014 ; Simonyan et al. , 2013 ; Mahendran & Vedaldi , 2015 ) , only a few ( Chen et al. , 2016 ; Bau et al. , 2019 ) address the understanding of generative ones . In this work , we propose the Random Path GAN ( RPGAN ) — an alternative design of GANs that allows natural interpretability of the generator network . In traditional GAN generators , the stochastic component that influences individual samples is a noisy input vector , typically sampled from the standard Gaussian distribution . In contrast , RPGAN generators instead use stochastic routing during the forward pass as their source of stochasticity . In a nutshell , the RPGAN generator contains several instances of the corresponding layer . For each sample , only one random instance of each layer is activated during generation . The training of the RPGAN can then be performed in the same adversarial manner as in traditional GANs . In the sections below , we show how RPGAN allows to understand the factors of variation captured by the particular layer and reveals several interesting findings about the image generation process , e.g . that different layers are “ responsible for ” coloring or objection location . As a practical advantage , RPGANs can be efficiently updated to new data via the simple addition of new instances to the bucket , avoiding re-training the full model from scratch . Finally , we observe that RPGANs allow the construction of generative models without nonlinearities , which can significantly speed up the generation process for fully-connected layers . In summary , the main contributions of our paper are the following : • We introduce RPGAN — GAN with an alternative source of stochasticity , based on random routing . While being close to traditional GANs in terms of generation quality , RPGAN allows natural interpretability and efficient model updates with new data . • With extensive experiments on standard benchmarks we reveal several insights about the image generation process . Many of our insights confirm and extend recent findings from Bau et al . ( 2019 ) . Note , that our scheme is more general compared to the technique from Bau et al . ( 2019 ) as RPGAN does not require labeled datasets or pretrained segmentation models . • We open-source the PyTorch implementation of RPGAN with common generator architectures1 . The rest of this paper is organized as follows . In Section 2 we review relevant ideas from prior art . The proposed Random Path GAN design is described in Section 3 and experimentally evaluated in Section 4 . Section 5 concludes the paper and discusses possible directions for future work . 2 RELATED WORK . In this section we briefly describe connections of RPGAN to existing ideas from prior works Generative adversarial networks . GANs are currently one of the main paradigms in generative modelling . Since the seminal paper on GANs by Goodfellow et al . ( 2014 ) , a plethora of alternative loss functions , architectures , normalizations , and regularization techniques were developed ( Kurach et al. , 2019 ) . Today , state-of-the-art GANs are able to produce high-fidelity images , often indistinguishable from real ones ( Brock et al. , 2019 ; Karras et al. , 2019 ) . In essence , GANs consist of two networks – a generator and a discriminator , which are trained jointly in an adversarial manner . In standard GANs , the generation stochasticity is provided by the input noise vector . In RPGANs , we propose an alternative source of stochasticity by using a fixed input but random routes during forward pass in the generator . Specific GAN architectures . Many prior works investigated different design choices for GANs , but to the best of our knowledge , none of them explicitly aimed to propose an interpretable GAN model . Hoang et al . ( 2018 ) proposed the use of several independent generators to address the mode collapse problem . Chavdarova & Fleuret ( 2018 ) employ several auxiliary local generators and discriminators to improve mode coverage as well . Huang et al . ( 2017 ) use layer-wise generators and discriminators to enforce hidden representations produced by layers of the generator to be similar to the corresponding representations produced by a reversed classification network . Important differences of RPGAN compared to the works described above is that it uses random routes as its latent space and does not enforce to mimic the latent representations of pretrained classifiers . Interpretability . While interpretability of models based on deep neural networks is an important research direction , most existing work addresses the interpretability of discriminative models . These works typically aim to understand the internal representations of networks ( Zeiler & Fergus , 2014 ; Simonyan et al. , 2013 ; Mahendran & Vedaldi , 2015 ; Dosovitskiy & Brox , 2016 ) or explain decisions produced by the network for particular samples ( Sundararajan et al. , 2017 ; Bach et al. , 2015 ; Simonyan et al. , 2013 ) . However , only a few works address interpretability of generative models . A related work by Bau et al . ( 2019 ) develops a technique that allows to identify which parts of the generator are responsible for the generation of different objects . In contrast , we propose GANs with alternative source of stochasticity that allows natural interpretation by design . Some of our findings confirm the results from Bau et al . ( 2019 ) , which provides stronger evidence about the responsibilities of different layers in the generation process . Note , that the technique ( Bau et al. , 2019 ) requires a pretrained segmentation network and can not be directly applied to several benchmarks , e.g . CIFAR10 or MNIST . In contrast , RPGAN does not require any auxiliary models or supervision and can be applied to any data . 1https : //github.com/rpgan-ICLR2020/RPGAN 3 RANDOM PATH GAN . 3.1 MOTIVATION . Before the formal description , we provide an intuition behind the RPGAN model . Several prior works have demonstrated that in discriminative convolutional neural networks different layers are “ responsible ” for different levels of abstraction ( Zeiler & Fergus , 2014 ; Babenko et al. , 2014 ) . For instance , earlier layers aim to detect small texture patterns , while activations in deeper layers typically correspond to semantically meaningful concepts . Similarly , in our paper we aim to understand the roles that different GAN layers play in image generation . Thus , we propose an architecture that provides a direct way to interpret the impact of individual layers . For a given generator architecture , we construct several copies of each layer in its architecture . During the forward pass we randomly choose a layer instance that will be used when generating a particular image . Therefore , we can analyze the role of each RPGAN layer by visualizing how different instances of that layer affect the generated image . 3.2 MODEL . Here we formally describe the structure of the RPGAN model . The model is highly flexible to the choice of generator and discriminator architectures as well as to the loss function and learning strategy . Similarly to the standard GAN architectures , our model consists of two networks – a generator and a discriminator . The RPGAN discriminator operates exactly like discriminators in common GANs , hence below we focus on the generator description . Unlike existing GANs , the RPGAN generator always receives a fixed input vector Z during forward pass and aims to produce an image from the real image distribution . The generator consists of several consequent buckets B1 , . . . , Bn . Each bucket is a union of independent blocks : Bi= { Bi1 , . . . , Bimi } , where each block is an arbitrary computational unit and mi=|Bi| . A typical example of a block is a ResNet block ( He et al. , 2015 ) , a convolutional layer with a nonlinearity or any other ( see Figure 1a ) layer type . In our experiments , all the units from the same bucket have the same architecture . For each i=1 , . . . , n − 1 a block from the bucket Bi produces an intermediate output tensor that is passed to a block from the next bucket Bi+1 . Typically we associate each bucket with a layer ( or several layers ) in the generator architecture , which we aim to interpret or analyze . A block from the first bucket B1 always receives a fixed input vector Z , which is the same for different forward passes . The stochasticity of the generator arises from a random path that goes from Z to an output image , using only a single block from each bucket . Formally , during each forward pass , we randomly choose indices s1 , . . . , sn with 1 ≤ si ≤ mi . The generator output is then computed as Bnsn ◦ · · ·B2s2 ◦ B1s1 ( Z ) ( see Figure 1b ) . Thus , the generator defines a map from the Cartesian product 〈m1〉 × 〈m2〉 × · · · × 〈mn〉 to the image space . Note that we can take an arbitrary existing GAN model , group its generator layers into buckets and replicate them into multiple blocks . In these terms , the original model can be treated as the RPGAN model with a single block in each bucket and random input noise . Note that during image generation we perform the same number of operations as in the standard GAN generator . By its design , RPGAN with buckets B1 , . . . , Bn and a constant input Z is able to generate at most |B1|×· · ·×|Bn| different samples were |Bk| is the number of blocks in the bucketBk . Nevertheless , this number is typically much larger compared to the training set size . We argue that the probability space of random paths can serve as a latent space to generate high-quality images , as confirmed by the experiments below . Block diversity loss . To guarantee that blocks in a particular bucket are different , we also add a specific diversity term in the generator loss function . The motivation for this term is to prevent blocksBki , Bkj from learning the same weights . LetW be the set of all parameters of the generator . For each parameter w ∈ W there is a set of its instances { w ( 1 ) , . . . w ( mw ) } in the RPGAN model . Then we enforce the instances to be different by the loss term − ∑ w∈W , i 6=j MSE ( w ( i ) sw , w ( j ) sw ) Here we also normalize by the standard deviation sw of all parameters from different blocks that correspond to the same layer . This normalization effectively guarantees that all buckets contribute to the diversity term .

Summary: This paper proposes the Random Path Generative Adversarial Network (RP-GAN) to serve as a tool for generative model analysis.  The main idea is to have several different buckets in each block of the generator and then train the generator with random paths. To interpret the features captured by each block, the authors unfreeze one block and show the variance of the generated images via different buckets.