Stochastic Conditional Generative Networks with Basis Decomposition

1 INTRODUCTION . Conditional image generation networks learn mappings from the condition domain to the image domain by training on massive samples from both domains . The mapping from a condition , e.g. , a map , to an image , e.g. , a satellite image , is essentially one-to-many as illustrated in Figure 1 . In other words , there exists many plausible output images that satisfy a given input condition , which motivates us to explore multi-mode conditional image generation that produces diverse images conditioned on one single input condition . One technique to improve image generation diversity is to feed the image generator with an additional latent code in the hope that such code can carry information that is not covered by the input condition , so that diverse output images are achieved by decoding the missing information conveyed through different latent codes . However , as illustrated in the seminal work Isola et al . ( 2017 ) , encoding the diversity with an input latent code can lead to unsatisfactory performance for the following reasons . While training using objectives like GAN loss Goodfellow et al . ( 2014 ) , regularizations like L1 loss Isola et al . ( 2017 ) and perceptual loss Wang et al . ( 2018 ) are imposed to improve both visual fidelity and correspondence to the input condition . However , no similar regularization is imposed to enforce the correspondence between outputs and latent codes , so that the network is prone to ignore input latent codes in training , and produce identical images from an input condition even with different latent codes . Several methods are proposed to explicitly encourage the network to take into account input latent codes to encode diversity . For example , Mao et al . ( 2019 ) explicitly maximizes the ratio of the distance between generated images with respect to the corresponding latent codes ; while Zhu et al . ( 2017b ) applies an auxiliary network for decoding the latent codes from the generative images . Although the diversity of the generative images is significantly improved , these methods experience drawbacks . In Mao et al . ( 2019 ) , at least two samples generated from the same condition are needed for calculating the regularization term , which multiplies the memory footprint while training each mini-batch . Auxiliary network structures and training objectives in Zhu et al . ( 2017b ) unavoidably increase training difficulty and memory footprint . These previously proposed methods usually require considerable modifications to the underlying framework . In this paper , we propose a stochastic model , BasisGAN , that directly maps an input condition to diverse output images , aiming at building networks that model the multi-mode intrinsically . The proposed method exploits a known observation that a well-trained deep network can converge to significantly different sets of parameters across multiple trainings , due to factors such as different parameter initializations and different choices of mini-batches . Therefore , instead of treating a conditional image generation network as a deterministic function with fixed parameters , we propose modeling the filter in each convolutional layer as a sample from filter space , and learning the corresponding filter space using a tiny network for efficient and diverse filter sampling . In Ghosh et al . ( 2018 ) , parameter non-uniqueness is used for multi-mode image generation by training several generators with different parameters simultaneously as a multi-agent solution . However , the maximum modes of Ghosh et al . ( 2018 ) are restricted by the number of agents , and the replication increases memory as well as computational cost . Based on the above parameters non-uniqueness property , we introduce into a deep network stochastic convolutional layers , where filters are sampled from learned filter spaces . Specifically , we learn the mapping from a simple prior to the filter space using neural networks , here referred to as filter generators . To empower a deterministic network with multi-mode image generation , we divide the network into a deterministic sub-model and a stochastic sub-model as shown in Figure 1 , where standard convolutional layers and stochastic convolutional layers with filter generators are deployed , respectively . By optimizing an adversarial loss , filter generators can be jointly trained with a conditional image generation network . In each forward pass , filters at stochastic layers are sampled by filter generators . Highly diverse images conditioned on the same input are achieved by jointly sampling of filters in multiple stochastic convolutional layers . However , filters of a convolutional layer are usually high-dimensional while being together written as one vector , which makes the modeling and sampling of a filter space highly costly in practice in terms of training time , sampling time , and filter generator memory footprint . Based on the low-rank property observed from sampled filters , we decompose each filter as a linear combination of a small set of basis elements Qiu et al . ( 2018 ) , and propose to only sample low-dimensional spatial basis elements instead of filters . By replacing filter generators with basis generators , the proposed method becomes highly efficient and practical . Theoretical arguments are provided on how perturbations introduced by sampling basis elements can propagate to the appearance of generated images . The proposed BasisGAN introduces a generalizable concept to promote diverse modes in the conditional image generation . As basis generators act as plug-and-play modules , variants of BasisGAN can be easily constructed by replacing in various state-of-the-art conditional image generation net- works the standard convolutional layers by stochastic layers with basis generators . Then , we directly train them without additional auxiliary components , hyperparameters , or training objectives on top of the underlying models . Experimental results consistently show that the proposed BasisGAN is a simple yet effective solution to multi-mode conditional image generation . We further empirically show that the inherent stochasticity introduced by our method allows training without paired samples , and the one-to-many image-to-image translation is achieved using a stochastic auto-encoder where stochasticity prevents the network from learning a trivial identity mapping . Our contributions are summarized as follows : • We propose a plug-and-played basis generator to stochastically generate basis elements , with just a few hundred of parameters , to fully embed stochasticity into network filters . • Theoretic arguments are provided to support the simplification of replacing stochastic filter generation with basis generation . • Both the generation fidelity and diversity of the proposed BasisGAN with basis generators are validated extensively , and state-of-the-art performances are consistently observed . 2 RELATED WORK . Conditional image generation . Parametric modeling of the natural image distribution has been studied for years , from restricted Boltzmann machines Smolensky ( 1986 ) to variational autoencoders Kingma & Welling ( 2013 ) ; in particular variants with conditions Oord et al . ( 2016 ) ; Sohn et al . ( 2015 ) ; Van den Oord et al . ( 2016 ) show promising results . With the great power of GANs Goodfellow et al . ( 2014 ) , conditional generative adversarial networks ( cGANs ) Isola et al . ( 2017 ) ; Pathak et al . ( 2016 ) ; Sangkloy et al . ( 2017 ) ; Wang et al . ( 2018 ) ; Xian et al . ( 2018 ) ; Zhu et al . ( 2017a ) achieve great progress on visually appealing images given conditions . However , the quality of images and the loyalty to input conditions come with sacrifice on image diversity as discussed in Zhu et al . ( 2017b ) , which is addressed by the proposed BasisGAN . Multi-mode conditional image generation . To enable the cGANs with multi-mode image generation , pioneer works like infoGAN Chen et al . ( 2016 ) and pix2pix Isola et al . ( 2017 ) propose to encode the diversity in an input latent code . To enforce the networks to take into account input latent codes , Zhu et al . ( 2017b ) deploys auxiliary networks and training objectives to impose the recovery of the input latent code from the generated images . MSGAN Mao et al . ( 2019 ) and DSGAN Yang et al . ( 2019 ) propose regularization terms for diversity that enforces a larger distance between generated images with respect to different input latent codes given one input condition . These methods require considerable modifications to the underlying original framework . Neural network parameters generating and uncertainty . Extensive studies have been conducted for generating network parameters using another network since Hypernetworks Ha et al . ( 2016 ) . As a seminal work on network parameter modeling , Hypernetworks successfully reduce learnable parameters by relaxing weight-sharing across layers . Followup works like Bayesian Hypernetworks Krueger et al . ( 2017 ) further introduce uncertainty to the generated parameters . Variational inference based methods like Bayes by Backprop Blundell et al . ( 2015 ) solve the intractable posterior distribution of parameters by assuming a prior ( usually Gaussian ) . However , the assumed prior unavoidably degrades the expressiveness of the learned distribution . The parameter prediction of neural network is intensively studied under the context of few shot learning Bertinetto et al . ( 2016 ) ; Qiao et al . ( 2018 ) ; Wang et al . ( 2019 ) , which aims to customize a network to a new task adaptively and efficiently in a data-driven way . Apart from few shot learning , Denil et al . ( 2013 ) suggests parameter prediction as a way to study the redundancy in neural networks . While studying the representation power of random weights , Saxe et al . ( 2011 ) also suggests the uncertainty and non-uniqueness of network parameters . Another family of network with uncertainty is based on variational inference Blundell et al . ( 2015 ) , where an assumption of the distribution on network weights is imposed for a tractable learning on the distribution of weights . Works on studying the relationship between local and global minima of deep networks Haeffele & Vidal ( 2015 ) ; Vidal et al . ( 2017 ) also suggest the non-uniqueness of optimal parameters of a deep network . 3 STOCHASTIC FILTER GENERATION . A conditional generative network ( cGAN ) Mirza & Osindero ( 2014 ) learns the mapping from input condition domain A to output image domain B using a deep neural network . The conditional image generation is essentially a one-to-many mapping as there could be multiple plausible instances B ∈ B that map to a condition A ∈ A Zhu et al . ( 2017b ) , corresponding to a distribution p ( B|A ) . However , the naive mapping of the generator formulated by a neural network G : A → B is deterministic , and is incapable of covering the distribution p ( B|A ) . We exploit the non-uniqueness of network parameters as discussed above , and introduce stochasticity into convolutional filters through plug-and-play filter generators . To achieve this , we divide a network into two sub-models : • A deterministic sub-model with convolutional filters φ that remain fixed after training ; • A stochastic sub-model whose convolutional filters w are sampled from parameter spaces modeled by neural networks T , referred to as filter generators , parametrized by θ with inputs z from a prior distribution , e.g. , N ( 0 , I ) for all experiments in this paper . Note that filters in each stochastic layer are modeled with a separate neural network , which is not explicitly shown in the formulation for notation brevity . With this formulation , the conditional image generation becomes Gφ , θ : A → B , with stochasticity achieved by sampling filters w = Tθ ( z ) for the stochastic sub-model in each forward pass . The conditional GAN loss Goodfellow et al . ( 2014 ) ; Mirza & Osindero ( 2014 ) then becomes min G max D V ( D , G ) =EA∼p ( A ) , B∼p ( B|A ) [ log ( D ( A , B ) ) ] + EA∼p ( A ) , z∼p ( z ) [ log ( 1−D ( A , Gφ , θ ( A ; Tθ ( z ) ) ) ) ] , ( 1 ) where D denotes a standard discriminator . Note that we represent the generator here as Gφ , θ ( A ; Tθ ( z ) ) to emphasize that the generator uses stochastic filters w = Tθ ( z ) . Given a stochastic generative network parametrized by φ and θ , and input condition A , the generated images form a conditional probability qφ , θ ( B|A ) , so that ( 1 ) can be simplified as min G max D V ( D , G ) =EA∼p ( A ) , B∼p ( B|A ) logD ( A , B ) + EA∼p ( A ) , B∼qφ , θ ( B|A ) log [ 1−D ( A , B ) ] . ( 2 ) When the optimal discriminator is achieved , ( 2 ) can be reformulated as C ( G ) = max D V ( D , G ) = EA∼p ( A ) [ − log ( 4 ) + 2 · JSD ( p ( B|A ) ||qφ , θ ( B|A ) ) ] , ( 3 ) where JSD is the Jensen-Shannon divergence ( the proof is provided in the supplementary material ) . The global minimum of ( 3 ) is achieved when given every sampled condition A , the generator perfectly replicates the true distribution p ( B|A ) , which indicates that by directly optimizing the loss in ( 1 ) , conditional image generation with diversity is achieved with the proposed stochasticity in the convolutional filters . To optimize ( 1 ) , we train D as in Goodfellow et al . ( 2014 ) to maximize the probability of assigning the correct label to both training examples and samples fromGφ , θ . Simultaneously , we trainGφ , θ to minimize the following loss , where filter generators Tθ are jointly optimized to bring stochasticity : L = EA∼p ( A , B ) , z∼p ( z ) [ log ( 1−D ( A , Gφ , θ ( A ; Tθ ( z ) ) ) ) ] . ( 4 ) We describe in detail the optimization of the generator parameters { φ , θ } in supplementary material Algorithm 1 . Discussions on diversity modeling in cGANs . The goal of cGAN is to model the conditional probability p ( B|A ) . Previous cGAN models Mao et al . ( 2019 ) ; Mirza & Osindero ( 2014 ) ; Zhu et al . ( 2017b ) typically incorporate randomness in the generator by setting B = G ( A , z ) , z ∼ p ( z ) , where G is a deep network with deterministic parametrization and the randomness is introduced via z , e.g. , a latent code , as an extra input . This formulation implicitly makes the following two assumptions : ( A1 ) The randomness of the generator is independent from that of p ( A ) ; ( A2 ) Each realization B ( ω ) conditional on A can be modeled by a CNN , i.e. , B = Gω ( A ) , where Gω is a draw from an ensemble of CNNs , ω being the random event . ( A1 ) is reasonable as long as the source of variation to be modeled by cGAN is independent from that contained in A , and the rational of ( A2 ) lies in the expressive power of CNNs for image to image translation . The previous model adopts a specific form of Gω ( A ) via feeding random input z ( ω ) to G , yet one may observe that the most general formulation under ( A1 ) , ( A2 ) would be to sample the generator itself from certain distribution p ( G ) , which is independent from p ( A ) . Since generative CNNs are parametrized by convolutional filters , this would be equivalent to set B = G ( A ; w ) , w ∼ p ( w ) , where we use “ ; ” in the parentheses to emphasize that what after is parametrization of the generator . The proposed cGAN model in the current paper indeed takes such an approach , where we model p ( w ) by a separate filter generator network .

Summary: In this paper, the authors introduce BasisGAN, a novel method for introducing stochasticity in conditional GANs, i.e., a way of conducting one-to-many mappings. This is a good addition in the literature as: (a) most of the widely-used conditional GANs such as pix2pix (Isola et al., 2016) or pix2pixHD (Wang et al., 2018) are deterministic (i.e., for a specific input a single output is always generated), (b) it improves upon the current SOTA in one-to-many mappings, (c) it is very useful application-wise. As also stated in the paper, there is a number of applications where this method is handy (e.g., converting a sketch to images varying in colors, etc.).