Robust Conditional Generative Adversarial Networks

1 INTRODUCTION . Image-to-image translation and more generally conditional image generation lie at the heart of computer vision . Conditional Generative Adversarial Networks ( cGAN ) ( Mirza & Osindero , 2014 ) have become a dominant approach in the field , e.g . in dense1 regression ( Isola et al. , 2017 ; Pathak et al. , 2016 ; Ledig et al. , 2017 ; Bousmalis et al. , 2016 ; Liu et al. , 2017 ; Miyato & Koyama , 2018 ; Yu et al. , 2018 ; Tulyakov et al. , 2018 ) . They accept a source signal as input , e.g . prior information in the form of an image or text , and map it to the target signal ( image ) . The mapping of cGAN does not constrain the output to the target manifold , thus the output can be arbitrarily off the target manifold ( Vidal et al. , 2017 ) . This is a critical problem both for academic and commercial applications . To utilize cGAN or similar methods as a production technology , we need to study their generalization even in the face of intense noise . Similarly to regression , classification also suffers from sensitivity to noise and lack of output constraints . One notable line of research consists in complementing supervision with unsupervised learning modules . The unsupervised module forms a new pathway that is trained with the same , or different data samples . The unsupervised pathway enables the network to explore the structure that is not present in the labelled training set , while implicitly constraining the output . The addition of the unsupervised module is only required during the training stage and results in no additional computational cost during inference . Rasmus et al . ( 2015 ) and Zhang et al . ( 2016 ) modified the original bottom-up ( encoder ) network to include top-down ( decoder ) modules during training . However , in dense regression both bottom-up and top-down modules exist by default , and such methods are thus not trivial to extend to regression tasks . Motivated by the combination of supervised and unsupervised pathways , we propose a novel conditional GAN which includes implicit constraints in the latent subspaces . We coin this new model ‘ Robust Conditional GAN ’ ( RoCGAN ) . In the original cGAN the generator accepts a source signal and maps it to the target domain . In our work , we ( implicitly ) constrain the decoder to generate samples that span only the target manifold . We replace the original generator , i.e . encoder-decoder , 1The output includes at least as many dimensions as the input , e.g . super-resolution , or text-to-image translation . with a two pathway module ( see Fig . 1 ) . The first pathway , similarly to the cGAN generator , performs regression while the second is an autoencoder in the target domain ( unsupervised pathway ) . The two pathways share a similar network structure , i.e . each one includes an encoder-decoder network . The weights of the two decoders are shared which promotes the latent representations of the two pathways to be semantically similar . Intuitively , this can be thought of as constraining the output of our dense regression to span the target subspace . The unsupervised pathway enables the utilization of all the samples in the target domain even in the absence of a corresponding input sample . During inference , the unsupervised pathway is no longer required , therefore the testing complexity remains the same as in cGAN . In the following sections , we introduce our novel RoCGAN and study their ( theoretical ) properties . We prove that RoCGAN share similar theoretical properties with the original GAN , i.e . convergence and optimal discriminator . An experiment with synthetic data is designed to visualize the target subspaces and assess our intuition . We experimentally scrutinize the sensitivity of the hyper-parameters and evaluate our model in the face of intense noise . Moreover , thorough experimentation with both images from natural scenes and human faces is conducted in two different tasks . We compare our model with both the state-of-the-art cGAN and the recent method of Rick Chang et al . ( 2017 ) . The experimental results demonstrate that RoCGAN outperform the baseline by a large margin in all cases . Our contributions can be summarized as following : • We introduce RoCGAN that leverages structure in the target space . The goal is to promote robustness in dense regression tasks . • We scrutinize the model performance under ( extreme ) noise and adversarial perturbations . To the authors ’ knowledge , this robustness analysis has not been studied previously for dense regression . • We conduct a thorough experimental analysis for two different tasks . We outline how RoCGAN can be used in a semi-supervised learning task , how it performs with lateral connections from encoder to decoder . Notation : Given a set of N samples , s ( n ) denotes the nth conditional label , e.g . a prior image ; y ( n ) denotes the respective target image . Unless explicitly mentioned otherwise || · || will declare an ` 1 norm . The symbols L∗ define loss terms , while λ∗ denote regularization hyper-parameters optimized on the validation set . 2 RELATED WORK . Conditional image generation is a popular task in computer vision , dominated by approaches similar to cGAN . Apart from cGAN , the method by Isola et al . ( 2017 ) , widely known as ‘ pix2pix ’ , is the main alternative . Pix2pix includes three modifications over the baseline cGAN : i ) lateral skip connections between the encoder and the decoder network are added in the generator , ii ) the discriminator accepts pairs of source/gt and source/model output images , iii ) additional content loss terms are added . The authors demonstrate how those performance related modifications can lead to an improved visual outcome . Despite the improved performance , the problem with the additional guarantees remains the same . That is we do not have any direct supervision in the process , since both the latent subspace and the projection are learned ; the only supervision is provided by the ground-truth ( gt ) signal in the generator ’ s output . Adding regularization terms in the loss function can impose stronger supervision , thus restricting the output . The most frequent regularization term is feature matching , e.g . perceptual loss ( Ledig et al. , 2017 ; Johnson et al. , 2016 ) , or embeddings for faces ( Schroff et al. , 2015 ) . Feature matching minimizes the distance between the projection of generated and ground-truth signals . However , the pre-defined feature space is restrictive . The method introduced by Salimans et al . ( 2016 ) performs feature matching in the discriminator ; the motivation lies in matching the low-dimensional distributions created by the discriminator layers . Matching the discriminator ’ s features has demonstrated empirical success . However , this does not affect the generator and its latent subspaces directly . A new line of research that correlates with our goals is that of adversarial attacks ( Szegedy et al. , 2014 ; Yuan et al. , 2017 ; Samangouei et al. , 2018 ) . It is observed that perturbing input samples with a small amount of noise , often imperceptible to the human eye , can lead to severe classification errors . There are several techniques to ‘ defend ’ against such attacks . A recent example is the Fortified networks of Lamb et al . ( 2018 ) which uses Denoising Autoencoders ( Vincent et al. , 2008 ) to ensure that the input samples do not fall off the target manifold . Kumar et al . ( 2017 ) estimate the tangent space to the target manifold and use that to insert invariances to the discriminator for classification purposes . Even though RoCGAN share similarities with those methods , the scope is different since a ) the output of our method is high-dimensional2 and b ) adversarial examples are not extended to dense regression3 . Except for the study of adversarial attacks , combining supervised and unsupervised learning has been used for enhancing the classification performance . In the Ladder network , Rasmus et al . ( 2015 ) modify a typical bottom-up network for classification by adding a decoder and lateral connections between the encoder and the decoder . During training they utilize the augmented network as two pathways : i ) labelled input samples are fed to the initial bottom-up module , ii ) input samples are corrupted with noise and fed to the encoder-decoder with the lateral connections . The latter pathway is an autoencoder ; the idea is that it can strengthen the resilience of the network to samples outside the input manifold , while it improves the classification performance . Our core goal consists in constraining the model ’ s output . Aside from deep learning approaches , such constraints in manifolds were typically tackled with component analysis . Canonical correlation analysis ( Hotelling , 1936 ) has been extensively used for finding common subspaces that maximally correlate the data ( Panagakis et al. , 2016 ) . The recent work of Murdock et al . ( 2018 ) combines the expressiveness of neural networks with the theoretical guarantees of classic component analysis . 3 METHOD . In this section , we elucidate our proposed RoCGAN . To make the paper self-contained we first review the original conditional GAN model ( sec . 3.1 ) , before introducing RoCGAN ( sec . 3.2 ) . Sequentially , we pose the modifications required in case of shortcut connections from the encoder to the decoder ( sec . 3.3 ) . In sec . 3.4 we assess the intuition behind our model with synthetic data . The core idea in RoCGAN is to leverage structure in the output space of the model . We achieve that by 2In the classification tasks studied , e.g . the popular Imagenet ( Deng et al. , 2009 ) , there are up to a thousand classes , while our output includes tens or hundreds of thousands of dimensions . 3The robustness in our case refers to being resilient to changes in the distribution of the labels ( label shift ) and training set ( covariance shift ) ( Wang et al. , 2017 ) . replacing the single pathway in the generator with two pathways . In the appendix , we study the theoretical properties of our method and prove that RoCGAN share the same properties as the original GAN ( Goodfellow et al. , 2014 ) . 3.1 CONDITIONAL GAN . GAN consist of a generator and a discriminator module commonly optimized with alternating gradient descent methods . The generator samples z from a prior distribution pz , e.g . uniform , and tries to model the target distribution pd ; the discriminator D tries to distinguish between the samples generated from the model and the target ( ground-truth ) distributions . Conditional GAN ( cGAN ) ( Mirza & Osindero , 2014 ) extend the formulation by providing the generator with additional labels . In cGAN the generator G typically takes the form of an encoder-decoder network , where the encoder projects the label into a low-dimensional latent subspace and the decoder performs the opposite mapping , i.e . from low-dimensional to high-dimensional subspace . If we denote s the conditioning label and y a sample from the target distribution , the adversarial loss is expressed as : Ladv ( G , D ) = Es , y∼pd ( s , y ) [ logD ( y|s ) ] + Es∼pd ( s ) , z∼pz ( z ) [ log ( 1−D ( G ( s , z ) |s ) ) ] ( 1 ) by solving the following min-max problem : min wG max wD Ladv ( G , D ) = min wG max wD Es , y∼pd ( s , y ) [ logD ( y|s , wD ) ] + Es∼pd ( s ) , z∼pz ( z ) [ log ( 1−D ( G ( s , z|wG ) |s , wD ) ) ] where wG , wD denote the generator ’ s and the discriminator ’ s parameters respectively . To simplify the notation , we drop the dependencies on the parameters and the noise z in the rest of the paper . The works of Salimans et al . ( 2016 ) and Isola et al . ( 2017 ) demonstrate that auxiliary loss terms , i.e . feature matching and content loss , improve the final outcome , hence we consider those as part of the vanilla cGAN . The feature matching loss ( Salimans et al. , 2016 ) is : Lf = N∑ n=1 ||π ( G ( s ( n ) ) ) − π ( y ( n ) ) || ( 2 ) where π ( ) extracts the features from the penultimate layer of the discriminator . The final loss function for the cGAN is the following : L = Ladv + λc · N∑ n=1 ||G ( s ( n ) ) − y ( n ) ||︸ ︷︷ ︸ content−loss +λπ · Lf ( 3 ) where λc , λπ are hyper-parameters to balance the loss terms .

Summary: In general, this is a well-written paper. This work focuses on the robustness of conditional GAN(RoC-GAN) when facing the noise. The authors claim the generator of RoC-Gan will span the target manifold, even in the presence of large amounts of noise. The main contribution of the paper is to introduce a two-pathway model, where one of them is used to perform regression as ordinary GAN while the other one helps the whole model span the target domain.