Robust Conditional Generative Adversarial Networks

1 INTRODUCTION . Image-to-image translation and more generally conditional image generation lie at the heart of computer vision . Conditional Generative Adversarial Networks ( cGAN ) ( Mirza & Osindero , 2014 ) have become a dominant approach in the field , e.g . in dense1 regression ( Isola et al. , 2017 ; Pathak et al. , 2016 ; Ledig et al. , 2017 ; Bousmalis et al. , 2016 ; Liu et al. , 2017 ; Miyato & Koyama , 2018 ; Yu et al. , 2018 ; Tulyakov et al. , 2018 ) . They accept a source signal as input , e.g . prior information in the form of an image or text , and map it to the target signal ( image ) . The mapping of cGAN does not constrain the output to the target manifold , thus the output can be arbitrarily off the target manifold ( Vidal et al. , 2017 ) . This is a critical problem both for academic and commercial applications . To utilize cGAN or similar methods as a production technology , we need to study their generalization even in the face of intense noise . Similarly to regression , classification also suffers from sensitivity to noise and lack of output constraints . One notable line of research consists in complementing supervision with unsupervised learning modules . The unsupervised module forms a new pathway that is trained with the same , or different data samples . The unsupervised pathway enables the network to explore the structure that is not present in the labelled training set , while implicitly constraining the output . The addition of the unsupervised module is only required during the training stage and results in no additional computational cost during inference . Rasmus et al . ( 2015 ) and Zhang et al . ( 2016 ) modified the original bottom-up ( encoder ) network to include top-down ( decoder ) modules during training . However , in dense regression both bottom-up and top-down modules exist by default , and such methods are thus not trivial to extend to regression tasks . Motivated by the combination of supervised and unsupervised pathways , we propose a novel conditional GAN which includes implicit constraints in the latent subspaces . We coin this new model ‘ Robust Conditional GAN ’ ( RoCGAN ) . In the original cGAN the generator accepts a source signal and maps it to the target domain . In our work , we ( implicitly ) constrain the decoder to generate samples that span only the target manifold . We replace the original generator , i.e . encoder-decoder , 1The output includes at least as many dimensions as the input , e.g . super-resolution , or text-to-image translation . with a two pathway module ( see Fig . 1 ) . The first pathway , similarly to the cGAN generator , performs regression while the second is an autoencoder in the target domain ( unsupervised pathway ) . The two pathways share a similar network structure , i.e . each one includes an encoder-decoder network . The weights of the two decoders are shared which promotes the latent representations of the two pathways to be semantically similar . Intuitively , this can be thought of as constraining the output of our dense regression to span the target subspace . The unsupervised pathway enables the utilization of all the samples in the target domain even in the absence of a corresponding input sample . During inference , the unsupervised pathway is no longer required , therefore the testing complexity remains the same as in cGAN . In the following sections , we introduce our novel RoCGAN and study their ( theoretical ) properties . We prove that RoCGAN share similar theoretical properties with the original GAN , i.e . convergence and optimal discriminator . An experiment with synthetic data is designed to visualize the target subspaces and assess our intuition . We experimentally scrutinize the sensitivity of the hyper-parameters and evaluate our model in the face of intense noise . Moreover , thorough experimentation with both images from natural scenes and human faces is conducted in two different tasks . We compare our model with both the state-of-the-art cGAN and the recent method of Rick Chang et al . ( 2017 ) . The experimental results demonstrate that RoCGAN outperform the baseline by a large margin in all cases . Our contributions can be summarized as following : • We introduce RoCGAN that leverages structure in the target space . The goal is to promote robustness in dense regression tasks . • We scrutinize the model performance under ( extreme ) noise and adversarial perturbations . To the authors ’ knowledge , this robustness analysis has not been studied previously for dense regression . • We conduct a thorough experimental analysis for two different tasks . We outline how RoCGAN can be used in a semi-supervised learning task , how it performs with lateral connections from encoder to decoder . Notation : Given a set of N samples , s ( n ) denotes the nth conditional label , e.g . a prior image ; y ( n ) denotes the respective target image . Unless explicitly mentioned otherwise || · || will declare an ` 1 norm . The symbols L∗ define loss terms , while λ∗ denote regularization hyper-parameters optimized on the validation set . 2 RELATED WORK . Conditional image generation is a popular task in computer vision , dominated by approaches similar to cGAN . Apart from cGAN , the method by Isola et al . ( 2017 ) , widely known as ‘ pix2pix ’ , is the main alternative . Pix2pix includes three modifications over the baseline cGAN : i ) lateral skip connections between the encoder and the decoder network are added in the generator , ii ) the discriminator accepts pairs of source/gt and source/model output images , iii ) additional content loss terms are added . The authors demonstrate how those performance related modifications can lead to an improved visual outcome . Despite the improved performance , the problem with the additional guarantees remains the same . That is we do not have any direct supervision in the process , since both the latent subspace and the projection are learned ; the only supervision is provided by the ground-truth ( gt ) signal in the generator ’ s output . Adding regularization terms in the loss function can impose stronger supervision , thus restricting the output . The most frequent regularization term is feature matching , e.g . perceptual loss ( Ledig et al. , 2017 ; Johnson et al. , 2016 ) , or embeddings for faces ( Schroff et al. , 2015 ) . Feature matching minimizes the distance between the projection of generated and ground-truth signals . However , the pre-defined feature space is restrictive . The method introduced by Salimans et al . ( 2016 ) performs feature matching in the discriminator ; the motivation lies in matching the low-dimensional distributions created by the discriminator layers . Matching the discriminator ’ s features has demonstrated empirical success . However , this does not affect the generator and its latent subspaces directly . A new line of research that correlates with our goals is that of adversarial attacks ( Szegedy et al. , 2014 ; Yuan et al. , 2017 ; Samangouei et al. , 2018 ) . It is observed that perturbing input samples with a small amount of noise , often imperceptible to the human eye , can lead to severe classification errors . There are several techniques to ‘ defend ’ against such attacks . A recent example is the Fortified networks of Lamb et al . ( 2018 ) which uses Denoising Autoencoders ( Vincent et al. , 2008 ) to ensure that the input samples do not fall off the target manifold . Kumar et al . ( 2017 ) estimate the tangent space to the target manifold and use that to insert invariances to the discriminator for classification purposes . Even though RoCGAN share similarities with those methods , the scope is different since a ) the output of our method is high-dimensional2 and b ) adversarial examples are not extended to dense regression3 . Except for the study of adversarial attacks , combining supervised and unsupervised learning has been used for enhancing the classification performance . In the Ladder network , Rasmus et al . ( 2015 ) modify a typical bottom-up network for classification by adding a decoder and lateral connections between the encoder and the decoder . During training they utilize the augmented network as two pathways : i ) labelled input samples are fed to the initial bottom-up module , ii ) input samples are corrupted with noise and fed to the encoder-decoder with the lateral connections . The latter pathway is an autoencoder ; the idea is that it can strengthen the resilience of the network to samples outside the input manifold , while it improves the classification performance . Our core goal consists in constraining the model ’ s output . Aside from deep learning approaches , such constraints in manifolds were typically tackled with component analysis . Canonical correlation analysis ( Hotelling , 1936 ) has been extensively used for finding common subspaces that maximally correlate the data ( Panagakis et al. , 2016 ) . The recent work of Murdock et al . ( 2018 ) combines the expressiveness of neural networks with the theoretical guarantees of classic component analysis . 3 METHOD . In this section , we elucidate our proposed RoCGAN . To make the paper self-contained we first review the original conditional GAN model ( sec . 3.1 ) , before introducing RoCGAN ( sec . 3.2 ) . Sequentially , we pose the modifications required in case of shortcut connections from the encoder to the decoder ( sec . 3.3 ) . In sec . 3.4 we assess the intuition behind our model with synthetic data . The core idea in RoCGAN is to leverage structure in the output space of the model . We achieve that by 2In the classification tasks studied , e.g . the popular Imagenet ( Deng et al. , 2009 ) , there are up to a thousand classes , while our output includes tens or hundreds of thousands of dimensions . 3The robustness in our case refers to being resilient to changes in the distribution of the labels ( label shift ) and training set ( covariance shift ) ( Wang et al. , 2017 ) . replacing the single pathway in the generator with two pathways . In the appendix , we study the theoretical properties of our method and prove that RoCGAN share the same properties as the original GAN ( Goodfellow et al. , 2014 ) . 3.1 CONDITIONAL GAN . GAN consist of a generator and a discriminator module commonly optimized with alternating gradient descent methods . The generator samples z from a prior distribution pz , e.g . uniform , and tries to model the target distribution pd ; the discriminator D tries to distinguish between the samples generated from the model and the target ( ground-truth ) distributions . Conditional GAN ( cGAN ) ( Mirza & Osindero , 2014 ) extend the formulation by providing the generator with additional labels . In cGAN the generator G typically takes the form of an encoder-decoder network , where the encoder projects the label into a low-dimensional latent subspace and the decoder performs the opposite mapping , i.e . from low-dimensional to high-dimensional subspace . If we denote s the conditioning label and y a sample from the target distribution , the adversarial loss is expressed as : Ladv ( G , D ) = Es , y∼pd ( s , y ) [ logD ( y|s ) ] + Es∼pd ( s ) , z∼pz ( z ) [ log ( 1−D ( G ( s , z ) |s ) ) ] ( 1 ) by solving the following min-max problem : min wG max wD Ladv ( G , D ) = min wG max wD Es , y∼pd ( s , y ) [ logD ( y|s , wD ) ] + Es∼pd ( s ) , z∼pz ( z ) [ log ( 1−D ( G ( s , z|wG ) |s , wD ) ) ] where wG , wD denote the generator ’ s and the discriminator ’ s parameters respectively . To simplify the notation , we drop the dependencies on the parameters and the noise z in the rest of the paper . The works of Salimans et al . ( 2016 ) and Isola et al . ( 2017 ) demonstrate that auxiliary loss terms , i.e . feature matching and content loss , improve the final outcome , hence we consider those as part of the vanilla cGAN . The feature matching loss ( Salimans et al. , 2016 ) is : Lf = N∑ n=1 ||π ( G ( s ( n ) ) ) − π ( y ( n ) ) || ( 2 ) where π ( ) extracts the features from the penultimate layer of the discriminator . The final loss function for the cGAN is the following : L = Ladv + λc · N∑ n=1 ||G ( s ( n ) ) − y ( n ) ||︸ ︷︷ ︸ content−loss +λπ · Lf ( 3 ) where λc , λπ are hyper-parameters to balance the loss terms .

Summary: This manuscript proposes a robust version of conditional GAN (named RoC-GAN) that leverage the intrinsic structure in the output space. To achieve robustness, the authors replace the single pathway in the generator with two different pathways that partially share weights. The authors study the theoretical properties of RoC-GAN and prove that it shares the same properties as the vanilla GAN. For quantitative evaluations, the authors use two datasets of natural scenes and faces and evaluate denoising and sparse inpainting using the SSIM metric.