COCO-GAN: Conditional Coordinate Generative Adversarial Network

1 INTRODUCTION . This paper explores the idea of enforcing both the generator and the discriminator of generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) to deal with only partial views via conditional coordinating . Via training and inference with partial views only , the minimum memory requirement can be largely reduced . However , as shown in Section 3.3 , naive approaches fail to generate high quality images , either having clear seams or totally failing to generate reasonable structures . We investigate this problem and propose a new GAN architecture : COnditional COordinate GAN ( COCO-GAN ) . Given a latent vector and multiple spatial positions , the generator of COCO-GAN learns to produce image patches independently according to the spatial positions . On the other hand , the discriminator learns to judge whether adjacently generated patches are structurally sound and visually homogeneous . During the inference phase , the generated patches can directly be used to compose a complete full-image without further post-processing . Owing to the adversarial loss provided by the discriminator , the composed full-image is locally smooth and globally convincing . We show several randomly-selected full-images generated by COCO-GAN in Figure 2a . In Section 3.2 , we visualize the interpolation between two spatial positions , showing COCO-GAN has inter-class continuity as common conditional GANs do . Further quantitative evaluations with “ Frchet Inception Distance ” ( FID ) ( Heusel et al. , 2017 ) score are presented in Table 1 . Without additional hyper-parameter tuning , the evaluations on CelebA ( Liu et al. , 2015 ) and LSUN ( Yu et al. , 2015 ) datasets suggest that COCO-GAN is competitive with other state-of-the-art GANs . To further demonstrate the effectiveness of COCO-GAN , we perform ablation study in Section 3.3 . In Section 3.4 , we demonstrate COCO-GAN is a flexible coordinate-system-aware framework that is suitable for different spatial coordinate systems . Common learning frameworks are confined to The code and data used in the paper will be made available immediately upon publication . Cartesian coordinate system , which restricts those frameworks from learning the characteristics of certain image formats . For instance , panoramas should be trained with a cylindrical coordinate system , which has a “ cyclic topology ” in the horizontal direction . In comparison , COCO-GAN directly learns with a cylindrical coordinate system . In Figure 6 and Figure 14 , the generated full scenes are naturally cylindrical and continuous while crossing the left and right borders . Besides , we find COCO-GAN has multiple interesting applications and characteristics . We select three representative new applications as case studies : Patch-Inspired Image Generation takes a real image patch as input . COCO-GAN can be inspired by the given patch and further generates a full-image proposal . This proposal is partially similar to the given patch , while globally realistic and reasonable . This setting is different from conventional image completion/reconstruction since the information loss during cropping is extreme and the spatial position information is not provided to the model . Therefore , COCO-GAN has to infer the position of the given patch before generating the whole image . Further experiments are presented in Section 3.5 . Partial-Scene Generation shows that COCO-GAN can generate partial scenes without spending additional computation outside certain designated regions . This capability is exclusively beneficial to applications that are only interested in partial information of the full scene . For instance , virtual reality ( VR ) is only interested in the user ’ s viewport direction , and COCO-GAN can seamlessly adapt to such viewport-aware settings . Computation-Friendly Generation demonstrates the computational related merits . First , since the patches are produced independently , the generator is able to generate patches with high parallelism . Second , as the full-image generation is decomposed to patches generation , the minimum memory requirement of generating images can be reduced . This characteristic enables COCO-GAN to generate images of very high resolution or containing much more complex structures . 2 COCO-GAN . Overview . COCO-GAN consists of three networks : a generator G , a discriminator D , and a auxiliary head Q . These three networks are trained with four loss terms : patch Wasserstein loss LW , patch gradient penalty loss LGP , spatial consistency loss LS , and content consistency loss LC . Compared to conventional GANs that use full images as input for both G and D , COCO-GAN only uses micro patches for G , and macro patches for D. The details of micro and macro patches will be described in the paragraph “ Spatial coordinate system design ” . D has two auxiliary prediction heads : the content vector prediction head ( Q ) and the spatial condition prediction head . Q is trained with an extra optimizer independent to G and D. It aims to minimize LC and is used to estimate the original latent vector of the given sample . The spatial condition prediction head , on the other hand , is jointly trained with the discriminator . It aims at minimizing LS and is used to estimate the macro spatial position of the given sample . Both of the two auxiliary prediction heads are simple feed-forward networks that take inputs from one of the high-level feature maps of the discriminator . COCO-GAN considers two coordinate systems : a micro coordinate system m , which refers to the annotated component that related to the finer coordinate system , on the generator ’ s side , and a macro coordinate systemM , which is related to the coarser macro coordinate system on the discriminator ’ s side . The discriminator learns to distinguish between the generated macro patch p̃M and the real macro patch pM . The discriminator also learns to predict two auxiliary outputs : a predicted macro coordinate c̃M and a predicted latent vector z̃M . Outputs from these two networks are then used to compute two auxiliary losses : Spatial Consistency Loss ( LS ) and Content Consistency Loss ( LC ) . The objective function of the discriminator D is LW +LGP +LS +LC , the generator G is−LW + LS + LC , and the content vector prediction head Q is −LW + LC . Spatial coordinate system . Before presenting the details of these four loss terms , we introduce our notations first . We start with designing two spatial coordinate systems , a micro coordinate system Cm for the generator G and a macro coordinate system CM for the discriminator D. Let SN be a space of spatial position sequences , each spatial position sequence s = 〈cmi 〉Ni=1 ∈ SN is an ordered sequence , which cmi ∈ Cm . During COCO-GAN training , R is some predefined spatial constraints for sampling s from a uniform distribution U ( SN , R ) . The generator G is conditioned by each spatial position cmi , and learns to accordingly produce micro patches p̃ m i = G ( z|cmi ) . The 〈p̃mi 〉Ni=1 = 〈G ( z|cmi ) 〉Ni=1 is a sequence of micro patches produced independently while sharing the same latent vector z across the spatial position sequence s. The design of R may need to be slightly changed with respect to the selection of CM and Cm . The design principle of R is that the accordingly generated micro patches 〈p̃mi 〉Ni=1 should be spatially close to each other . Then the micro patches are merged by a merging function T ( m→M ) to form a complete macro patch p̃M = T ( m→M ) ( 〈p̃mi 〉Ni=1 ) as a coarser partial-view of the full-scene x̃ . Meanwhile , we assign p̃M with a new spatial position cM under the macro coordinate system for s. In Figure 1 , we illustrate one of the simplest design for the above heuristic functions that we have adopted throughout our experiments . The four micro patches are always a neighbor of each other and can be directly combined into a square macro patch with T ( m→M ) , which is just simple concatenation . Figure 3 shows some examples of micro and macro patches generation . On the real samples side , we also sample s ∼ S , but we directly infer s to macro position cM . Then we design a transformation T ( X→M ) with respect to T ( m→M ) to transform a real full-image x to a real macro patch pM by T ( X→M ) ( x|cM ) . The shape of pM should be the same as p̃M . In our simplest experiment setting , T ( X→M ) is a simple cropping function that crops x into pM , which has the same shape as p̃M . During the testing phase , depending on the design of Cm , we can directly infer a corresponding spatial position sequence 〈cmj 〉Kj=1 . It is used to independently produce spatially-disentangled patches that constitute the full-image . Figure 9 demonstrates how the full-image is generated during the testing phase . Figure 2a shows some examples of the full-image generation . Loss functions . The patch Wasserstein loss LW is a patch-level Wasserstein distance loss similar to Wasserstein-GAN ( Arjovsky et al. , 2017 ) loss . It forces the discriminator to discriminate between the real macro patch pM and the generated macro patch p̃M , and on the other hand , encourages the generator to confuse the discriminator with seemingly realistic p̃M . Its complete form is LW = E [ D ( T ( X→M ) ( x|cM ) ) ] − E [ D ( T ( m→M ) ( 〈G ( z|cmi ) 〉Ni=1 ) ) ] , ( 1 ) where x ∼ Pr and 〈cmi 〉Ni=1 ∼ U ( SN , R ) . Note that Pr is the real data distribution . We also apply Gradient Penalty ( Gulrajani et al. , 2017 ) to the patches generation : LGP = E [ ( ‖∇p̃MD ( p̃M ) ‖2 − 1 ) 2 ] , ( 2 ) where p̃M ∼ Pg . Note that Pg is the generator distribution . The spatial consistency loss LS is similar to ACGAN loss ( Odena et al. , 2017 ) . A slight difference is that cmi has relatively more continuous values than the discrete setting of ACGAN . As a result , we apply a distance measurement loss for LS , which is an L2-loss between c̃M and cM . It aims to train the generator of COCO-GAN to generate the corresponding micro patch by G ( z|cmi ) with respect to the given spatial condition cmi . The spatial consistency loss is LS = E [ ‖cM − c̃M‖2 ] . ( 3 ) On the other hand , the content consistency loss LC is similar to a hybrid of infoGAN loss ( Chen et al. , 2016 ) and the latent space constraint loss ( Chang et al. , 2018 ) . The former one uses a separate optimizer to optimize an auxiliary network Q , which aims to reconstruct the original latent vector . The latter one suggests that the latent space consistency loss can be a distance measurement instead of minimizing the KL-divergence as the original infoGAN does . In our experiments , we train the extra Q network with a separate optimizer and directly minimize L1-loss between z̃ and z. LC aims to force the generator to produce shared context between patches that share the same latent vector but locate at different micro coordinate positions . The content consistency loss is defined by LC = E [ ‖z − z̃‖1 ] . ( 4 ) To ensure that Z , Cm , and CM share the similar scale , which are directly concatenated and feed to G. We evaluate the maximum possible pixel position of Cm and CM , then normalize the range into [ −1 , 1 ] . For the latent space Z , although uniform sampling between [ −1 , 1 ] should be numerically more compatible with the normalized spatial condition space , we empirically do not observe significant differences even if we switch to random sampling with a zero-mean and unit-variance normal distribution . For simplicity , we adopt uniform sampling strategy throughout our experiments . Training details . Our generator and discriminator architectures follow the idea of projection discriminator ( Miyato & Koyama , 2018 ) , both with ResNet ( He et al. , 2016 ) based architecture and adding class-projection to the discriminator . All convolutional and feed-forward layers of generator and discriminator are added with the spectral-normalization scheme ( Miyato et al. , 2018 ) as suggested in ( Zhang et al. , 2018 ) . A more detailed architecture diagram is illustrated in Appendix B . We also add conditional-batch-normalization ( CBN ) ( Dumoulin et al. , 2016 ) to the generator . In our design , CBN is conditioned on the given spatial positions and the input latent vector . It learns to normalize the feature maps with respect to the given conditions . However , our implementation has a crucial difference from the one described in ( Miyato & Koyama , 2018 ) : our spatial positions are real values rather than discrete classes . We alternatively adopt similar strategy to ( de Vries et al. , 2017 ) with a slight modification . Instead of using MLPs to produce ∆γ and ∆β , we make MLPs directly output γ and β . For a K-channel input feature map iK with mean recorder µK and variance recorder σK , we creates two learnable MLP layers , MLPγ and MLPβ . The output feature map oM is calculated as oM = ( ( iK − µK ) /σK ) ∗MLPγ ( iK ) + MLPβ ( iK ) . We use Adam ( Kingma & Ba , 2014 ) optimizer with β1 = 0 and β2 = 0.999 for both the generator and the discriminator . The learning rates are based on the Two Time-scale Update Rule ( TTUR ) ( Heusel et al. , 2017 ) , setting 0.0001 for the generator and 0.0004 for the discriminator . We do not specifically balance the generator and the discriminator by manually setting how many iterations to update the generator once as described in the WGAN paper ( Arjovsky et al. , 2017 ) .

Summary: The paper describes a GAN architecture and training methodology where a generator is trained to generate "micro-" patches, being passed as input a latent vector and patch co-ordinates. Micro-patches generated for different adjacent locations with the same latent vector are combined to generate a "macro" patch. This "macro" output is trained against a discriminator that tries to label this output as real and fake, as well as predict the location of the macro patch and the value of the latent vector. The generator is trained to fool the discriminator's label, and minimize the error in the prediction of location and latent vector information.