COCO-GAN: Conditional Coordinate Generative Adversarial Network

1 INTRODUCTION . This paper explores the idea of enforcing both the generator and the discriminator of generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) to deal with only partial views via conditional coordinating . Via training and inference with partial views only , the minimum memory requirement can be largely reduced . However , as shown in Section 3.3 , naive approaches fail to generate high quality images , either having clear seams or totally failing to generate reasonable structures . We investigate this problem and propose a new GAN architecture : COnditional COordinate GAN ( COCO-GAN ) . Given a latent vector and multiple spatial positions , the generator of COCO-GAN learns to produce image patches independently according to the spatial positions . On the other hand , the discriminator learns to judge whether adjacently generated patches are structurally sound and visually homogeneous . During the inference phase , the generated patches can directly be used to compose a complete full-image without further post-processing . Owing to the adversarial loss provided by the discriminator , the composed full-image is locally smooth and globally convincing . We show several randomly-selected full-images generated by COCO-GAN in Figure 2a . In Section 3.2 , we visualize the interpolation between two spatial positions , showing COCO-GAN has inter-class continuity as common conditional GANs do . Further quantitative evaluations with “ Frchet Inception Distance ” ( FID ) ( Heusel et al. , 2017 ) score are presented in Table 1 . Without additional hyper-parameter tuning , the evaluations on CelebA ( Liu et al. , 2015 ) and LSUN ( Yu et al. , 2015 ) datasets suggest that COCO-GAN is competitive with other state-of-the-art GANs . To further demonstrate the effectiveness of COCO-GAN , we perform ablation study in Section 3.3 . In Section 3.4 , we demonstrate COCO-GAN is a flexible coordinate-system-aware framework that is suitable for different spatial coordinate systems . Common learning frameworks are confined to The code and data used in the paper will be made available immediately upon publication . Cartesian coordinate system , which restricts those frameworks from learning the characteristics of certain image formats . For instance , panoramas should be trained with a cylindrical coordinate system , which has a “ cyclic topology ” in the horizontal direction . In comparison , COCO-GAN directly learns with a cylindrical coordinate system . In Figure 6 and Figure 14 , the generated full scenes are naturally cylindrical and continuous while crossing the left and right borders . Besides , we find COCO-GAN has multiple interesting applications and characteristics . We select three representative new applications as case studies : Patch-Inspired Image Generation takes a real image patch as input . COCO-GAN can be inspired by the given patch and further generates a full-image proposal . This proposal is partially similar to the given patch , while globally realistic and reasonable . This setting is different from conventional image completion/reconstruction since the information loss during cropping is extreme and the spatial position information is not provided to the model . Therefore , COCO-GAN has to infer the position of the given patch before generating the whole image . Further experiments are presented in Section 3.5 . Partial-Scene Generation shows that COCO-GAN can generate partial scenes without spending additional computation outside certain designated regions . This capability is exclusively beneficial to applications that are only interested in partial information of the full scene . For instance , virtual reality ( VR ) is only interested in the user ’ s viewport direction , and COCO-GAN can seamlessly adapt to such viewport-aware settings . Computation-Friendly Generation demonstrates the computational related merits . First , since the patches are produced independently , the generator is able to generate patches with high parallelism . Second , as the full-image generation is decomposed to patches generation , the minimum memory requirement of generating images can be reduced . This characteristic enables COCO-GAN to generate images of very high resolution or containing much more complex structures . 2 COCO-GAN . Overview . COCO-GAN consists of three networks : a generator G , a discriminator D , and a auxiliary head Q . These three networks are trained with four loss terms : patch Wasserstein loss LW , patch gradient penalty loss LGP , spatial consistency loss LS , and content consistency loss LC . Compared to conventional GANs that use full images as input for both G and D , COCO-GAN only uses micro patches for G , and macro patches for D. The details of micro and macro patches will be described in the paragraph “ Spatial coordinate system design ” . D has two auxiliary prediction heads : the content vector prediction head ( Q ) and the spatial condition prediction head . Q is trained with an extra optimizer independent to G and D. It aims to minimize LC and is used to estimate the original latent vector of the given sample . The spatial condition prediction head , on the other hand , is jointly trained with the discriminator . It aims at minimizing LS and is used to estimate the macro spatial position of the given sample . Both of the two auxiliary prediction heads are simple feed-forward networks that take inputs from one of the high-level feature maps of the discriminator . COCO-GAN considers two coordinate systems : a micro coordinate system m , which refers to the annotated component that related to the finer coordinate system , on the generator ’ s side , and a macro coordinate systemM , which is related to the coarser macro coordinate system on the discriminator ’ s side . The discriminator learns to distinguish between the generated macro patch p̃M and the real macro patch pM . The discriminator also learns to predict two auxiliary outputs : a predicted macro coordinate c̃M and a predicted latent vector z̃M . Outputs from these two networks are then used to compute two auxiliary losses : Spatial Consistency Loss ( LS ) and Content Consistency Loss ( LC ) . The objective function of the discriminator D is LW +LGP +LS +LC , the generator G is−LW + LS + LC , and the content vector prediction head Q is −LW + LC . Spatial coordinate system . Before presenting the details of these four loss terms , we introduce our notations first . We start with designing two spatial coordinate systems , a micro coordinate system Cm for the generator G and a macro coordinate system CM for the discriminator D. Let SN be a space of spatial position sequences , each spatial position sequence s = 〈cmi 〉Ni=1 ∈ SN is an ordered sequence , which cmi ∈ Cm . During COCO-GAN training , R is some predefined spatial constraints for sampling s from a uniform distribution U ( SN , R ) . The generator G is conditioned by each spatial position cmi , and learns to accordingly produce micro patches p̃ m i = G ( z|cmi ) . The 〈p̃mi 〉Ni=1 = 〈G ( z|cmi ) 〉Ni=1 is a sequence of micro patches produced independently while sharing the same latent vector z across the spatial position sequence s. The design of R may need to be slightly changed with respect to the selection of CM and Cm . The design principle of R is that the accordingly generated micro patches 〈p̃mi 〉Ni=1 should be spatially close to each other . Then the micro patches are merged by a merging function T ( m→M ) to form a complete macro patch p̃M = T ( m→M ) ( 〈p̃mi 〉Ni=1 ) as a coarser partial-view of the full-scene x̃ . Meanwhile , we assign p̃M with a new spatial position cM under the macro coordinate system for s. In Figure 1 , we illustrate one of the simplest design for the above heuristic functions that we have adopted throughout our experiments . The four micro patches are always a neighbor of each other and can be directly combined into a square macro patch with T ( m→M ) , which is just simple concatenation . Figure 3 shows some examples of micro and macro patches generation . On the real samples side , we also sample s ∼ S , but we directly infer s to macro position cM . Then we design a transformation T ( X→M ) with respect to T ( m→M ) to transform a real full-image x to a real macro patch pM by T ( X→M ) ( x|cM ) . The shape of pM should be the same as p̃M . In our simplest experiment setting , T ( X→M ) is a simple cropping function that crops x into pM , which has the same shape as p̃M . During the testing phase , depending on the design of Cm , we can directly infer a corresponding spatial position sequence 〈cmj 〉Kj=1 . It is used to independently produce spatially-disentangled patches that constitute the full-image . Figure 9 demonstrates how the full-image is generated during the testing phase . Figure 2a shows some examples of the full-image generation . Loss functions . The patch Wasserstein loss LW is a patch-level Wasserstein distance loss similar to Wasserstein-GAN ( Arjovsky et al. , 2017 ) loss . It forces the discriminator to discriminate between the real macro patch pM and the generated macro patch p̃M , and on the other hand , encourages the generator to confuse the discriminator with seemingly realistic p̃M . Its complete form is LW = E [ D ( T ( X→M ) ( x|cM ) ) ] − E [ D ( T ( m→M ) ( 〈G ( z|cmi ) 〉Ni=1 ) ) ] , ( 1 ) where x ∼ Pr and 〈cmi 〉Ni=1 ∼ U ( SN , R ) . Note that Pr is the real data distribution . We also apply Gradient Penalty ( Gulrajani et al. , 2017 ) to the patches generation : LGP = E [ ( ‖∇p̃MD ( p̃M ) ‖2 − 1 ) 2 ] , ( 2 ) where p̃M ∼ Pg . Note that Pg is the generator distribution . The spatial consistency loss LS is similar to ACGAN loss ( Odena et al. , 2017 ) . A slight difference is that cmi has relatively more continuous values than the discrete setting of ACGAN . As a result , we apply a distance measurement loss for LS , which is an L2-loss between c̃M and cM . It aims to train the generator of COCO-GAN to generate the corresponding micro patch by G ( z|cmi ) with respect to the given spatial condition cmi . The spatial consistency loss is LS = E [ ‖cM − c̃M‖2 ] . ( 3 ) On the other hand , the content consistency loss LC is similar to a hybrid of infoGAN loss ( Chen et al. , 2016 ) and the latent space constraint loss ( Chang et al. , 2018 ) . The former one uses a separate optimizer to optimize an auxiliary network Q , which aims to reconstruct the original latent vector . The latter one suggests that the latent space consistency loss can be a distance measurement instead of minimizing the KL-divergence as the original infoGAN does . In our experiments , we train the extra Q network with a separate optimizer and directly minimize L1-loss between z̃ and z. LC aims to force the generator to produce shared context between patches that share the same latent vector but locate at different micro coordinate positions . The content consistency loss is defined by LC = E [ ‖z − z̃‖1 ] . ( 4 ) To ensure that Z , Cm , and CM share the similar scale , which are directly concatenated and feed to G. We evaluate the maximum possible pixel position of Cm and CM , then normalize the range into [ −1 , 1 ] . For the latent space Z , although uniform sampling between [ −1 , 1 ] should be numerically more compatible with the normalized spatial condition space , we empirically do not observe significant differences even if we switch to random sampling with a zero-mean and unit-variance normal distribution . For simplicity , we adopt uniform sampling strategy throughout our experiments . Training details . Our generator and discriminator architectures follow the idea of projection discriminator ( Miyato & Koyama , 2018 ) , both with ResNet ( He et al. , 2016 ) based architecture and adding class-projection to the discriminator . All convolutional and feed-forward layers of generator and discriminator are added with the spectral-normalization scheme ( Miyato et al. , 2018 ) as suggested in ( Zhang et al. , 2018 ) . A more detailed architecture diagram is illustrated in Appendix B . We also add conditional-batch-normalization ( CBN ) ( Dumoulin et al. , 2016 ) to the generator . In our design , CBN is conditioned on the given spatial positions and the input latent vector . It learns to normalize the feature maps with respect to the given conditions . However , our implementation has a crucial difference from the one described in ( Miyato & Koyama , 2018 ) : our spatial positions are real values rather than discrete classes . We alternatively adopt similar strategy to ( de Vries et al. , 2017 ) with a slight modification . Instead of using MLPs to produce ∆γ and ∆β , we make MLPs directly output γ and β . For a K-channel input feature map iK with mean recorder µK and variance recorder σK , we creates two learnable MLP layers , MLPγ and MLPβ . The output feature map oM is calculated as oM = ( ( iK − µK ) /σK ) ∗MLPγ ( iK ) + MLPβ ( iK ) . We use Adam ( Kingma & Ba , 2014 ) optimizer with β1 = 0 and β2 = 0.999 for both the generator and the discriminator . The learning rates are based on the Two Time-scale Update Rule ( TTUR ) ( Heusel et al. , 2017 ) , setting 0.0001 for the generator and 0.0004 for the discriminator . We do not specifically balance the generator and the discriminator by manually setting how many iterations to update the generator once as described in the WGAN paper ( Arjovsky et al. , 2017 ) .

Summary: This paper proposes to constrain the Generator of a WGAN-GP on patches locations to generate small images (“micro-patches”), with an additional smoothness condition so these can be combined into full images. This is done by concatenating micro-patches into macro patches, that are fed to the Discriminator.  The discriminator aims at classifying the macro-patches as fake or real, while additionally recovering the latent noise used for generation as well as the spatial prior.