Rethinking conditional GAN training: An approach using geometrically structured latent manifolds

1 Introduction . Generative adversarial networks ( GAN ) are a family of deep generative models that learn to model data distribution Y from random latent inputs z ∼ Z using a stochastic generator functionG : Z → Y [ 1 ] . A seemingly natural extension from unconditional GANs to conditional GANs ( cGAN ) can be achieved via conditioning both the discriminator and the generator on a conditioning signal x ∼ X . However , such a straightforward extension can cause the models to disregard x [ 2 , 3 , 4 , 5 ] . To overcome this unsought behavior , a reconstruction loss is typically added to the objective function to penalise the model when it deviates from x . This approach has been widely adapted for diverse tasks including image-to-image translation [ 6 , 2 ] , style transfer [ 7 , 8 ] and inpainting [ 9 , 3 , 10 ] , and super-resolution [ 11 , 12 , 13 , 14 ] . However , in spite of the wide usage , naively coupling the reconstruction and the adversarial objectives entails undesirable outcomes as discussed next . Many conditional generation tasks are ill-posed ( many possible solutions exist for a given input ) , and an ideal generator should be able to capture one-to-many mappings between the input and output domains . Note that the stochasticity of G typically depends on two factors , first the randomness of z and second the dropout [ 15 ] . However , empirical evidence suggests the composition of reconstruction and adversarial losses leads to a limited diversity , despite the random seed z . In fact , many prior works have reported that the generator often tends to ignore z , and learns a deterministic mapping 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . from X to Y , leaving dropout as the only source of stochasticity [ 2 , 4 , 3 , 5 ] . Additionally , [ 16 ] and [ 17 ] demonstrated that from a geometrical perspective , latent spaces of generative models ( e.g. , cGANs ) tend to give a distorted view of the generated distribution , thus , the Euclidean paths on the latent manifold do not correspond to the geodesics ( shortest paths ) on the output manifold . This hinders many possibilities such as clustering in the latent space , better interpolations , higher interpretability and ability to manipulate the outputs . We show that the foregoing problems can be direct consequences of the conventional training approach . Moreover , the naive coupling of regression loss and the adversarial loss can also hamper the visual quality of the generated samples due to contradictory goals of the two objective functions ( see Sec . 2.1 ) . The aforementioned drawbacks have led multi-modal conditional generation approaches to opt for improved objective functions [ 18 , 19 ] , and even complex architectures compared to vanilla cGANs [ 20 , 5 , 4 ] . However , in Sec . 2 , we show that while the existing solutions may improve the diversity and address the loss mismatch , they can also aggravate the topology mismatch and distortion between the latent and output manifolds . In contrast , we argue that these issues are not a consequence of the model capacities of vanilla cGANs [ 2 , 3 , 21 , 6 ] , rather a result of sub-optimal training procedures that are insensitive to their underlying geometry . As a remedy , we show that the foregoing problems can be addressed by systematically encouraging a structured bijective and a continuous mapping , i.e. , a homeomorphism , between the latent and the generated manifolds . Furthermore , the structure of the latent space can be enhanced by enforcing bi-lipschitz conditions between the manifolds . To this end , we introduce a novel training procedure and an optimization objective to encourage the generator and the latent space to preserve a bi-lipschitz mapping , while matching the Euclidean paths in the latent space to geodesics on the output manifold . We choose Pix2Pix [ 2 ] , a vanilla cGAN , and modify its training procedure to demonstrate that the proposed mapping improves the realism of the outputs by removing the loss mismatch , enhances the structure of the latent space , and considerably improves the output diversity . As the formulation of our conditional generation approach is generic , we are able to evaluate the modified Pix2Pix model , dubbed Pix2Pix-Geo , on a diverse set of popular image-to-image translation tasks . We show that with the modified training approach , our Pix2Pix-Geo significantly improves the prediction diversity of the cGAN compared to the traditional baseline procedure and achieves comparable or better results than the more sophisticated state-of-the-art models . Most importantly , our modifications are purely aimed at the optimization procedure , which demands no architectural modifications to vanilla cGANs . 2 Motivation . In conditional generative modeling , the ground truth ( output ) data distribution Y ⊆ RM is conditioned on an input distribution X ⊆ Rd . Consider the data distribution Y|xp ⊂ Y conditioned on xp ∈ X . Then , the following adversarial objective function is used to optimize the generator G by playing a min-max game against a discriminator D , thereby approximating the distribution Y|xp , Ladv = G min D max E y∼Y [ Φ ( D ( xp , y ) ] ) + E z∼ζ [ Φ ( 1−D ( xp , G ( xp , z ) ) ] , ( 1 ) where Φ is a suitably chosen monotone function , y ∼ Y and z ∈ Rk is a latent vector sampled from a prior distribution ζ . It has been widely observed that using the above objective function in isolation , pushes the models to generate samples that are not strongly conditioned on the input signal xp [ 2 , 18 , 4 , 20 ] . Hence , the conventional cGAN loss couples a reconstruction loss Lr ( typically ` 1 or ` 2 ) with Eq . 1 . However , as alluded in Sec . 1 , this entails several drawbacks : a ) contradictory goals of the loss components , b ) conditional mode collapse , and c ) insensitivity to the underlying manifold geometry . Below , we explore these issues in detail and contrast our method against several recent attempts towards their resolution . From this point onwards , our analysis is focused on the conditional setting and we do not explicitly denote the conditioning signal x in our notations , to avoid clutter . 2.1 Mismatch b/w adversarial & reconstruction losses . Given the generated distribution pg and the ground truth distribution pd , the optimal generator G∗ for the adversarial loss can be formulated as , G∗ = argmin G ( JSD [ pg ( ȳ ) ‖pd ( y ) ] ) , ( 2 ) where JSD is the Jensen–Shannon divergence , y is the ground-truth and ȳ = G ( z ) is the output . Let us also consider the expected ` 1 loss Lr = Ey , z|y − ȳ| . App . F shows that Lr is minimized when , ∫ ȳ −∞ pd ( y ) dy = ∫ ∞ ȳ pd ( y ) dy . ( 3 ) This shows the probability mass to the left of ȳ is equal to the probability mass of right of ȳ , i.e , ȳ is the median of y . Therefore , the optimal generator obtained from minimizing Lr does not equal to G∗ , except for the rare case where pd ( y ) is unimodal with a sharp peak . With a similar approach , it can be shown that ` 2 concentrates pg near the average of the ground truth distribution . Hence , these contradictory goals of Lr and Ladv force the model to reach a compromise , thereby settling in a sub-optimal position in the parameter space . On the contrary , this mismatch can be removed by our proposed training approach by encouraging a homeomorphism between the latent and output spaces ( App . F ) . This argument is empirically backed by our experiments , as we show that the realism of the outputs of the Pix2Pix [ 2 ] model can be significantly improved using the proposed method . Both Bicycle-GAN [ 20 ] and MR-GAN [ 4 ] remove this loss mismatch using a bijective mapping and by matching the moments of the generated and target distributions , respectively . However , their training procedures can disrupt the structure of the latent space ( see Sec . 2.3 ) . 2.2 Conditional mode collapse . ( Conditional ) mode collapse is a commonly observed phenomenon in cGANs [ 2 , 4 , 3 , 5 ] . In this section , we discuss how the traditional training procedure may cause mode collapse and show that the existing solutions tend to distort the structure of the latent manifold . Definition 1 [ 18 ] . A mode H is a subset of Y s.t . maxy∈H‖y−y∗‖ < α for an output y∗ and α > 0 . Then , at the training phase , z1 is attracted to H by from an optimization step if ‖y∗−Gθ ( t+1 ) ( z1 ) ‖+ < ‖y∗−Gθ ( t ) ( z1 ) ‖ , where θ ( t ) are the parameters of G at time t. Proposition 1 [ 18 ] . Suppose z1 is attracted toH by . Then , there exists a neighbourhood N ( z1 ) of z1 , such that z is attracted toH by /2 , ∀z ∈ N ( z1 ) . Furthermore , the radius of N ( z1 ) is bounded by an open ball of radius r where the radius is defined as , r = ( 4 inf z { max ( τ ( t ) , τ ( t+ 1 ) ) } ) −1 , where τ ( t ) = ∥∥Gθ ( t ) ( z1 ) −Gθ ( t ) ( z ) ∥∥ ‖z1 − z‖ . ( 4 ) Proposition 1 yields that by maximizing τ ( t ) at each optimization step , one can avoid mode collapse . Noticeably , the traditional training approach does not impose such a constraint . Thus , ‖z1 − z‖ can be arbitrary large for a small change in the output and the model is prone to mode collapse . As a result , DSGAN [ 18 ] , MS-GAN [ 19 ] and MR-GAN [ 4 ] ( implicitly ) aim to maximize τ . Although maximizing τ improves the diversity , it also causes an undesirable side-effect , as discussed next . 2.3 Loss of structure b/w output & latent manifolds . A sufficiently smooth generative model G ( z ) can be considered as a surface model [ 22 ] . This has enabled analyzing latent variable generative models using Riemannian geometry [ 23 , 24 , 16 , 25 ] . Here , we utilize the same perspective : a generator can be considered as a function that maps low dimensional latent codes z ∈ Mz⊆Rk to a data manifoldMy in a higher dimensional space RM whereMz andMy are Riemannian manifolds , i.e. , z encodes the intrinsic coordinates ofMy . Note that increasing τ in an unconstrained setting does not impose any structure in the latent space . That is , since the range of ∥∥G ( z1 ) −G ( z ) ∥∥ is arbitrary in different neighbourhoods , stark discontinuities in the output space can occur , as we move alongMz . Further note that Bicycle-GAN also does not impose such continuity on the mapping . Thus , the distance between two latent codes onMz may not yield useful information such as the similarity of outputs . This is a significant disadvantage , as we expect the latent space to encode such details . Interestingly , if we can induce a continuous and a bijective mapping , i.e. , a homeomorphism betweenMy andMz , while maximizing τ , the structure of the latent space can be preserved to an extent . However , a homeomorphism does not reduce the distortion ofMy with respect toMz . In other words , although the arc length between z1 and z is smoothly and monotonically increasing with the arc length between G ( z1 ) and G ( z ) under a homeomorphism , it is not bounded . This can cause heavy distortions between the manifolds . More formally , maximizing τ encourages maximizing the components of the Jacobian Jd×k = ∂G∂z at small intervals . If G is sufficiently smooth , the Riemannian metric M = JTJ can be obtained , which is a positive definite matrix that varies smoothly on the latent space . Further , by the Hadamard inequality , det ( M ) ≤ k∏ i=0 ‖Ji‖2 , ( 5 ) where Ji are the columns of J . This leads to an interesting observation . In fact , det ( M ) can be seen as a measure of distortion of the output manifold with respect to the latent manifold . Therefore , although maximizing τ acts as a remedy for mode collapse , even under a homeomorphism , it can increase the distortion betweenMz andMy . In conditional generation tasks , it is useful to reduce the distortion between the manifolds . Ideally , we would like to match the Euclidean paths onMz to geodesics onMy , as it entails many advantages ( see Sec . 1 ) . Consider a small distance ∆z onMz . Then , the corresponding distance inMy can be obtained using Taylor expansion as , G ( ∆z ) = J∆z + Θ ( ‖∆z‖ ) ≈ J∆z , ( 6 ) where Θ ( ‖∆z‖ ) is a function which approaches zero more rapidly than ∆z . It is evident from Eq . 6 that the corresponding distance onMy for ∆z is governed by J . Ideally , we want to constrain J in such a way that small Euclidean distances ∆z encourage the output to move along geodesics inMy . However , since random sampling does not impose such a constraint on J , the traditional training approach and the existing solutions fail at this . Interestingly , it is easy to deduce that geodesics avoid paths with high distortions [ 26 ] . Recall that minimizing τ along optimization curves reduces the distortion ofMy , thus , encourages ∆z to match geodesics onMy . However , minimizing τ can also lead to mode collapse as discussed in Sec . 2.2 . Although the above analysis yields seemingly contradictory goals , one can achieve both by establishing a bi-lipschitz mapping between My andMz , as it provides both an upper and a lowerbound for τ . Such a mapping betweenMz and My provides a soft bound for det ( M ) , and prevents mode collapse while preserving structure of the latent manifold . Remark 1 : An ideal generator function should be homeomorphic to its latent space . The structure of the latent space can be further improved by inducing a bi-lipschitz mapping between the latent space and generator function output.1 Based on the above Remark , we propose a training approach that encourages a structured bilipschitz mapping between the latent and the generated manifolds and show that in contrast to the existing methods , the proposed method is able to address all three issues mentioned above .

Summary: This paper attempts to address poor sample quality and sample diversity in image conditioned generative models by addressing the topology mismatch between latent space and image space. It introduces two new loss functions: one which encourages Euclidean paths in the latent space to map to geodesic paths in image space, and another which helps to enforce a bijective mapping between the latent manifold and the output image manifold. Experimental results demonstrate quantitative and qualitative improvement over previous methods.