Rethinking conditional GAN training: An approach using geometrically structured latent manifolds

Abstract:
Conditional GANs (cGAN), in their rudimentary form, suffer from critical drawbacks such as the lack of diversity in generated outputs and distortion between the latent and output manifolds. Although efforts have been made to improve results, they can suffer from unpleasant side-effects such as the topology mismatch between latent and output spaces. In contrast, we tackle this problem from a geometrical perspective and propose a novel training mechanism that increases both the diversity and the visual quality of a vanilla cGAN, by systematically encouraging a bi-lipschitz mapping between the latent and the output manifolds. We validate the efficacy of our solution on a baseline cGAN (i.e., Pix2Pix) which lacks diversity, and show that by only modifying its training mechanism (i.e., with our proposed Pix2Pix-Geo), one can achieve more diverse and realistic outputs on a broad set of image-to-image translation tasks. Code available at: https://github.com/samgregoost/Rethinking-CGANs

heading: 1 Introduction
text: Generative adversarial networks (GAN) are a family of deep generative models that learn to model data distribution Y from random latent inputs z ∼ Z using a stochastic generator functionG : Z → Y [1]. A seemingly natural extension from unconditional GANs to conditional GANs (cGAN) can be achieved via conditioning both the discriminator and the generator on a conditioning signal x ∼ X . However, such a straightforward extension can cause the models to disregard x [2, 3, 4, 5]. To overcome this unsought behavior, a reconstruction loss is typically added to the objective function to penalise the model when it deviates from x. This approach has been widely adapted for diverse tasks including image-to-image translation [6, 2], style transfer [7, 8] and inpainting [9, 3, 10], and super-resolution [11, 12, 13, 14]. However, in spite of the wide usage, naively coupling the reconstruction and the adversarial objectives entails undesirable outcomes as discussed next.
Many conditional generation tasks are ill-posed (many possible solutions exist for a given input), and an ideal generator should be able to capture one-to-many mappings between the input and output domains. Note that the stochasticity of G typically depends on two factors, first the randomness of z and second the dropout [15]. However, empirical evidence suggests the composition of reconstruction and adversarial losses leads to a limited diversity, despite the random seed z. In fact, many prior works have reported that the generator often tends to ignore z, and learns a deterministic mapping
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
from X to Y , leaving dropout as the only source of stochasticity [2, 4, 3, 5]. Additionally, [16] and [17] demonstrated that from a geometrical perspective, latent spaces of generative models (e.g., cGANs) tend to give a distorted view of the generated distribution, thus, the Euclidean paths on the latent manifold do not correspond to the geodesics (shortest paths) on the output manifold. This hinders many possibilities such as clustering in the latent space, better interpolations, higher interpretability and ability to manipulate the outputs. We show that the foregoing problems can be direct consequences of the conventional training approach. Moreover, the naive coupling of regression loss and the adversarial loss can also hamper the visual quality of the generated samples due to contradictory goals of the two objective functions (see Sec. 2.1).
The aforementioned drawbacks have led multi-modal conditional generation approaches to opt for improved objective functions [18, 19], and even complex architectures compared to vanilla cGANs [20, 5, 4]. However, in Sec. 2, we show that while the existing solutions may improve the diversity and address the loss mismatch, they can also aggravate the topology mismatch and distortion between the latent and output manifolds. In contrast, we argue that these issues are not a consequence of the model capacities of vanilla cGANs [2, 3, 21, 6], rather a result of sub-optimal training procedures that are insensitive to their underlying geometry. As a remedy, we show that the foregoing problems can be addressed by systematically encouraging a structured bijective and a continuous mapping, i.e., a homeomorphism, between the latent and the generated manifolds. Furthermore, the structure of the latent space can be enhanced by enforcing bi-lipschitz conditions between the manifolds. To this end, we introduce a novel training procedure and an optimization objective to encourage the generator and the latent space to preserve a bi-lipschitz mapping, while matching the Euclidean paths in the latent space to geodesics on the output manifold.
We choose Pix2Pix [2], a vanilla cGAN, and modify its training procedure to demonstrate that the proposed mapping improves the realism of the outputs by removing the loss mismatch, enhances the structure of the latent space, and considerably improves the output diversity. As the formulation of our conditional generation approach is generic, we are able to evaluate the modified Pix2Pix model, dubbed Pix2Pix-Geo, on a diverse set of popular image-to-image translation tasks. We show that with the modified training approach, our Pix2Pix-Geo significantly improves the prediction diversity of the cGAN compared to the traditional baseline procedure and achieves comparable or better results than the more sophisticated state-of-the-art models. Most importantly, our modifications are purely aimed at the optimization procedure, which demands no architectural modifications to vanilla cGANs.

heading: 2 Motivation
text: In conditional generative modeling, the ground truth (output) data distribution Y ⊆ RM is conditioned on an input distribution X ⊆ Rd. Consider the data distribution Y|xp ⊂ Y conditioned on xp ∈ X . Then, the following adversarial objective function is used to optimize the generator G by playing a min-max game against a discriminator D, thereby approximating the distribution Y|xp ,
Ladv = G min D max E y∼Y [Φ(D(xp, y)]) + E z∼ζ [Φ(1−D(xp, G(xp, z))], (1)
where Φ is a suitably chosen monotone function, y ∼ Y and z ∈ Rk is a latent vector sampled from a prior distribution ζ. It has been widely observed that using the above objective function in isolation, pushes the models to generate samples that are not strongly conditioned on the input signal xp [2, 18, 4, 20]. Hence, the conventional cGAN loss couples a reconstruction loss Lr (typically `1 or `2) with Eq. 1. However, as alluded in Sec. 1, this entails several drawbacks: a) contradictory goals of the loss components, b) conditional mode collapse, and c) insensitivity to the underlying manifold geometry. Below, we explore these issues in detail and contrast our method against several recent attempts towards their resolution. From this point onwards, our analysis is focused on the conditional setting and we do not explicitly denote the conditioning signal x in our notations, to avoid clutter.

heading: 2.1 Mismatch b/w adversarial & reconstruction losses
text: Given the generated distribution pg and the ground truth distribution pd, the optimal generator G∗ for the adversarial loss can be formulated as,
G∗ = argmin G
( JSD [ pg(ȳ)‖pd(y) ]) , (2)
where JSD is the Jensen–Shannon divergence, y is the ground-truth and ȳ = G(z) is the output. Let us also consider the expected `1 loss Lr = Ey,z|y − ȳ|. App. F shows that Lr is minimized when,∫ ȳ
−∞ pd(y)dy = ∫ ∞ ȳ pd(y)dy. (3)
This shows the probability mass to the left of ȳ is equal to the probability mass of right of ȳ, i.e, ȳ is the median of y. Therefore, the optimal generator obtained from minimizing Lr does not equal to G∗, except for the rare case where pd(y) is unimodal with a sharp peak. With a similar approach, it can be shown that `2 concentrates pg near the average of the ground truth distribution. Hence, these contradictory goals of Lr and Ladv force the model to reach a compromise, thereby settling in a sub-optimal position in the parameter space. On the contrary, this mismatch can be removed by our proposed training approach by encouraging a homeomorphism between the latent and output spaces (App. F). This argument is empirically backed by our experiments, as we show that the realism of the outputs of the Pix2Pix [2] model can be significantly improved using the proposed method. Both Bicycle-GAN [20] and MR-GAN [4] remove this loss mismatch using a bijective mapping and by matching the moments of the generated and target distributions, respectively. However, their training procedures can disrupt the structure of the latent space (see Sec. 2.3).

heading: 2.2 Conditional mode collapse
text: (Conditional) mode collapse is a commonly observed phenomenon in cGANs [2, 4, 3, 5]. In this section, we discuss how the traditional training procedure may cause mode collapse and show that the existing solutions tend to distort the structure of the latent manifold.
Definition 1 [18]. A mode H is a subset of Y s.t. maxy∈H‖y−y∗‖<α for an output y∗ and α>0. Then, at the training phase, z1 is attracted to H by from an optimization step if ‖y∗−Gθ(t+1)(z1)‖+ < ‖y∗−Gθ(t)(z1)‖, where θ(t) are the parameters of G at time t. Proposition 1 [18]. Suppose z1 is attracted toH by . Then, there exists a neighbourhood N (z1) of z1, such that z is attracted toH by /2,∀z ∈ N (z1). Furthermore, the radius of N (z1) is bounded by an open ball of radius r where the radius is defined as,
r = (
4 inf z
{ max(τ(t), τ(t+ 1)) })−1 , where τ(t) = ∥∥Gθ(t)(z1)−Gθ(t)(z)∥∥ ‖z1 − z‖ . (4)
Proposition 1 yields that by maximizing τ(t) at each optimization step, one can avoid mode collapse. Noticeably, the traditional training approach does not impose such a constraint. Thus,‖z1 − z‖ can be arbitrary large for a small change in the output and the model is prone to mode collapse. As a result, DSGAN [18], MS-GAN [19] and MR-GAN [4] (implicitly) aim to maximize τ . Although maximizing τ improves the diversity, it also causes an undesirable side-effect, as discussed next.

heading: 2.3 Loss of structure b/w output & latent manifolds
text: A sufficiently smooth generative model G(z) can be considered as a surface model [22]. This has enabled analyzing latent variable generative models using Riemannian geometry [23, 24, 16, 25]. Here, we utilize the same perspective: a generator can be considered as a function that maps low dimensional latent codes z ∈ Mz⊆Rk to a data manifoldMy in a higher dimensional space RM whereMz andMy are Riemannian manifolds, i.e., z encodes the intrinsic coordinates ofMy . Note that increasing τ in an unconstrained setting does not impose any structure in the latent space. That is, since the range of
∥∥G(z1)−G(z)∥∥ is arbitrary in different neighbourhoods, stark discontinuities in the output space can occur, as we move alongMz . Further note that Bicycle-GAN also does not impose such continuity on the mapping. Thus, the distance between two latent codes onMz may not yield useful information such as the similarity of outputs. This is a significant disadvantage, as we expect the latent space to encode such details. Interestingly, if we can induce a continuous and a bijective mapping, i.e., a homeomorphism betweenMy andMz , while maximizing τ , the structure of the latent space can be preserved to an extent.
However, a homeomorphism does not reduce the distortion ofMy with respect toMz . In other words, although the arc length between z1 and z is smoothly and monotonically increasing with the arc length between G(z1) and G(z) under a homeomorphism, it is not bounded. This can cause heavy distortions between the manifolds. More formally, maximizing τ encourages maximizing
the components of the Jacobian Jd×k = ∂G∂z at small intervals. If G is sufficiently smooth, the Riemannian metric M = JTJ can be obtained, which is a positive definite matrix that varies smoothly on the latent space. Further, by the Hadamard inequality,
det(M) ≤ k∏ i=0 ‖Ji‖2 , (5)
where Ji are the columns of J. This leads to an interesting observation. In fact, det(M) can be seen as a measure of distortion of the output manifold with respect to the latent manifold. Therefore, although maximizing τ acts as a remedy for mode collapse, even under a homeomorphism, it can increase the distortion betweenMz andMy . In conditional generation tasks, it is useful to reduce the distortion between the manifolds. Ideally, we would like to match the Euclidean paths onMz to geodesics onMy, as it entails many advantages (see Sec. 1). Consider a small distance ∆z onMz . Then, the corresponding distance inMy can be obtained using Taylor expansion as,
G(∆z) = J∆z + Θ(‖∆z‖) ≈ J∆z, (6) where Θ(‖∆z‖) is a function which approaches zero more rapidly than ∆z. It is evident from Eq. 6 that the corresponding distance onMy for ∆z is governed by J. Ideally, we want to constrain J in such a way that small Euclidean distances ∆z encourage the output to move along geodesics inMy . However, since random sampling does not impose such a constraint on J, the traditional training approach and the existing solutions fail at this. Interestingly, it is easy to deduce that geodesics avoid paths with high distortions [26]. Recall that minimizing τ along optimization curves reduces the distortion ofMy , thus, encourages ∆z to match geodesics onMy . However, minimizing τ can also lead to mode collapse as discussed in Sec. 2.2.
Although the above analysis yields seemingly contradictory goals, one can achieve both by establishing a bi-lipschitz mapping between My andMz , as it provides both an upper and a lowerbound for τ . Such a mapping betweenMz and My provides a soft bound for det(M), and prevents mode collapse while preserving structure of the latent manifold.
Remark 1: An ideal generator function should be homeomorphic to its latent space. The structure of the latent space can be further improved by inducing a bi-lipschitz mapping between the latent space and generator function output.1
Based on the above Remark, we propose a training approach that encourages a structured bilipschitz mapping between the latent and the generated manifolds and show that in contrast to the existing methods, the proposed method is able to address all three issues mentioned above.

heading: 3 Methodology
text: Our approach is based on three goals. R1) A bi-lipschitz mapping must exist betweenMz andMy ,
1 C dMz (z p, zq) ≤ dMy (φ −1(zp),φ−1(zq)) ≤ CdMz (z p, zq), (7)
where d·(·) is the geodesic distance in the denoted manifold, zp and zq are two latent codes, andC is a constant. Further, φ :My→Mz is a continuous global chart map with its inverse φ−1. R2) Euclidean distances inMz should map to geodesics inMy for better structure. R3) The geodesic distance between two arbitrary points onMy should correspond to a meaningful metric, i.e., pixel distance (note the loss mismatch is implicitly resolved by R1). Next, we explain our training procedure.
1Note that every bi-lipschitz mapping is a homeomorphism.

heading: 3.1 Geodesics and global bi-lipschitz mapping
text: Here, we discuss the proposed training procedure. Consider a map γMz : I →Mz , that parameterizes a curve onMz using t ∈ I ⊂ R. Then, there also exists a map (G ◦ γMz ) ≡ γMy : I →My. If γMy is a geodesic, this mapping can be uniquely determined by a p ∈My and an initial velocity V ∈ TpMy, where TpMy is the tangent space ofMy at p (see App. C)2. This is a useful result, as we can obtain a unique point p′ ∈My only by defining an initial velocity and following γMy for a time T (note we do not consider the unlikely scenario where two geodesics may overlap at t = T ).
To find the geodesic between two points on a Riemannian manifold, γMz is usually constrained as, γ̈Mz = − 12 M −1 [ 2(Ik ⊗ γ̇TMz ) ∂vec(M) ∂γMz γ̇Mz − [ ∂vec(M) ∂γMz ]T (γ̇Mz ⊗ γ̇Mz ) ] ,
where Mk×k = JTφ−1Jφ−1 is the metric tensor, Jφ−1 is the Jacobian, ⊗ is the outer product, dot operator is the first-order gradient and the double dot operator is the second-order gradient [17]. This approach is expensive, as it requires calculating the Jacobians in each iteration and moreover, causes unstable gradients. In practice, an exact solution is not needed, hence, we adapt an alternate procedure to encourage γMy to be a geodesic, and use Eq. 8 only for evaluation purposes in Sec. 4. Since geodesics are locally length minimizing paths on a manifold, we encourage the model to minimize the curve length L(γMy (t)) onMy in the range t = [0, T ]. L(γMy (t)) is measured as:
L(γMy (t)) = ∫ 1 0 ∥∥∥∥∂G ◦ γMz (t)∂t ∥∥∥∥ dt = ∫ 1 0 ∥∥∥∥∂G ◦ γMz (t)∂γMz (t) ∂γMz (t)∂t ∥∥∥∥ dt. (8)
Eq. 8 can be expressed using the Jacobian Jφ−1 as,
= ∫ 1 0 ∥∥∥∥Jφ−1 ∂γMz (t)∂t ∥∥∥∥ dt = ∫ 1 0 √[ Jφ−1 ∂γMz (t) ∂t ]T Jφ−1 ∂γMz (t) ∂t dt.
Since Mk×k = JTφ−1Jφ−1 ,
= ∫ 1 0 √[∂γMz (t) ∂t ]T M∂γMz (t) ∂t dt.
Considering small ∆t = TN ,
≈ N∑ i=0 √[∂γMz (t) ∂t ]T M∂γMz (t) ∂t ∆t = N−1∑ i=0 √ żTi Mżi∆t. (9)
Further, ∥∥G(∆z)∥∥ = ∆zTM∆z > 0,∀∆z > 0, i.e., M is positive definite (since dGdz 6= 0, which is discussed next). By Hadamard inequality (Eq. 5), it can be seen that we can minimize the ∂G∂z , in order for L(γMy (t)) to be minimized. But on the other hand, we also need γMy (T ) = y. Therefore, we minimize ∂G∂z at small intervals along the curve by updating the generator at each ti = i∆t,
Lgh(ti, zti , y, x) =‖[α(ti) · y − (1− α(ti)) ·G(zt0 , x)]−G(zti , x)‖, (10) where i = 0, 1, . . . , N, and α(·) is a monotonic function under the conditions α(0) = 0 and α(T ) = T . Another perspective for the aforementioned procedure is that the volume element of
My can be obtained as = √∣∣det(M)∣∣dz. Therefore, det(M) is a measure of the distortion in My with respect toMz and geodesics prefer to avoid regions with high distortions. The procedure explained so far encourages a bi-lipschitz mapping as in Eq. 7 (proof in App. D), and satisfies R1. Further, we show that the enforced bijective mapping removes the loss mismatch between the adversarial and reconstruction losses, hence, improves the visual quality (see Fig. 3 and App. F).
According to R2, the proposed training mechanism should map Euclidean paths onMz to geodesics onMy . Therefore, we move z along Euclidean paths when minimizing Lgh, which also ensures that Mz ⊆ Rk. Furthermore, we constrain ż to be a constant for simplicity. Since we ensure that the distortion ofMy along the paths of z are minimum, in practice, it can be observed that the Euclidean paths on the latent space are approximately matched to the geodesics on the output manifold (Fig. 6).
2V depends on p and hence the dependency of the mapping γMp on p does need to be explicitly denoted.
Further, let γV (t) be a geodesic curve with an initial velocity V . Then, it can be shown, γcV (t) = γV (ct), (11)
where c is a constant (proof in App. E). This is an important result, since it immediately follows that ∥∥ż1t0∥∥ > ∥∥ż2t0∥∥ =⇒ L(γż1(T )) > L(γż2(T )). Following these intuitions, we define ż = ∇z ∥∥y −G(zt0)∥∥. This yields an interesting advantage, i.e.,‖ż‖ (hence L(γż(T ))) tends to be large
for high ∥∥y −G(zt0)∥∥, which corresponds to R3.

heading: 3.2 Encouraging the local bijective conditions
text: The approach described in Sec. 3.1 encourages a global bi-lipschitz mapping between My and Mz . However, we practically observed that imposing bijective conditions in local neighborhoods in conjunction leads to improved performance. Thus, we enforce a dense bijective mapping between My and Mz near γMy (T ). Let zT and y be the latent code at γMy (T ) and the ground truth, respectively. We generate two random sets Z̃ and Ỹ using the distribution,
Z̃ = N (zT ; 2) and Ỹ = Ψ(y), (12) where Ψ(·) applies random perturbations such as brightness, contrast and small noise, and 0 < 2 < 1. One trivial method to ensure that a bijective mapping exists is to apply a loss function
∑∥∥yi −G(zi)∥∥, ∀zi ∈ Z̃, yi ∈ Ỹ to update the generator. However, we empirically observed that the above loss function unnecessarily applies a hard binding between the perturbations and the generated data. Therefore, we minimize the KL-distance between G and Ỹ up to second order moments. One possible way to achieve this is to model each pixel as a univariate distribution (App G). However in this case, since the generator cannot capture the correlations between different spatial locations, unwanted artifacts appear on the generated data. Therefore, we treat G and Ỹ as M -dimensional multivariate distributions (M = image height× image width). Then, the KL-distance between the distributions up to the second order of moments can be calculated using the following equation,
Llh(y, z, x) = 1
2 [ log |ΣG∗ |∣∣ΣỸ ∣∣ −M + tr(Σ−1G ΣỸ) + (µG − µỸ)TΣ−1G (µG − µỸ) ] , (13)
Algorithm 1: Training algorithm sample inputs {x1, x2, ..., xJ} ∼ X ; sample outputs {y1, y2, ..., yJ} ∼ Y ; for k epochs do
for x in χ do z ∼ Bkr //Sample z from k-ball with a
small radius r V ← ∇z ∥∥y −G(zt0)∥∥ t← 0 for T steps do
sample noise: e ∼ N (0, 1); 1 1 update G: ∇wLgh(y, z, x, t) update z: z ← z + ηV + e update t: t← t+ 1
update G: ∇w[Llh(y, z, x)+LR(y, z, x)+Ladv(y, z, x)]
where Σ and µ denote the correlation matrices and the means (App. H). However, using the above loss (Eq. 13) in its original form yields practical obstacles: for instance, the correlation matrices have the dimension M ×M , which is infeasible to handle. Therefore, following [27], we use a random projection matrix RM×h;h M to project the images to a h−dimensional space, where Ri,j ∼ p(x); p( √
3) = 16 , p(0) = 2 3 , p(− √ 3) = 16 (we empirically justify this reduction method using an ablation study in Sec. 4). Moreover, numerically calculating |Σ| and Σ−1 causes unstable gradients which hinders the generator optimization. We address this issue by adapting the approximation technique proposed in [28]:
log(|Σ|) ≈ − N∑ i=1 tr(Ci) i , (14)
where C = I− Σ. Further, Σ−1 can be calculated as, Vi+1 = Vi(3I− ΣVi(3I− ΣVn)), i = 1, 2, . . . , N, (15) where Li et al. [29] proved that Vi → Σ−1 as i → ∞, for a suitable approximation of V0. They further showed that a suitable approximation should be V0 = αΣT , 0 < α < 2/ρ(ΣΣT ), where ρ(·) is the spectral radius. Our final loss function Ltotal consists of four loss components:
Ltotal = β0Lgh + β1Llh + β2Lr + β3Ladv, (16) where β0 . . . β3 are constant weights learned via cross-validation. Further, Llh is estimated per mini-batch. Algorithm 1 shows overall training.

heading: 4 Experiments
text: In this section, we demonstrate the effectiveness of the proposed training scheme using qualitative and quantitative experiments. First, we illustrate the generalizability of our method by comparing against the state-of-the-art methods across a diverse set of image-to-image translation tasks. Then, we explore the practical implications of geometrically structuring the latent manifold. Finally, we conduct an ablation study to compare the effects of the empirical choices we made in Sec. 3. In all the experiments, we use Pix2Pix [2] as our model architecture, and use the same model trained using the traditional training approach as the main baseline. We use the official implementation of other comparable methods to benchmark their performance against ours. For a fair comparison, we use their pre-trained models wherever available, otherwise train their model from scratch, strictly following the authors’ instructions to the best of our ability. For further details on the datasets and hyper-parameter settings, see App. I.
Image-to-image translation: We compare our method against state-of-the-art models that focus on multimodal image-to-image translation. Fig. 2 shows the qualitative results on landmarks→ faces, sketch→ anime and BW → color. As evident, our training mechanism increases the diversity and the visual quality of the baseline P2P model significantly, and shows better performance compared to other models. Fig. 3 shows qualitative comparison against the baseline. Table 1 depicts the quantitative results. As shown, our model exhibits a higher diversity and a higher realism on multiple
datasets. In all the cases, we outperform our baseline by a significant margin. Fig. 4 compares color distribution in BW2color task.
Geometrical interpretations: A key implication of our training scheme is that the Euclidean shortest paths onMz map to geodesics onMy , which preserves better structure. We conduct an experiment to empirically validate the aforementioned attribute. First, we travel along Euclidean paths onMz and measure the corresponding curve length LE on the data manifold. Second, we calculate the actual
geodesic distance LG between the same two points onMy using Eq. 8 in discrete intervals. We travel in 10 random directions starting from random initial points, and obtain LGi for evenly spaced LE ∈ {10, 20, 30, . . . 90}. Then, we obtain a set of the means and standard deviations of LG for the corresponding LE . Fig. 6 illustrates the distribution. As evident, our model exhibits a significantly high overlap with the ideal curve, i.e., LE = E(LG) compared to DS-GAN and Bicycle-GAN. A useful attribute of travelling along the geodesics on the output manifold (My) is to obtain smooth interpolations, since the geodesics tend to avoid regions with high distortions, i.e., rapid changes. However, Euclidean shortest paths in the latent spaces (Mz) of cGANs often do not correspond to geodesics on theMy. Therefore, in order to travel along geodesics, it is required to numerically obtain the geodesic paths using Eq. 8, which requires extra computation. In contrast, the proposed training method encourages the generator to map the Euclidean paths onMz to geodesics onMy. Therefore, smooth interpolations can be obtained by traveling between two latent codes in a straight path. To evaluate this, we compare the interpolation results between Bicycle-GAN, DS-GAN and our model. Fig. 5 shows a qualitative example, along with a quantitative evaluation. As visible, our model exhibits smooth transition from the starting point to the end point. In comparison, Bicycle-GAN shows abrupt and inconsistent changes along the path. DS-GAN does not show any significant variance in the beginning and shows sudden large changes towards the end. We also quantify this comparison using the velocity on the data manifold: since the curve length onMy can be calculated using Eq. 9, it is easy to see that the velocity onMy can be obtained using √ żTi Mżi. Fig. 5 illustrates the change in the velocity, corresponding to the given qualitative examples. Our model demonstrates an approximately constant velocity (geodesics have constant velocities), while the other models show sudden velocity changes. We did not include CGML in these evaluations (App. I).
Ablation study: We conduct an ablation study to compare the different variants of the proposed technique. Table 2 depicts the results. First, we compare different distance functions used to calculate Llh. As expected, naive maximization of the distances between the generated samples without any constraints increases the diversity, but reduces the visual quality drastically. Further, we observed unwanted artifacts when modeling each pixel as a univariate distribution, as the model then cannot capture dependencies across spatial locations. Then, we compare different down-sampling methods that can be used for efficient calculation of the correlation matrices, where random projection performed the best. Interestingly, we observed a reduction of the visual quality when the dimension of the latent code is increased. In contrast, the diversity tends to improve with the latter. We chose dim(z) = 64 as a compromise. Finally, we
compare the effects of different combinations of the loss components.
Generalizability: To demonstrate the generalizability of the proposed algorithm across different loss functions and architectures, we employ it on three classic networks: Pathak et al. [3], Johnson et al. [30], and Ronneberger et al. [31]. These networks use a masked reconstruction loss with the
adversarial loss, perception loss from pre-trained networks, and a reconstruction loss, respectively. Further, in the original form, these networks only learn one-to-one mappings. As depicted in Fig. 7, our algorithm increases the diversity of the models and obtains one-to-many mappings with no changes to the architecture (for fair comparison, we concatenate a latent code at the bottlenecks during both the original and proposed training). Further, since our algorithm is architecture agnostic, it can be easily injected into any suitable model. To show this, we replace the Lds of StarGANv2 [32] with our loss Lgh. In StarGANv2, the images are generated as a composition of two functions: the style generator (F (·)) and the image generator (G(·, ·)). We enforce a bi-lipschitz constraint between the latent space and the image generator outputs. For continuously differentiable functions F and G, if the composition G ◦ F is homeomorphic, F and G are individually homeomorphic. Hence, the image generator output becomes homeomorphic to the style space. As shown in Table 3, our approach can improve the performance of StarGANv2, without any modifications to the architecture.

heading: 5 Conclusion
text: We show that the cGANs, in their basic form, suffer from significant drawbacks in-terms of diversity and realism. We propose a novel training algorithm that can increase both realism and the diversity of the outputs that are generated by cGANs while preserving the structure of the latent manifold. To this end, we enforce a bi-lipschitz mapping between the latent and generated output manifolds while encouraging Euclidean shortest paths on the latent manifold to be mapped to the geodesics on the generated manifold. We establish the necessary theoretical foundation and demonstrate the effectiveness of the proposed algorithm at a practical level, using a diverse set of image-to-image translation tasks, where our model achieves compelling results.

Review summaries:
This paper attempts to address poor sample quality and sample diversity in image conditioned generative models by addressing the topology mismatch between latent space and image space. It introduces two new loss functions: one which encourages Euclidean paths in the latent space to map to geodesic paths in image space, and another which helps to enforce a bijective mapping between the latent manifold and the output image manifold. Experimental results demonstrate quantitative and qualitative improvement over previous methods.

To tackle the problem of mode collapse in conditional generative adversarial networks (C-GAN), the paper introduces a new loss function and an associated training procedure, inspired by differential geometric considerations. In particular, the mapping learnt by the generator network is required to be bi-Lipschitz so that it preserves the local Euclidean geometry between the latent and data spaces (the tangent space of the manifold) at every point. The proposed approach is empirically validated on a variety of image to image translation tasks.

This paper tries to improve the current limitation of conditional GANs with a novel topological space matching view. The authors claim to solve the main drawbacks of existing cGANs that cause the GAN to lack diversity and randomness. Basically, they propose an optimization algorithm to encourage the generator and the latent space to preserve a bi-lipschitz mapping between generated space and GT space. The authors compare evaluate their method on image translation task and show higher quality of generator comparing with existing SOTA cGANs.