Rethinking conditional GAN training: An approach using geometrically structured latent manifolds
Authors: Sameera Ramasinghe
Moshiur R Farazi
Salman Khan
Nick Barnes
Stephen Gould
Venue: NeurIPS 2021 Poster (2021)
Keywords: Generative modeling
Multimodal output spaces
Conditional generative models

Abstract: Conditional GANs (cGAN), in their rudimentary form, suffer from critical drawbacks such as the lack of diversity in generated outputs and distortion between the latent and output manifolds.  Although efforts have been made to improve results, they can suffer from unpleasant side-effects such as the topology mismatch between latent and output spaces. In contrast, we tackle this problem from a geometrical perspective and propose a novel training mechanism that increases both the diversity and the visual quality of a vanilla cGAN, by systematically encouraging a bi-lipschitz mapping between the latent and the output manifolds. We validate the efficacy of our solution on a baseline cGAN (i.e., Pix2Pix) which lacks diversity, and show that by only modifying its training mechanism (i.e., with our proposed Pix2Pix-Geo), one can achieve more diverse and realistic outputs on a broad set of image-to-image translation tasks.

Rethinking conditional GAN training: An approach using geometrically structured latent manifolds
Sameera Ramasinghe
Moshiur Farazi
The Australian National University, Dataó1-CSIRO
Data61-CSIRO
sameera. x rag.errar.mmiinneeaaar..coc
Salman Khan
Nick Barnes The Australian National University
Mohamed Bin Zayed University of AI
Stephen Gould
The Australian National University
Abstract
Conditional GANs (cGAN), in their rudimentary form, suffer from critical draw- backs such the lack of diversity in generated outputs and distortion between the latent and output manifolds. Although efforts have been made to improve results, they can suffer from unpleasant side-effects such the topology mismatch between latent and output spaces. In contrast, we tackle this problem from a geometrical perspective and propose a novel training mechanism that increases both the diversity and the visual quality of vanilla cGAN, by systematically encouraging bi-lipschitz mapping between the latent and the output manifolds. We validate the efficacy of our solution on a baseline cGAN (i.e., Pix2Pix) which lacks diversity, and show that by only modifying its training mechanism (i.e., with our proposed Pix2Pix-Geo), one can achieve more diverse and realistic outputs on a broad set image-to-image translation tasks. Code available at: https://see //gtth/b/hhtbb com/ wwwgrrggss///teei/nnlggggeee
1 Introduction
Generative adversarial networks (GAN) are family of deep generative models that learn to model data distribution ) from random latent inputs Z using stochastic generator function G Z y [1]. A seemingly natural extension from unconditional GANs to conditional GANs (cGAN) can be achieved via conditioning both the diseriminator and the generator However, such a straightforward extension can cause the models to disregard x [2, 3, 4, 5]. To overcome this unsought behavior, reconstruction loss is typically added to the objective function to penalise the model when it deviates from æ. This approach has been widely adapted for diverse tasks including image-to-image translation [6, 2]. style transfer [7, 8] and inpainting [9, 10], and super-resolution [11, 12, 13, 14]. However, in spite of the wide usage, naively coupling the reconstruction and the adversarial objectives entails undesirable outcomes discussed next.
conditioning signal x X.
Many conditional generation tasks are ill posed (many possible solutions exist for given input), and an ideal generator should be able to capture one-to-many mappings between the input and output domains. Note that the stochasticity of G typically depends on two factors, first the randomness of and second the dropout [15]. However, empirical evidence suggests the composition of reconstruction and adversarial losses leads to a limited diversity, despite the random seed z. In fact, many prior works have reported that the generator often tends to ignore and learns deterministic mapping
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

from A' to y' leaving dropout as the only source of stochasticity [2, 4, 3, 5]. Additionally, [16] and [17] demonstrated that from a geometrical perspective, latent spaces of generative models (e.g.. cGANs) tend to give a distorted view of the generated distribution, thus, the Fuclidean paths on the latent manifold do not correspond to the geodesics (shortest paths) on the output manifold. This hinders many possibilities such as clustering in the latent space, better interpolations, higher interpretability and ability to manipulate the outputs. We show that the foregoing problems can be direct consequences of the conventional training approach. Moreover, the naive coupling of regression loss and the adversarial loss can also hamper the visual quality of the generated samples due to contradictory goals of the two objective functions (see Sec. 2.1).
The aforementioned drawbacks have led multi- conditional generation approaches to opt for improved objective functions [18, 19], and even complex architectures compared to vanilla cGANs [20, 5, 4]. However, Sec. 2, we show that while the existing solutions may improve the diversity and address the loss mismatch, they can also aggravate the topology mismatch and distortion between the latent and output manifolds. In contrast, we argue that these issues are not a consequence of the model capacities of vanilla cGANs [2, 3, 21, 6], rather result of sub-optimal training procedures that are insensitive to their underlying geometry. As remedy, we show that the foregoing problems can be addressed by systematically encouraging a structured bijective and continuous mapping, i.e., a homeomorphism, between the latent and the generated manifolds. Furthermore, the structure of the latent space can be enhanced by enforcing bi-lipschitz conditions between the manifolds, To this end, we introduce a novel training procedure and an optimization objective to encourage the generator and the latent space to preserve a bi-lipschitz mapping, while matching the Euclidean paths in the latent space to geodesics on the output manifold.
We choose Pix2Pix [2], vanilla cGAN, and modify its training procedure to demonstrate that the proposed mapping improves the realism of the outputs by removing the loss mismatch, enhances the structure of the latent space, and considerably improves the output diversity. As the formulation of our conditional generation approach is generie, we are able to evaluate the modified Pix2Pix model, dubbed Pix2Pix-Geo on a diverse set of popular image-to-image translation tasks. We show that with the modified training approach, our Pix2Pix- Geo significantly improves the prediction diversity of the cGAN compared to the traditional baseline procedure and achieves comparable better results than the more sophisticated state-of oftt--ffth-arrr models. Most importantly. our modifications are purely aimed at the optimization procedure, which demands architectural modifications to vanilla cGANs.
2 Motivation
In conditional generative modeling, the ground truth (output) data distribution )' C R is conditioned on an input distribution X' C Rdd. Consider the data distribution Vrr, C conditioned on Tp E X. Then, the following adversarial objective function is used to optimize the generator G by playing min-max game against discriminator D, thereby approximating the distribution ) 10p
Lode min max E (D(((((ppp))))) v) + . w--
[0(1 D(xp, G(zp, 2])],
(I)
where is a suitably chosen monotone function, y ~ y and z E I is a latent vector sampled from a prior distribution Ç. It has been widely observed that using the above objective function in isolation, pushes the models to generate samples that are not strongly conditioned on the input signal Ip [2, 18, 4, 20J. Hence, the conventional cGAN loss couples reconstruction loss L, (typically f or E2) with Eq. 1. However, as alluded in Sec. 1, this entails several drawbacks: a) contradictory goals of the loss components, b) conditional mode collapse, and c) insensitivity to the underlying manifold geometry. Below, we explore these issues in detail and contrast our method against several recent attempts towards their resolution. From this point onwards, our analysis is focused on the conditional setting and we do not explicitly denote the conditioning signal x in our notations, to avoid clutter.
2.1 Mismatch b/w adversarial & reconstruction losses
Given the generated distribution Py and the ground truth distribution pa, the optimal generator G* for the adversarial loss can be formulated as,
G argmin JSD pp ())lla()))
(2)
G
2

---

where JSD is the Jensen-Shannen Shannon divergence, is the ground-truth and y = G(z) is the output. Let us also consider the expected & loss L, yy y 7. App. F shows that L, is minimized when,
L Pa(U)dy = S, pa(u)dy.
This shows the probability mass to the left of y is equal to the probability mass of right of g. i., g is the median of y. Therefore, the optimal generator obtained from minimizing L, does not equal to G", except for the rare case where pa(y) is unimodal with a sharp peak. With a similar approach, can be shown that &g concentrates Py near the average of the ground truth distribution. Hence, these contradictory goals of L, and Ladv force the model to reach a compromise, thereby settling in a sub-optimal position in the parameter space. On the contrary, this mismatch can be removed by our proposed training approach by encouraging a homeomorphism between the latent and output spaces (App. F). This argument is empirically backed by our experiments, as we show that the realism of the outputs of the Pix2Pix [2] model can be significantly improved using the proposed method. Both Bicycle-GAN [20] and MR-GAN [4] remove this loss mismatch using a bijective mapping and by matching the moments of the generated and target distributions, respectively. However, their training procedures can disrupt the structure of the latent space (see Sec. 2.3).
2.2 Conditional mode collapse
(Conditional) mode collapse is a commonly observed phenomenon in cGANs [2, 4, 3, 5]. In this section, we discuss how the traditional training procedure may cause mode collapse and show that the existing solutions tend to distort the structure of the latent manifold.
Definition [18]. A mode H. is a subset of y s.t. max <a for an output y" and a>0., Then, at the training phase, z1 is attracted to H by e from an optimization step if Ily ( <
Co(o (2)) where (t) are the parameters of G at time t.
Proposition 1 [18]. Suppose is attracted to H by Then, there exists neighbourhood M(z)) of 21, such that is attracted to H by e/2, Vz (2)). Furthermore, the radius of N(z)) is bounded by an open ball of radius r where the radius is defined as,
r= (4 maxxm(xxx T( ((+ 1))) where 7(1) = |[Gocs(as) - Gote (3)
r=
(4)
|| - =||
Proposition yields that by maximizing r(t) each optimization step, one avoid mode collapse. Noticeably, the traditional training approach does not impose such a constraint. Thus, || - =|l can be arbitrary large for small change in the output and the model is prone mode collapse. As result, DSGAN [18]. MS-GAN [19] and MR-GAN [4] (implicitly) aim to maximize r. Although maximizing improves the diversity, it also causes undesirable side-effect, as discussed next.
2.3 Loss of structure b/w output & latent manifolds
A sufficiently smooth generative model G(z) can be considered as a surface model [22]. This has enabled analyzing latent variable generative models using Riemannian geometry [23, 24, 16, 25]. Here, we utilize the same perspective: a generator can be considered as function that maps low dimensional latent codes 2 E R data manifold M, in higher dimensional space RM where M, and My are Riemannian manifolds, i.e.. encodes the intrinsic coordinates of MMy Note that increasing T in an unconstrained setting does not impose any structure in the latent space. That is, since the range of G=)]] is arbitrary in different neighbourhoods, stark discontinuities in the output space can occur, as we move along Mg. Further note that Bicycle-GAN also does not impose such continuity the mapping. Thus, the distance between two latent codes on M, may not yield useful information such as the similarity of outputs. This is a significant disadvantage, as we expect the latent space to encode such details. Interestingly, if we can induce continuous and bijective mapping, i.e., a homeomorphism between My and M,, while maximizing т, the structure of the latent space can be preserved to an extent.
However, a homeomorphism does not reduce the distortion of My with respect to M2. other words, although the are length between 2 and = is smoothly and monotonically increasing with the are length between G(21) and G(2) under a homeomorphism, it is not bounded. This can cause heavy distortions between the manifolds. More formally, maximizing 7 encourages maximizing
3

---

the components of the Jacobian Jdx& = 8G 0s at small intervals. If G is sufficiently smooth, the Riemannian metric M J can be obtained, which is positive definite matrix that varies smoothly on the latent space. Further, by the Hadamard inequality.
A det(M) - III.
(5)
where J; are the columns of J. This leads to an interesting observation. In fact, det(M) can be seen as a measure of distortion of the output manifold with respect to the latent manifold. Therefore, although maximizing r acts as a remedy for mode collapse, even under a homeomorphism, it can increase the distortion between M2 and My.
In conditional generation tasks, it is useful to reduce the distortion between the manifolds. Ideally, we would like to match the Euclidean paths on M, to geodesics on My. as it entails many advantages (see Sec. 1). Consider small distance A on Mg. Then, the corresponding distance in My can be obtained using Taylor expansion as,
G(A=) JAz + @(|A=|)> JAz,
(6)
where 00 a=]) is function which approaches zero more rapidly than Az. It is evident from Eq. 6 that the corresponding distance on My for A= is governed by J. Ideally, we want constrain J in such a way that small Euclidean distances Az encourage the output to move along geodesics in My. However, since random sampling does not impose such constraint on J. the traditional training approach and the existing solutions fail at this. Interestingly, it is easy to deduce that geodesics avoid paths with high distortions [26]. Recall that minimizing 7 along optimization curves reduces the distortion of My, thus, encourages A to match geodesics on My. However, minimizing 7 can also lead to mode collapse as discussed in Sec. 2.2.
Although the above analysis yields seemingly contradictory goals, one can achieve both by tablishing a bi-lipschitz mapping between M, and M2. as it provides both an upper and a lower- bound for 7. Such a mapping between M2 and My provides soft bound for det(M), and pre- vents mode collapse while preserving structure of the latent manifold.
drwwre))
0.2
(PtaPPt-Gen)
NR-GAN
DS-GAN
(Higher LPIPS
6.
Blajes-GAN
FID
(Lower more realistic)
Remark 1: An ideal generator function should be homeomorphic to its latent space. The struc- ture of the latent space can be further improved by inducing bi- bi-lipschitz mapping between the latent space and generator function output.'
Latent Manifold
Output Manifoid
Eunoe distnne
bi-Lisschilz mapping
Condens pas
RM
Based on the above Remark, we propose a train- ing approach that encourages a structured bi- lipschitz mapping between the latent and the gen- crated manifolds and show that in contrast to the existing methods, the proposed method is able to address all three issues mentioned above.
Figure 1: Approach Overview. Our training proce- dure encourages a bi-lipschitz mapping between the latent and generated output manifolds, while mapping the Euclidean shortest paths in the latent manifold to geodesics on the generated output manifold, which allows better diversity and structure. We gain con- siderable improvement in both visual quality and the image diversity over our baseline Pix2Pix [2], using the same network architecture (landmark + faces image- to to-image translation task).
3 Methodology
Our approach is based on three goals. RI) A bi-lipschitz mapping must exist between M: and My:
1 dd (# 20) S dm (0-
S CdM, (z))
where d. d..)) is the geodesic distance in the denoted manifold, 2n and 29 two latent codes, and is constant. Further, Φ +M is a continuous global chart map with its inverse o- R2) Euclidean distances in M, should map to geodesics in My for better structure. R3) The geodesic distance between two arbitrary points on M,, should correspond to a meaningful metric, i.e., pixel distance (note the loss mismatch is implicitly resolved by R1). Next, we explain our training procedure.
Note that every bi- lipschiitz mapping is a homeomorphism.

---

3.1 Geodesics and global bi-lipschitz mapping
Here, we discuss the proposed training procedure. Consider a map 7M, - M,, that parameter- izes a curve on M, using E I C IR. Then, there also exists map (G o 7MM, ) = 'M, + My. If J,M, a geodesic, this mapping can be uniquely determined by a E My and an initial velocity V E My, where M,,,, is the tangent space of M, at (see App. C)². This is a useful result, as we can obtain a unique point p' E My only by defining initial velocity and following 7.M, for time T (note we do not consider the unlikely scenario where two geodesics may overlap at = T).
To find the geodesic between two points on Riemannian manifold, YM, is usually constrained as,
Ry. -MM-
2( @ (im)) (vee(MM iM. -
(v)c 0YM=
1 (f... @ fm.i]
where M = J J is the metric tensor, Jg-1 the Jacobian, @ the outer product, dot operator is the first-order gradient and the double dot operator is the second-order gradient [17]. This approach is expensive, as it requires calculating the Jacobians in each iteration and moreover, causes unstable gradients. In practice, an exact solution is not needed, hence, we adapt an alternate procedure to encourage 7,M, to be a geodesic, and use Eq. 8 only for evaluation purposes in Sec. 4. Since geodesics are locally length minimizing paths on manifold, we encourage the model to minimize the curve length (7M,, (f)) on My in the range & = [0, T]. L(TM, (1)) is measured as:
S.1]
8G TMM (M,
dt = [.'l 8G YMa (1) 0Ym. (t) 0rm. (t)
L(M,, (t))
dt.
(8)
Ot
8t
Eq. 8 can be expressed using the Jacobian Jo as,
% Jo- (t) Øt
dt 1'' v V
drm, (f)
dy..
Ot J Ot
dt.
Since kx = JJ J0-1
(t)
T Orm, (f)
Ot M
dt.
Ot
Considering small A N
Tyym (t)
T grm, (t) Ot M
Al
AM. Σ
(9)
Ot
Further, ||G(A=)) = AZ TT MA: > O, VA > 0, i.c. M is positive definite (since dG * 0, which is ds discussed next). By Hadamard inequality (Eq. 5), it can be seen that we can minimize the ag äz in order for L(MM, (t)) to be minimized. But on the other hand, we also need YM, (T) = y. Therefore, we minimize ac 82
at small intervals along the curve by updating the generator at each t2 = itt,
Ligh (li, zt, x) =I((a(t)) y - a(4)) G( a)] G(
(10)
where i 0, 0, .N, and a(-) is a monotonic function under the conditions a()) and (T) = T. Another perspective for the aforementioned procedure is that the volume element c of
My can be obtained as / det dz. Therefore, det(M) is measure of the distortion in M, with respect to M2 and geodesics prefer to avoid regions with high distortions. The procedure explained so far encourages bi-lipschitz mapping as in Eq. 7 (proof in App. D), and satisfies RL. Further, we show that the enforced bijective mapping removes the loss mismatch between the adversarial and reconstruction losses, hence, improves the visual quality (see Fig. 3 and App. F).
According to R2, the proposed training mechanism should map Euclidean paths on M, to geodesies on MG. Therefore, we move z along Euclidean paths when minimizing Lgh, which also ensures that M2 C I2 Furthermore, we constrain E to be a constant for simplicity. Since we ensure that the distortion of My along the paths of 2 are minimum, practice, it can be observed that the Euclidean paths on the latent space are approximately matched to the geodesics on the output manifold (Fig. 6).
V depends p and hence the dependency of the mapping 7M, on p does need to be explicitly denoted.
5

---

Further, let yV (t) be geodesic curve with an initial velocity V. Then, it can be shown,
Yav (1) = (et),,
(11)
where c is a constant (proof in App. E). This is an important result, since immediately follows
that | >
2 = I( (T)) > ((yaa (T)). Following these intuitions, we define á =
V y G(tta)) I|. This yields an interesting advantage. i.e., (hence L(r((T)) tends to be large
for high y (( which corresponds to R3.
3.2 Encouraging the local bijective conditions
The approach described in Sec. 3.1 encourages global bi-lipschitz mapping between My and M,. However, practically observed that imposing bijective conditions in local neighborhoods in conjunction leads to improved performance. Thus, we enforce dense bijective mapping between My and Ma near T.M, (T') Let zT and y be the latent code T.M, (T) and the ground truth, respectively. We generate two random sets and y using the distribution,
- = (2) and Y (v))
(12)
where (-) applies random perturbations such as brightness, contrast and small noise, and 0 < E2 < 1. One trivial method to ensure that a bijective mapping exists is to apply a loss function Σ| G(
Vzz E 2. y E Y to update the generator. However, we empirically observed that the above loss function unnecessarily applies a hard binding between the perturbations and the generated data. Therefore, we minimize the KL-distance between G and Y up to second order moments. One possible way to achieve this is to model each pixel univariate distribution (App G). However in this case, since the generator cannot capture the correlations between different spatial locations, unwanted artifacts appear on the generated data. Therefore, we treat G' and y as M -ddimensional multivariate distributions (M image height image width). Then, the KL-distance between the distributions up to the second order of moments can be calculated using the following equation,
E.. Σ3
M tr(2o' + (pg u55) 2g (pc us)
Cu. 2, x)
log
(13)
where Σ and denote the correlation matri- ces and the means (App. H). However, using the above loss (Eq. 13) in its original form yields practical obstacles: for instance, the cor- relation matrices have the dimension M x M, which infeasible to handle. Therefore, fol- lowing [27], we use random projection ma- trix RM Maxh h < A to project the images to h didimensional space, where Ri, ~ p(x); (we em-
Algorithm 1: Training algorithm
sample inputs {x1, X2, .., 2.) ~ X; sample outputs {ur, 32, ys} y; for epochs do
for in x do
z B //Sample from k-ball with
small radius
V V-lly-G((a
p( p(v3) 5..p(0 (0) 2 p(-33)
for ''' steps do
pirically justify this reduction method using an ablation study in Sec. 4). Moreover, numerically calculating |كك] and E causes unstable gradi- ents which hinders the generator optimization. We address this issue by adapting the approxi- mation technique proposed in [28]:
sample noise: e NN(0, &
update G: V Luhh (g. 2, 2,
update 2: 2 f- 2 + V/ +
update t: l t- 1+1
update G:
V. Cin 2, +)
Lode (V,
x]
log (2)) tr((")
(14)
where C' I-I-1 Σ. Further, E-1 1 can be calculated as,
V4+1 V3(31 (31 VV.)), 1,2, N,
(15)
where Li et al. [29] proved that V, - Σー as i - xo, for a suitable approximation of Vy- They further showed that a suitable approximation should be Vi a'T, O < a < 2/p(E>"), where p(-) is the spectral radius. Our final loss function Liotal consists of four loss components:
Ltokal = Lo + Lihh + 32h, + BoLado,
(16)
where 3o 3o... B3 are constant weights learned via gross-validation. Further, Tan estimated per mini- Algorithm I shows overall training.

---

Face
Facial Landmark
Skathh
Anime
B5A7 Image
Input:
Color imag
Durs
PIzzF
Birycic SAN DS GAN
OuIs
PIZZPIX
B Bicycle GAN
DS GAN
Durs
PaZP BRycie GAN DS-GAN
Ginerated Samples
Figure 2: Qualitative comparison with state-of-the-art eGANs on three 12I translation tasks. We compare our model with the bascline Pix2Pix [2], Bicycle-GAr [20] and DS-GA DS-GAN[18]. It can be seen that samples generated by model clearly more diverse (c.g., color and subtle structural variation) and realistic (e.g.. shape and color) compared other models in all tasks. Note that model has the same architecture Pix2Pix.
Method
Aacader pppoter.ploto
edges2rlocr
edgerZbogs
DlvtolZaaeune PIPS
BWFlcooor
FID
Inceaaces
Sacess
JIPPS
nightzday
BAcyll--ANN [20]
0.112
58.21
0.109
$4.21
0.139
21.49
0.184
23.33
0.026
75.33
0.00
78.13
0.125
72.93
0.05
98,208
0.103
120.63
DS-GAN [18]
0.181
39.43
0.128
48.13
0.126
27.44
0.113
26.66
0.006
67.41
0.02
71.56
0.168
88.31
0.01
92.14
0.101
137.9
[1]
CGML [5]
0.191
46.2
0.143
42.11
0.13
20.38
0.19
20.43
61.40
0.092
51.4
0.190
73.40
0.14]
51.33
0.100
127.8
Baseline (P2P)
0.011
92.06
0.014
$8.33
0.016
34.50
0.012
32.11
0.001
93.47
0.002
97.14
0.009
121.69
0.021
151.4
0.008
157.3
Ours(P2P 0.148
63.27
0.141
20.48
19.31
0.086
61.33
67.82
0.156
Table Quantitative comparison with the stale-of-the-art on (nine) challenging datasets. * denotes cases where we were not able to make the models converge. higher LPIPS similarity score means more diversity and lower FID score signifies more realism in the generated samples. Our approach gives consistent improvements over the baseline.
Experiments
In this section, we demonstrate the effectiveness of the proposed training scheme using qualitative and quantitative experiments. First, we illustrate the generalizability of our method by comparing against the state-of-the-an methods across a diverse set of image-to poyeetto-maae translation tasks. Then, we explore the practical implications of geometrically structuring the latent manifold. Finally, we conduct ablation study to compare the effects of the empirical choices we made in Sec. 3. In all the experiments, we use Pix2Pix [2] as our model architecture, and use the same model trained using the traditional training approach the main baseline. We use the official implementation of other comparable methods to benchmark their performance against ours. For a fair comparison, we use their pre-trained models wherever available, otherwise train their model from scratch, strictly following the authors' instructions to the best of our ability. For further details on the datasets and hyper-parameter settings, see App.
We compare our method against sfate-of-the-ar models that focus on multimodal image-to- taage-to-image translation. Fig. 2 shows the qualitative results on landmarks - faces, sketch - anime and BW - color. As evident, our training mechanism increases the diversity and the visual quality of the baseline P2P model significantly, and shows better performance compared to other models. Fig. shows qualitative comparison against the baseline. Table depicts the quantitative results. As shown, our model exhibits higher diversity and higher realism multiple
Image-to-image translation:
7

---

datasets. In all the cases, we outperform our baseline by a significant margin. Fig. 4 compares color distribution in BW2color task.
Input
GT
Ours (Pix2Pix Geo)
Bas eine (Pix2Pis)
Edgas
XEX 138
XX
Facardes
Photo
Faces
Generated Samples
Generated Samples
Figure Qualitative comparisons with baseline Pix2Pix [2] model. Our proposed model consistently
generates diverse and realistic samples compared to its baseline Pix2Pix model.
Figure 4: Colour distribution comparison BW color dataset. left: a-plane and right: b-plane in Lab color space, Our model exhibits closest color distribution compared to the ground truth. Furthermore, our model is able to generate rare colors which implies more diverse colorization.
Velocity
(P2P
DS DS-GAN
Bicyclo-GAN
0.3
0.2
0.1
Samples-
9
10
11
Ours (P2P Geo)
Ground
DS-GAN
Input
Bicycle-GAN
Figure 5: A visual example of interpolation along an Euclidean shortest path on the latent manifold. Top row: the velocity = VEMÉ change on My across the samples. Bottom three rows: the corresponding interpolated samples in Bicycle-GAN, DS-GAN, and P2P Geo (Ours). As evident, our model exhibits smooth interpolation along with an approximately constant velocity on M, compared to the other networks, implying that model indeed tends to along geodesics. The total standard deviations of the V for 100 random interpolations for Bicycle-GAN, DS-GAN, and P2P Geo (Ours) are 0.056 0.067, and 0.011, respectively.
Geometrical interpretations: key implication of our training scheme that the Euclidean shortest paths on MM map to geodesics on MM, which preserves better structure. We conduct experiment to empirically validate the aforementioned attribute. First, we travel along Fuclidean paths on M, and measure the corresponding curve length LE on the data manifold. Second, calculate the actual

---

Figure 6: Euclidean path vs. geodesic comparison. We travel along a Euclidean shortest path on Mr and measure the cor- responding curve distance LG on M2 (1m2faces). Then, we traverse between the same two points along the numerically cal- culated geodesic and measure the corresponding curve length Lg- TR{ [.g) vs Lg is illustrated with the corresponding standard devi- ation obtained along 10 random paths. Our model closer to the oracle case (Le = B(Le)). ). We were not able to obtain distance greater than ~ 60 for DS-GAN and Bicyle-GAN which implies that our model generates more diverse data. Further, Pix2Pix did not produce enough diversity for this comparison.
....
Dtack coe PIP Gio (Dum))
Bayd&GGA
DIKIIAN
Elcal (
60-
43-
geodesic distance LG between the same two points on My using Eq. 8 in discrete intervals. We travel in 10 random directions starting from random initial points, and obtain La, for evenly spaced LE E {10, 20, 30, ... 903. Then, we obtain set of the means and standard deviations of LG for the corresponding Lg. Fig. 6 illustrates the distribution. As evident, our model exhibits a significantly high overlap with the ideal curve, i.e., L TE( [Ga) compared to DS-GAN and Bicycle-GAN.
A useful attribute of travelling along the geodesics on the output manifold (My) is to obtain smooth interpolations, since the geodesics tend avoid regions with high distortions, i.e., rapid changes. However, Euclidean shortest paths in the latent spaces (M22) of cGANs often do not correspond to geodesics on the Myy Therefore, in order to travel along geodesics, is required obtain the geodesic paths using Eq. 8, which requires extra computation. In contrast, the proposed training method encourages the generator to map the Euclidean paths on M, to geodesics on My Therefore, smooth interpolations can be obtained by traveling between two latent codes in straight path. To evaluate this, we compare the interpolation results between Bicycle- GAN, DS-GAN and our model. Fig. 5 shows qualitative example, along with quantitative evaluation. As visible. our model exhibits smooth transition from the starting point to the end point. In comparison, Bicycle- shows abrupt and inconsistent changes along the path. DS-GAN does not show any significant variance in the beginning and shows sudden large changes towards the end. We also quantify this
numerically
comparison using the velocity on the data manifold: since the curve length on My can be calculated
using Eq. 9, it easy to see that the velocity on My can be obtained using v V Må.. Fig. illustrates the change in the velocity, corresponding to the given qualitative examples. Our model demonstrates an approximately constant velocity (geodesics have constant velocities), while the other models show sudden velocity changes. We did not include CGML in these evaluations (App. 1).
We conduct ablation study to compare the different variants of the proposed technique. Table 2 depicts the results. First, we compare different distance functions used to calculate Li., As expected, naive maximiza- tion of the distances between the generated sam- ples without any constraints increases the diver- sity, but reduces the visual quality drastically. Further, we observed unwanted artifacts when modeling each pixel as a univariate distribution, as the model then cannot capture dependencies across spatial locations. Then, we compare dif- ferent hown-sampling methods that can be used for efficient calculation of the correlation ma- trices, where random projection performed the best. Interestingly, the visual quality when the dimension of the latent code is increased. In contrast, the diver- sity tends to improve with the latter. We chose dim(2) 64 compromise. Finally, we compare the effects of different combinations of the loss components.
Ablation study:
Variant type
Model
FID
LPIPS
MMD 2nd moment (univaritate)
66.31
0.188
117.53
0.201
Co
Maxinizing distance
132.91
0,232
2nd moment (multivariate)
67.82
0.197
Mean pool Max pool CNN
75.41
0.192
82.42
0.162
Downsampling
77.93
0.191
Random Projection
0.197
16
65.32
0.172
dim(z)
32
67.11
0.188
64
67.82
0.197
128
82.33
0. 0.166
L, + Luss
Training loss
Ligh + Li + Ludt
63.11
0.151
Lik + Li + Ludto
91.3
0.055
Las Lgb + La Lade
67.82
0.197
Table 2: Ablation study. Ablation study with different variants of our model on landmark > faces dataset re- porting FID score (lower more realistic) and I.PPPS (higher = more diverse).
observed reduction of
Generalizability: To demonstrate the generalizability of the proposed algorithm across different loss functions and architectures, we employ it on three classic networks: Pathak et al. [3], Johnson et al. [30], and Ronneberger et al. [31]. These networks use masked reconstruction loss with the

---

Neural Style Transfer
Image Inpainting
Segmentation Mask to Face
Johnson el
Ronneberger el
Domain
solution to improve diversity Solution
Ours (580 Johnsson ml)
Ours (Sco Pathak ot al)
Ours (G00
Hanneberger etali
generalizabel
Figure 7: We apply algorithm to three classic networks and obtain increased diversity with no architectural modifications. Note that the original networks only learn one-to-one mappings.
CelebA-HQ
AFHQ
Method
FID
LPIPS
FID
LPIPS
StarGANv2
13.7
0.452
16.2
0,450
StarGANy2 (modified)
13.1
0.488
15.7
0.477
Table 3: We apply our algorithm to StarGANy2 [32] which improves both realism and diversity.
adversarial loss, perception loss from pre-trained networks, and a reconstruction loss, respectively. Further, in the original form, these networks only learn one-to-one mappings. As depicted in Fig. 7, our algorithm increases the diversity of models and obtains one-to-many mappings with no changes to the architecture (for fair comparison, we concatenate latent code at the bottlenecks during both the original and proposed training). Further, since our algorithm architecture agnostic, it can be easily injected into any suitable model. To show this, replace the Las of StarGANv2 [32] with our loss Ligh In StarGANv2, the images are generated as a composition two functions: the style generator (F(+)) and the image generator (G..-)). We enforce bi-lipschitz constraint between the latent space and the image generator outputs. For continuously differentiable functions and G, if the composition G F is homeomorphic, F and G are individually homeomorphic. Hence, the image generator output becomes homeomorphic to the style space. As shown in Table 3, our approach can improve the performance of StarGANv2, without any modifications the architecture.
5 Conclusion
We show that the cGANs, in their basic form, suffer from significant drawbacks in-terms of diversity and realism. We propose a novel training algorithm that can increase both realism and the diversity of the outputs that are generated by cGANs while preserving the structure of the latent manifold. To this end, we enforce bi-lipschitz mapping between the latent and generated output manifolds while encouraging Euclidean shortest paths on the latent manifold to be mapped to the geodesics on the generated manifold. We establish the necessary theoretical foundation and demonstrate the effectiveness of the proposed algorithm practical level, using diverse set of image-to-image translation tasks, where our model achieves compelling results.
10

---

References
[1] I. Goodfellow, J. Pouget-Abauce Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial nets," in Advances in neural information processing systems, 2014, pp. 2672-2680. 1, 19
[2] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, "Immagetto-image translation with conditional adversarial networks," in Proceedings of the IEEE conference computer vision and pattern
recognition, 2017, pp. 1125-1134. 1, 2, 3, 4, 7,8, 18, 20
[3] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros, "Context encoders: Feature learning by inpainting," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2536-2544, 2, 3,9, 19
[4] S. Lee. J. Ha, and G. Kim, "Harmonizing maximum likelihood with gans for multimodal conditional generation," arXiv preprint urXiv:/902.0990: 1902.09225, 2019. 1, 2, 3, 20
[5] S. Ramasinghe, K. N. Ranasinghe, S. Khan, N. Barnes, and S. Gould, "Conditional generative modeling via learning the latent space," in International Conference on Learning Representations, 2021. [Online]. Available: http://ppeeeeeeeeee////oomn/////
1, 2,
3,7
[6] T.-C. Wang. M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro, "High-resolution image synthesis and semantic manipulation with conditional gans," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 8798-8807. 1,2
[7] J.-Y. Zhu, T. Park, P. Isola, and A. Efros, "Unpaired image-to-imaan translation using cycle-consistent adversarial networks," in Proceedings of the IEEE international conference computer vision, 2017, pp. 2223-2232.
[8] A. Junginger, M. Hanselmann. T. Strauss, S. Boblest, J. Buchner, and H. Ulmer, "Unpaired high-resolution and scalable style transfer using generative adversarial networks," arXiv preprint arXiv:1810.05524. 2018.
(9] Y. Zeng, J. Fu, H. Chao, and B. Guo, "Learning pyramid-context encoder network for high- quality image inpainting." in Proceedings of the IEEE conference on computer vision and pattern recognition, 2019, pp. 1486-1494. 1
(10) C. Yang, X. Lu, Z. Lin, E. Shechtman, O. Wang, and H. Li, "High-resolution image inpainting using multi-scale neural patch synthesis," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 6721-6729.
[11] S. Maeda, "Unpaired image super-resolution using pscudo-supervision." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 291-300. 1
[12] A. Bulat, J. Yang, and G. Tzimiropoulos, "To learn image super-resolution, use gan to learn how to do image degradation first," in Proceedings of the European conference computer vision (ECCV), 2018, pp. 185-200.
[13] T. Zhao, W. Ren, C. Zhang. D. Ren, and Q. Hu, "Unsupervised degradation learning for single image super-resolution." " arXiv preprint arXiv:1812.0422 042.842..444. 2018.
[14] A. Lugmayr, M. Danelljan, and R. Timofte, "Unsupervised learning for real-world super- resolution," in 20/9 JEEE/CVF International Conference on Computer Vision Workshop (IC- CVW). IEEE, 2019, pp. 3408-3416.
[15] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: simple way to prevent neural networks from overfitting." The journal of machine learning research, vol. 15, no. 1, pp. 1929 1958, 2014.
[16] H. Shao, A. Kumar, and P. Thomas Fletcher, "The riemannian geometry of deep generative models," in Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition Workshops, 2018, pp. 315-323. 2, 2, 20
[17] G. Arvanitidis, L. K. Hansen, and S. Hauberg. "Latent space oddity: on the curvature of deep generative models," arXiv preprint arXiv:
2017. 20
[18] D. Yang, S. Hong, Y. Jang, T. Zhao, and H. Lee, DDiversity-sensitii conditional generative adversarial networks," arXiv preprint arXiv: 1rXiv:1901.09024 2019. 2, 7, 20

---

[19] Q. Mao, H.-Y. Lee, H.-Y. Tseng, S. Ma, and M.-H. Yang, "Mode seeking generative adversarial networks for diverse image synthesis." " in Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2019, pp. 1429-1437. 2, 3, 20
[20] J.-Y. Zhu, R. Zhang, D. Pathak, T. Darrell, A. A. Efros, O. Wang, and E. Shechtman, "Toward multimodal image-to-image translation," " in Advances in neural information processing systems,
2017, pp. 465-476. 2, 3, 7,20 20
[21] M. Mathieu, C. Couprie, and Y. LeCun, "Deep multi-scale video prediction beyond mean square error," arXiv preprint 1rXiv:151.005440 2015. 2
[22] C. F. Gauss, Disquisitiones generales circa superficies curvas.
Typis Dieterichianis, 1828,
vol. 3
[23] G. Arvanitidis, S. Hauberg, and B. Schölkopf, "Geometrically enriched latent spaces," arXiv preprint S0l::0008,00660 2020. 3, 20
[24] F. Wang, H. Liu, D. Samaras, and C. Chen, "Topogan: A topology-awe aware generative adversarial network," in European Conference Computer Vision, 2020. 3, 20
[25] V. Khrulkov and I. Oseledets, "Geometry score: A method for comparing generative adversarial 2018. 3, 20
networks," arXiv preprint
[26] S. Gallot, D. Hulin, and J. Lafontaine, Riemannian geometry. Springer, 1990, vol. 2. 4
[27] D. Achlioptas, "Database-friendll random projections," in Proceedings of the twentieth ACM STGMOD-SIGACT-SSIGARR symposium Principles of database systems, 2001, pp. 274-281.
6
[28] C. Boutsidis. P. Drineas, P. Kambadur, E.-M. Kontopoulou, and A. Zouzias, "A randomized algorithm for approximating the log determinant of a symmetric positive definite matrix," " Linear Algebra and its Applications, vol. 533, pp. 95-119, 117, 2017. 6
[29] H.-B. Li, T.-Z. Huang, Y. Zhang. X.PP. Liu, and T.-X. Gu, "Chebyshev-type methods and preconditioning techniques," Applied Mathematics and Computation, vol. 218, no. 2, pp. 260- 270, 2011. 6
[30] L. A. Gatys, A. S. Ecker, and M. Bethge, "A neural algorithm of artistic style," arXiv preprint arXiv: 06576, 2015. 9
[31] O. Ronneberger, P. Fischer, and Brox, "U-net: Convolutional networks for biomedical image segmentation," in International Conference on Medical image computing and compuler-assisted intervention. Springer, 2015, pp. 234-241. 9, 19
[32] Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha, "Stargan v2: Diverse image synthesis for multiple domains." " in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2020, pp. 8188-8197. 10
[33] Z. Zhang, Y. Song, and H. Qi, "Age progressionnregression by conditional adversarial autoen- coder," in IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE,
2017. 18
[34] C.-H. Lee, Z. Liu, L. Wu, and P. Luo, "Maskgan: Towards diverse and interactive facial image manipulation," in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
18
[35] A. Coates, A. Ng, and H. Lee, "An analysis of single-layer networks in unsupervised feature learning," in Proceedings of the fourteenth international conference on artificial intelligence and statistics, 2011, pp. 215-223, 18
[36] T. Kim, Anime Sketch Colorization Pair, 2014.
Available: https:/wwww.aggle..oom/
haacbum/anime-ta http///mmmmmesetteeeeeeeeeeeeeee colorization- pair 18
[37] Johnson, A. Alahi, and L. Fei-Fei, "Pereptulll losses for real-time style transfer and super- resolution," in European conference on computer vision. Springer, 2016, pp. 694-711. 19
[38] D. P. Kingma and M. Welling, "Auto-encoding variational bayes," arXiv preprint arXiv: 1312.61 14, 2013. 19
[39] R. Zhang, P. Isola, and A. A. Efros, "Colorful image colorization," in European conference on computer vision. Springer, 2016, pp. 649-666 20
12

---

[40] S. Ramasinghe, S. Khan, N. Barnes, and S. Gould, "Spectral gans for high-resolution 3d point-cloud generation," " arXiv preprint arXiv:1912.01 2019. 20
-gpettal-gans
[41] Bao, D. Chen, F. Wen, H. Li, and G. Hua, "Cvae-gan: Fine-grained image generation through asymmetric training," in The IEEE International Conference on Computer Vision (ICCV), Oct
2017. 20
[42] H.-Y. Lee, H.-Y. Tseng, J.-B. Huang, M. Singh, and M.-H. Yang, "Diverse image-to-image translation via disentangled representations, in The European Conference on Computer Vision (ECCV), September 2018. 20
[43] Y. Zeng, J. Fu, H. Chao, and B. Guo, "Learning pyramid-context encoder network for high- quality image inpainting," " in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 1486-1494. 20
[44] S. Chang, S. Park, J. Yang, and N. Kwak, Symm-aaraetterized dynamic inference for mixed- domain image translation," " in Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 4803-4811. 20
[45] L. Maaløe, C. K. Sonderby, S. K. Sønderby, and O. Winther, "Auxiliary deep generative models," arXiv preprint arXiv: 1602.0642..,, 2016. 20
[46] D. J. Rezende and S. Mohamed, "Variational inference with normalizing flows," arXiv preprint arXiv:1505.00577, 2015.
13


Reviews:
[{"review_id": "mlShiTtZ6LM", "reviewer": "Reviewer_q3Au", "summary": "To tackle the problem of mode collapse in conditional generative adversarial networks (C-GAN), the paper introduces a new loss function and an associated training procedure, inspired by differential geometric considerations. In particular, the mapping learnt by the generator network is required to be bi-Lipschitz so that it preserves the local Euclidean geometry between the latent and data spaces (the tangent space of the manifold) at every point. The proposed approach is empirically validated on a variety of image to image translation tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#1. Originality/ Novelty: The approach builds upon the work in [18] by analyzing the issue of manifold distortion to propose a novel loss-function to encourage local bijectivity and global bi-Lipschitz mapping to simultaneously address the problems of mode collapse and latent space distortion. The proposed methodology is novel to my knowledge.    \n\n#2. Prior Art: The prior art is adequately cited. \n\n#3. Technical Soundness: The technical formulation appears okay subject to the following queries which the authors should address in the response or fix in the paper. I did check the math in the paper and the Appendix but not completely and rigorously.  \n\nQ1. (line 17-20) Shouldn't the reconstruction loss be between y and G(x,z)?  \n\nQ2: (line 31-35) It's not clear what is meant here:  Aren't the latent space variables the chart variables for the manifold in the data space and for a bijective onto map, locally the map is well approximated as an invertible affine transformation,  i.e. locally, the z-space and the tangent space look very similar to each other and shortest distances will map to each other ? Conversely, if 'all' shortest distances on the data manifold correspond to shortest distance between the pre-images in the latent space, then the mapping function has to be linear (affine)?  \n\nQ3. (lines 36-38) \"naïve coupling of regression and adversarial loss...\" This should be carefully framed. Objectives having multiple terms mostly lead to a compromise.   \n\nQ4: (lines 45-51) Shouldn't bijective homeomorphism imply that dim(Y) = dim(X) + dim(Z)? In other words, the dimensionality of the latent space, by design, should exactly equal the different between the intrinsic dimensionality of the output and input spaces? If so, how is this ensured? Also, since a locally bijective homeomorphic will preserve shortest distances (locally), is the goal to have this property preserved somewhat non-locally (it can't be globally ensured)?   \n\nQ5: Section 2.1, Eqn. (3) and Proof in Appendix D: \\bar{y}(z)  is a function (z). It seems like it is being treated erroneously as independent of z and therefore \\bar{y}(z) appears in the limits of the outer integral with the integrating variable y while the inner integrating variable is z. This doesn't seem correct.   \n\nQ6: Section 2.3 lines 112-120: Generators with a continuous activation function are compositions of continuous functions and hence continuous. Furthermore, bijectivity is enforced by the Bicycle-GANs, and hence Bicycle-GANs should be homeomorphic as well. However, it is correct that the local gradients and the Lipschitz constant of the G network are not explicitly bounded. \n\nQ7: Since locally bi-Lipschitz mapping would suffice to reduce mode collapse as well as bound the distortion, why does a simpler loss function with an additional term similar to (but in the other direction) Equation (2) in [18] doesn't suffice and the mechanism in 3.1 is needed? Even if needed, shouldn't the simpler approach be tried as a baseline? \n\n#4. Technical details and reproducibility: Reasonable technical details are provided in the paper. The methodology is fairly involved and may not be reproducible by itself. However, the authors provide the code (which I have not tested) which should help reproduce the results. \n\n#5. Experimental Evaluation: Results on a variety of tasks are provided. They seem promising though not compelling. For example, the quantitative results in Table 1 are promising (better in majority of cases) but not conclusive with respect to MR-GAN [4], DS-GAN [18] and Bicycle-GAN [20]. Images are too small for qualitative evaluations. Interpolations and the ablation study bolsters the presented approach.\n\n#6. Clarity: The paper needs to be written more carefully (see my queries in point #3 above).  \n\n- It is stated in line 160 that R1 implicitly resolves the loss mismatch, while there are details with regards to this in the supplementary material, an intuitive explanation of the same in the main paper would be helpful given that it is an important motivation for the proposed method.  \n\n- In section-2.2, since the definition and proposition are exactly the same as presented in [18], it might be more useful to present an intuitive explanation of the relevant concepts here, rather than reproducing the definition/proposition verbatim. \n\nMinor typos: \n-- Capital letters should be used for acronyms in the references.  \n-- line 14: 'learns' --> 'learn' \n-- line 121: M_y instead of M_z \n\n#7. Significance:  Using insights from differential geometry to analyze the mode collapse and the manifold distortion problem is good addition to this line of work. The approach is reasonable and the experimental results promising. As the community verifies and reproduces the results, the paper has a potential to be a significant addition to the field. \n\n#8. Preliminary rating and justification: A good paper addressing an important problem in the space of generative models with potentially wide scope for impact. Most ideas presented are well motivated; the proposed method seems novel and is well supported by relevant experimental results. Though there is a need to improve clarity and address my concerns and queries about technical soundness in #3. Hence, I recommend the acceptance of the paper subject to the author response and other reviews. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "lGbPbjIc0FD", "reviewer": "Reviewer_agx2", "summary": "This paper tries to improve the current limitation of conditional GANs with a novel topological space matching view. The authors claim to solve the main drawbacks of existing cGANs that cause the GAN to lack diversity and randomness. Basically, they propose an optimization algorithm to encourage the generator and the latent space to preserve a bi-lipschitz mapping between generated space and GT space. The authors compare evaluate their method on image translation task and show higher quality of generator comparing with existing SOTA cGANs.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ This paper theoretically analysis the main drawback of existing cGANs in view of space matching. It is very interesting to explain the current drawback of GANs (mode collapse, insensitivity to randomness) using Riemannian geometry.\n+ The authors provide detailed proof of their claim and theory.\n+ The experimental results show significant improvement on image translation tasks comparing with existing major cGANs,.\n \nThis is a good paper. There are still some flaws in this paper.\n\t1. This paper lacks essential visualization of helping to understand. Some parts of their explanation are not easy to follow.\n\t2. Some notations in section 3.1 are not cleared. It is suggested to summarize all notations in a list before going into the math part.\n\t3. It is encouraged to evaluate the method on wider datasets and tasks. For example, evaluate it on sentence generation or cross-domain generation tasks.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper potentially has impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "YntqwiPX2cL", "reviewer": "Reviewer_9CLc", "summary": "This paper suggests that the current way of training conditional generative models can be the root of the mode-collapsing problems and proposes a fix at a fundamental level. By imposing bi-Lipschitz constraint to the relationship between latent space and image space, the baseline model shows largely improved diversity and quality without any architectural modifications. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a very nice work with impressive analyses and the paper is very well written. \n\n\nIn particular, I am very impressed with how systematically the authors investigate the deficiencies of the current state-of-the-art. The way that the authors analyze the problem is very convincing, and the theoretical analyses provide a good insight on the prevailing problem in conditional generative models at a fundamental level. I believe this insight will be useful for the future developments of the field. \n\n\nMy main concern is on the quality of the results and the baseline models that the authors used. Although Pix2Pix is a good baseline to showcase the prominent problem that is being addressed, the results would have looked more persuasive if they were compared with more recent image-to-image translation models that addressed the intra/inter domains diversity such as, MUNIT, StarGANv2 , etc. (since the method can be applied generally.) The quantitative and qualitative results largely fall behind to those of the current state-of-the-arts, which keeps me from giving a strong score to this paper. (since the practical impact is also a very important aspect in this field.) \n\n\nRelated to the previous comment, I am curious if this would be possible to be applied to more aggressive translations such as among different domains? (Since it is constraining the relationship quite strongly, I am not sure if this can be applicable to add any new features that exist in the new domain, e.g., to translate a dog to a lion, one needs to draw its mane.)\n\n\nLine 121 the distortion of M_z with respect to M_z. -> I guess one of these two must be M_y?\n\n\nLine 162 _I_ is not defined. I guess, this stands for an interval?\n\n---\n\nAfter rebuttal (update): I raise the score to 7. \n\n---\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "no comments", "review_text": "", "extra_scores": {"ethical_concerns": "no comments", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "Ngqd5ZTgPvw", "reviewer": "Reviewer_X71S", "summary": "This paper attempts to address poor sample quality and sample diversity in image conditioned generative models by addressing the topology mismatch between latent space and image space. It introduces two new loss functions: one which encourages Euclidean paths in the latent space to map to geodesic paths in image space, and another which helps to enforce a bijective mapping between the latent manifold and the output image manifold. Experimental results demonstrate quantitative and qualitative improvement over previous methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n - Good theoretical motivation for the proposed loss functions.\n - Well written and fairly easy to follow.\n - I enjoyed the analysis of model properties such as colour distribution and interpolation behaviour. Figure 4 in particular does a great job of demonstrating why we might want good mappings between Euclidean latent paths to geodesics in image space.\n\nWeaknesses:\n- The main contribution of matching Euclidean paths in the latent manifold to geodesic paths in the image manifold seems to have been proposed before as \"path length regularization\" in StyleGAN2 [1]. Can the authors comment on the differences between path length regularization and their method?\n- Performance improvement over existing methods is not clear. Proposed methods achieves better image quality on only 4 of 9 datasets, and better diversity on 6 of 9 datasets. That being said, performance improvement over the baseline is significant. Could the proposed method be combined with other SOTA methods to achieve better performance across all datasets?\n\n\nQuestions:\n- The loss in Equation 10 appears to require several iterations for each training batch. How does this affect training time compared to the baseline model?\n- Random perturbations such as brightness and contrast are applied to real images in Equations 12 and 13 in order to improve sample diversity. How does this compare to simply applying these augmentations to the full training set?\n\n\nOther comments:\n- Line 27: May want to explain more about the significance of dropout in case readers are not aware of its function in Pix2Pix.\n- Line 157: Potential typo: \"...is a continues global chart map\". Should \"continues\" be \"continuous\"?\n- Table 1: It seems weird to me to use the incomplete acronym LPIP (Learned Perceptual Image Patch). Using the full acronym LPIPS (Learned Perceptual Image Patch Similarity) would make more sense.\n- Figure 1 does not include comparison to CGML despite it being one of the better models being compared against. It may be good to include some comparison with CGML as well so as to not mistakenly mislead readers about the quality improvements compared to previous methods.\n\nOverall, I thought this was a good paper. My main concern is about how similar the proposed method is to path length regularization. If the authors could demonstrate that their method outperforms path length regularization I would feel much better about accepting this paper.\n\n[1] Karras, Tero, et al. \"Analyzing and improving the image quality of stylegan.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\n---\nAfter rebuttal:  \nMy concerns have been addressed. I have increased my score from 5 to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Paper does not discuss negative societal impact of work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were in agreement that the method is of broad interest to the community, and is sufficiently general to improve many types of GANs. In the initial reviews, the author's noted that the GAN used was not sufficiently recent. The authors addressed this and other concerns more than satisfactorily, and all reviewers agree that this paper should be accepted. I agree. Nice work!
Decision: Accept (Poster)