Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: To tackle the problem of mode collapse in conditional generative adversarial networks (C-GAN), the paper introduces a new loss function and an associated training procedure, inspired by differential geometric considerations. In particular, the mapping learnt by the generator network is required to be bi-Lipschitz so that it preserves the local Euclidean geometry between the latent and data spaces (the tangent space of the manifold) at every point. The proposed approach is empirically validated on a variety of image to image translation tasks.
main_review: #1. Originality/ Novelty: The approach builds upon the work in [18] by analyzing the issue of manifold distortion to propose a novel loss-function to encourage local bijectivity and global bi-Lipschitz mapping to simultaneously address the problems of mode collapse and latent space distortion. The proposed methodology is novel to my knowledge.    

#2. Prior Art: The prior art is adequately cited. 

#3. Technical Soundness: The technical formulation appears okay subject to the following queries which the authors should address in the response or fix in the paper. I did check the math in the paper and the Appendix but not completely and rigorously.  

Q1. (line 17-20) Shouldn't the reconstruction loss be between y and G(x,z)?  

Q2: (line 31-35) It's not clear what is meant here:  Aren't the latent space variables the chart variables for the manifold in the data space and for a bijective onto map, locally the map is well approximated as an invertible affine transformation,  i.e. locally, the z-space and the tangent space look very similar to each other and shortest distances will map to each other ? Conversely, if 'all' shortest distances on the data manifold correspond to shortest distance between the pre-images in the latent space, then the mapping function has to be linear (affine)?  

Q3. (lines 36-38) "naïve coupling of regression and adversarial loss..." This should be carefully framed. Objectives having multiple terms mostly lead to a compromise.   

Q4: (lines 45-51) Shouldn't bijective homeomorphism imply that dim(Y) = dim(X) + dim(Z)? In other words, the dimensionality of the latent space, by design, should exactly equal the different between the intrinsic dimensionality of the output and input spaces? If so, how is this ensured? Also, since a locally bijective homeomorphic will preserve shortest distances (locally), is the goal to have this property preserved somewhat non-locally (it can't be globally ensured)?   

Q5: Section 2.1, Eqn. (3) and Proof in Appendix D: \bar{y}(z)  is a function (z). It seems like it is being treated erroneously as independent of z and therefore \bar{y}(z) appears in the limits of the outer integral with the integrating variable y while the inner integrating variable is z. This doesn't seem correct.   

Q6: Section 2.3 lines 112-120: Generators with a continuous activation function are compositions of continuous functions and hence continuous. Furthermore, bijectivity is enforced by the Bicycle-GANs, and hence Bicycle-GANs should be homeomorphic as well. However, it is correct that the local gradients and the Lipschitz constant of the G network are not explicitly bounded. 

Q7: Since locally bi-Lipschitz mapping would suffice to reduce mode collapse as well as bound the distortion, why does a simpler loss function with an additional term similar to (but in the other direction) Equation (2) in [18] doesn't suffice and the mechanism in 3.1 is needed? Even if needed, shouldn't the simpler approach be tried as a baseline? 

#4. Technical details and reproducibility: Reasonable technical details are provided in the paper. The methodology is fairly involved and may not be reproducible by itself. However, the authors provide the code (which I have not tested) which should help reproduce the results. 

#5. Experimental Evaluation: Results on a variety of tasks are provided. They seem promising though not compelling. For example, the quantitative results in Table 1 are promising (better in majority of cases) but not conclusive with respect to MR-GAN [4], DS-GAN [18] and Bicycle-GAN [20]. Images are too small for qualitative evaluations. Interpolations and the ablation study bolsters the presented approach.

#6. Clarity: The paper needs to be written more carefully (see my queries in point #3 above).  

- It is stated in line 160 that R1 implicitly resolves the loss mismatch, while there are details with regards to this in the supplementary material, an intuitive explanation of the same in the main paper would be helpful given that it is an important motivation for the proposed method.  

- In section-2.2, since the definition and proposition are exactly the same as presented in [18], it might be more useful to present an intuitive explanation of the relevant concepts here, rather than reproducing the definition/proposition verbatim. 

Minor typos: 
-- Capital letters should be used for acronyms in the references.  
-- line 14: 'learns' --> 'learn' 
-- line 121: M_y instead of M_z 

#7. Significance:  Using insights from differential geometry to analyze the mode collapse and the manifold distortion problem is good addition to this line of work. The approach is reasonable and the experimental results promising. As the community verifies and reproduces the results, the paper has a potential to be a significant addition to the field. 

#8. Preliminary rating and justification: A good paper addressing an important problem in the space of generative models with potentially wide scope for impact. Most ideas presented are well motivated; the proposed method seems novel and is well supported by relevant experimental results. Though there is a need to improve clarity and address my concerns and queries about technical soundness in #3. Hence, I recommend the acceptance of the paper subject to the author response and other reviews. 
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper tries to improve the current limitation of conditional GANs with a novel topological space matching view. The authors claim to solve the main drawbacks of existing cGANs that cause the GAN to lack diversity and randomness. Basically, they propose an optimization algorithm to encourage the generator and the latent space to preserve a bi-lipschitz mapping between generated space and GT space. The authors compare evaluate their method on image translation task and show higher quality of generator comparing with existing SOTA cGANs.
main_review: + This paper theoretically analysis the main drawback of existing cGANs in view of space matching. It is very interesting to explain the current drawback of GANs (mode collapse, insensitivity to randomness) using Riemannian geometry.
+ The authors provide detailed proof of their claim and theory.
+ The experimental results show significant improvement on image translation tasks comparing with existing major cGANs,.
 
This is a good paper. There are still some flaws in this paper.
	1. This paper lacks essential visualization of helping to understand. Some parts of their explanation are not easy to follow.
	2. Some notations in section 3.1 are not cleared. It is suggested to summarize all notations in a list before going into the math part.
	3. It is encouraged to evaluate the method on wider datasets and tasks. For example, evaluate it on sentence generation or cross-domain generation tasks.

limitations_and_societal_impact: This paper potentially has impacts.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper suggests that the current way of training conditional generative models can be the root of the mode-collapsing problems and proposes a fix at a fundamental level. By imposing bi-Lipschitz constraint to the relationship between latent space and image space, the baseline model shows largely improved diversity and quality without any architectural modifications. 
main_review: This is a very nice work with impressive analyses and the paper is very well written. 


In particular, I am very impressed with how systematically the authors investigate the deficiencies of the current state-of-the-art. The way that the authors analyze the problem is very convincing, and the theoretical analyses provide a good insight on the prevailing problem in conditional generative models at a fundamental level. I believe this insight will be useful for the future developments of the field. 


My main concern is on the quality of the results and the baseline models that the authors used. Although Pix2Pix is a good baseline to showcase the prominent problem that is being addressed, the results would have looked more persuasive if they were compared with more recent image-to-image translation models that addressed the intra/inter domains diversity such as, MUNIT, StarGANv2 , etc. (since the method can be applied generally.) The quantitative and qualitative results largely fall behind to those of the current state-of-the-arts, which keeps me from giving a strong score to this paper. (since the practical impact is also a very important aspect in this field.) 


Related to the previous comment, I am curious if this would be possible to be applied to more aggressive translations such as among different domains? (Since it is constraining the relationship quite strongly, I am not sure if this can be applicable to add any new features that exist in the new domain, e.g., to translate a dog to a lion, one needs to draw its mane.)


Line 121 the distortion of M_z with respect to M_z. -> I guess one of these two must be M_y?


Line 162 _I_ is not defined. I guess, this stands for an interval?

---

After rebuttal (update): I raise the score to 7. 

---

limitations_and_societal_impact: no comments
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper attempts to address poor sample quality and sample diversity in image conditioned generative models by addressing the topology mismatch between latent space and image space. It introduces two new loss functions: one which encourages Euclidean paths in the latent space to map to geodesic paths in image space, and another which helps to enforce a bijective mapping between the latent manifold and the output image manifold. Experimental results demonstrate quantitative and qualitative improvement over previous methods.
main_review: Strengths:
 - Good theoretical motivation for the proposed loss functions.
 - Well written and fairly easy to follow.
 - I enjoyed the analysis of model properties such as colour distribution and interpolation behaviour. Figure 4 in particular does a great job of demonstrating why we might want good mappings between Euclidean latent paths to geodesics in image space.

Weaknesses:
- The main contribution of matching Euclidean paths in the latent manifold to geodesic paths in the image manifold seems to have been proposed before as "path length regularization" in StyleGAN2 [1]. Can the authors comment on the differences between path length regularization and their method?
- Performance improvement over existing methods is not clear. Proposed methods achieves better image quality on only 4 of 9 datasets, and better diversity on 6 of 9 datasets. That being said, performance improvement over the baseline is significant. Could the proposed method be combined with other SOTA methods to achieve better performance across all datasets?


Questions:
- The loss in Equation 10 appears to require several iterations for each training batch. How does this affect training time compared to the baseline model?
- Random perturbations such as brightness and contrast are applied to real images in Equations 12 and 13 in order to improve sample diversity. How does this compare to simply applying these augmentations to the full training set?


Other comments:
- Line 27: May want to explain more about the significance of dropout in case readers are not aware of its function in Pix2Pix.
- Line 157: Potential typo: "...is a continues global chart map". Should "continues" be "continuous"?
- Table 1: It seems weird to me to use the incomplete acronym LPIP (Learned Perceptual Image Patch). Using the full acronym LPIPS (Learned Perceptual Image Patch Similarity) would make more sense.
- Figure 1 does not include comparison to CGML despite it being one of the better models being compared against. It may be good to include some comparison with CGML as well so as to not mistakenly mislead readers about the quality improvements compared to previous methods.

Overall, I thought this was a good paper. My main concern is about how similar the proposed method is to path length regularization. If the authors could demonstrate that their method outperforms path length regularization I would feel much better about accepting this paper.

[1] Karras, Tero, et al. "Analyzing and improving the image quality of stylegan." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

---
After rebuttal:  
My concerns have been addressed. I have increased my score from 5 to 7.
limitations_and_societal_impact: Paper does not discuss negative societal impact of work.
review_title: null

Meta-review: The reviewers were in agreement that the method is of broad interest to the community, and is sufficiently general to improve many types of GANs. In the initial reviews, the author's noted that the GAN used was not sufficiently recent. The authors addressed this and other concerns more than satisfactorily, and all reviewers agree that this paper should be accepted. I agree. Nice work!
Decision: Accept (Poster)