Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
Summary:

The paper proposes an analysis way of a latent space of GAN in a GAN architecture agnostic way. They use the Riemannian manifold analysis to investigate image manifold, which leads to a simple algorithm with eigen-decomposition of the Hessian matrix of a local point.

Unfortunately, many ideas and some of the findings (Hessian-based GAN exploration and anisotropy of Hessian in the latent space) are already well explored in [C1] with much higher quality and various modality applications.
But still, there are interesting ideas contained in this paper: 1) the latent space is homogeneous (in a sense that the major directions obtained by a latent position are shared at different positions with similar semantic meanings), and 2) extensive eigenvalue analyses of Hessian.

Based on the findings, they show interesting applications of efficient GAN Inversion, gradient-free search in Image space, and unsupervised discovery of interpretable axes. However, these applications are also already explored by [C1] in a visually pleasing way.

[C1] Human-in-the-Loop Differential Subspace Search in High-Dimensional
Latent Space, SIGGRAPH 2020.

Reasons for score: 

I like the idea that leverages the findings of the homogeneous property of Hessian and improves the efficiency of GAN inversion with Hessian preconditioning. Also, I appreciate the analysis of eigenvalues and correlations.
However, unfortunately, most of the contributions the authors argued are largely overlapped with [C1], of which citation is missed by the authors. Thus, with these remaining contributions, this reviewer could not vote for the acceptance of this work at this point.
This reviewer would like to ask the significant different contribution from [C1] that this reviewer may miss.

Pros:
- The paper shows interesting interpretation and ideas, and its applications.
- The paper is written very clearly and readily.
- The authors validate the ideas with enough effort in the experiments.

Cons:
- The GAN architecture agnostic latent space exploration with Riemannian interpretation is already done in [C1] in a more general way. 


Detail comments:
- While [C1] presents mainly about the SVD of Jacobian (but they also show Hessian interpretation (metric tensor) in Sec. 4.1 as well), it is essentially the same because of Eq. (3) in this paper. Also, [C1] suggested much more diverse exploration methods with diverse search methods. 
Thus, this reviewer would like to listen to the authors' responses about this overlap.
review_title: Significant overlap of the contributions with [Chiu et al., SIGGRAPH 2020]

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a method for finding the axes of largest variation in the latent space of a generative model. This can be leveraged for better generator inversion and explainability. Several experiments are done to evaluate the latent vectors used by the method quantitatively and qualitatively.

##########################################################################

Reasons for score: 

The analyses are interesting and the method seems novel. This method could prove useful in analysis of latent space properties. Its full significance and practicality could be clearer, though.

##########################################################################

Pros:
- The problem of understanding the latent space structure better is relevant and timely in the research on generative models equipped with a continuous latent space.
- Experiments seem to demonstrate successful dimensional truncation, GAN inversion, and that the eigenvectors behave consistently

##########################################################################

Cons:
- I am not sure I see whether each experiment actually supports the corresponding claim. Especially, the claim that interpretable axes are found seems central but it is not much supported in the experiments. It seems mostly represented in Fig. 9 in the Appendix, but I do not fully understand how we are supposed interpret the result in this figure.
- Fig 2 is hard to parse and clearly too tightly packed. Something should be done about it.

##########################################################################

Questions during rebuttal period:
- It is easy to get lost in the math and in the various measurements. Could the authors summarize, in practise and plain English, how their method should be used to find the most intepretable axes of variation for a new generative model, and how to confirm and measure that we have indeed found them?

Typo: on page 7 around "(Fig 5.)"

##########################################################################

Update after rebuttal discussions:
- In the light of the considerable overlap with [Chiu et al., 2020] pointed out by the other reviewers, I decreased my score. I have familiarized myself with it and can confirm the said overlap. However, given remaining differences, I do not find it unreasonable to consider this paper as "complementary" to [Chiu et al., 2020], *provided that the authors explicitly address the similarities in the final version*.
- I consider the sum total of contributions of the paper still tilting towards being sufficient for publication.
review_title: Review: THE GEOMETRY OF DEEP GENERATIVE IMAGE MODELS AND ITS APPLICATIONS

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper performs the analysis of the GAN latent spaces from the geometric perspective, inducing a metric tensor in the latent space from the LPIPS distance in the image space. The main authors' finding is that under such metric, the latent spaces of typical GANs are highly anisotropic, which can be exploited for more effective GAN inversion. Furthermore, the authors show that eigen vectors of the metric tensor often correspond to interpretable latent transformations.

Pros:

1) The paper is exceptionally well-written and provides a very interesting read. While the performed analysis is simple and natural, it does reveal several interesting findings about typical latent spaces: LPIPS-anisotropy, global consistency of the metric tensor.

2) The authors confirm the usefulness of their analysis by providing immediate practical benefits: more effective GAN inversion, which accounts for the latent space anisotropy.

Cons:

1) Missing work on interpretable GAN directions:

[A] The Hessian Penalty: A Weak Prior for Unsupervised Disentanglement, ECCV 2020

2) In my opinion, the authors do not provide enough support for their claim "This finding unifies previous unsupervised methods that discover interpretable axes in the GAN space".

- While the proposed method does seem to generalize both Ha ̈rko ̈nen et al., 2020 and Shen & Zhou, 2020, I do not see, how it captures Voynov & Babenko, 2020 and Pebbles et al (see above [A]). Furthermore, I believe that such claims should be supported by the experiments.  Could the authors experimentally confirm that their method results in the same set of directions as the existing methods?

Overall, I am positive about this submission, since the main analysis is both interesting and practically useful. My main criticism is that in terms of discovery of interpretable directions, the methods should be experimentally compared to existing alternatives. If it does provide a super-set of directions, obtained by existing methods, this would make the submission much stronger. Otherwise, the claim about unification should be toned down in my opinion.

======== AFTER REBUTTAL ========

I appreciate the authors' efforts on additional thorough comparison to existing works on interpretable axes discovery. From the updated manuscript, however, it is not clear what method is superior and the authors' approach appears to be a yet another method for this task rather than generalization of previous ones. Overall, I am still on the positive side since the observed findings deliver a clear profit for GAN inversion. But I am not increasing my score given that the "interpretable axes" part has become less impressive (in terms of weaker claims and conclusions) and the competing SIGGRAPH work.   
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: _Summary_:
This work intends to explore the geometry of the latent space and proposes to define the distance in latent space as the distance between the corresponding generated images and use the Hessian of that squared distance as metric tensor to define the manifold. Using Learned Perceptual Image Patch Similarity (LPIPS), they show that the Hessian can either be computed through backpropagation or if that is not efficient, it is sufficient to iteratively compute the eigenvectors corresponding to the largest eigenvalues. With the proposed method, the empirical observations showed 1) the impact of those eigenvectors through examples, 2) consistent geometric local changes over different positions in the latent space, and 3) the impact of top eigenvectors on particular layers. Further, the authors discuss three areas of possible application (gradient-based GAN inversion, gradient-free image search, interpretable axes discovery).

_Strengths_:
- The paper addresses an interesting topic (understanding the latent space of GANs) and proposes a straightforward method.
- The empirical observations demonstrates the advantages of using an image similarity metric as latent space distance and the correlation between eigenvalues corresponding to the largest eigenvalues and image perception.

_Weaknesses_:
- The method is depending heavily on the distance metric employed, however, does not discuss in what way this could influence the outcome. In this paper, only LPIPS was considered as metric. What are the advantages and disadvantages? How would the analysis and outcome change when other distance metrics are used? These are questions that might benefit the work to discuss.
- The actual method was barely discussed in the paper, but rather moved to the appendix. I think the authors may want to restructure the paper to include how to compute the eigenvectors are computed.
- The actual application of the proposed method was rather discussed than actually performed. I do think this is a very interesting proposal, however, with only showing empirical observations, I believe the scope of this paper is too little. It would have been better if the authors picked one of the applications and applied their method.

For more details see below.

_Overall assessment_: 
Overall, I find the proposed method very interesting and the empirical observations compelling. However, I find the scope of the paper not sufficient as it would have been nice to see that one of the application with the usage of the Hessian metric as distance function would have worked. 

_Detailed questions and comments_:
- Learned Perceptual Image Patch Similarity (LLIPS, Section 3 "Numerical Method"): As this is the main distance metric being used, would it be possible to briefly introduce and define it in the paper?
- Requirement for understanding the latent space (abstract): The abstract mentions inversion and interpretability as requirement for understanding the latent space. It was not clear to me how the method presented is addressing each of the requirement. Further, the abstract also claims "This geometric understanding unifies previous results of GAN inversion and interpretation.". Can you clarify these claims?
- Appendix A.2 Methods for computing the hessian: Parts of how the Hessian is computed should be in the main paper as this is the main method for this paper. The explanation in Section 3 "Numerical Method" was superficial and I could have not understood the methodology without reading the the appendix.
- Spectrum Structure of GANs (Section 4, Figure 2): Can you clarify what has been exactly used to plot Figure 2? In the paragraph, Figure 2 was plotted using the mean and 90% confidence interval, Figure 2 y-axis label says $\log(eig/eigmax)$.
- Figure 3D: The figure is hard to read, with many data points overlapping each other. I am also confused what the two lines crossing each data point represents.
- "Then we explored linearly the latent space along the eigenvector" (Section 4): Why is the exploration linearly? Does this conform with the manifold? How is $\mu_i$ defined? The footnote also says that spherical linear exploration is used some spaces. Can you elaborate more on how you performed exploration?
- Top Eigenvectors Capture Perceptual Relevant Changes: Would it be able to quantify this? As only four samples were shown, how would we know that this generalizes for all samples?
- [1]-[4] might be also relevant to the topic of latent space exploration. They are not necessarily w.r.t. to manifold learning, however, with respect to applications in Section 6, they look very close. 

_Post-rebuttal_:
I do highly appreciate the authors trying to answer all our questions and adding more details and experiments. However, after also reading through [Chiu et al.,  SIGGRAPH 2020] I do find that this paper has a large overlap with the one mentioned. Therefore, agreeing with Reviewer #2, the contribution is reduced to applying the idea to GANs. Therefore I am keeping my recommendation.

[1] Lipton, Z.C. and Tripathi, S., 2017. Precise Recovery of Latent Vectors from Generative Adversarial Networks. ICLR 2017 workshop.
[2] Albright, M. and McCloskey, S., 2019, May. Source Generator Attribution via Inversion. In CVPR Workshops (Vol. 7).
[3] Webster, R., Rabin, J., Simon, L. and Jurie, F., 2019. Detecting overfitting of deep generative networks via latent recovery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 11273-11282).
[4] Bojanowski, P., Joulin, A., Lopez-Pas, D. and Szlam, A., 2018, July. Optimizing the Latent Space of Generative Networks. In International Conference on Machine Learning (pp. 600-609).
review_title: Official Blind Review #1

Meta-review: This paper presents several analyses on the geometry of GAN generators through the lens of Riemannian geometry: showing interpretability of the leading eigenvectors of the Hessian, homogeneity of the space, and more efficient latent-space inference through preconditioning. Reviewers found the (revised) paper well-written and clear, with a thorough set of experiments to support their main claims. While there were several concerns around the generality of the approach, the authors performed several experiments in the rebuttal period to address many of the reviewer’s concerns (robustness of findings with different image distance functions, inversion on additional GANs and datasets, user study of perceptual properties of axes, and comparison to previous methods for intepretable axes discovery). I found these experiments extensive and convincing, supporting the claims around robustness of the approach to different image distance metrics, GAN architectures, and interpretability of the axes. 

There were also strong concerns around similarity with recent work (Chiu et al., SIGGRAPH 2020 and Peebles et al., ECCV 2020), but both of these papers were published at most 1 month before the ICLR submission deadline, and thus should be considered as concurrent work. 

Given the strong set of additional experiments and interesting empirical observations, I recommend accepting this paper.

There remain concerts around the extent to which the findings “unify” previous approaches on interpretable axes, and we encourage the authors to update the paper before the camera ready to address these and additional reviewer concerns (especially expanding the discussion of the relationship with concurrent work in Chiu et al. and Peebles et al.).
Decision: Accept (Poster)