Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a geometric method for image editing with pretrained StyleGAN2. The idea is simple. Editing is performed by constraining the editing path on the manifold. The simplest way of making this is to use the tangent space. A natural realization of the tangent space is the principal subspace of the Jacobian. The authors harness the Jacobian regarding the mapping from $z$ to $w$. The extensive experimental results verify the effectiveness of the method.
main_review: The idea is simple and convincing. The key is to use the principal subspace of the Jacobian to extract  attribute representations. The novelty lies in two aspects: 1) the authors study the Jacobian of the mapping network, as opposed to previous works and 2) the editing path is iteratively formed by taking the most similar direction  of adjacent tangent spaces (principal subspaces of the Jacobians). 

A very relevant NeurIPS work is missed.

Low-Rank Subspaces in GANs
https://arxiv.org/abs/2106.04488

In this paper, the authors reviewed the related work using the Jacobians. Please cite those related papers. In this work, the authors show that the principal subspace of the Jacobian may fail to capture disentangled attribute directions and result in artifacts.  They use the null space of the Jacobian with respect to complementary region  to improve the performance. Please discuss this problem in your paper.
summary_of_the_review: A simple and effective algorithm supported by geometry. But the author did not seriously discuss the drawback revealed by the relevant paper. This part should be improved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper propose to explore disentanglement locally through the top singular vectors of the Jacobian of the generator of a GAN. This contrast the popular view that disentanglement should follow Euclidean coordinate axes in the latent space. An algorithm for traversing the latent space is proposed.
main_review: Disentanglement of a representation often stems from intuitions gathered from linear models, namely that we should expect that different aspects of data may change as we move along a chosen Euclidean coordinate axis (similar to principal components). This paper challenges this view and argue that we should only locally follow a specific direction, and that this direction should come from the subspace spanned by the top singular vectors of the Jacobian of the generator (the specific paper investigate GANs, but the idea is general). It is trivially seen (Proposition 1) that if we consider samples from a 'small' Gaussian in latent space, then looking at the directions of the top singular vectors of the Jacobian amounts to local PCA in observation space (follows from Taylor's theorem).

This analysis is sound and reasonable. It is, however, not particularly novel. In the context of VAEs (rather than GANs), there is a large body of work that equip the latent space with the Riemannian pull-back metric associated with the generator, i.e. $G(z) = J(z)^T J(z)$, where $J$ is the Jacobian, and $G$ is then the metric tensor. Existing work then study the general identifiability problem associated with latent variable models; this is arguably a more general problem than the disentanglement problem. Riemannian quantities such as geodesics then serve similar purposes to the traversels proposed in the present paper, though geodesics are perhaps better understood. I speculate if indeed the proposed traversel method isn't just a first-order Euler method for solving the initial value problem associated with geodesics.

Related work in this regard (these groups of authors has published significantly more on the topic, but I just list the most commonly cited papers):
1. Latent Space Oddity: on the Curvature of Deep Generative Models. G. Arvanitidis, LK. Hansen & S. Hauberg. ICLR 2018. This was the first paper to study the Riemannian geometry of VAEs; a key technical difference to the paper at hand is that Arvanitidis et al. considered stochastic generators, while the current paper considers deterministic generators.
2. N. Chen, A. Klushyn, R. Kurle, X. Jiang, J. Bayer & P. Smagt. Metrics for deep generative models. AISTATS 2018. Similar to the ICLR paper above except it considers deterministic generators.
3. H. Shao, A. Kumar & PT. Fletcher. The Riemannian geometry of deep generative models. CVPR Workshops 2018. Similar to paper 2.

These papers focus on the identifiability problem and not the disentanglement problem, so in that regard the present paper has a novel contribution. This, however, seems rather minor as identifiability is arguable a more general and harder problem.

I have already mentioned that the traversal algorithm seems quite similar to geodesic computations, e.g. akin to the algorithm proposed in paper 3 above. It also seems very similar to the Brownian motion / diffusion sampling proposed in this paper:

4. Variational Autoencoders with Riemannian Brownian Motion Priors. D. Kalatzis, D. Eklund, G. Arvanitidis & S. Hauberg. ICML 2020

The main difference seems to be that paper 4 solves and SDE (they add a bit of Gaussian noise in each step), while the present seems more like an ODE solver.

The current paper, like so many other disentanglement papers, has a somewhat ad hoc empirical evaluation strategy. This is fair given the complexity of the topic. Results seems encouraging, but it's really difficult to get a feel for which approaches are better. I don't think this is the authors fault, but it still seems a bit disappointing.

I should also note that I did not really understand the point of the warpage evaluation. It isn't clear to me what the authors aim to achieve in this part of the paper.

All that being said, I think the paper is nicely presented and the empirical results are generally quite nice (unlike the above papers that are more proof-of-concepts).

*Post-rebuttal*
I think the authors answered my key concerns during our discussions, and I trust that the paper will be adjusted accordingly. I have therefore bumped my score up a bit to recommend acceptance.
summary_of_the_review: The key ideas behind the paper has appeared in a series of papers elsewhere, and these arguably provide a deeper understanding of the problem than what is discussed here. That makes it hard to recommend acceptance. That being said, the paper is well-written and the empirical results are interesting, hence the borderline/reject score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors proposed Local Basis: a purposeful traversal direction based on the local-geometry of the intermediate latent space of a GAN. To support the theory of Local Basis, the global-geometry of the latent space, along with an iterative traversal method to trace the latent space, moves apart from the space in a piece-wise, step-by-step manner. Their results show that Local Basis decomposes an image's semantics, providing more stability in transforming images via the proposed iterative traversal. Following the suggested evaluation of the W-space of StyleGAN2 reveals that the W-space is distorted at the global level: the global method, thus, limits global consistency while in W-space.

main_review: Overall, they did well in this work and are high in potential. 

STRENGTHS:
- End-to-end, the paper stays focused on a consistent theme via a well-organized summary of their work.
- Visualizations show concepts well
- There is a nice distribution of research paper components (e.g., figures, tables, proper intro, background, conclusion, and such).
- The idea is interesting and relevant.

WEAKNESSES:
- Figures have text that is difficult to read (text in figures should match the font that is used in the paper's body, in style and size).
- Equations look okay, but there are several components that should not be variable (i.e., italicized). For instance, "Local Basis (...)..." in Eq 4 should be wrapped in \text{Local Basis}-- there are many instances like this. Make sure only variables are italicized in equations.
- Fig 1, 4, 6, 8 are low in quality. Increase the resolution and be sure to use vectorized images (e.g., PDF).
- They do not reference Figure 1 in the text. Also, the first figure referenced in the text is in the supplemental material. Referring to supplemental to further support claims is okay, however, it seems like Fig 9 should be in the major paper (i.e., or they should change the paragraph referring to it not to depend on the supplemental)
- Figures are not well explained
   - the conditions for which they interpolated other methods are unclear-- why some cases there are clear variables being swept (e.g., pose of the face), but in others, it is not quite clear. Why does this vary from method to method to the extent it does? Sure, the proposed might work well at staying within photorealistic ranges but is the reason that the other methods do not be that sweeping used was not optimal for each other SOA method
-- 
summary_of_the_review: The paper is a marvelous piece: well put together, professionally presented, relevant and interesting topic, excessive experiments and analysis, and, mostly, well written.

DISCLAIMER: the mathematics in this paper I am not too familiar with-- there appeared to be consistent use in variables, each properly defined, and expressed in the body of text coherently. If another review can validate the math, it would be great (i.e., my review does not include deeply validating the math, for low-rank I have minimal experience with.

My primary concern is that the analysis is limited (not in figures, but in an explanation). Also, the story could be told better. Note that the two aforementioned concerns go hand in hand: with a better description of the image montages (i.e., what to look for, highlight the pros, also show some failure modes). Share thoughts on why results are better; same with failure modes (which are not currently highlighted).

Thus, the figures are sufficient, but the intuitive insight by breaking the figures and plots down is severally lacking (both in-text body, caption, and in the visual themselves). This would likely yield a better story, end-to-end, for it might bring the motivation full circle: as is, I am not seeing it go full circle, as all the best stories do.

With that, assuming the math is correct (per another reviewer validating), the pros slightly outweigh the cons, so I am neutral (or slightly above, since there is no middle chose :)
review_title: null

Meta-review: The initial reviews for this paper were diverging. After the rebuttal all reviewers have reached the consensus of recommending the paper's acceptance. Some reviewers have concerns regarding the novelty of the paper, however they appreciate that the paper is ell written and the empirical results are interesting. Following the reviewers recommendation, the meta reviewer recommends acceptance. In the final version of the paper the authors are encouraged to strengthen the weaknesses discussion as requested by one of the reviewers.
Decision: Accept (Poster)