Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed a local intrinsic dimension estimation algorithm for the intermediate latent
space in a pre-trained GAN. Using this algorithm, we analyzed the intermediate layers in the mapping
network of StyleGANs on various datasets.

The paper is very straight-forward. It raised a concept to evaluate GAN by exploring the latent space of Generative model.
strength_and_weaknesses: The strength of paper: 

The paper look into style-GAN and explore a new metric named as Disentanglement Score. This is a groundbreaking method.
Also this paper illustrate a good method to evaluate GAN by a deep look into latent space.

clarity,_quality,_novelty_and_reproducibility: The paper is in good clarity and quality. 

The novelty is good, however, I doubt if this is an interesting topic for the field now.
summary_of_the_review: I think this is a ground breaking paper for evaluating and improving the robustness of GAN. The authors have applied strict method to evaluate that. The only concern is that the authors should show more to illustrate if this method can contribute to the field now.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose an unsupervised, geometrically aware local intrinsic dimension estimation algorithm for latent space manipulation of GANs, along with a metric called “Distortion” as a global disentanglement score which is constructed with the help of the local intrinsic dimension. They build their work on a previously proposed method called “Local Basis” which gives local, semantically disentangled directions for style space W of StyleGANs. While these direction vectors are obtained from the solution of low-rank approximation to the linear map df_z between the tangent spaces of Z and W, they are already shown to demonstrate meaningful, stable manipulations. However, a lower dimensional approximation might be possible. Therefore, the authors of this paper presents a method to estimate this intrinsic dimension. As the vectors of Local Basis are obtained from the Jacobian, the authors ﬁrst eliminate the singular values of the Jacobian which cause overestimation of the intrinsic rank. Then, they apply Pseudorank algorithm to the Jacobian to diﬀerentiate meaningful components from the noisy components where the rank of the Jacobian indicates this diﬀerence. They experimentally validate their estimations of the intermediate layers in the mapping network and visually show manipulation results in the image space. Besides, they calculate a global disentanglement score which is related to the inconsistency of intrinsic tangent spaces. They experimentally show the correlation of this metric with the global disentanglement.
strength_and_weaknesses: Strengths:

1. The ﬂow of the paper is well designed and easy to follow. 

2. Theoretical parts seem plausible. Most of the methodologies are also explained intuitively. Experimental results support both the paper’s hypothesis and the previous literature.

3. The proposed metric Distortion might be useful to validate the global disentanglement of a learnt latent space for the upcoming works.
 
 Weaknesses:

1. At the end of “Understanding Latent Semantics” part in Related Work, the authors state that the global methods showed promising results, but also said that they were successful in a limited area. This sentence deserves further explanation.


2. In “Sparsity Constraint” part, the authors compare their method with the LowRankGAN’s rank estimation method, and state that “saturation should occur to ﬁnd an intrinsic rank”. I’m not convinced enough with this explanation. I think a clearer explanation should be made.


3. In Figure 5(b) and and Figure 13(b-e), traversals along the (d-1)-th and d-th axis where d indicates the estimated dimension are shown. It would be good to see the visual impacts of traversing along the axis greater than d since the authors present d as an upper bound on the number of perturbations. Would it display random/noisy traversals, as those dimensions are redundant ?
clarity,_quality,_novelty_and_reproducibility: This work is well written, experimentally supported and clear to follow. It is originally built over an existing method, and proposes new solutions to the latent space manipulation problem of GANs. The codes are shared.I ﬁnd most of the intuitive and theoretical parts plausible. The general structure of the paper is consistent. Even though I ﬁnd the work worthy, especially the Distortion metric, I still have minor doubts about the local intrinsic dimension estimation in terms of its novelty and necessity for the literature.

summary_of_the_review: I ﬁnd most of the intuitive and theoretical parts plausible. The general structure of the paper is consistent. Even though I ﬁnd the work worthy, especially the Distortion metric, I still have minor doubts about the local intrinsic dimension estimation in terms of its novelty and necessity for the literature.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method for estimating local intrinsic dimensions in the intermediate layers of pretrained GANs, specifically with the mapping network of a stylegan. It also introduces a metric based of the method, which is used for analysis and comparison against similar metrics.
strength_and_weaknesses: Strengths:
- The method for analyzing the mapping network seems novel.
- The paper provides a rigorous detailing of the pseudorank algorithm where its conclusion leads naturally to the Distortion metric.
- The proposed distortion metric is a an interesting and novel analysis tool for understanding GANs


Weaknesses:
- The evaluation could be more comprehensive. I'd like to see the distortion metric results on a variety of Karras et al.'s pretrained GANs versus FID.
- The lack of generated images is a little concerning. Figure 9 has some examples, but I am not quite sure what I am looking at. What attributes are being shown exactly? When the image travels from row 1 to row 5, I'm not sure that's a semantically meaningful attribute. The multi-dimensional travel is non-standard, maybe the center image could be highlighted in some what to indicate that it's the starting point. I'd also like to see examples from non-human faces GANs

Minor: 
- I think a small diagram of a styleGAN to show the mapping network is where all the work is done may be illuminating for first time readers, as "intermediate layers of styleGAN" typically refer to the layers of the synthesis network. 
- The notation abuse made my initial - skim a bit confusing. Perhaps calling it W- space (as opposed to W+ space) would be sufficient.
clarity,_quality,_novelty_and_reproducibility: The paper is somewhat hard to follow. For example the following sentence "The local intrinsic dimension is the dimension of approximating submanifold that well describes the latent space locally." is self referential. The paper could use another round of edits for clarity/being concise. The paper jumps directly into somewhat complex math that is not often found in GAN papers, a more gentle introduction to Local Basis (for example), would add to the clarity.
summary_of_the_review: The paper is the first of its kind to use the intermediate layers of a stylegan mapping network to understand latent semantics. While the qualitative examples could use some work and while paper can, at times, be hard to follow I think the proposed analysis and metric are of interest to the broader community. Therefore I recommend weak accept.

Post-rebuttal:
I must admit, during my initial review I was uncertain about this paper. I believed my confusion regarding various mathematical concepts was due to my own lack of knowledge in this area. However, upon seeing the discussion with other reviews, it seems my confusion about clarity is not just related to myself. I do believe the content of the paper may be interesting to the community, but given the state of the writing I believe other readers will have as much of a difficulty as I did (even with the updates). I, unfortunately, must lower my score to weak reject.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new metric for GAN latent space estimation, called Distortion. The metric builds on the Local Basis method of (Choi, 2022), and is shown to be correlated with the "global-basis-compatibility" and supervised disentanglement score from (Eastwood & Williams, 2018). It provides an estimate of "intrinsic dimensionality" of each layer in a network.
strength_and_weaknesses: Strengths:
- The paper deals with an important problem of illuminating the high-dimensional W space of StyleGAN.
- The proposed local intrinsic dimensionality estimation scheme appears original and could prove useful in latent space analysis.
- Experiments support the claims (with disclaimers below) and support the hypothesis that the proposed metric is correlated with the other two scores (global basis compatibility and supervised disentanglement score)

Weaknesses:
- The relevance of the main claims is not clearly explained. The significance of the proposed metric, in comparison to e.g. (Choi, 2022) appears to be that it is unsupervised, but if this is really the key benefit, I could not find this point clearly explained or compared to (Choi, 2022).
- The paper as a whole is difficult to follow. The writing is part of the problem, with phrases such as "The compatibility with the global basis represents how well the (optimal) global basis can work on the target latent space", "an unsupervised selection of globally disentangled
latent space among the intermediate latent spaces in a GAN", or "the number of semantically disentangled local perturbation from w".
- For the same reasons, it is not clear whether the experiments are sufficient to justify the main claims. For instance, for visual comparison, there are a few traversed layers shown in Fig. 9, it's not clear how much ground does this kind of visualization intend to cover. 
- In the empirical section, the explanations are hard to follow, with sentences like "The global basis proposed in GANSpace is PCA components of latent variable samples", or "Because this optimal global basis is also the local disentangled perturbations at all latent variables".

clarity,_quality,_novelty_and_reproducibility: - The work seems original and a non-trivial extension of Choi, 2022.
- The clarity of presentation needs much improvement.
- The lack of clarity makes it hard to fully judge the substance of the paper.
- The methods appear to be reproducible, but code was not provided.
summary_of_the_review: In terms of technical contents, the paper appears potentially valuable as a contribution to GAN latent space analysis.

However, without serious improvements in the clarity, I do not believe it meets the quality criteria for publishing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors propose a method for estimating the number of disentangled local perturbations. Strengths include a relatively well motivated method, and the proposal of a distortion metric that correlates well with supervised disentanglement score. The major weakness of the paper, as noted by the reviewers, is the limited empirical evaluation. It is difficult to know whether the latent space traversals correspond to anything semantically meaningful, and whether these are better than predecessors. Furthermore, it's applicability to only StyleGAN models limits its applicability. Finally, as some reviewers noted, the presentation could be clearer
justification_for_why_not_higher_score: The empirical evaluation was weak, with limited insight into whether the latent space traversals correspond to anything semantically meaningful.
Decision: Reject