Analyzing the Latent Space of GAN through Local Dimension Estimation
Authors: Jaewoong Choi
Geonho Hwang
Hyunsoo Cho
Myungjoo Kang
Venue: Submitted to ICLR 2023 (2023)
Keywords: generative adversarial network
disentanglement
semantic factorization
dimension estimation
grassmannian

Abstract: The impressive success of style-based GANs (StyleGANs) in high-fidelity image synthesis has motivated research to understand the semantic properties of their latent spaces. Recently, a close relationship was observed between the semantically disentangled local perturbations and the local PCA components in the learned latent space $\mathcal{W}$. However, understanding the number of disentangled perturbations remains challenging. Building upon this observation, we propose a local dimension estimation algorithm for an arbitrary intermediate layer in a pre-trained GAN model. The estimated intrinsic dimension corresponds to the number of disentangled local perturbations. In this perspective, we analyze the intermediate layers of the mapping network in StyleGANs. Our analysis clarifies the success of $\mathcal{W}$-space in StyleGAN and suggests a method for finding an alternative. Moreover, the intrinsic dimension estimation opens the possibility of unsupervised evaluation of global-basis-compatibility and disentanglement for a latent space. Our proposed metric, called Distortion, measures an inconsistency of intrinsic tangent space on the learned latent space. The metric is purely geometric and does not require any additional attribute information. Nevertheless, the metric shows a high correlation with the global-basis-compatibility and supervised disentanglement score. Our findings pave the way towards an unsupervised selection of globally disentangled latent space among the intermediate latent spaces in a GAN.


Reviews:
[{"review_id": "jysOg79XI-e", "reviewer": "Reviewer_dnDf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose an unsupervised, geometrically aware local intrinsic dimension estimation algorithm for latent space manipulation of GANs, along with a metric called “Distortion” as a global disentanglement score which is constructed with the help of the local intrinsic dimension. They build their work on a previously proposed method called “Local Basis” which gives local, semantically disentangled directions for style space W of StyleGANs. While these direction vectors are obtained from the solution of low-rank approximation to the linear map df_z between the tangent spaces of Z and W, they are already shown to demonstrate meaningful, stable manipulations. However, a lower dimensional approximation might be possible. Therefore, the authors of this paper presents a method to estimate this intrinsic dimension. As the vectors of Local Basis are obtained from the Jacobian, the authors ﬁrst eliminate the singular values of the Jacobian which cause overestimation of the intrinsic rank. Then, they apply Pseudorank algorithm to the Jacobian to diﬀerentiate meaningful components from the noisy components where the rank of the Jacobian indicates this diﬀerence. They experimentally validate their estimations of the intermediate layers in the mapping network and visually show manipulation results in the image space. Besides, they calculate a global disentanglement score which is related to the inconsistency of intrinsic tangent spaces. They experimentally show the correlation of this metric with the global disentanglement.", "summary_of_the_review": "I ﬁnd most of the intuitive and theoretical parts plausible. The general structure of the paper is consistent. Even though I ﬁnd the work worthy, especially the Distortion metric, I still have minor doubts about the local intrinsic dimension estimation in terms of its novelty and necessity for the literature.\n", "strength_and_weaknesses": "Strengths:\n\n1. The ﬂow of the paper is well designed and easy to follow. \n\n2. Theoretical parts seem plausible. Most of the methodologies are also explained intuitively. Experimental results support both the paper’s hypothesis and the previous literature.\n\n3. The proposed metric Distortion might be useful to validate the global disentanglement of a learnt latent space for the upcoming works.\n \n Weaknesses:\n\n1. At the end of “Understanding Latent Semantics” part in Related Work, the authors state that the global methods showed promising results, but also said that they were successful in a limited area. This sentence deserves further explanation.\n\n\n2. In “Sparsity Constraint” part, the authors compare their method with the LowRankGAN’s rank estimation method, and state that “saturation should occur to ﬁnd an intrinsic rank”. I’m not convinced enough with this explanation. I think a clearer explanation should be made.\n\n\n3. In Figure 5(b) and and Figure 13(b-e), traversals along the (d-1)-th and d-th axis where d indicates the estimated dimension are shown. It would be good to see the visual impacts of traversing along the axis greater than d since the authors present d as an upper bound on the number of perturbations. Would it display random/noisy traversals, as those dimensions are redundant ?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TlZBaxkQdC", "reviewer": "Reviewer_RU6v", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed a local intrinsic dimension estimation algorithm for the intermediate latent\nspace in a pre-trained GAN. Using this algorithm, we analyzed the intermediate layers in the mapping\nnetwork of StyleGANs on various datasets.\n\nThe paper is very straight-forward. It raised a concept to evaluate GAN by exploring the latent space of Generative model.", "summary_of_the_review": "I think this is a ground breaking paper for evaluating and improving the robustness of GAN. The authors have applied strict method to evaluate that. The only concern is that the authors should show more to illustrate if this method can contribute to the field now.", "strength_and_weaknesses": "The strength of paper: \n\nThe paper look into style-GAN and explore a new metric named as Disentanglement Score. This is a groundbreaking method.\nAlso this paper illustrate a good method to evaluate GAN by a deep look into latent space.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JlDe6l_96j", "reviewer": "Reviewer_STT2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for estimating local intrinsic dimensions in the intermediate layers of pretrained GANs, specifically with the mapping network of a stylegan. It also introduces a metric based of the method, which is used for analysis and comparison against similar metrics.", "summary_of_the_review": "The paper is the first of its kind to use the intermediate layers of a stylegan mapping network to understand latent semantics. While the qualitative examples could use some work and while paper can, at times, be hard to follow I think the proposed analysis and metric are of interest to the broader community. Therefore I recommend weak accept.\n\nPost-rebuttal:\nI must admit, during my initial review I was uncertain about this paper. I believed my confusion regarding various mathematical concepts was due to my own lack of knowledge in this area. However, upon seeing the discussion with other reviews, it seems my confusion about clarity is not just related to myself. I do believe the content of the paper may be interesting to the community, but given the state of the writing I believe other readers will have as much of a difficulty as I did (even with the updates). I, unfortunately, must lower my score to weak reject.", "strength_and_weaknesses": "Strengths:\n- The method for analyzing the mapping network seems novel.\n- The paper provides a rigorous detailing of the pseudorank algorithm where its conclusion leads naturally to the Distortion metric.\n- The proposed distortion metric is a an interesting and novel analysis tool for understanding GANs\n\n\nWeaknesses:\n- The evaluation could be more comprehensive. I'd like to see the distortion metric results on a variety of Karras et al.'s pretrained GANs versus FID.\n- The lack of generated images is a little concerning. Figure 9 has some examples, but I am not quite sure what I am looking at. What attributes are being shown exactly? When the image travels from row 1 to row 5, I'm not sure that's a semantically meaningful attribute. The multi-dimensional travel is non-standard, maybe the center image could be highlighted in some what to indicate that it's the starting point. I'd also like to see examples from non-human faces GANs\n\nMinor: \n- I think a small diagram of a styleGAN to show the mapping network is where all the work is done may be illuminating for first time readers, as \"intermediate layers of styleGAN\" typically refer to the layers of the synthesis network. \n- The notation abuse made my initial - skim a bit confusing. Perhaps calling it W- space (as opposed to W+ space) would be sufficient.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JNffp5AtLB0", "reviewer": "Reviewer_1W36", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new metric for GAN latent space estimation, called Distortion. The metric builds on the Local Basis method of (Choi, 2022), and is shown to be correlated with the \"global-basis-compatibility\" and supervised disentanglement score from (Eastwood & Williams, 2018). It provides an estimate of \"intrinsic dimensionality\" of each layer in a network.", "summary_of_the_review": "In terms of technical contents, the paper appears potentially valuable as a contribution to GAN latent space analysis.\n\nHowever, without serious improvements in the clarity, I do not believe it meets the quality criteria for publishing.", "strength_and_weaknesses": "Strengths:\n- The paper deals with an important problem of illuminating the high-dimensional W space of StyleGAN.\n- The proposed local intrinsic dimensionality estimation scheme appears original and could prove useful in latent space analysis.\n- Experiments support the claims (with disclaimers below) and support the hypothesis that the proposed metric is correlated with the other two scores (global basis compatibility and supervised disentanglement score)\n\nWeaknesses:\n- The relevance of the main claims is not clearly explained. The significance of the proposed metric, in comparison to e.g. (Choi, 2022) appears to be that it is unsupervised, but if this is really the key benefit, I could not find this point clearly explained or compared to (Choi, 2022).\n- The paper as a whole is difficult to follow. The writing is part of the problem, with phrases such as \"The compatibility with the global basis represents how well the (optimal) global basis can work on the target latent space\", \"an unsupervised selection of globally disentangled\nlatent space among the intermediate latent spaces in a GAN\", or \"the number of semantically disentangled local perturbation from w\".\n- For the same reasons, it is not clear whether the experiments are sufficient to justify the main claims. For instance, for visual comparison, there are a few traversed layers shown in Fig. 9, it's not clear how much ground does this kind of visualization intend to cover. \n- In the empirical section, the explanations are hard to follow, with sentences like \"The global basis proposed in GANSpace is PCA components of latent variable samples\", or \"Because this optimal global basis is also the local disentangled perturbations at all latent variables\".\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject