Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a method to exploit the low-rank structure of the Jacobian of a deep generative model to perform precise image editing. The Jacobian corresponding to a region of an image is constructed and its null space is computed. By varying the input to the generator in the null space along the eigenvectors, only one attribute can be changed at a time, leaving the rest of the image unchanged. Experiments show that the method outperforms strong baselines in terms of semantic editing. 
main_review: I think this paper has many strengths:
- The overall method is quite simple and easy to implement.
- The method is unsupervised which is attractive.
- That finding the relevant directions in the latent space using only image, and using it for other images is very interesting and could be useful.
- Both the qualitative and quantitative results are very promising. The experiments have been well done in general, the user study in particular is an important addition to understand image quality. Editing is shown on many different attributes on several datasets.
- The paper is well-written and easy to understand.

Questions and suggestions for improvement:
- I understand the algorithm and why the eigenvectors are calculated. However, what is the guarantee that every eigenvector corresponds to a single semantic attribute? I can easily imagine a case where the eigenvector could change both the hair color and the eye color, for example, depending on the correlations in the dataset. In other words, what are the assumptions made on the dataset used to train the generator?

- What is the step size taken in the direction of the eigenvectors? Can the authors show results where the step size goes from a larger negative number to a larger positive number in an experiment involving zooming, pose changes and eye opening/closing? 

- The experimental results in Figure 5 are very interesting and the author mention that this suggests that the method could be used for data augmentation. Would it be possible for the authors to substantiate this claim with a small experiment. This can make the paper even stronger. 
limitations_and_societal_impact: I am not that the authors have discussed limitations of their work. Perhaps they can outline it in the rebuttal and add them to the paper.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors propose an approach to identify meaningful latent directions in a GAN latent space. First, they factorize the Jacobian of a generator map $G: z \to \mathrm{RGB}$, treating its principal axis as the desired directions. Then they fine-tune them in a supervised regime to guarantee only the regions of the interest being edited. Though some of the results look intriguing, a lot of relevant comparisons with prior works is missed and the experiments are not sufficient.
main_review: Strengths:
- some of the results look promising, especially, the transferability of the directions discovered by a single image to the whole dataset.

Weakness:
- As for me, it was really hard to read the paper. Its structure is unclear, the introduction gives just a vague idea of the contribution. The experiments have no implementation details which makes them unreproducible as the authors do not supply the code.
- Though the authors state that one of the main contributions is the locality of the editings produced by their directions, the comparison with the prior works is unfair as they use additional supervision provided by either segmentation masks or region boundaries. It would be reasonable to provide the editing comparison of the proposed approach with e.g. [1, 2].
- It remains unclear how a direction discovered for a particular $z$ is translated to the whole latent space. Do the authors use always the same z to discover the interpretable directions?
- On line 85 the authors state that "to our best knowledge, we are the first to demonstrate the effectiveness of low-rank factorization...". The very close idea to utilize a generator's Hessian eigenvectors as interpretable directions was first proposed in [3] which is not cited. A close idea to use SVD or Hessian-based editable weights decomposition was also discussed in [4].
- line 157: it is unclear, how exactly the authors determine if a particular vector influences region $A$.
- The only quantitive evaluation is presented in Table 1. The authors perform a comparison with unsupervised approaches ignoring supervision approaches. It is also unclear, how do they scale the magnitudes of the latent shifts for different methods. For a particular latent direction $h$, I would be happy to see a plot of points [ $\mathrm{FID}(G(z + \alpha \cdot h))$, $\mathbb{E}~ \mathrm{LPIPS}(G(z), G(z + \alpha \cdot h))$ ] with different shift scales $\alpha$. The first coordinate is the FID of the shifted latent distribution and the second coordinate is the expectation of an image deformation in terms of the lpips-distance.
- line 251: "Thereby, a potential application of our algorithm could be data augmentation." it would be preferable if the authors could demonstrate this experimentally.

[1] Editing in Style: Uncovering the Local Semantics of GANs, Edo Collins et al.

[2] Mask-Guided Discovery of Semantic Manifolds in Generative Models, Mengyu Yang et al.

[3] The Hessian Penalty: A Weak Prior for Unsupervised Disentanglement, William Peebles et al.

[4] Navigating the GAN Parameter Space for Semantic Image Editing, Anton Cherepkov et al.

---------
UPD: I raise my score 3 -> 4
I still think that the quantitive evaluation is insufficient as it consists of a table in the main text with only two attributes and one figure in the response for a single attribute. In my opinion, some of the baselines (though, with higher supervision) should also be reported.
limitations_and_societal_impact: There is no limitations section.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes using low-rank factorization of the Jacobian of generator’s selected output region with respect to its latent noise source in order to perform precise edits of generated images. Specifically, a latent vector in the top-subspace (imply maximium edit impact) of the output region is identified, after which, it is then projected onto the null-space (no edit impact) of the negative region (the region not selected).

Contributions:
1) Examine the low-rank subspaces of of the generator output w.r.t. the latent space and show how to perform edits in an unsupervised manner.
2) Demonstrate that this process need not be repeated on each image individually, but instead can be performed just once and generalize to many images due to the semantic organization of the generator’s latent space.
3) Perform several experiments on various image datasets, comparing against other state of the art models, both qualitatively and quantitatively.
main_review: Originality:
To my knowledge, this approach is novel and also appears to be much simpler than previous approaches.

Quality: The paper is of good quality.

Clarity: There are some small typos in places, but otherwise the paper is clearly written.

Significance: Precise editing of generated / synthetic images is of great important to the computer vision and graphics communities.


Overall, I liked the simplicity of the approach and the results in several cases are impressive. To improve the paper, I would suggest:
- Figure 2 is visually a bit disappointing relative to the rest of in the paper. What are the units on the $l_1$-loss heatmaps? Is it the sum of the absolute values of the differences across the rgb channels $[0, 255]$? I can see the cloud is slightly larger in (b) than (a), but I can’t visually tell the any difference between (d) and (e). Please comment on any visual difference we should be noticing.
- Figure 3 (bubble): In order to confirm that the “white dots” were not there before, we need to see a zoomed-out version of the target image.
- Could you include Figure 4 repeated but with $v$ (the original attribute vector) instead of $p$ (the vector projected into the null-space)?
- Figure 6 is interesting but reveals a downside to the approach. The fact that different masks lead to similar edits means a lack of precise control of edits. Can you please discuss this limitation?


Minor:
- I don’t have a counterexample, but this sentence raised alarm bells for me: “first to demonstrate effectiveness of low-rank factorization in deep generative models”. It seems like an overstatement.
limitations_and_societal_impact: I included a suggestion for discussion of a specific limitation in my review above. The authors might also want to add a comment on the use of GAN editing for producing fake news among other things.
review_title: null

Meta-review: The paper used the Jacobean of the generator and Robust PCA (on covariance of Jacobean) to find directions for editing a region A that keep a region B in an image intact. By basically projecting the robust PCA eigenvectors of region A on the null space of robust PCA in region B. The idea is intuitive and makes sense and works in practice. 

After reading the reviews and the paper. I have few suggestions to clarify the paper. Although reviewers said the paper is easy to follow, and I am familiar with works in the topic  I found the description of the method to be quite unclear and not precise. 

Please include a sketch of an algorithm in the paper that makes it clear to the reader what is your method, and please also make your code available since one can not reproduce this paper in its current version. 

is  your method  I) 
1. The method starts by finding $z_A$ and $z_{B}$ , to produce $x_{A},x_{B}$  and compute Robust PCA of $J^{\top}J$ at $z_{A}$ and $z_{B}$ get $V_{A}, V_{B}$
2. Project $V_{A}$ on null space of $V_{B}$, get $P_{A}$
3. $G(z_{A}+ \alpha P_{A,j})$  and merge with $x_{B}$ to produce new images

or II) 
1. Find $z$ to reproduce image then compute robust PCA on $JG_{A}(z)^{\top}JG_{A}(z)$ , by restricting the covariance to pixels in region A , same for region B
2. Continue with steps 2 and 3 from above using latent $z$
 

I am left with a question to the authors that you should clarify in the final manuscript . I may have overlooked it in appendix but I did not see it. The method works because of projection to the null space of region B or because of  the use of Robust PCA. 
What if you replaced in your method Robust PCA with just PCA , I think this ablation  should be included in the paper.

I recommend a weak accept but I ask the authors to include a sketch of clear procedure of how their method is applied, open source their code, and to do the ablation robust PCA versus PCA. 

Decision: Accept (Poster)