Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: 
This paper studies transformations in GAN latent space that map to meaningful transformations in the generated data. The main contribution is to derive closed form methods for discovering latent transformations that correspond to 1) geometric changes and 2) changes that capture principle components of model variation. The paper also contributes new methods for nonlinear latent transformations, disentangled transformations, and an application to attribute transfer.

I really like this paper. The capabilities demonstrated here aren’t dramatically new — other methods can achieve similar effects — but this paper achieves these effects in a new way, which has its own advantages.

The positives I see are:
+ Simple and elegant alternative to prior work on finding latent transformations
+ Nice qualitative and quantitative results
+ Practical benefits including speed up, analytical transformation end points, and better disentanglement

My main criticisms are:
- toward the end of the paper it’s a bit of a hodgepodge of ideas
- the ideas and methods in Section 3 are mostly disjoint from those in Section 2
- the attribute transfer application especially feels tangential, and receives minimal analysis or evaluation
- the experiment in Figure 7 is not fully convincing

I think everything in this paper is interesting, but the different sections don’t fully cohere together. Section 2 is great and seems to tell a complete story, about closed form solutions to finding geometric transformations. Section 3 then diverges into a few different directions, which don’t directly build on each other. When the circular trajectories are introduced, I’m left wondering: why not use these in Section 2? For example, would it be possible to derive closed form solutions for great circles that map to target transformations $\mathbf{P}$?

I do like the story about first and second order biases, which somewhat connects Sections 2 and 3. However I would like to see more connections made at the methodological level, or a clear justification for why methods from Section 2 where not used in Section 3 and vice versa.

Aside from the coherence of the story, there are a few smaller things that could be improved:

In Figure 4, it would be nice to show the data distribution, or the distribution from the non-transformed GAN samples. Otherwise it’s hard to tell if the transformations had an effect.

In Section 3.1 I think an equation would help clarify things, e.g., write out the SVD and refer to the right singular vectors by an algebraic symbol. Perhaps also contrast this with the equation for PCA from Harkonen et al.

The experiment in Figure 7 is not convincing to me. From the plots it’s hard to tell if the proposed methods are actually incurring less shift. Maybe plotting the delta from the original distribution would make the plot clearer? Most of all, I think some numerical metric should be defined to quantify the second-order biases, rather than just requiring visual inspection of the plots. It is convincing that the FID improves for the proposed methods, but not that the level of disentanglement between zoom and shift improves.

Minor comments:
1. Page 1: “the precise same effect” — I think this is an overstatement. There are substantial differences between how a single direction affects different classes. For example see Fig 4 of Jahanian et al. 2020. I agree that it is interesting that the directions have _similar_ effects across classes.
2. In Eqn 3, what if the matrix is singular? Do you use the pseudoinverse?

review_title: A fun paper that presents elegant new methods with the one drawback being that it is a bit of a hodgepodge of ideas

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper is clearly written. Its significance lies in transferring both user-prescribed transformations and unsupervisingly discovered transformations over a pre-trained generator’s weights to the generator output.  Also, it discovers endpoints for latent space trajectories.

Pros: 
One nice feature of the  proposed technique is that it works on a pre-trained generator, and no extra training or optimization is required, therefore, it is efficient.
Endpoints of the walks are investigated through analyzing the convergence of the walks.
Further demonstrations with walks on the great circle and small circle on the sphere through an initial point  and a  principal direction are presented that give insights into the transformations obtained in the generated images.

Cons: 
- Results in the paper display artifacts (e.g Figure 3, Figure 6), therefore raises the question of carefully picked results being displayed, indeed it’s hard to predict the overall quality of generations.
- Section 3.4 where the attribute transfer is described is not clearly written. Reading this paper, one is not able to understand how background or texture  transfer is obtained through this work. I believe that part is based on a recent work by (Voynov, Babenko 2020). They showed a technique to discover meaningful latent directions in an unsupervised way, which included similar attribute transfers this paper has shown. I believe this paper relies on that work in finding the  directions that correspond to semantically meaningful attributes, or this point was not very clearly explained.  If that’s the case, the contribution of this paper mainly boils down to certain geometric transforms or principal directions.

The method of (Voynov and Babenko 2020) also operated on a pre-trained generator and obtained semantically meaningful directions in an unsupervised way through only optimizing two other simple networks to obtain meaningful directions. As far as I understand their optimization is not very demanding at all, therefore, is also efficient. Hence, the point made here in terms of ultimate efficiency with no training leads to a trade-off in not being able to discover a richer set of transformations.
review_title: A simple approach to find closed form expressions for latent space directions corresponding to prescribed geometric transformations such as a shift or a zoom is presented. Also extended to curved trajectories in the latent space. Both linear and nonlinear latent space walks as well as walks corresponding to principal directions of the first layer weight matrix are obtained. The paper also makes use of semantic directions such as background effects, color, and texture changes.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose two new techniques that extract interpretable directions from latent spaces of pretrained GAN generators. Both techniques are very efficient and are shown to work with the state-of-the-art BigGAN models. Furthermore, the authors describe additional details of the method, like determining the transformation end-points, which are important for usage in the practical visual editing.

Strengths:

1. The paper tackles the important problem, provides a thorough description of the field, demonstrates an in-depth understanding of the area. The important small details (determining the end-points, entangled transformations) are only slightly addressed in the existing literature, but are crucial for editing applications. This paper convincingly addresses this gap.

2. The proposed techniques are both simple and efficient, much faster compared to existing methods.

Weaknesses:

1. What upset me most was a certain amount of overclaiming about the main contributions.

"Second, it detects many more semantic directions than other methods." - I cannot agree with this statement.

a) For user-specified transformations, the method finds the same directions as  Jahanian et al. (2020).

b) For unsupervisedly discovered transformations, I did not find in the text any examples of directions, that are not covered by the:

Voynov & Babenko, 2020

Ha ̈rko ̈nen et al. (2020), 2020,

The Hessian Penalty: A Weak Prior for Unsupervised Disentanglement, ECCV 2020 (which is a missing related work, btw)

In my opinion, the statement is misleading, which is not acceptable when listing contributions.

2. I am not convinced by the significance of the search of user-specified geometric transformations, which take the largest part of the submission. Simple transformations, like zoom/shift/brightness, can be easily obtained automatically in editing applications, do we really need GANs for them?

3. "[Ha ̈rko ̈nen et al. (2020)] obtain a set of non-orthogonal latent-space directions that correspond to repeated effects. In contrast, our directions are orthogonal by construction, and therefore capture a super-set of the effects found by Ha ̈rko ̈nen et al. (2020)"

This sound like a bold claim. First, directions from Ha ̈rko ̈nen et al. (2020), while not being orthogonal by construction, de-facto still can be close to orthogonal. Second, from both the main text and appendix, I did not understand, why the authors' method captures a super-set of directions, no quantitative comparison is provided.

To sum up, my current evaluation is (5). The proposed method is obviously more efficient, but I do not consider this advantage as a very important one, since transformation search is performed only once. Given the weaknesses listed above, I cannot recommend acceptance.

AFTER REBUTTAL:

The authors have toned down some of their claims and I am increasing my score accordingly. 
review_title: Paper review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: I am a picky reviewer in most cases. This submission is one of the highest-quality manuscripts I have reviewed. The problem setup is clear, the survey is comprehensive, and the contrastive to the prior arts is clear. I am pretty enjoyed reading this paper. 

The task of finding meaningful steerability within the latent space is relatively a new research area. The authors have made a fairly complete review and investigation on the related works, then propose an unsupervised (meanwhile, still providing slight user controllability with a small set of transformations) and training-free algorithm with high-diversity and high-quality (in terms of the robustness of the trajectory found by the algorithm). Besides, the authors propose the concept of the first-order and second-order dataset biases, and tackles the problem with non-linear trajectories discovery. Also, highlight that the proposed method is significantly faster than the prior arts as shown in Table 1. 

Experiment-wise, the authors provide a huge set of qualitative results in the appendix. Though the results are already convincing enough, I would still recommend the authors to provide some quantitative numbers for the comparisons on the diversity of latent directions against other methods (Figure 34-38). The visual differences are obvious, but a quantitative number can further showcase the universality of higher diversity. Some simple measurements like a mean over the variance of LPIPS among all dimensions [1] may be fair enough.

Despite the submission is pretty strong and I pretty much has no problem with it, I ended up rating it with an 8, as the impact is relatively narrow in a specific area/task. But I am willing to see the feedback from other reviewers and give it a certain level of adjustment.


[1] To avoid ambiguity, I mean 
```
all_vars = []
for trajectory in all_trajectories:
	img_pairs = # interpolate the trajectory
	cur_var = np.var([lpips(*img_pair) for img_pairs in latent_pairs])
	all_vars.append(cur_var)
mean_var = np.mean(all_vars)
```
This is just a demonstration, any implementation with a similar concept is fine.


**[Minor comments]**
1. While mentioning other related methods in the paper (e.g., Table 1), the references are made with plain text (without hyperlinks) and mostly using the authors' names in replacement of the method names. It is quite hard to keep tracking which reference corresponds to which method. It would be great if the authors can investigate this problem a bit.

2. Typo: Frobenious => Frobenius

**[Discussion]**
I do not count this section as a part of the review, so it wouldn't affect my scoring if the authors do not reply. 
What is the main application of the dataset summarization technique (let's exclude image editing and content creation, which can be achieved with other techniques)? It is interesting to visualize these properties of the dataset, however, the real-world application of such a technique is not obvious and carefully investigated. 
A quick answer might be finding the dataset bias or domain shifting, and we may explicitly fix it when these problems are revealed by these dataset summarization techniques. However, the works on dataset summarization have never explicitly shown if these applications are viable with some experimental setup. 
I am interested in what is the authors' perspective on the applications, and whether showing these applications is an important part of the dataset summarization research direction, as the applications often influence the impact and research direction of proposed new settings and new components.
review_title: Strong submission with clear novelties and comprehensive experiments

Meta-review: The paper gives an elegant and efficient closed form solution for steering directions in the latent space of a pretrained GAN to to produce transformations in the image domain such as scaling and rotation etc,  this also extended to attribute transfer. The new method leads to "speed up, analytical transformation end points, and better disentanglement"  w.r.t to competitive methods.  All reviewers agreed on the merits of this work, and the good qualitative and quantitative results . The rebuttal addressed reviewers questions and concerns regarding the structure of the paper and its coherence.  Accept
Decision: Accept (Spotlight)