Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a way to manipulate intermediate latent features of pre-trained generative models to edit local parts of the image.
Specifically, they decompose a tensor of sampled latent features into two tensors, in which one could be interpreted as the set of appearance features and the other as the set of saliency maps that represent how much a specific appearance feature spans over the pixel-level space (coined as parts feature).
This is induced by constraining the decomposition procedure to make the parts feature nonnegative.
The decomposed features enable the manipulation of an original image by adjusting the attribute feature for the specific parts feature of interest.
As suggested by the results, the proposed method can be applied well to various pre-trained generative models with different architectures and training datasets.

strength_and_weaknesses: Strengths
- Dissecting a pre-trained generative model is an important and interesting task, and proposed method is novel and strong enough both in terms of its formulation and the quality of manipulation results.
- The goal of strict localized editing is well demonstrated qualitatively, and also quantitative evaluated with an appropriate evaluation metric.
- The authors present experimental results, and the changing trend in the decomposed features according to the hyperparameters’ change vastly in the supplementary material.

Weaknesses
- This method would only work for the models that are trained on images of single object type.

clarity,_quality,_novelty_and_reproducibility: Though the method would not be applicable for models trained on general scene images like MS-COCO, the proposed method is clearly stated and novel enough, and its empirical results are impressive.
summary_of_the_review: This paper tackles an important problem which is helpful for both interpreting and further utilizing pre-trained generative models.
The proposed method is clear and novel, and its efficacy is well proved with various experimental results.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a factorization of features maps in GANs such that spatial location is disentangled from appearance. Using NMF, the methods obtains a set of appearance prototypes and a set of non-negative spatial activations which, when combined with instance specific coefficients, reconstruct the feature map. Results show that this obtains localized compact regions corresponding to parts which are uncovered without supervision. These parts can be used to perform unsupervised object detection or even local editing by change the appearance prototype that is used at that location.
strength_and_weaknesses: This is a very interesting paper that follows recent work on analyzing what happens in the feature maps of GANs. The idea of decomposing the feature map using an additive process is very sound. The results are also very good: parts are clear and the editing are OK.

That being said, there are some questions about this work:
1. The process is not entirely additive: the coefficient in $\Lambda$ can be negative, which means that some appearance and/or some spatial locations can be removed. Why not just making this non-negative as well? Sure it renders the optimization problem more complex, but it also corresponds more to the intuition given as motivation.
2. $\Lambda$ is arbitrarily set as $A^\top Z_i P$. Why? Why not optimize $\Lambda_i$ for each image? In classical NMF, instance coefficient are usually optimized along the atoms in the dictionary. And since you perform a few step of instance specific optimization for $P$ anyway, why not optimize it also.
3. In the editing, a rank 1 element is added to the original feature map. Why not removal as well? Why not reducing the weights of the appearance vectors that where at the spatial location as well?
4. Global spatial decomposition is problematic because it assumes a fixed layout, hence the instance specific optimization. Did you try a translation invariant formulation by, e.g., using a convolution operator in the decomposition? (this is very successful in NMF)
5. In 4.1.1, it is stated "frequently learns an appearance vector for a high-level background concept". Only one? So a single vector encodes all the possible backgrounds of the generator? That sounds implausible.
clarity,_quality,_novelty_and_reproducibility: The paper is clear and should be easily reproduced.
summary_of_the_review: The idea is very sound and has practical application. There are a few question as to why the extra mile was not made in order to have a much more intuitive method.


-----

The rebuttal discuss most of my concerns which were more remarks and discussion material than actual requests for change.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work introduces a novel way of performing localized edits in the latent space of convolutional GANs like. The authors propose first to identify visual concepts by factorizing network activations at a certain layer into a set of appearance factors applied at certain spatial locations encoded by part factors. Once the factorization has been computed the learned appearance factors act as templates of visual concepts and can be used to identify parts of the generated image corresponding to a high level semantic concepts like “background” or “nose”. By swapping features in the latent space according to the localization provided by the appearance factors it is possible to apply local edits to images. The method is fast, architecture agnostic and by design limits the edits to only a user specified small portion of the image.
strength_and_weaknesses: ## Strengths

+ The proposed method does not make any assumption on the architecture of the GAN (except for the generator to be convolutional). As such it can be applied to a vast array of existing models without requiring modifications.

+ The completely unsupervised nature of the formulation, together with the fast convergence time makes it very flexible and easy to be applied to many different generators and datasets (the authors tried it on 5 generators trained on 5 different datasets with good results).

## Weaknesses

1. The proposed method is remarkably similar to previous works in the literature (Collins 2020), with the main difference being the method used to cluster activations of the neural network. While Collins used k-Means, the authors proposed to use matrix factorization to identify patterns in the feature space. From my perspective the two approaches achieve a very similar outcome with a slightly different formulation. The authors of this work however explore the application of this clustering discovery more deeply than what was done in [Collins 2020]. I don’t agree with the point raised by the authors at the end of the related work section where they say that “Collins is limited to part swapping”, while their method is not. From my perspective their method is also doing part swapping but allows the extra flexibility of either swapping features between two images or using one of the learnt appearance factors. 

2. The proposed method is completely unsupervised and task agnostic and as such can be applied out of the box to many different models and datasets. However the formulation does come with some hyperparameter that drastically can change the outcome: mostly the number of factors to extract and the layer of the generator to factorize. The ablation study shows that these two components, as expected, have a big impact on the factorization extracted and as such will need to be retuned for a new task and generator. Moreover once the factorization has been computed there’s still the need of some “human supervision” to map the learned factors to semantic concepts. As such this method might in the end require more human labor for practical application than competing works relying on pre-trained classifiers.

3. The proposed formulation implicitly assumes that the images are aligned while factorizing activations to compute the activations factors. While the authors show that the part factors can be aligned to a specific image with an ad hoc optimization, this is something done only in a second phase but not during the main optimization. From my perspective there’s no reason (except the computation cost) to not model separate part factors for each one of the activation maps used during the main factorization process. This will make the method more robust to datasets which are not very well aligned. With the current formulation the method cannot explain features that are not very well aligned across different samples. 
A question I have for the authors is if there is anything preventing the use of a per sample part factorization during the main optimization?

4. The fact that the method performs very localized edits compared with the competitors can be seen both as a pro and as a cons. In particular it might be a limitation in the sense that it cannot adapt the full image to the particular edit being introduced. A practical example of this could be removing an object from an image and replacing it with background, but not removing the shadow casted by it. Some discussion about the limitations of the method along this direction will make the paper stronger.

clarity,_quality,_novelty_and_reproducibility: ## Clarity

The paper is quite clear and easy to follow.

## Quality

The results achieved by the model are quite good however some limitation do exist, see weakness (4) and some manual hyperparameter tuning is required (2)

## Novelty

As mentioned in weakness (1) I think the paper proposes a novel formulation but not a completely novel idea. 

## Reproducibility

The paper should be reproducible.

summary_of_the_review: The idea behind the work is not novel, but the formulation is and the results are quite convincing. The method has some limitations but overall I feel like the authors made a good job at proving their contribution experimentally and providing a detailed analysis of the method through ablation studies. The main weakness of the paper is that it makes some strong assumptions in the factorization of the activations (weakness b). Lifting these would make the case for a stronger submission.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a method to decompose the feature maps of a GAN generator into spatial and semantic components. This decomposition allows editing the output image at the level of parts. The experiments show that the method can be applied to several different GAN architectures and datasets. Additionally, due to the simplicity of the approach, the method is very fast.
strength_and_weaknesses: ## Strengths
The method is very fast and simple as it is based on a linear decomposition of feature maps of the generator. This means that no network training is needed to discover the parts and appearances. 

It is possible to finetune the decomposition for individual images to allow for more precise edits. Additionally, as the decomposition is spatial, the edits can be more targeted that editing the latent space of the GAN alone. 

The method has been evaluated on a range of GAN architectures and datasets which shows the generality of the method and its independence from specific generators.

## Weaknesses

A main weakness of the paper is the evaluation. Although evaluating semantic editing of GAN generated images is a difficult problem, it is necessary to support the claims of the paper.
### W1 Semanticity
It is not clear if the part decomposition is indeed semantic and not spatial. It is likely that the linear decomposition prefers decomposing into spatial regions and relies on the alignment of samples across the dataset. See e.g. Fig 9 columns 4,7,8 where instance alignment is not present. For conditional GANs, can semantic parts be derives across classes to find correspondences between parts (e.g. to replace dog legs with cat legs)?

### W2 Background
The paper emphasizes the ability of the model to identify the background/foreground of images. The fact that this is possible has been shown not only by (Voynov&Babenko 2020) but also in (Melas-Kyriazi et al, 2021) for a large variety of generative models. The quality of the background discovery could be measured possibly in two ways. Either through inversion of the GAN on real images with associated ground truth on a saliency segmentation dataset or relative to a supervised model, directly on the generated images. This would be helpful, as currently the results look more similar to a simple spatial decomposition rather than a semantic one. (e.g. c,d in Fig 4, Fig 19) similar to the decomposition on face datasets seemingly strongly relying on the alignment between images. 

### W3 Hyper-parameter and ablation
The method contains several hyper-parameters and specific choices. It would be valuable to understand their influence on the model. E.g.: How well does the model work just with SVD alone? What is the influence of the number of iterations T? Are there substantial differences between different GANs trained on the same data - do certain architectures lead to feature maps that are easier to decompose? Can appearance vectors of the same part be swapped between instances? How does the layer choice matter? Does the user need to select the layer for decomposition or is there an automated way to choose? Other forms of clustering could be compared, e.g. k-means.

clarity,_quality,_novelty_and_reproducibility: The paper is missing several many important explanations to understand the method. It is not explained how the editing works. Does the user inspect all discovered parts and then edit their spatial or semantic components? How does changing semantics work? Where does the changed appearance vector come from?
summary_of_the_review: Overall the paper presents a simple approach to enable more fine-grained editing of GAN images by decomposing feature maps. Despite the strengths, the current state of the evaluation makes it difficult to assess the benefits of this method over existing techniques.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a way to manipulate intermediate latent features of pre-trained generative models to edit local parts of the image. Specifically, they decompose a tensor of sampled latent features into two tensors, in which one could be interpreted as the set of appearance features and the other as the set of saliency maps that represent how much a specific appearance feature spans over the pixel-level space. This is induced by constraining the decomposition procedure to make the parts feature nonnegative. The decomposed features enable the manipulation of an original image by adjusting the attribute feature for the specific parts feature of interest. The proposed method can be applied well to various pre-trained generative models with different architectures and training datasets.
All reviewers agree on the good empirical performance of the method and its simplicity. The rebuttal submitted by the authors clarified the relation of the present work to Collins 2020, as well as its applicability beyond single object image. 

justification_for_why_not_higher_score: 
The method presented is similar to previous approaches e.g., Collins et al. 2020 about clustering feature activations of generative networks.
justification_for_why_not_lower_score: 
All reviewers agree on the good empirical performance of the method and its simplicity.
Decision: Accept: poster