Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to use the self-attention mechanism (in particular, Transformer) to enhance image editing. Experiments show promising performance.
main_review: This paper proposes to use the self-attention mechanism (in particular, Transformer) to enhance image editing. Experiments show promising performance.

This paper falls out of my research field. I would try my best to offer some comments.

First, the idea of using Transformer, a recently popular module, to enhance the existing problems (e.g. missing global information) is fine. However, this idea seems to have been proposed by the Taming paper (which is cited frequently in this work). Based on this, I am a bit confused of the contribution of this work. In particular, regarding the issues mentioned in the abstract (missing global information, and information leakage of local guidance), how are they solved by the Transformer design, and can both of them be solved by the Transformer design? In particular, the issue of information leakage issue - is it alleviated by random masking? In summary, this paper should mention clearly about the difference between this paper and the Taming paper, and the additional contribution beyond the Taming paper.

Regarding the network design, since I am not an expert, I can only say that everything looks fine, and I will wait for other reviewers' comments for further judgment.

Regarding the experiments, the shown results seem good, but I have two minor questions in Fig 4. (1) In the left part, why is the resolution of some methods significantly different from others? (2) In the right part, why the proposed method change some non-edited contents in the original image (e.g. in the 3rd row, the letter of S in the right border of the image) ?

Overall, this is a well prepared paper. I give a borderline score and will wait for additional information to make the final decision.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper aims to utilize the auto-regressive generative model to achieve the local image editing based on previous state-of-the-art method *taming transformer*.  Specifically, to prevent the information leakage while encoding the input image into discrete sequences, a very straightforward two-stream convolution is proposed. Besides, this paper also presents a new local auto-regressive transformer to benefit from the capability of generative AR model and the global information simultaneously. The experiments show   some improvements over compared methods.
main_review: 

What this paper wants to achieve, i.e. editing any regions of input image while maintaining the original information, is really interesting, but there indeed are some aspects should be explained and clarified.

* The **real effectiveness** of two-stream convolution is not claimed well. In my understanding, the ultimate target of VQ-VAE is to *faithfully reconstruct the input image*, even though there are masks in the input. However, none experiments analyze why the designed convolution is better or could help this point. By contrast, the main effective part of Sec 3.1 may be Equation 4. However, in Line157-Line159, it claims that such simple trick makes AR inference more efficient, which also seems inaccurate. In my opinion, this finetuning trick actually aims to preserve the boundary textures as much as possible, and the efficiency improvements come from the subsequent design of local auto-regressive transformer. Even so, I believe that there are still cases in which the original local contents could not be preserved well while giving arbitrary downsampled masks. This aspect also should be analyzed in details.

* Sec 3.2 introduces the local autoregressive transformer learning, but is it the strict auto-regressive model? To predict next tokens, you need to rely on the conditions of both past tokens and *future* global tokens, which actually breaks the fixed-order decomposed joint distributions of pixels.

* The comparison with *taming transformer* is not fair. You should re-train the baseline, i.e., conditioned on the edited input to regress the final results. For the unmasked region, pick the top-1 token while sampling to reserve the original content.

* Why does this paper mainly focus on editing? The best way to show the improvements of the proposed method is to conduct experiments on image inpainting task. The rests are just the controllable applications of image inpainting.

Based on these considerations, I think currently this submission could not reach the acceptance bar of NeurIPS venue. My rating is rejection now.

limitations_and_societal_impact: Please see the above mentioned comments.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a transformer-based model for local image editing. The key challenges tackled by the paper is the need to integrate global information, while also preventing leakage of local guidance information (i.e. the local editing should only affect the designated region). To accomplish this, a combination of discrete representation learning (VQVAE) and an autoregresive transformer model is used. The local guidance is enforced through the use of a masked convolution in the feature learning and a local attention mask in the transformer.
main_review: **TL;DR:** The paper introduces a useful technique for local image editing / synthesis, but the writing could be substantially improved to make the paper clearer. In the main review below I have labelled points as a strength (+), weakness (-) or mixed (+/-).

**Originality**: 

(+) The method proposed in the paper has some novelty. Particularly, I’m not aware of an existing approach that generates local image edits using a transformer model. A transformer does seem ideally suited for this task as they are able to operate on a set of discrete tokens, which means that only the relevant parts of the image can be generated. The proposed components, such as the local attention mask for the transformer and the masked convolution are also to some extent novel, and useful for the task.

(+/-) The authors have described how the work differs from generally related work in a satisfactory manner. However, it would be advisable to add a description of how the proposed method compares to existing transformer-based image synthesis approaches such as [A] and [14] “Taming transformers”. I know [14] is included in the numerical comparison, but it is not clear what parts of the proposed method improve upon [14]. For example, the related work states “recovering images from the discrete codebook still causes blur and artifacts in complex scene” - but the proposed method also uses a discrete codebook. What part of the proposed method addresses this?

[A] Parmar, Niki, et al. "Image transformer." International Conference on Machine Learning. PMLR, 2018.

**Quality**: 

(+) The submission appears to be technically sound and I cannot see any major errors or flaws in the method.
 
(+) The claims made in the paper are largely supported by the experimental results. Specifically, the results show that the proposed approach significantly improves upon the state of the art both qualitatively and quantitatively. The generated images show a significant improvement in detail on both the PA dataset for pose-guided image generation and the face editing task using the FFHQ dataset

(+/-) I would regard the work as complete in terms of the method being well-developed and the convincing results. However, the paper itself can do with some editing as many parts are a bit difficult to read and not expressed very well (see below). 

**Clarity**: 

(-) The submission is not that clearly written and some explanations are very difficult to follow. For example, at the start of the intro (line 20) the authors state “most works can only handle the images of ‘icon-view’ foreground, rather than the image synthesis of ‘non-iconic view’ foreground”. However, as a reader I have no idea what non-iconic view foreground means. The reference provided ([23] - Microsoft COCO) explains this concept to some extent, but it is by no means a general term. I would suggest the authors add some explanation to the paper to clarify it. Some further parts that need to be clarified:

- What are “respective fields” (ln 50,55,61,310)? 

- What does “to lighten the negative influence to the normal CNN learning.” (ln 135) mean?

- What does “condition” and “target” mean in “condition to condition (C2C), condition to target (C2T), target to condition (T2C), and target to target (T2T)”?

- Also, the aspect ratio of some of the figures is severely distorted (especially Figure 4). For a paper about generating high quality images, I would at least expect the basic concepts of image editing to be done correctly.

**Significance**: 

(+) In my view the results are quite important as guided image generation is a challenging problem and the proposed approach show a marked improvement over the current methods in the literature. It would, however, be nice to see some higher resolution editing results as the current images seem to be limited to resolutions of 256x256 which is not that useful for real-world use cases.
The idea of editing and synthesising local parts of the image using a transfomer is definitely an idea that others would be keen to build upon. I can see how this could be applied to other problems too (for example in 3D scene and object synthesis).

# Post author response and discussion
After considering the other reviews and the authors' responses, I remain positive about this paper. The authors have promised to address some of the repeated typos and overall, I think this is a good contribution in terms of image editing.The impact of the paper could be improved by some more visually-appealing examples (such as showcasing high-resolution editing), but this is not a fundamental limitation. Therefore, my final rating is "6: Marginally above the acceptance threshold".

limitations_and_societal_impact: The paper is about synthesising realistic media which has many potential negative societal consequences, however no mention of these are made anywhere in the paper. I strongly suggest that a note about the potential negative impacts is added to the paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper focuses on AutoRegressive (AR) models for the image generation empowered by transformers as they show comparable results compared to Generative Adversarial Networks (GANs). Directly applying such AR models to edit/change local image regions, may suffer from the problems of missing global information, slow inference speed, and information leakage of local guidance. This paper focuses on these problems with a novel model – image Local Autoregressive Transformer (iLAT). The proposed method learns the novel local discrete representations, by the newly proposed local lautoregressive (LA) transformer of the attention mask and convolution mechanism. iLAT can efficiently synthesize the local image regions by key guidance and can perform well in various tasks.
main_review: 1/ The paper focuses on important problems in AR models such as local generation, semantical consistency, generation of only necessary regions (filling missing regions).

2/ A novel attention mask is proposed to incorporate respective fields of both AR and AE to fit LA generation.

3/ The proposed iLAT dramatically reduces the inference time for local generation, since the only masked region will be generated autoregressively. 

4/ Two-stream convolution and a local causal attention mask mechanism are proposed for discrete image encoder and transformer respectively.

5/ The choice of compared works is not clear as there are many GAN-based techniques for missing region completion.

6/ In pose-to-image task, compared to Taming better results are obtained, can authors state the main reasons behind this performance.


limitations_and_societal_impact: Face editing may create some issues such as identity theft so it must be approached carefully.
review_title: null

Meta-review: Three reviewers are positive about the paper, while one is more sceptical. As pointed out by the authors, the latter may be due to some confusion about the task addressed: image editing vs. image inpainting.
So the AC follows the advise of the positive reviewers, and decides to accept the paper for publication in Neurips.
The authors are encouraged to work on the paper to improve the clarity, based on the constructive feedback given by the reviewers.
Decision: Accept (Poster)