The Image Local Autoregressive Transformer
Authors: Chenjie Cao
Yuxin Hong
Xiang Li
Chengrong Wang
Chengming Xu
Yanwei Fu
Xiangyang Xue
Venue: NeurIPS 2021 Poster (2021)
Keywords: Autoregressive
Image synthesis
Generative model
Transformer

Abstract: Recently, AutoRegressive (AR) models for the whole image generation empowered by transformers have achieved comparable or even better performance compared to Generative Adversarial Networks (GANs). Unfortunately, directly applying such AR models to edit/change local image regions, may suffer from the problems of missing global information, slow inference speed, and information leakage of local guidance. To address these limitations, we propose a novel model -- image Local Autoregressive Transformer (iLAT), to better facilitate the locally guided image synthesis. Our iLAT learns the novel local discrete representations, by the newly proposed local autoregressive (LA) transformer of the attention mask and convolution mechanism. Thus iLAT can efficiently synthesize the local image regions by key guidance information. Our iLAT is evaluated on various locally guided image syntheses, such as pose-guided person image synthesis and face editing. Both quantitative and qualitative results show the efficacy of our model.

The Image Local Autoregressive Transformer
Chenjic Cao, Yuxin Hong, Xiang Li, Chengrong Wang, Chengming Xu, Yanwei Fu, Xiangyang Xue
School Data Science Fudan University
[2011080001..ameeiffuaua leraan.edd.ccm
Abstract
Recently, AutoRegressive (AR) models for the whole image generation empowered by transformers have achieved comparable even better performance compared to Generative Adversarial Networks (GANs). Unfortunately, directly applying such AR models to edit/change local image regions, may suffer from the problems of missing global information, slow inference speed, and information leakage of local guidance. To address these limitations, we propose novel model image Local Autoregressive Transformer (iLAT), to better facilitate the locally guided image synthesis. Our iLAT learns the novel local discrete representations, by the newly proposed local autoregressive (LA) transformer of the attention mask and convolution mechanism. Thus iLAT can efficiently synthesize the local image regions by key guidance information. Our iLAT is evaluated on various locally guided image syntheses, such as pose-guided person image synthesis and face editing. Both quantitative and qualitative results show the efficacy of our model.
Introduction
Generating realistic images has been attracting ubiquitous research attention of the community for a long time. In particular, those image synthesis tasks involving persons or portrait [6] 28 29]
can be applied in wide variety of scenarios, such as advertising, games, and motion capture, etc. Most real-world image synthesis tasks only involve the local generation, which means generating pixels in certain regions, while maintaining the semantic consistency, e.g., face editing [19] 40),
pose guiding 33 53 471. and image inpainting [51] 30 49 53). Unfortunately. most works can only handle the well aligned images of 'icon-view' foregrounds, rather than the image synthesis of 'non-iconic view' foregrounds [47] (24), i.e., person instances with arbitrary poses in cluttered scenes, which is concerned in this paper. Even worse, the global semantics tend to be distorted during the generation of previous methods, even if subtle modifications are applied to a local image region. Critically, given the local editing/guidance such as sketches of faces, skeleton of bodies in the first column of Fig. AA),, it is imperative to design our new algorithm for locally guided image synthesis.
Generally, several inherent problems exist in previous works for such task. For example, despite impressive quality of images are generated,
methods [5T] 47 19 30
18 are inclined to synthesize blurr local regions, in Fig. A()oooo(().. Furthermore, some inspiring
autoregressive (AR) methods, such as PixeICNN [32 41 23] and recent transformers [8] 14], should
efficiently model the joint image distribution (even in very complex background B2] for whole
image generation as Fig. BB)-ro(bb). These AR models, however, are still not ready for locally guided image synthesis, several reasons. (1) Missing global information. As in Fig. B vanilla AR models take the top-to-down and left-to-right sequential generation with limited receptive fields for the initial generating (top left corner), which incapable of directly modeling global information. Additionally, the sequential AR models suffer from exposure bias [2], which may
Correponding
LSTBI Algorithm
insireed Inselligence, Zhejiang Univensity,
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Inputs
Outputs
Autrenenderr (AU))
(
Askorerratte (AR)
18)
with Informttion information
6O Local Aumeegrosiive Tranfomeer (ILAT)
roelhed
reaak Myy
(A) Inguts and outputs of local generation compared with previous weeks
(R) Comparison of differentt genenative modes
Figure The illustration of (A) influence of missing semantic consistency, the information leak, and blur in AE based method in the local generation, and (B) comparison of AE, AR, and our iLAT for different conditional image generations. Our method more efficient for locally guided image synthesis by keeping both global semantics and local guidance.
predict future pixels conditioned on the past ones with mistakes, due to the discrepancy between training and testing in AR. This makes small local guidance unpredictable changes to the whole image, resulting in inconsistent semantics as Fig. A))owa(()) (2) Slow inference speed. The AR models have to sequentially predict the pixels in the testing with notoriously slow inference speed, especially for high-resolution image generation. Although the parallel training techniques are used in pixelCNN [2] and transformer [14], the conditional probability sampling fails to work in parallel during the inference phase. (3) Information leakage of local guidance. As shown in Fig. [ (r)o the local guidance should be implemented with specific masks to ensure the validity of the local AR learning. During the sequential training process, pixels from masked regions may be exposed to AR models by convolutions with large kernel sizes inappropriate attention masks in the transformer. We call it information leakage [44] [6] of local guidance, which makes models overfit the masked regions, and miss detailed local guidance in Fig. (o))r.(()).
To this end, we propose a novel image Local Autoregressive Transformer (iLAT) for the task of locally guided image synthesis. Our key idea lies in learning the local discrete representations effectively. Particularly, we tailor the receptive fields of AR models to local guidance, achieving semantically consistent and visually realistic generation results. Furthermore, local autoregressive (LA) transformer with the novel LA attention mask and convolution mechanism is proposed to enable successful local generation of images with efficient inference time, without information leakage.
Formally, we propose the iLAT model with several novel components. (1) We complementarily incorporate receptive fields of both AR and AE to fit LA generation with a novel attention mask as shown Fig. TB)_BB)-B roww(c)) In detail, local discrete representation is proposed to represent those masked regions, while the unmasked areas are encoded with continuous image features. Thus, we achieve favorable results with both consistent global semantics and realistic local generations. (2) Our iLAT dramatically reduces the inference time for local generation, since only masked regions will be generated autoregressively. (3) A simple effective two-stream convolution and a local causal attention mask mechanism are proposed for discrete image encoder and transformer respectively, with which information Icakage is prevented without detriment to the performance.
We make several contributions in this work. (1) A novel local discrete representation learning is proposed to efficiently help to learn our iLAT for the local generation. (2) We propose image local autoregressive transformer for local image synthesis, which enjoys both semantically consistent and realistic generative results. (3) Our iLAT only generates necessary regions autoregressively. which much faster than vanilla AR methods during the inference. (4) We propose a two-stream convolution and a LA attention mask to prevent both convolutions and transformer from information leakage thus improving the quality of generated images. Empirically, we introduce several locally guidance tasks, including pose-guided image generation and face editing tasks; and extensive experiments are conducted on the corresponding dataset to validate the efficacy of our model.

---

2 Related Work
Conditional Image Synthesis. Some conditional generation models are designed to globally generate images with pre-defined styles based on user-provided references, such as poses and face sketches. These previous synthesis efforts are made on Variational auto-encoder (VAB) 100 13]. AR Model [48]
39]. , and AE with adversarial training [51
49 53 Critically, it is very non-trivial for all these methods to generate images of locally guided image synthesis of the non-iconic foreground. Some tentative attempts have been conducted in pose-guided synthesis [42 47 with person existing in non-ironic views. On the other hand, face editing methods are mostly based on adversarial AE based inpainting 30 5T 19] and GAN inversion based methods [53 40 M. Rather than synthesizing the whole image, our iLAT generates the local regions of images autoregressively, which not only improves the stability with the well-optimized joint conditional distribution for large masked regions but also maintains the global semantic information.
Autoregressive Generation. The deep AR models have achieved great success recently in the community [35
Conditional (3). Gated PixelCNN (32], and WaveNet [3T] Recently, transformer based AR models 12 have achieved excellent results in many machine learning tasks. Unfortunately, the
15
32
9 38).
Some well known works include PixeIRNN 43
PixelCNN
38
5
common troubles of these AR models are the expensive inference time and potential exposure bias, as AR models sequentially predict future values from the given past values. The inconsistent receptive fields for training and testing will lead accumulated errors and unreasonable generated results [ Our iLAT is thus designed to address these limitations.
Visual Transformer. The transformer takes advantage of the self-attention module 44 and shows impressive expressive power in many Natural Language Processing (NLP) [38
11 5 and vision
tasks I2 7 25
With costly time and space complexity of (n?²) in transformers, Parmar et al. (34
utilize local self-attention to achieve AR generated results. Chen et al. [8] and Kumar el al. 22 autoregressively generate pixels with simplified discrete color palettes to save computations. But limited by the computing power, they still generate low-rcsolution images. To address this, some works have exploited the discrete representation learning, e.g. dVAE 3 and VQGAN (4. It not only reduces the sequence length of image tokens but also shares perceptually rich latent features in image synthesis as the word embedding in NLP. However, recovering images from the discrete codebook still causes blur and artifacts in complex scenes. Besides, vanilla convolutions of the discrete encoder may leak information among different discrete codebook tokens. Moreover, local conditional generation based on VQGAN [14] suffers from poor semantical consistency compared with other unchanged image regions. To end these, the iL proposes the novel discrete representation to improve the model capability of local image synthesis.
3 Approach
Given the conditional image L. and target image L,, our image Local Autoregressive Transformer (iLAT) aims at producing the output image I, of semantically consistent and visually realistic. The key foreground objects (e.g., the skeleton of body), or the regions of interest (e.g., sketches of facial regions) extracted from Ic, are applied to guide the synthesis of output image I.- Essentially, the background and the other non-key foreground image regions of I, should be visually similar to I. As shown in Fig. 2 our iLAT includes the branches of Two-Stream convolutions based Vector Quantized GAN (TS- (OS-VQAAN) for the discrete representation learning and a transformer for the AR generation with Local Autoregressive (LA) attention mask. Particularly, our iLAT firstly encodes I, and I,, into codebook vectors =g.e and zq by TS-VQGAI
(Sec. 3.1] without local information leakage. Then the index of the masked vectors dg,t will be predicted by the transformer autoregressively with LA attention mask (Sec. (.2). During the test phase, the decoder of TS-VQGAN takes the combination of a..t in masked regions and 2 in unmasked regions to achieve the final result.
3.1 Local Discrete Representation Learning
We propose a novel local discrete representation learning in this section. Since it inefficient to learn the generative transformer model through pixels directly, inspired by VQGAN [4), we incorporate the VQVAE mechanism [ into the proposed iLATT for the discrete representation learning. The VQVAE is consisted of an encoder E, a decoder D, and a learnable discrete codebook Ž = K
3

---

Ouipu Image
Loes (Training)
Inferczse only
M. + (1
TS-VQGAN (1aaass))
only
Combine
Inferenee
MLP hend
Cren
Masked Transformer
Postioa Enobelding
Conditional Image l
CodeBook
Pomkil P
TS-VQGAN Encoder
TS-VIGAA
Eneeder
(egavittie
(nining)
Pose Galllegolly
Target
(Inerenne)
Figure 2: The overview of iLAT with 3 of latent quantization. The target image I, is encoded by TS VS-VQGAN with a binary mask M, while conditional image L, is directly encoded without mask. As explained in Sec(3.2] the guided information of poses and sketches are utilized in our two tasks. For pose guidence, target pose coordinates are encoded by PointMLP into sequential features, which are concatenated with conditional tokens (c(s), Co Cg Cpose }. The TS-VQGAN decoder takes the combination of yg,t converted from generated target tokens { f2 ..., } in masked regions and directly encoded features in unmasked regions to get the final result.
CIK EE
Lecked
Quancized M.
Mork
1000000
E3Caee candiioe DA pe to
Combine
Input
Global sub musk Mgr
GOB Aisonn
35
Musked festure Bi
The T2T part Ma
fenture F2
Local Autoregresaive
Caucal sub-mask M.
(LA) attention mask M..
(a) The two stressstaammmmm convolation
(b) The local autoregreaivee atoention sazek.
Figure 3: Illustration of (a) the two-stream convolution in the TS-VQGAN encoder and (b) the LA attention mask MLA. In (a), after the feature F is convoluted, the 3 x 3 convolution kernel spreads the features from the masked regions to unmasked regions. Patches with leaked information are circled with red. For (b), a 3 x 3 quantized mask M, is assumed as the input mask. MLA can be divided into the global sub-mask Mgs and the causal sub-mask M... Then, t(s), , 13. is can be categorized into global pixel tokens, while others are casual tokens: t1, f2, ts. f7 are masked tokens tm. (Mg.0 = 1), and fo. f1, f4, is are tokens f that need predict masked ones. The global tokens can be attended with all tokens. And causal tokens attend to the targets with a lower triangular matrix. All colored tokens are valued and white tokens are MG.
where K means the total number of codebook vectors. Given image I E Xxxx3 E encodes the image into latent features 8 = EII) E R xaxce, where ce indicates the channel of the encoder outputs. Then, the spatially unfolded ENOw E 1ee,(h' h, w' w) are replaced with the closest codebook vectors as
(g) arg min || zk|l TR 2ho
2LE2
24 fold(= 1' E wh w) gixwxx
where cy indicates the codebook channels. However, VQVAE will suffer from obscure information leakage for the local image synthesis, if the receptive field (kernel size) of vanilla convolution is larger than features to the outside border of the mask. Furthermore, multi-convolutional based E accumulates the information leakage, which makes the model learn the local generating with unreasonable confidence, leading to model overfitting (see Sec. 4.3
as shown Fig. a). Intuitively, each x 3 convolution layer spreads the masked

---

To this end, we present two-stream convolutions in TS-VQGAN as shown in Fig. Bla). Since the masked information is only leaked to a circle around the mask with each 3 x convolution layer. we can just replace the corrupt features for each layer with masked ones. Thus, the influence of information leakage will be eliminated without hurting the integrity of both masked and unmasked features. Specifically, for the given image mask M E IR. Hx that 1 means masked regions, and O means unmasked regions, it should be resized into M' with max- max-pooling to fit the feature size. The two-stream convolution converts the input feature F into the masked feature Fm. = F o (1 - -M',, where O is element-wise multiplication. Then, both and Fm are convoluted with shared weights and combined according to the leaked regions M,, which can be obtained from the difference of convoluted mask as
M clip(conv) (M'' 0, 1) M', M 0] 11
where convi implemented with an all-one 3 x 3 kernel. Therefore, the output of two-stream convolution can be written as
F = conv (F) O (1 - M,) + (Fnv(F O M,.
(3)
So the leaked regions are replaced with features that only depend on unmasked regions. Besides, masked features can be further leveraged for AR learning without any limitations. Compared with VQVAE, we replace all vanilla convolutions with two-stream convolutions in the encoder of TS- VQGAN. Note that the decoder D is unnecessary to prevent information leakage at all. Since the decoding process is implemented after the AR generating of the transformer as shown in Fig. 2
For the VQVAE, the decoder D decodes the codebook vectors 2q got from Eq. and reconstructs the output image as I, = D(=g). Although VQGAN [14] can generate more reliable textures with adversarial training, handling complex real-world backgrounds and precise face details is still tough to the existed discrete learning methods. In TS- VOQAN,, we further finetune the model with local
quantized learning, which can be written as
I. = D2g O Mg + 2 (1 (1- M,,),
(4)
where Mg E R is the resized mask for quantized vectors, and & is the output of the encoder. In Eq. 4. unmasked features directly encoded from the encoder, while masked features are replaced with the codebook vectors, which works between AE and VQVAE. This simple trick effectively maintains the fidelity of the unmasked regions and reduces the number of quantized vectors that have to be generated autoregressively, which also leads to a more efficient local AR inference. Note that the bropaproon of Eq. 4) is implemented with the straight-through gradient estimator 4
3.2 Local Autoregressive Transformer Learning
From the discrete representation learning in Sec. 3.1 we can get the discrete codebook vectors Eq.es Zy,t Rxxxxx for conditional images and target images respectively. Then the conditional and target image tokens {c,, ty hew fij=== {0, 1,.., K 1} can be converted from the index- representation of zq.es Zq,t in the codebook with length hws where K indicates the all number of codebook vectors. For the resized target mask M, E gxx the second stage needs to learn the AR likelihood for the masked target tokens } where Mg. = with conditional tokens (c) and other unmasked target tokens {tu} where My.0 = 0 as
p(4m /c, bu) = II [p((mm buy ((m,<))))
(5)
Benefits from Eq. (4), iLAT only needs to generate masked target tokens (tmm) rather than all. Then, the negative log likelihood (NLL) loss can be optimized as
LNLI BB,,.-p(t. le,t.) (ogp((m,cc, lu).
We use decoder-only transformer to handle the AR likelihood. As shown in Fig. 2 two special tokens c[s), t[s] are concatenated {c,) and {tz} as start tokens respectively. Then, the trainable position embedding [1]] is added to the token embedding to introduce the position information to the self-attention modules. According to the self- attention mechanism in the transformer, the attention mask is the key factor to achieve parallel training without information leakage. As shown in Fig Bbb)
propose a novel LA attention mask MLA with four sub-masks, which indicate receptive fields
5

---

of condition to condition (C2C), condition to target (C2T), target to condition (T2C), and target to target (T2T) respectively. All conditional tokens { can be attended by themselves and targets. So C2C and T2C should be all-one matrices. We think that the conditional tokens are unnecessary to attend targets in advance, so C2T is set as the all-zero matrix. Therefore, the LA attention mask Mr can be written as
1. MAA
1
M
1,
where MLA indicates the T2T LA mask. To leverage the AR generation and maintain the global information simultaneously, the target tokens are divided into two groups called the global group and the causal group. Furthermore, the causal group includes masked targets
- 1) and (fm-1} } that need to predict them, because the labels need to be shifted to the right with one position for the AR learning. Besides, other tokens are classified into the global group. Then, the global attention sub-mask Mgs can be attended to all tokens to share global information and handle the semantic consistency. On the other hand, the causal attention sub-mask M.s constitutes the local AR generation. Note that Mgs can not attend any masked tokens to avoid information leakage. The T2T LA mask can be got with MA = Mgs + M A more intuitive example is shown in Fig. b)) Therefore, for the given feature h, the self-attention in iLAT can be written as
SelfAttention ()) = softmax( QKT - (1 (1- -o)V,
(8)
vd
where Q, K, V are h encoded with different weights of d channels. We make all masked elements to -oo before the softmax. During the inference, all generated target tokens {fm} } are converted back to codebook vectors 8g.t- Then, they are further combined with encoded unmasked features & and decoded with Eq. 4 as shown in Fig. 2
To highlight the difference of our proposed mask MLA, other common attention masks are shown in Fig. D The Vanilla AR mask MAn is widely used in the AR transformer [14 g] but they fail to maintain the semantic consistency and cause unexpected identities for the face synthesis. AE mask MAE is utilized in some attention based image inpainting tasks (50] 49). Although MAE enjoys good receptive fields, the masked regions are completely corrupted in the AE, which is much more unstable to restructure a large hole, Our method is an in-between strategy with both their superiorities mentioned above.
3.3 Implement Details for Different Tasks
Non-Iconic Posed-Guiding. The proposed TS- VO-AN is also learned with adversarial training. For the complex non-iconic pose guiding, we finetune the pretrained open-source ImageNet based VQGAN weights with the two-stream convolution strategy. To avoid adding too many sequences with excessive computations, we use the coordinates of 13 target pose landmarks as the supplemental condition to the iLAT. They are encoded with 3 fully connected layers with ReLU. As shown in Fig. 2 both the condition and the target are images, which have different poses in the training phase, and the same pose in the inference phase. Besides, we use the union of conditional and target masks got by dilating the poses with different kernel sizes according to the scenes to the target image.
Face Editing. In face editing, we find that the adaptive GAN learning weight À makes the results unstable, is replaced with fixed À = 0.1. Besides, the TS- TQ-AN is simplified compared to the one used in the pose guiding. Specifically, all attention layers among the encoder and decoder are removed. Then, all Group Normalizations are replaced with Instance Normalization save memory without a large performance drop. The conditions are composed of the sketch images extracted with the XDoG [46), while the targets are face images. The training masks for the face editing are COCO masks 24 and simulated irregular masks [5I. while the test ones are drawn manually.
4 Experiments
In this section, we present experimental results on pose- aase-guided generation of Penn Action (PA) [22 and Synthetic DeepFashion (SDF) (26), face editing of CelebA [2] and FFHQ 20 compared with other competitors and variants of iLAT.
2 More about the expansion of LA attention mask are discussed in the supplementary.
Details about the mask generation illustrated in the supplementary.

---

Table 1: Quantitative results in PA (left) and SDF (right). 1 means larger is better while | means lower is better. iLAT indicates that iLAT trained without two-stream convolutions.
PATN
PN-GAN
Posewarp
MR-Net
Taming
iLAT*
iLAT
Taming
iLAT
PSNRT 20.83
21.36
21.76
21.79
21.43
21.68
22.94
16,25
16.71
SSIM
0.744
0.761
0.794
0.792
0.746
0.748
0.800
0.539
0.599
MAEL
0.062
0.062
0.053
0.066
0.057
0.056
0.046
0.107
0.096
FID 82.79
64.43
93.61
79.50
33.53
31.83
27.36
72.77
70.58
Datasets. For the pose guiding. PA dataset [52], which contains 2,326 video sequences of 15 action classes in non-iconic views is used in this section. Each frame from the video is annotated with 13 body landmarks consisted of 2D locations and visibility. The resolution of PA is resized into 256 x 256 during the preprocessing. We randomly gather pairs of the same video sequence in the training phase dynamically and select 1,000 testing pairs in the remaining videos. Besides, the SDF is synthesized with DeepFashion (26] images as foregrounds and Places2 [54] images as backgrounds. Since only few images of DeepFashion have related exact segmentation masks, we select 4,500/285 pairs from it for training and testing respectively. Each pair of them contains two images of the same person with different poses and randomly chosen backgrounds. The face editing dataset consists of Hlick--Faces-t
dataset (FFHQ) [20] and CelebA-Ho HO [27]. FFHQ is high-quality image dataset with 70,000 human faces. We resize them from 1024 1024 into 256 x 256 and use 68,000 of them for the training. The CelebA is only used for testing in this section for the diversity. Since face editing has no paired ground truth, we randomly select 68 images from the rest of FFHQ and all CelebA, and draw related sketches for them.
Implementation Details. Our method is implemented in PyTorch in 256 256 image size. For the TS-VQGAN training, we use the Adam optmizer [21] with 31= 0.5 and B- = 0.9. For the pose guiding, the TS-V TS-VQGAN is finetuned from the ImageNet pretrained VQGAN 14. while is completely retrained for FFHQ. TS- TS-VQGAN is trained with 150k steps without masks at first, and then it is trained with another 150k steps with masks in batch size 16. The initial learning rates of pose guiding and face editing are Se-5 and 2e-4 respectively, which are decayed by 0.5 for every 50k: steps. For the transformer training, we use Adam with 31 = 0.9 and 33 0.95 with initial learning rate 5e-5 and 0.01 weight decay. Besides, we warmup the learning rate with the first 10k steps, then it is linearly decayed to for 300k iterations with batch size 16. During the inference, we simply use top-1 sampling for our iLAT.
The model proposed in 14] is abbreviated as Taming transformer (Taming) in this section. For fair comparisons, VQGAN used in Taming is finetuned for pose guiding, and retrained for face editing with the same steps as TS- VQGAN. For the pose guiding, we compare the proposed iLAT with other of-te-of-the-arr methods retrained in the PA dataset, which include PATN [56), PN-GAN 37 Pose Warp 3, MR-Net [47] and Taming 14). As the image size of Warp and MR-Net is 128 x 128, we resized the outputs for the comparison. For the face editing, we compare
Competitors.
the iLATT with inpainting based SC-FEGAN [19] and Taming 14. We also test the Taming results in
our LA attention mask as Taming* (without retraining).
Ponmary
2
2
(A) PoeeCarnee Generiion ItL.
FFIQ (ree 1,30 and CalabA gree
Figure 4: Qualitative results. Targets in (B) are combined with masks and XDoG sketches. Taming* * means that the Taming transformer tested with our LA attention mask. Please zoom-in for details.
7

---

Table Average inference time (sec/image) in PA, SDF, and FFHQ of the vanilla AR transformer based generation (Taming) and iLAT. We also show the average masked rate of three datasets.
masked rate
Taming
iLAT
PA
31.97%
8.551
3,426
SDF
28.09%
8.372
3.898
FFHQ
6.64%
8.183
1.180
4.1 Quantitative Results
Pose-Guided Comparison.
Quantitative results in PA and SDF datasets of our proposed iLAT and other competitors are presented in Tab. Peak signal-to-noise ratio (PSNR), Structural Similarity (SSIM) [45] Mean Average Error (MAE) and Fréchet Inception Distance (FID) [7] are employed to measure the quality of results. We also add the results of iLAT*, which is implemented without the two-stream convolutions. The results in Tab. [ clearly show that our proposed method outperforms other methods in all metrics, especially for the FID, which accords with the human perception. The good scores of iLAT indicate that the proposed iLAT can generate more convincing and photo realisie images locally guided image synthesis of the non-iconie foreground. For the more challenging SDF dataset, iLAT still achieves better results compared with Taming.
We also record the average inference times in PA, SDE, and FFHQ as showed in Tab. Except for the better quality of generated images over Taming as discussed above, our iLAT costs less time for the local synthesis task according to the masking rate of the inputs. Low masking rates can achieve dramatic speedup, e.g., face editing.
Inference Time.
4.2 Qualitative Results
Non-Iconic Pose Guiding, Fig. 4(A) shows qualitative results in the non-iconic pose- gess-ggided image synthesis task. Compared to other competitors, it is apparent that our method generate more reasonable target images both in human bodies and backgrounds, while images generated by other methods suffer from either wrong poses or deformed backgrounds. Partieularly, PATN collapses in most cases. PN-GAN and PoseWarr only copy the reference images as the target ones, which fails to be guided by the given poses due to the challenging PA dataset. Moreover, MR-Net and Taming* can indeed generate poses that are similar to the target ones, but the background details of reference images are not transferred properly. Especially for the results in column (g), Taming fails to synthesize complicated backgrounds, such as noisy audiences in rows 2 and 3 and the gym with various fitness equipment in row 4. Compared others, our proposed iLAT can capture the structure of human bodies given the target poses as well as retaining the vivid backgrounds, which demonstrate the efficacy of our model in synthesizing high-quality images in the non-iconic pose guiding. Besides, for the pose guiding with synthetic backgrounds of SDF, iLAT can still get more reasonable and stable backgrounds and foregrounds compared with Taming as in Fig. 5C).
Face Editing. Since there are no ground truth face editing targets, we only compared the qualitative results shown in Fig. 4 of FFHQ and CelebA. Note that the Taming results in column (c) fail to preserve the identity information in both FFHQ and CelebA compared with the reference. For example, in rows 1, 2 and 3, the skin tones of Taming results are different from the original ones. And in row 4, Taming generates absolutely another person with contrasting ages, which indicates that vanilla AR is unsuited to the local face editing. When Taming is tested with our LA attention mask, column (d) shows that Taming* can retain the identities of persons. However, rows I and 2 demonstrate that Taming* fails properly generate the target faces according to guided sketches, while in rows 3 and 4 some generations have obvious artifacts without consistency. Besides, inpainting- based SC-FEGAN achieves unstable results in rows 3 and SC-FEGAN also strongly depends on the quality of input sketches, while unprofessional sketches lead to unnatural results as shown row 1. Besides, detailed textures of AE-based SC-FEGAN are unsatisfactory too. Compared with these methods, our iLAT can always generate correct and vivid human faces with identities retained. Furthermore, benefits from the discrete representation, iLATT enjoys robustness to the guided information.
8

---

Target
Taming
Ablation in pose guidlng
editing
nouls
Figure 5: Ablation study for two-stream convolutions (A, B) and qualitative results in SDF (C). iLAT* means iLAT without two-stream convolutions. Please zoom-in for details.
4.3 Further Discussions
Ablation Study. The effectiveness of our proposed two-stream convolution discussed in the ablation study. As we can find in Fig. 5] the woman face in row 1, (c) generated by iLAT* has residual face parts that conflict with the guided sketch. Moreover, in row 2, iLAT* without two-stream convolutions leaks information from sunglasses that lacks correct semantic features and leads to the inconsistent color of the generated face. For the pose-guided instance shown in the second row, is apparent that the man generated in column (c) has blurry leg positions. However, in column (d) the complete iLAT can always generate authentic and accurate images, validating the efficacy of our designed two-stream convolutions.
Sequential Generation. Our iLAT can also be extended to guide the video generation properly. We give qualitative example in this section. As shown in Fig. 6 given sequence of poses and one reference image, iLATT can forecast plausible motion of the person. And the results are robust for most kinds of activities in non-ironic views.
3.
Figure 6: Sequential generated results by iLAT different ironic views, where odd rows mean the generated images, while even rows indicate the guided poses.
5
Conclusion
This paper proposes a transformer based AR model called iLAT to solve local image generation tasks. This method leverages a novel LA attention mask to enlarge the receptive fields of AR, which achieves not only semantically consistent
also realistic generative results and accelerates inference speed of AR. Besides, TS- VQGAN is proposed to learn discrete representation learning without information leakages. Such model can get superior performance detail editing. Extensive experiments validate the efficacy of our iLAT for local image generation.

---

Social Impacts
This paper exploited the image editing with transformers. Since face editing may causes some privacy issues, we sincerely remind users to pay attention for it. Our method only focuses on technical aspects. The images used in this paper are all open soureed.
Acknowledgements
This work was supported in part by NSFC Project (62076067, 62176061), Science and Technology Commission of Shanghai Municipality Projects
2021SHZDDZ00133)
References
[1] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan++: How edit the embedded images? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8296-8305, 2020.
[2] Kenan Ak, Ning Xu, Zhe Lin, and Yilin Wang. Incorporating reinforced adversarial learning in autoregressive image generation. arXiv preprint uxXiv:2007009223 2020.
[3] Guha Balakrishnan, Amy Zhao, Adrian V Dalca, Fredo Durand, and John Guttag. Synthesizing images humans in unseen poses. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages $340-8348, 2018.
[4] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 1308.34.. 3432, 2013.
[5] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
[6] Haoye Cai, Chunyan Bai, Yu-Wing Tai, and Chi-Keung Tang. Deep video generation, prediction and completion of human action sequences. In Proceedings of the European Conference Computer Vision (ECCV), pages 366-382, 2018
[7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference Computer Vision, pages 213-229. Springer, 2020.
[8] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pages 1691-1703. PMLR, 2020.
[9] Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. Pixelsnail: An improved autoregressive generative model. In International Conference on Machine Learning, pages 864-872, PMLR, 2018.
[10] Yen-Chi Cheng, Hsin-Ying Ying Lee, Min Sun, and Ming-Hsuan Yang. Controllable image synthesis via segvae. In ECCV. Springer, 2020.
[11] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXiv:1810.04800 2018.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprins arXiv:2010.1 1929, 2020.
[13] Yuki Endo and Yoshihiro Kanamori. Diversifying semantic image synthesis and editing via class-and layer-wise vaes. In Computer Graphics Forum, volume 39, pages 519-530. Wiley Online Library, 2020.
10

---

[14] Patrick Esser, Robin Rombach, and Bjöm Ommer. Taming transformers for high-resolution image synthesis. arXiv preprint arXiv:2012.098411 2020.
[15] Klaus Greff, Rupesh K Srivastava, Jan Koutník, Bas R Steunebrink, and Jürgen Schmidhuber. Lstm: search space odyssey. IEEE transactions on neural networks and learning systems, (8(10))2222-2232 2016.
[16] Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text generation via adversarial training with leaked information. AAAI, 2018.
[17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. arXiv preprint Xiv: ww..vvv:060000 1706.08000,00 2017.
[18] Kun Huang, Yifan Wang. Zihan Zhou, Tianjiao Ding, Shenghua Gao, and Yi Ma. Learning to parse wireframes in images of man-made environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[19] Youngjoo and Jongyoul Park. Sc-fegan: face editing generative adversarial network with user's s sketch and color. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1753-1753, 2019.
[20] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2019.
[21] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1121v:1412.6980, 2014.
[22] Manoj Kumar, Dirk Weissenborn, and Nal Kalchbrenner. Colorization transformer. arXiv preprint arXiv:2102.04432 2021.
[23] Henry Leopold, Jeff Orchard, John S Zelek, and Vasudevan Lakshminarayanan. Pixelbnn: Augmenting the pixelenn with batch normalization and the presentation of a fast architecture for retinal vessel segmentation. Journal of Imaging, 5(2):26, 2019.
[24] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[25] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103. 14030, 2021.
[26] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[27] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 7730-3738, 2015.
[28] Liqian Ma, Xu Jia. Qianru Sun, Bernt Schiele. Tinne Tuytelaars, and Luc Van Gool. Pose guided person image generation. arXiv preprini arXiv: 1705. 09368, 2017.
[29] Liqian Ma, Qianru Sun, Stamatios Georgoulis, Luc Van Gool, Bernt Schiele, and Mario Fritz. Disentangled person image generation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 99-108, 2018.
[30] Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Qureshi, and Mehran Ebrahimi. Edgeconnect: Structure guided image inpainting using edge prediction. In Proceedings of the IEEF/CVF International Conference Computer Vision Workshops, pages 0-0, 2019.
11

---

[31] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint (609. 2016.
[32] Aaron van den Oord, Nal Kalchbrenner. Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with pixelenn decoders. arXiv preprint 1rXiv:1606.05322 2016.
[33] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. arXiv preprint arXiv: arXiv:1711.00933 2017.
[34] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz. Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference Machine Learning, pages 4055 4064. PMLR, 2018.
[35] Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Ken- ton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05335. 2018.
[36] Albert Pumarola, Antonio Agudo, Alberto Sanfeliu, and Francesc Moreno-Noguer. Unsuper- vised person image synthesis in arbitrary poses. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8620-8628, 2018
[37] Xuelin Qian, Yanwei Fu, Tao Xiang, Wenxuan Wang. Jie Qiu, Yang Wu, Yu-Gang Jiang, and Xiangyang Xue. Pose-normalized image generation for person re-identification. In Proceedings of the European conference on computer vision (ECCV), pages 650-667, 2018.
[38] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training 2018.
[39] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102. 12092. 2021.
[40] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding style: stylegan encoder for image-to-image translation. arXiv preprint arXiv:2008.0099 2020.
[41] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelemn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint 1701.0/70 2017.
[42] Sijie Song, Wei Zhang, Jiaying Liu, Zongming Guo, and Tao Mei. Unpaired person image generation with semantic parsing transformation. IEEE transactions on puttern analysis and machine intelligence, 2020.
[43] Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, pages 1747-1756. PMLR, 2016.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv: arXiv: 770700060777 2017.
[45] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
[46] Holger Winnemöller, Jan Eric Kyprianidis, and Sven C Olsen. Xdog: an extended difference- of- of-gaassiias compendium including advanced image stylization. Computers Graphics 36(6):740-753, 2012.
[47] Chengming Xu, Yanwei Fu, Chao Wen, Ye Pan, Yu-Gang Jiang, and Xiangyang Xue. Pose- guided person image synthesis in the non-iconic views. IEEE Transactions on Image Processing, 29:9060-9077 9072, 2020.
12

---

[48] Jingyu Yang, Xinchen Ye, Kun Li, Chunping Hou, and Yao Wang. Color-guided depth recovery from rgb-d data using an adaptive autoregressive model. IEEE transactions on image processing. 23(8)3344-3453 34488434488,,, 2014.
[49] Zili Yi, Qiang Tang, Shekoofeh Azizi, Daesik Jang, and Zhan Xu. Contextual residual aggrega- tion for ultra high- nigh-resolution image impainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7508-7517. 2020.
[50] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5505-5514, 2018.
[51] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2019.
[52] Weiyu Zhang, Menglong Zhu, and Konstantinos G Derpanis. From actemes to action: strongly-supervised representation for detailed action understanding. In Proceedings of the IEEE International Conference on Computer Vision, pages 2248-22225522 2013.
[53] Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong. Xiao Liang, Eric Chang, and Yan Xu. Large scale image completion via co-modulated generative adversarial networks. arXiv preprint arXiv:2103. 10428, 2021.
[54] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
[55] Xingran Zhou, Siyu Huang, Bin Li, Yingming Li, Jiachen Li, and Zhongfei Zhang. Text guided person image synthesis. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3663-3672, 2019.
[56] Zhen Zhu, Tengteng Huang, Baoguang Shi, Miao Yu, Bofei Wang, and Xiang Bai. Progressive pose attention transfer for person image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2347-2356, 2019.
13


Reviews:
[{"review_id": "x6FBzCK9mVz", "reviewer": "Reviewer_yqns", "summary": "This paper aims to utilize the auto-regressive generative model to achieve the local image editing based on previous state-of-the-art method *taming transformer*.  Specifically, to prevent the information leakage while encoding the input image into discrete sequences, a very straightforward two-stream convolution is proposed. Besides, this paper also presents a new local auto-regressive transformer to benefit from the capability of generative AR model and the global information simultaneously. The experiments show   some improvements over compared methods.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n\nWhat this paper wants to achieve, i.e. editing any regions of input image while maintaining the original information, is really interesting, but there indeed are some aspects should be explained and clarified.\n\n* The **real effectiveness** of two-stream convolution is not claimed well. In my understanding, the ultimate target of VQ-VAE is to *faithfully reconstruct the input image*, even though there are masks in the input. However, none experiments analyze why the designed convolution is better or could help this point. By contrast, the main effective part of Sec 3.1 may be Equation 4. However, in Line157-Line159, it claims that such simple trick makes AR inference more efficient, which also seems inaccurate. In my opinion, this finetuning trick actually aims to preserve the boundary textures as much as possible, and the efficiency improvements come from the subsequent design of local auto-regressive transformer. Even so, I believe that there are still cases in which the original local contents could not be preserved well while giving arbitrary downsampled masks. This aspect also should be analyzed in details.\n\n* Sec 3.2 introduces the local autoregressive transformer learning, but is it the strict auto-regressive model? To predict next tokens, you need to rely on the conditions of both past tokens and *future* global tokens, which actually breaks the fixed-order decomposed joint distributions of pixels.\n\n* The comparison with *taming transformer* is not fair. You should re-train the baseline, i.e., conditioned on the edited input to regress the final results. For the unmasked region, pick the top-1 token while sampling to reserve the original content.\n\n* Why does this paper mainly focus on editing? The best way to show the improvements of the proposed method is to conduct experiments on image inpainting task. The rests are just the controllable applications of image inpainting.\n\nBased on these considerations, I think currently this submission could not reach the acceptance bar of NeurIPS venue. My rating is rejection now.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see the above mentioned comments.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "dk1emrjpw0y", "reviewer": "Reviewer_eaP9", "summary": "This paper proposes a transformer-based model for local image editing. The key challenges tackled by the paper is the need to integrate global information, while also preventing leakage of local guidance information (i.e. the local editing should only affect the designated region). To accomplish this, a combination of discrete representation learning (VQVAE) and an autoregresive transformer model is used. The local guidance is enforced through the use of a masked convolution in the feature learning and a local attention mask in the transformer.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**TL;DR:** The paper introduces a useful technique for local image editing / synthesis, but the writing could be substantially improved to make the paper clearer. In the main review below I have labelled points as a strength (+), weakness (-) or mixed (+/-).\n\n**Originality**: \n\n(+) The method proposed in the paper has some novelty. Particularly, I’m not aware of an existing approach that generates local image edits using a transformer model. A transformer does seem ideally suited for this task as they are able to operate on a set of discrete tokens, which means that only the relevant parts of the image can be generated. The proposed components, such as the local attention mask for the transformer and the masked convolution are also to some extent novel, and useful for the task.\n\n(+/-) The authors have described how the work differs from generally related work in a satisfactory manner. However, it would be advisable to add a description of how the proposed method compares to existing transformer-based image synthesis approaches such as [A] and [14] “Taming transformers”. I know [14] is included in the numerical comparison, but it is not clear what parts of the proposed method improve upon [14]. For example, the related work states “recovering images from the discrete codebook still causes blur and artifacts in complex scene” - but the proposed method also uses a discrete codebook. What part of the proposed method addresses this?\n\n[A] Parmar, Niki, et al. \"Image transformer.\" International Conference on Machine Learning. PMLR, 2018.\n\n**Quality**: \n\n(+) The submission appears to be technically sound and I cannot see any major errors or flaws in the method.\n \n(+) The claims made in the paper are largely supported by the experimental results. Specifically, the results show that the proposed approach significantly improves upon the state of the art both qualitatively and quantitatively. The generated images show a significant improvement in detail on both the PA dataset for pose-guided image generation and the face editing task using the FFHQ dataset\n\n(+/-) I would regard the work as complete in terms of the method being well-developed and the convincing results. However, the paper itself can do with some editing as many parts are a bit difficult to read and not expressed very well (see below). \n\n**Clarity**: \n\n(-) The submission is not that clearly written and some explanations are very difficult to follow. For example, at the start of the intro (line 20) the authors state “most works can only handle the images of ‘icon-view’ foreground, rather than the image synthesis of ‘non-iconic view’ foreground”. However, as a reader I have no idea what non-iconic view foreground means. The reference provided ([23] - Microsoft COCO) explains this concept to some extent, but it is by no means a general term. I would suggest the authors add some explanation to the paper to clarify it. Some further parts that need to be clarified:\n\n- What are “respective fields” (ln 50,55,61,310)? \n\n- What does “to lighten the negative influence to the normal CNN learning.” (ln 135) mean?\n\n- What does “condition” and “target” mean in “condition to condition (C2C), condition to target (C2T), target to condition (T2C), and target to target (T2T)”?\n\n- Also, the aspect ratio of some of the figures is severely distorted (especially Figure 4). For a paper about generating high quality images, I would at least expect the basic concepts of image editing to be done correctly.\n\n**Significance**: \n\n(+) In my view the results are quite important as guided image generation is a challenging problem and the proposed approach show a marked improvement over the current methods in the literature. It would, however, be nice to see some higher resolution editing results as the current images seem to be limited to resolutions of 256x256 which is not that useful for real-world use cases.\nThe idea of editing and synthesising local parts of the image using a transfomer is definitely an idea that others would be keen to build upon. I can see how this could be applied to other problems too (for example in 3D scene and object synthesis).\n\n# Post author response and discussion\nAfter considering the other reviews and the authors' responses, I remain positive about this paper. The authors have promised to address some of the repeated typos and overall, I think this is a good contribution in terms of image editing.The impact of the paper could be improved by some more visually-appealing examples (such as showcasing high-resolution editing), but this is not a fundamental limitation. Therefore, my final rating is \"6: Marginally above the acceptance threshold\".\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper is about synthesising realistic media which has many potential negative societal consequences, however no mention of these are made anywhere in the paper. I strongly suggest that a note about the potential negative impacts is added to the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "QHLg5ZbxVqG", "reviewer": "Reviewer_BKtm", "summary": "This paper focuses on AutoRegressive (AR) models for the image generation empowered by transformers as they show comparable results compared to Generative Adversarial Networks (GANs). Directly applying such AR models to edit/change local image regions, may suffer from the problems of missing global information, slow inference speed, and information leakage of local guidance. This paper focuses on these problems with a novel model – image Local Autoregressive Transformer (iLAT). The proposed method learns the novel local discrete representations, by the newly proposed local lautoregressive (LA) transformer of the attention mask and convolution mechanism. iLAT can efficiently synthesize the local image regions by key guidance and can perform well in various tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1/ The paper focuses on important problems in AR models such as local generation, semantical consistency, generation of only necessary regions (filling missing regions).\n\n2/ A novel attention mask is proposed to incorporate respective fields of both AR and AE to fit LA generation.\n\n3/ The proposed iLAT dramatically reduces the inference time for local generation, since the only masked region will be generated autoregressively. \n\n4/ Two-stream convolution and a local causal attention mask mechanism are proposed for discrete image encoder and transformer respectively.\n\n5/ The choice of compared works is not clear as there are many GAN-based techniques for missing region completion.\n\n6/ In pose-to-image task, compared to Taming better results are obtained, can authors state the main reasons behind this performance.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Face editing may create some issues such as identity theft so it must be approached carefully.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 hour"}, "extra_text": {}}, {"review_id": "Ix_uzpnXcwg", "reviewer": "Reviewer_je6a", "summary": "This paper proposes to use the self-attention mechanism (in particular, Transformer) to enhance image editing. Experiments show promising performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes to use the self-attention mechanism (in particular, Transformer) to enhance image editing. Experiments show promising performance.\n\nThis paper falls out of my research field. I would try my best to offer some comments.\n\nFirst, the idea of using Transformer, a recently popular module, to enhance the existing problems (e.g. missing global information) is fine. However, this idea seems to have been proposed by the Taming paper (which is cited frequently in this work). Based on this, I am a bit confused of the contribution of this work. In particular, regarding the issues mentioned in the abstract (missing global information, and information leakage of local guidance), how are they solved by the Transformer design, and can both of them be solved by the Transformer design? In particular, the issue of information leakage issue - is it alleviated by random masking? In summary, this paper should mention clearly about the difference between this paper and the Taming paper, and the additional contribution beyond the Taming paper.\n\nRegarding the network design, since I am not an expert, I can only say that everything looks fine, and I will wait for other reviewers' comments for further judgment.\n\nRegarding the experiments, the shown results seem good, but I have two minor questions in Fig 4. (1) In the left part, why is the resolution of some methods significantly different from others? (2) In the right part, why the proposed method change some non-edited contents in the original image (e.g. in the 3rd row, the letter of S in the right border of the image) ?\n\nOverall, this is a well prepared paper. I give a borderline score and will wait for additional information to make the final decision.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three reviewers are positive about the paper, while one is more sceptical. As pointed out by the authors, the latter may be due to some confusion about the task addressed: image editing vs. image inpainting.
So the AC follows the advise of the positive reviewers, and decides to accept the paper for publication in Neurips.
The authors are encouraged to work on the paper to improve the clarity, based on the constructive feedback given by the reviewers.
Decision: Accept (Poster)