The Image Local Autoregressive Transformer

1 Introduction . Generating realistic images has been attracting ubiquitous research attention of the community for a long time . In particular , those image synthesis tasks involving persons or portrait [ 6 , 28 , 29 ] can be applied in a wide variety of scenarios , such as advertising , games , and motion capture , etc . Most real-world image synthesis tasks only involve the local generation , which means generating pixels in certain regions , while maintaining the semantic consistency , e.g. , face editing [ 19 , 1 , 40 ] , pose guiding [ 36 , 55 , 47 ] , and image inpainting [ 51 , 30 , 49 , 53 ] . Unfortunately , most works can only handle the well aligned images of ‘ icon-view ’ foregrounds , rather than the image synthesis of ‘ non-iconic view ’ foregrounds [ 47 , 24 ] , i.e. , person instances with arbitrary poses in cluttered scenes , which is concerned in this paper . Even worse , the global semantics tend to be distorted during the generation of previous methods , even if subtle modifications are applied to a local image region . Critically , given the local editing/guidance such as sketches of faces , or skeleton of bodies in the first column of Fig . 1 ( A ) , it is imperative to design our new algorithm for locally guided image synthesis . Generally , several inherent problems exist in previous works for such a task . For example , despite impressive quality of images are generated , GANs/Autoencoder ( AE ) -based methods [ 51 , 47 , 19 , 30 , 18 ] are inclined to synthesize blurry local regions , as in Fig . 1 ( A ) -row ( c ) . Furthermore , some inspiring autoregressive ( AR ) methods , such as PixelCNN [ 32 , 41 , 23 ] and recent transformers [ 8 , 14 ] , should efficiently model the joint image distribution ( even in very complex background [ 32 ] ) for whole image generation as Fig . 1 ( B ) -row ( b ) . These AR models , however , are still not ready for locally guided image synthesis , as several reasons . ( 1 ) Missing global information . As in Fig . 1 ( B ) -row ( b ) , vanilla AR models take the top-to-down and left-to-right sequential generation with limited receptive fields for the initial generating ( top left corner ) , which are incapable of directly modeling global information . Additionally , the sequential AR models suffer from exposure bias [ 2 ] , which may ∗ Corresponding author . Dr. Fu is also with Fudan ISTBI—ZJNU Algorithm Centre for Brain-inspired Intelligence , Zhejiang Normal University , Jinhua , China . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . predict future pixels conditioned on the past ones with mistakes , due to the discrepancy between training and testing in AR . This makes small local guidance unpredictable changes to the whole image , resulting in inconsistent semantics as in Fig . 1 ( A ) -row ( a ) . ( 2 ) Slow inference speed . The AR models have to sequentially predict the pixels in the testing with notoriously slow inference speed , especially for high-resolution image generation . Although the parallel training techniques are used in pixelCNN [ 32 ] and transformer [ 14 ] , the conditional probability sampling fails to work in parallel during the inference phase . ( 3 ) Information leakage of local guidance . As shown in Fig . 1 ( B ) -row ( c ) , the local guidance should be implemented with specific masks to ensure the validity of the local AR learning . During the sequential training process , pixels from masked regions may be exposed to AR models by convolutions with large kernel sizes or inappropriate attention masks in the transformer . We call it information leakage [ 44 , 16 ] of local guidance , which makes models overfit the masked regions , and miss detailed local guidance as in Fig . 1 ( A ) -row ( b ) . To this end , we propose a novel image Local Autoregressive Transformer ( iLAT ) for the task of locally guided image synthesis . Our key idea lies in learning the local discrete representations effectively . Particularly , we tailor the receptive fields of AR models to local guidance , achieving semantically consistent and visually realistic generation results . Furthermore , a local autoregressive ( LA ) transformer with the novel LA attention mask and convolution mechanism is proposed to enable successful local generation of images with efficient inference time , without information leakage . Formally , we propose the iLAT model with several novel components . ( 1 ) We complementarily incorporate receptive fields of both AR and AE to fit LA generation with a novel attention mask as shown in Fig . 1 ( B ) -row ( c ) . In detail , local discrete representation is proposed to represent those masked regions , while the unmasked areas are encoded with continuous image features . Thus , we achieve favorable results with both consistent global semantics and realistic local generations . ( 2 ) Our iLAT dramatically reduces the inference time for local generation , since only masked regions will be generated autoregressively . ( 3 ) A simple but effective two-stream convolution and a local causal attention mask mechanism are proposed for discrete image encoder and transformer respectively , with which information leakage is prevented without detriment to the performance . We make several contributions in this work . ( 1 ) A novel local discrete representation learning is proposed to efficiently help to learn our iLAT for the local generation . ( 2 ) We propose an image local autoregressive transformer for local image synthesis , which enjoys both semantically consistent and realistic generative results . ( 3 ) Our iLAT only generates necessary regions autoregressively , which is much faster than vanilla AR methods during the inference . ( 4 ) We propose a two-stream convolution and a LA attention mask to prevent both convolutions and transformer from information leakage , thus improving the quality of generated images . Empirically , we introduce several locally guidance tasks , including pose-guided image generation and face editing tasks ; and extensive experiments are conducted on the corresponding dataset to validate the efficacy of our model . 2 Related Work . Conditional Image Synthesis . Some conditional generation models are designed to globally generate images with pre-defined styles based on user-provided references , such as poses and face sketches . These previous synthesis efforts are made on Variational auto-encoder ( VAE ) [ 10 , 13 ] , AR Model [ 48 , 39 ] , and AE with adversarial training [ 51 , 49 , 53 ] . Critically , it is very non-trivial for all these methods to generate images of locally guided image synthesis of the non-iconic foreground . Some tentative attempts have been conducted in pose-guided synthesis [ 42 , 47 ] with person existing in non-ironic views . On the other hand , face editing methods are mostly based on adversarial AE based inpainting [ 30 , 51 , 19 ] and GAN inversion based methods [ 53 , 40 , 1 ] . Rather than synthesizing the whole image , our iLAT generates the local regions of images autoregressively , which not only improves the stability with the well-optimized joint conditional distribution for large masked regions but also maintains the global semantic information . Autoregressive Generation . The deep AR models have achieved great success recently in the community [ 35 , 15 , 32 , 9 , 38 ] . Some well known works include PixelRNN [ 43 ] , Conditional PixelCNN [ 32 ] , Gated PixelCNN [ 32 ] , and WaveNet [ 31 ] . Recently , transformer based AR models [ 38 , 5 , 12 ] have achieved excellent results in many machine learning tasks . Unfortunately , the common troubles of these AR models are the expensive inference time and potential exposure bias , as AR models sequentially predict future values from the given past values . The inconsistent receptive fields for training and testing will lead to accumulated errors and unreasonable generated results [ 2 ] . Our iLAT is thus designed to address these limitations . Visual Transformer . The transformer takes advantage of the self-attention module [ 44 ] , and shows impressive expressive power in many Natural Language Processing ( NLP ) [ 38 , 11 , 5 ] and vision tasks [ 12 , 7 , 25 ] . With costly time and space complexity of O ( n2 ) in transformers , Parmar et al . [ 34 ] utilize local self-attention to achieve AR generated results . Chen et al . [ 8 ] and Kumar et al . [ 22 ] autoregressively generate pixels with simplified discrete color palettes to save computations . But limited by the computing power , they still generate low-resolution images . To address this , some works have exploited the discrete representation learning , e.g . dVAE [ 39 ] and VQGAN [ 14 ] . It not only reduces the sequence length of image tokens but also shares perceptually rich latent features in image synthesis as the word embedding in NLP . However , recovering images from the discrete codebook still causes blur and artifacts in complex scenes . Besides , vanilla convolutions of the discrete encoder may leak information among different discrete codebook tokens . Moreover , local conditional generation based on VQGAN [ 14 ] suffers from poor semantical consistency compared with other unchanged image regions . To end these , the iLAT proposes the novel discrete representation to improve the model capability of local image synthesis . 3 Approach . Given the conditional image Ic and target image It , our image Local Autoregressive Transformer ( iLAT ) aims at producing the output image Io of semantically consistent and visually realistic . The key foreground objects ( e.g. , the skeleton of body ) , or the regions of interest ( e.g. , sketches of facial regions ) extracted from Ic , are applied to guide the synthesis of output image Io . Essentially , the background and the other non-key foreground image regions of Io should be visually similar to It . As shown in Fig . 2 , our iLAT includes the branches of Two-Stream convolutions based Vector Quantized GAN ( TS-VQGAN ) for the discrete representation learning and a transformer for the AR generation with Local Autoregressive ( LA ) attention mask . Particularly , our iLAT firstly encodes Ic and It , into codebook vectors zq , c and zq , t by TS-VQGAN ( Sec . 3.1 ) without local information leakage . Then the index of the masked vectors ẑq , t will be predicted by the transformer autoregressively with LA attention mask ( Sec . 3.2 ) . During the test phase , the decoder of TS-VQGAN takes the combination of ẑq , t in masked regions and ẑ in unmasked regions to achieve the final result . 3.1 Local Discrete Representation Learning . We propose a novel local discrete representation learning in this section . Since it is inefficient to learn the generative transformer model through pixels directly , inspired by VQGAN [ 14 ] , we incorporate the VQVAE mechanism [ 33 ] into the proposed iLAT for the discrete representation learning . The VQVAE is consisted of an encoder E , a decoder D , and a learnable discrete codebook Z = { zk } Kk=1 , where K means the total number of codebook vectors . Given an image I ∈ RH×W×3 , E encodes the image into latent features ẑ = E ( I ) ∈ Rh×w×ce , where ce indicates the channel of the encoder outputs . Then , the spatially unfolded ẑh′w′ ∈ Rce , ( h′ ∈ h , w′ ∈ w ) are replaced with the closest codebook vectors as z ( q ) h′w′ = arg minzk∈Z ||ẑh′w′ − zk|| ∈ Rcq , zq = fold ( z ( q ) h′w′ , h ′ ∈ h , w′ ∈ w ) ∈ Rh×w×cq , ( 1 ) where cq indicates the codebook channels . However , VQVAE will suffer from obscure information leakage for the local image synthesis , if the receptive field ( kernel size ) of vanilla convolution is larger than 1× 1 as shown in Fig . 3 ( a ) . Intuitively , each 3× 3 convolution layer spreads the masked features to the outside border of the mask . Furthermore , multi-convolutional based E accumulates the information leakage , which makes the model learn the local generating with unreasonable confidence , leading to model overfitting ( see Sec . 4.3 ) . To this end , we present two-stream convolutions in TS-VQGAN as shown in Fig . 3 ( a ) . Since the masked information is only leaked to a circle around the mask with each 3 × 3 convolution layer , we can just replace the corrupt features for each layer with masked ones . Thus , the influence of information leakage will be eliminated without hurting the integrity of both masked and unmasked features . Specifically , for the given image mask M ∈ RH×W that 1 means masked regions , and 0 means unmasked regions , it should be resized into M′ with max-pooling to fit the feature size . The two-stream convolution converts the input feature F into the masked feature Fm = F ( 1−M′ ) , where is element-wise multiplication . Then , both F and Fm are convoluted with shared weights and combined according to the leaked regions Ml , which can be obtained from the difference of convoluted mask as Ml = clip ( conv1 ( M ′ ) , 0 , 1 ) −M′ , Ml [ Ml > 0 ] = 1 , ( 2 ) where conv1 implemented with an all-one 3 × 3 kernel . Therefore , the output of two-stream convolution can be written as Fc = conv ( F ) ( 1−Ml ) + conv ( Fm ) Ml . ( 3 ) So the leaked regions are replaced with features that only depend on unmasked regions . Besides , masked features can be further leveraged for AR learning without any limitations . Compared with VQVAE , we replace all vanilla convolutions with two-stream convolutions in the encoder of TSVQGAN . Note that the decoder D is unnecessary to prevent information leakage at all . Since the decoding process is implemented after the AR generating of the transformer as shown in Fig . 2 . For the VQVAE , the decoder D decodes the codebook vectors zq got from Eq . ( 1 ) , and reconstructs the output image as Io = D ( zq ) . Although VQGAN [ 14 ] can generate more reliable textures with adversarial training , handling complex real-world backgrounds and precise face details is still tough to the existed discrete learning methods . In TS-VQGAN , we further finetune the model with local quantized learning , which can be written as Io = D ( zq Mq + ẑ ( 1−Mq ) ) , ( 4 ) where Mq ∈ Rh×w is the resized mask for quantized vectors , and ẑ is the output of the encoder . In Eq . ( 4 ) , unmasked features are directly encoded from the encoder , while masked features are replaced with the codebook vectors , which works between AE and VQVAE . This simple trick effectively maintains the fidelity of the unmasked regions and reduces the number of quantized vectors that have to be generated autoregressively , which also leads to a more efficient local AR inference . Note that the back-propagation of Eq . ( 4 ) is implemented with the straight-through gradient estimator [ 4 ] .

Summary: This paper aims to utilize the auto-regressive generative model to achieve the local image editing based on previous state-of-the-art method *taming transformer*.  Specifically, to prevent the information leakage while encoding the input image into discrete sequences, a very straightforward two-stream convolution is proposed. Besides, this paper also presents a new local auto-regressive transformer to benefit from the capability of generative AR model and the global information simultaneously. The experiments show   some improvements over compared methods.