Relative Pixel Prediction For Autoregressive Image Generation

1 INTRODUCTION . It is has long been appreciated in the field of image compression ( Harrison , 1952 ; Haskell & Netravali , 1997 ) that adjacent pixels in natural images tend to share similar colors and intensities . This is evidenced in Figure 1 , where we plot the marginal distribution of absolute sub-pixel ( i.e. , color channel ) intensities and the difference between the sub-pixel intensities to the corresponding sub-pixel immediately to its left . While the absolute pixel values tend to be distributed non-trivially across the whole range of possible sub-pixel values [ 0 , 255 ] with peaks around the values of 100 and 255 , the relative sub-pixel values closely resembles a Laplacian distribution with the mean at 0 as most pixels do not differ greatly from their neighbouring pixels ( Takamura , 1996 ) . This suggests that image generation process could mostly be done by predicting the relative distance between existing and new pixels , and absolute pixel prediction is only needed to generate more abrupt transitions , such as boundaries between two different objects . In predictive coding compression models , this fact is leveraged to achieve increased compression rates by learning to predict pixels as a delta between pixels rather than predicting their absolute values ( al Mahmood & Al-Rubaye , 2014 ) . Autoregressive models are one of the main forces driving research in image generation van den Oord et al . ( 2016a ) ; Salimans et al . ( 2017 ) ; Parmar et al . ( 2018 ) ; Chen et al . ( 2018 ) ; Child et al . ( 2019 ) . In contrast to other popular image models based on adversarial methods ( Goodfellow et al. , 2014 ; Radford et al. , 2015 ) or that use latent variables ( van den Oord et al. , 2017 ) , autoregressive models have a tractable likelihood function that decomposes image generation into a sequence of conditionally dependent pixel predictions . This is a desirable property that provides the grounds for applying the same principles of predictive coding in the image generation process . In this paper , we propose an image generation model where new pixels are generated by modeling the differences between new pixels and pre-existing ones . In contrast with existing approaches that tackle the challenging problem of learning to generate each pixel in terms of their absolute values ( van den Oord et al. , 2016b ) , we model gradually shifting pixel intensities in relative terms . From the modeling perspective , we propose a copy and adjustment mechanism , where new pixels are predicted by selecting an existing pixel , and adjusting its sub-pixel values to generate a new pixel . We show that new generation methodology generalizes better for multiple datasets in unconditional image generation and image translation tasks , namely colorization ( Cheng et al. , 2016 ; Zhang et al. , 2016 ; Guadarrama et al. , 2017 ; Iizuka et al. , 2016 ; Guadarrama et al. , 2017 ) and super-resolution ( Parmar et al. , 2018 ) . Additionally , we show that our mechanism can be adapted to copy and adjust pixels from the input image in image-to-image translation tasks . Finally , we show that the different generation mechanisms can be unified into a single loss function with tractable likelihood computation . The paper is organized as follows : Firstly , we describe our copy and adjustment model ( §2 ) . Then , we describe its application in image-to-image translation tasks ( §3 ) . Afterwards , we describe a mixture model that combines different forms of pixel prediction ( §4 ) . Then , we provide experimental results that support our claims ( §5 ) . Finally , we describe the positioning of our work with respect the previous research ( §6 ) and proceed to the conclusions ( §7 ) . 2 RELATIVE PIXEL PREDICTION . PixelCNN ( van den Oord et al. , 2016a ) models the generation of an image x = ( x1 , x2 , . . . , xn2 ) with width and height n as a probability density function p ( x ) that is factorized into an autoregressive prediction of pixels as follows1 : p ( x ) = n2∏ i=1 p ( xi | xk < i ) , ( 1 ) where xi , which consists of C subpixel channels ( xi,1 , xi,2 , . . . , xi , C ) , is conditioned on k previously generated pixels . Each pixel prediction p ( xi | xk < i ) is further factorized by the prediction of its C subpixel channels , p ( xi | xk < i ) = C∏ c=1 p ( xi , c | xi , m < c , xk < i ) . ( 2 ) Similarly , each new sub-pixel is conditioned on the previously k generated pixels and them generated sub-pixels . The probability density function of each sub-pixel xi , c is modelled by bucketing the range of continuous values in the range [ 0 , 1 ] into 256 evenly distributed classes ( van den Oord et al. , 2016a ) , which is modeled as a categorical distribution over Va = [ 0 , 255 ] . More formally , given a hidden state hi , c−1 , obtained by composing the history of generated sub-pixels , the following pixel is generated as follows : p ( xi , c | xi , m < c , xk < i ) = softmaxVa ( Wshi , c−1 ) . ( 3 ) Our work changes the probability density function in Equation 2 , which computes the subpixel values directly , with one that computes these values relative to previously generated pixels . More formally , rather than predicting xi , c , we compute the distance ∆ xi , c xr , c = xi , c − xr , c , where xr denotes the previously generated pixel at position r. Under this formulation , we rewrite Equation 2 as follows : 1Sequences of pixels representing either an image of part of an image are written as x , a vector representing a multi-channel pixel is written x , and scalars are written x . The ith scalar element of vector x is xi , and the jth vector element of a sequence x is xj . p ( xi | xk < i ) = p ( xr , ∆xi , ∗xr , c | xk < i ) = p ( xr | xk < i ) C∏ c=1 p ( ∆xi , cxr , c | xr , xi , m < c , xk < i ) . ( 4 ) Illustrated in Figure 2 , our relative pixel prediction can be interpreted as selecting a pixel to copy under the probability mass function p ( xr | xk < i ) and then predicting the adjustment needed to add to that pixel to obtain xi modelled by p ( ∆ xi , c xr , c | xr , xi , m < c , xk < i ) . We name the former the copy mechanism and the latter the adjustment mechanism . 2.1 COPY MECHANISM . The concept of learning to reference a pre-existing pixel under a given distribution p ( xr | xki ) is closely related to the copy mechanism frequently used in natural language generation models ( Ling et al. , 2016 ; Gu et al. , 2016 ; Merity et al. , 2016 ) , where observed words , either from the input or previously generated output , can be copied to generate new outputs using a pointer network ( Vinyals et al. , 2015 ) . Pointer networks define a probability distribution over a discrete space of units , in our case previously generated pixels , and compute p ( xr | xk < i ) as follows : ur′ = v T tanh ( Washi,0 + W a rhr′+1,0 + W a xxr′ ) ( 5 ) p ( xr | xk < i ) = softmaxUi ( u ) , ( 6 ) where ur′ ( for each r′ in Ui ) computes the affinity between current state hi,0 before any sub-pixels at position i are generated and state of a previously generated pixel hr′+1,0 after all pixels in candidate source position r′ have been generated with sub-pixel values xr . Parameters Was , W a r , W a x and v are used to model this function . The probability of choosing a given pixel xr is computed as a softmax over the affinities of all pixels that can be selected Ui . While applications of pointer networks in natural language processing can be applied to whole sentences , which contain perhaps 40–200 words , the large number of pixels in images ( e.g . 4096 in 64×64 images ) makes this practice prohibitive . Following ( Parmar et al. , 2018 ) , we restrict this choice to a predefined set of neighbouring pixels . Finally , near the edges of images , some positions in Ui will lie outside of the image . To address this case , we always can copy the pixel with value xnull = [ 0 , 0 , 0 ] and learn a special state hnull . This is needed for the top and left edges of the image , where no pixels have been previously generated . 2.2 ADJUSTMENT MECHANISM . The adjustment mechanism aims to learn a continuous variable ∆ba which measures the absolute distance between two sub-pixels a and b . In order to keep log-likelihood computations consistent with the buckets defined in previous work ( van den Oord et al. , 2016a ) , we compute the distances over the discretized pixel values . Thus , we model the distance function as a categorical distribution over integers in the range Vr = [ −255 , 255 ] . This quantifies the number of buckets that need to be adjusted with relation to the pixel to be copied . The computation of the distance distribution is performed as follows : a = Wpshi , c−1 + W p rhr , C + W p xxr ( 7 ) p ( ∆xi , cxr , c | xr , xi , m < c , xk < i ) = softmaxVr ( a ) . ( 8 ) Thus , the softmax that computes the required adjustment depends on the current hidden state hi , c−1 , and the state and the pixel to copy from hr , C and its values xr . It is worth mentioning that this softmax can be replaced by any mechanism that defines a categorical distribution , such as that described in ( Salimans et al. , 2017 ) . While the outcome of the relative predictor consists of a value in the interval [ −255 , 255 ] , for a given copied pixel , the number of valid adjustment values that situate the intensity within the valid range [ 0 , 255 ] remains as 256 . For instance , if the copied pixel ’ s value is 100 , the range of possible adjustment values is [ −100 , 155 ] . While the model can learn to avoid such mistakes from the data , any probability mass assigned to invalid adjustment values is penalized in terms of perplexity . Thereby , we ensure that the model avoids such errors by limiting softmax defined in Equation 7 to the range of valid values . 2.3 MARGINALIZATION OVER RELATIVE PIXELS . For likelihood computation , the choice of the relative pixel xr is unobserved and must be inferred during training . As we define a small range of possible pixels for xr , limited to neighbouring pixels , the marginal likelihood over the latent domain Ui is tractable and can be defined as follows : p ( xi | xk < i ) = ∑ xr∈Ui p ( xr , ∆ xi,1 xr , c , . . . , ∆ xi , C xr , c | xk < i ) ∑ xr∈Ui p ( xr | xk < i ) C∏ c=1 p ( ∆xi , cxr , c | xr , xi , m < c , xk < i ) For generation , we first sample the pixel to copy xr from the distribution p ( xr | xk < i ) , then we sample an adjustment from the distributions p ( ∆xi , cxr , c | xr , xi , m < c , xk < i ) for each sub-pixel value c , thereby generating xi .

Summary: In this paper the authors present a new way to use autoregressive modeling to generate images pixel by pixel where each pixel is generated by modeling the difference between the current pixel value and  the preexistent ones. In order to achieve that, the authors propose a copy and adjustment mechanism that select an existing pixel, and then adjust its sub-pixel (channel values) to generate the new pixel. The proposed model is demonstrated with a suite of experiments in classic image generation benchmark. The authors also demonstrate the use of their technique in Image to Image translation.