Relative Pixel Prediction For Autoregressive Image Generation

1 INTRODUCTION . It is has long been appreciated in the field of image compression ( Harrison , 1952 ; Haskell & Netravali , 1997 ) that adjacent pixels in natural images tend to share similar colors and intensities . This is evidenced in Figure 1 , where we plot the marginal distribution of absolute sub-pixel ( i.e. , color channel ) intensities and the difference between the sub-pixel intensities to the corresponding sub-pixel immediately to its left . While the absolute pixel values tend to be distributed non-trivially across the whole range of possible sub-pixel values [ 0 , 255 ] with peaks around the values of 100 and 255 , the relative sub-pixel values closely resembles a Laplacian distribution with the mean at 0 as most pixels do not differ greatly from their neighbouring pixels ( Takamura , 1996 ) . This suggests that image generation process could mostly be done by predicting the relative distance between existing and new pixels , and absolute pixel prediction is only needed to generate more abrupt transitions , such as boundaries between two different objects . In predictive coding compression models , this fact is leveraged to achieve increased compression rates by learning to predict pixels as a delta between pixels rather than predicting their absolute values ( al Mahmood & Al-Rubaye , 2014 ) . Autoregressive models are one of the main forces driving research in image generation van den Oord et al . ( 2016a ) ; Salimans et al . ( 2017 ) ; Parmar et al . ( 2018 ) ; Chen et al . ( 2018 ) ; Child et al . ( 2019 ) . In contrast to other popular image models based on adversarial methods ( Goodfellow et al. , 2014 ; Radford et al. , 2015 ) or that use latent variables ( van den Oord et al. , 2017 ) , autoregressive models have a tractable likelihood function that decomposes image generation into a sequence of conditionally dependent pixel predictions . This is a desirable property that provides the grounds for applying the same principles of predictive coding in the image generation process . In this paper , we propose an image generation model where new pixels are generated by modeling the differences between new pixels and pre-existing ones . In contrast with existing approaches that tackle the challenging problem of learning to generate each pixel in terms of their absolute values ( van den Oord et al. , 2016b ) , we model gradually shifting pixel intensities in relative terms . From the modeling perspective , we propose a copy and adjustment mechanism , where new pixels are predicted by selecting an existing pixel , and adjusting its sub-pixel values to generate a new pixel . We show that new generation methodology generalizes better for multiple datasets in unconditional image generation and image translation tasks , namely colorization ( Cheng et al. , 2016 ; Zhang et al. , 2016 ; Guadarrama et al. , 2017 ; Iizuka et al. , 2016 ; Guadarrama et al. , 2017 ) and super-resolution ( Parmar et al. , 2018 ) . Additionally , we show that our mechanism can be adapted to copy and adjust pixels from the input image in image-to-image translation tasks . Finally , we show that the different generation mechanisms can be unified into a single loss function with tractable likelihood computation . The paper is organized as follows : Firstly , we describe our copy and adjustment model ( §2 ) . Then , we describe its application in image-to-image translation tasks ( §3 ) . Afterwards , we describe a mixture model that combines different forms of pixel prediction ( §4 ) . Then , we provide experimental results that support our claims ( §5 ) . Finally , we describe the positioning of our work with respect the previous research ( §6 ) and proceed to the conclusions ( §7 ) . 2 RELATIVE PIXEL PREDICTION . PixelCNN ( van den Oord et al. , 2016a ) models the generation of an image x = ( x1 , x2 , . . . , xn2 ) with width and height n as a probability density function p ( x ) that is factorized into an autoregressive prediction of pixels as follows1 : p ( x ) = n2∏ i=1 p ( xi | xk < i ) , ( 1 ) where xi , which consists of C subpixel channels ( xi,1 , xi,2 , . . . , xi , C ) , is conditioned on k previously generated pixels . Each pixel prediction p ( xi | xk < i ) is further factorized by the prediction of its C subpixel channels , p ( xi | xk < i ) = C∏ c=1 p ( xi , c | xi , m < c , xk < i ) . ( 2 ) Similarly , each new sub-pixel is conditioned on the previously k generated pixels and them generated sub-pixels . The probability density function of each sub-pixel xi , c is modelled by bucketing the range of continuous values in the range [ 0 , 1 ] into 256 evenly distributed classes ( van den Oord et al. , 2016a ) , which is modeled as a categorical distribution over Va = [ 0 , 255 ] . More formally , given a hidden state hi , c−1 , obtained by composing the history of generated sub-pixels , the following pixel is generated as follows : p ( xi , c | xi , m < c , xk < i ) = softmaxVa ( Wshi , c−1 ) . ( 3 ) Our work changes the probability density function in Equation 2 , which computes the subpixel values directly , with one that computes these values relative to previously generated pixels . More formally , rather than predicting xi , c , we compute the distance ∆ xi , c xr , c = xi , c − xr , c , where xr denotes the previously generated pixel at position r. Under this formulation , we rewrite Equation 2 as follows : 1Sequences of pixels representing either an image of part of an image are written as x , a vector representing a multi-channel pixel is written x , and scalars are written x . The ith scalar element of vector x is xi , and the jth vector element of a sequence x is xj . p ( xi | xk < i ) = p ( xr , ∆xi , ∗xr , c | xk < i ) = p ( xr | xk < i ) C∏ c=1 p ( ∆xi , cxr , c | xr , xi , m < c , xk < i ) . ( 4 ) Illustrated in Figure 2 , our relative pixel prediction can be interpreted as selecting a pixel to copy under the probability mass function p ( xr | xk < i ) and then predicting the adjustment needed to add to that pixel to obtain xi modelled by p ( ∆ xi , c xr , c | xr , xi , m < c , xk < i ) . We name the former the copy mechanism and the latter the adjustment mechanism . 2.1 COPY MECHANISM . The concept of learning to reference a pre-existing pixel under a given distribution p ( xr | xki ) is closely related to the copy mechanism frequently used in natural language generation models ( Ling et al. , 2016 ; Gu et al. , 2016 ; Merity et al. , 2016 ) , where observed words , either from the input or previously generated output , can be copied to generate new outputs using a pointer network ( Vinyals et al. , 2015 ) . Pointer networks define a probability distribution over a discrete space of units , in our case previously generated pixels , and compute p ( xr | xk < i ) as follows : ur′ = v T tanh ( Washi,0 + W a rhr′+1,0 + W a xxr′ ) ( 5 ) p ( xr | xk < i ) = softmaxUi ( u ) , ( 6 ) where ur′ ( for each r′ in Ui ) computes the affinity between current state hi,0 before any sub-pixels at position i are generated and state of a previously generated pixel hr′+1,0 after all pixels in candidate source position r′ have been generated with sub-pixel values xr . Parameters Was , W a r , W a x and v are used to model this function . The probability of choosing a given pixel xr is computed as a softmax over the affinities of all pixels that can be selected Ui . While applications of pointer networks in natural language processing can be applied to whole sentences , which contain perhaps 40–200 words , the large number of pixels in images ( e.g . 4096 in 64×64 images ) makes this practice prohibitive . Following ( Parmar et al. , 2018 ) , we restrict this choice to a predefined set of neighbouring pixels . Finally , near the edges of images , some positions in Ui will lie outside of the image . To address this case , we always can copy the pixel with value xnull = [ 0 , 0 , 0 ] and learn a special state hnull . This is needed for the top and left edges of the image , where no pixels have been previously generated . 2.2 ADJUSTMENT MECHANISM . The adjustment mechanism aims to learn a continuous variable ∆ba which measures the absolute distance between two sub-pixels a and b . In order to keep log-likelihood computations consistent with the buckets defined in previous work ( van den Oord et al. , 2016a ) , we compute the distances over the discretized pixel values . Thus , we model the distance function as a categorical distribution over integers in the range Vr = [ −255 , 255 ] . This quantifies the number of buckets that need to be adjusted with relation to the pixel to be copied . The computation of the distance distribution is performed as follows : a = Wpshi , c−1 + W p rhr , C + W p xxr ( 7 ) p ( ∆xi , cxr , c | xr , xi , m < c , xk < i ) = softmaxVr ( a ) . ( 8 ) Thus , the softmax that computes the required adjustment depends on the current hidden state hi , c−1 , and the state and the pixel to copy from hr , C and its values xr . It is worth mentioning that this softmax can be replaced by any mechanism that defines a categorical distribution , such as that described in ( Salimans et al. , 2017 ) . While the outcome of the relative predictor consists of a value in the interval [ −255 , 255 ] , for a given copied pixel , the number of valid adjustment values that situate the intensity within the valid range [ 0 , 255 ] remains as 256 . For instance , if the copied pixel ’ s value is 100 , the range of possible adjustment values is [ −100 , 155 ] . While the model can learn to avoid such mistakes from the data , any probability mass assigned to invalid adjustment values is penalized in terms of perplexity . Thereby , we ensure that the model avoids such errors by limiting softmax defined in Equation 7 to the range of valid values . 2.3 MARGINALIZATION OVER RELATIVE PIXELS . For likelihood computation , the choice of the relative pixel xr is unobserved and must be inferred during training . As we define a small range of possible pixels for xr , limited to neighbouring pixels , the marginal likelihood over the latent domain Ui is tractable and can be defined as follows : p ( xi | xk < i ) = ∑ xr∈Ui p ( xr , ∆ xi,1 xr , c , . . . , ∆ xi , C xr , c | xk < i ) ∑ xr∈Ui p ( xr | xk < i ) C∏ c=1 p ( ∆xi , cxr , c | xr , xi , m < c , xk < i ) For generation , we first sample the pixel to copy xr from the distribution p ( xr | xk < i ) , then we sample an adjustment from the distributions p ( ∆xi , cxr , c | xr , xi , m < c , xk < i ) for each sub-pixel value c , thereby generating xi .

Summary: The paper bases its methodology on well known developments in image analysis/synthesis about similarity of pixel values in adjacent locations. Many techniques have been used for modelling this similarity, including predictive models, cliques and graphs. The paper uses a simple autoregressive model for generating pixel values based on the values of previously processed pixels, estimating the differences between these neighboring pixel values.