Relative Pixel Prediction For Autoregressive Image Generation
Authors: Wang Ling
Chris Dyer
Lei Yu
Lingpeng Kong
Dani Yogatama
Susannah Young
Venue:  (2020)
Keywords: Image Generation
Autoregressive

Abstract: In natural images, transitions between adjacent pixels tend to be smooth and gradual, a fact that has long been exploited in image compression models based on predictive coding. In contrast, existing neural autoregressive image generation models predict the absolute pixel intensities at each position, which is a more challenging problem. In this paper, we propose to predict pixels relatively, by predicting new pixels relative to previously generated pixels (or pixels from the conditioning context, when available). We show that this form of prediction fare favorably to its absolute counterpart when used independently, but their coordination under an unified probabilistic model yields optimal performance, as the model learns to predict sharp transitions using the absolute predictor, while generating smooth transitions using the relative predictor.
Experiments on multiple benchmarks for unconditional image generation, image colorization, and super-resolution indicate that our presented mechanism leads to improvements in terms of likelihood compared to the absolute prediction counterparts. 

Under review as a conference paper at ICLR 2020
RELATIVE PIXEL PREDICTION FOR AUTOREGRESSIVE IMAGE GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
In natural images, transitions between adjacent pixels tend to be smooth and gradual, fact that has long been exploited in image compression models based on predictive coding. In contrast, existing neural autoregressive image generation models predict the absolute pixel intensities at each position, which a more challenging problem. In this paper, we propose to predict pixels relatively, by predicting new pixels relative to previously generated pixels (or pixels from the conditioning context, when available). We show that this form of prediction fare favorably its absolute counterpart when used independently, but their coordination under an unified probabilistic model yields optimal performance, the model learns to predict sharp transitions using the absolute predictor, while generating smooth transitions using the relative predictor. Experiments multiple benchmarks for unconditional image generation, image colorization, and super-resolution indicate that our presented mechanism leads to improvements in terms of likelihood compared the absolute prediction counterparts.
INTRODUCTION
It is has long been appreciated in the field of image compression (Harrison) 1952; Haskell & Netravali (997) that adjacent pixels in natural images tend share similar colors and intensities. This is evidenced in Figure [ where we plot the marginal distribution of absolute sub uubpixee (i.e., color channel) intensities and the difference between the sub-pixel intensities to the corresponding sub-pixel immediately to its left. While the absolute pixel values tend to be distributed non-trivially across the whole range of possible sub-pixel values [0, 255) with peaks around the values of 100 and 255, the relative sub-pixet values closely resembles Laplacian distribution with the mean at 0 as most pixels do not differ greatly from their neighbouring pixels (Takamura) 1996). This suggests that image generation process could mostly be done by predicting the relative distance between existing and new pixels, and absolute pixel prediction is only needed to generate more abrupt transitions, such as boundaries between two different objects. In predictive coding compression models, this fact is leveraged to achieve increased compression rates by learning to predict pixels as a delta between pixels rather than predicting their absolute values (al Mahmood & Al-Rubaye 2014).
Autoregressive models are one of the main forces driving research in image generation van den Oord al. (2016a): Salimans et al. (2017); Parmar et al. (2018); Chen et al. (2018); Child al. (2019). In contrast to other popular image models based on adversarial methods (Goodfellow et al. 2014 Radford al. 2015) or that use latent variables (van den Oord al. 2017), autoregressive models have tractable likelihood function that decomposes image generation into a sequence of conditionally dependent pixel predictions. This is desirable property that provides the grounds for applying the same principles of predictive coding in the image generation process.
In this paper, we propose an image generation model where new pixels are generated by modeling the differences between new pixels and pre-existing ones. In contrast with existing approaches that tackle the challenging problem of learning to generate each pixel in terms of their absolute values (van den Oord et al. 2016b), we model gradually shifting pixel intensities in relative terms. From the modeling perspective, we propose a copy and adjustment mechanism, where new pixels are predicted by selecting existing pixel, and adjusting its sub-pixel values to generate that new generation methodology generalizes better for multiple datasets in unconditional image generation and image translation tasks, namely colorization (Cheng al. 2016 Zhang al. 2016
pixel. We show

---

Under review as a conference paper at ICLR 2020
10g
100
63
127
191
255
255
Marginal distribution of absolute and relative (to the left pixel) pixel intensities in the training set of CIFAR-10.
Figure 1:
Guadarrama et al. 2017 lizuka et al. 2016 Guadarrama et al. 2017) and super-resolution (Parmar et al. 2018). Additionally, we show that our mechanism can be adapted to copy and adjust pixels from the input image in image-to-image translation tasks. Finally, we show that the different generation mechanisms can be unified into single loss function with tractable likelihood computation.
The paper is organized as follows: Firstly, we describe copy and adjustment model (2)) Then, we describe its application image-to-image translation tasks (33) Afterwards, we describe mixture model that combines different forms of pixel prediction ($4) Then, we provide experimental
results that support our claims (5.). Finally, we describe the positioning of our work with respect the previous research (16) and proceed the conclusions
2 RELATIVE PIXEL PREDICTION
PixelCNN vann den Oord et al. 2016a) models the generation of an image (x1, X2 x,,) with width and height n as a probability density function p(x) that is factorized into an autoregressive prediction of pixels follow!
?
p(x)
II p(x: Bk<s))
where x,, which consists of C subpixel channels (Ir,, T1.2, ..., TF,C). is conditioned on previously generated pixels. Each pixel prediction p(æ;; (E is further factorized by the prediction of its C subpixel channels,
p(x:
II p( Ximaces Tkc)).
Similarly, each new sub-pixel is conditioned on the previously & generated pixels and the m generated sub-pixels. The probability density function of each sub-pixel hgec is modelled by bucketing the range of continuous values the range [0, 1] into 256 evenly distributed classes (van den Oord et al.) 2016a), which is modeled as categorical distribution over V, [0, 255]. More formally, given hidden state h obtained by composing the history of generated sub-pixels, the following pixel is generated as follows:
p(Zi,e Xi.m Tbess) softmaxy, (W (Wh...-.):
(3)
Our work changes the probability density function in Equation 2 which computes the subpixel values directly, with one that computes these values relative to previously generated pixels. More formally, rather than predicting Ti,c, we compute the distance A Ti.c Fr.c, where x, denotes the previously generated pixel at position r. Under this formulation, we rewrite Equation 2 follows:
'Squuencs of pixels representing either an image of part of an image are written as æ, a vector representing a muuti--cannell pixel is written x, and scalars are written The ath scalar element of vector is 3a and the jth vector element of sequence Xj-
2

---

Under review as a conference paper at ICLR 2020
e pl A
p(x4 @xci)) = p(x,, A Ane
Fkecs) = p(x, Dbcc)
II
si,o Ric
X,, Xp.Dimmes FRecs) (4)
c=1
Illustrated in Figure our relative pixel prediction can be interpreted as selecting pixel to copy under the probability mass function p(x, Tk<e) and then predicting the adjustment needed to add to that pixel to obtain x, modelled by p( p(Ar, Mice Fk<i). We name the former the copy mechanism and the latter the adjustment mechanism.
U:
Copy
Adjustment
0.1
0.4 0.3
12,00
0.2
Figure 2: Illustration of the copy and adjustment mechanism. The copy mechanism selects previously generated pixel and the adjustment mechanism modifies the pixel to generate the next pixel.
2.1
COPY MECHAANISM
The concept of learning to reference a pre-existing pixel under given distribution p(x, Xx. is closely related to the copy mechanism frequently used natural language generation models (Ling et al.
2016 Gu et al. 2016, Merity et al. 2016), where observed words, either from the input or previously generated output, can be copied to generate new outputs using a pointer network (Vinyals)
ct al. 2015). Pointer networks define probability distribution over discrete space of units, in
case previously generated pixels, and compute p(x, Økes) as follows:
Uy Va tanhh W W p(x, @ber) softmaxy, (u);
(5)
(6)
where u, (for each r' in U.) computes the affinity between current state ha before any sub-pixels at position i are generated and state of previously generated pixel h, after all pixels in candidate source position r' have been generated with sub-pixel values x,.. Parameters Wa Wa W and are used to model this function.
The probability of choosing given pixel x,- is computed softmax over the affinities of all pixels that can be selected U,. While applications pointer networks in natural language processing can be applied to whole sentences, which contain perhaps 40-200 words, the large number of pixels in images (e.g. 4096 in 64x64 64 images) makes this practice prohibitive. Following (Parmar et al. 2018). restrict this choice predefined set of neighbouring pixels.

---

Under review as a conference paper at ICLR 2020
Finally, near the edges of images, positions in U, will lie outside of the image. To address this case, we always can copy the pixel with value Xaull = [0, 0, 0] and learn a special state haull- This is needed for the top and left edges of the image, where no pixels have been previously generated.
2.2 ADJUSTMENT MECHANISM
The adjustment mechanism aims to learn continuous variable A which measures the absolute distance between two sub-pixels a and b. In order to keep log-likelihood computations consistent with the buckets defined in previous work (van den Oord et al.) 2016a), over the discretized pixel values. Thus, we model the distance function as a categorical distribution integers in the range V, [-255, 255]. This quantifies the number of buckets that need to be adjusted with relation to the pixel to be copied. The computation of the distance distribution is performed as follows:
compute the distances
Whh, c- + W + WTx, X,, Xi,mces Tkee) = softmaxy, (a).
pl p(Aie Ri.e Rr.e.
(8)
Thus, the softmax that computes the required adjustment depends on the current hidden state ha and the state and the pixel to copy from h,,c and its values x,- is worth mentioning that this softmax can be replaced by any mechanism that defines a categorical distribution, such as that described in (Salimans et al., 2017).
While the outcome of the relative predictor consists of value in the interval [-255, 255], for given copied pixel, the number of valid adjustment values that situate the intensity within the valid range [0, 255] remains 256. For instance, if the copied pixel's value is 100, the range of possible adjustment values [-100, 155]. While the model can learn to avoid such mistakes from the data, any probability mass assigned to invalid adjustment values is penalized in terms of perplexity. Thereby, we ensure that the model avoids such errors by limiting softmax defined Equation 7 to the range of valid values.
2.3 MARGINALIZATION OVER RELATIVE PIXELS
For likelihood computation, the choice of the relative pixel x, is unobserved and must be inferred during training. As we define small range of possible pixels for Ty, limited neighbouring pixels, the marginal likelihood over the latent domain U, is tractable and can be defined as follows:
p(x
Σ X.EUU
p(x,,
A
Dbcs)
Σ X
p(x,
@ies)
II Cml
p(
The
Xxy Xigmoces (kei)
For generation, we first sample the pixel copy x, from the distribution p(x, @xee)), ) then we sample an adjustment from the distributions p( A di. thereby generating x4.
Fr. Xr, mimee Tx)cs) for each sub-pixel value
3 HHAG-TTOIMMAGE TRANSLATION
the image generation process is conditioned on an input image y. the copy mechanism can also be applied copy pixels from the input image. This desirable in tasks, such image colorization and super-resolution, where the input image is structurally similar to the output image and we hypothesize that learning the adjustment in such tasks generalizes better than learning generate from seratch.
PixeICNN can be adjusted for image translation by conditioning on a fully visible input image More formally, each pixel p(x; # y) is now generated as follows:
p(x; Dbsss v) = softmaxy, (W,h W. W.g:)
4

---

Under review as a conference paper at ICLR 2020
where gi is the hidden state of the imput image y pixel i. This differs from the output image hidden states mim in that gi has visibility of the whole imput image.
Relative pixel prediction over the input image can be embedded by reformulating Equations 5 and as follows:
Up' vz tanh Wi + Wmg.r + Wyrr")
(9)
x(x, Tk v) softmax B3 (u)
(10)
= WTh W + Wyy, X,, Xi,m Tzci, = softmaxy, (a), Mise
(11)
p(4
(12)
where we replace the output pixels and their hidden states of the output pixels with those of the input, namely y and g. Secondly, we define B, as the set of pixels the input image y that are available to be copied from position i. For simplicity, we also assume that y and æ have the same dimensions. For image colorization, where the input image has only one channel, we simply copy the same channel three times. For super-resolution, a heuristic method (e.g. bilinear interpolation). These do not correspond to optimal architecture choices for these tasks, but still allow the fair evaluation between the absolute and relative predictors.
ensure this by first increasing to the desired dimensions using
UNIFIED PREDICTOR
The copy and adjust mechanism is self-sufficien for generating any image, and benefits from denser distribution when neighbouring pixels share similar sub-pixel values. However, when large color perturbations are observed and no correlations between neighbouring pixels can be found, using this mechanism is intuitively disadvantageous, since the range of differences hetween two sub-pixel values is larger than the range of sub-pixel values (511 vs. 256). Furthermore, in image-to-image tasks it would also be desirable to combine both forms of relative prediction (e.g. copying from the input and output). Our solution supporting both absolute and relative generation mechanisms is using latent predictors (Ling et al. (0166) We define our set of predictors P [Pabx. Pirel. P Parel), where Pats denotes the absolute pixel predictor computed according to Equation 2 and Piree and Powee denote the relative pixel predictors defined in Equation 4 and I1
At each generation step i, we define
the predictor probability p(r) (Kses) as follows:
p( x = softmax Paa (W,,,),
(13)
where W, defines the parameters used by the gating system that switches between the absolute and relative predictors.
is also beneficial to prune marginally less frequent adjustment values in the relative predictor by removing them from the softmax in Equations 7 as these generally correspond to sharp transitions between pixels, which are better handled by the absolute predictor. We do this by pruning the domain of the softmax over the interval [-255, 255] to the set of the & most frequently observed adjustment values the training data, where is a hyperparameter tuned on the validation set. This means that for pruned adjustment values, the relative predictor will predict the pixel with probability 0, which acceptable the unified predictor, as the absolute predictor would present a valid probabilistic interpretation for such pixels.
5 EXPERIMENTS
5.1
SETUP
Our experiments were performed on existing benchmarks for image generation, namely, CIFAR-
10 (Krizhevsky 2009) and downsampled ImageNet (Deng al. 2009 Chrabaszcz et al. 2017).
For unconditional image generation, hidden states h are obtained using gated convolutional lay- ers (yan den Oord et al. 20) More concretely, after discrediting the input image æ, apply convolutional layer with kernel size 7 and hidden size H₁. Then, we apply an additional D layers with kernel size 5 and hidden size Hi, with the exception of the final layer which has the hidden size
5

---

Under review as a conference paper at ICLR 2020
Table 1: Quantitative results using relative pixel prediction. Each cell reports the negative log- likelihoods in bits/dim on CIFAR-10's test set and ImageNet's validation set for unconditional image generation (U), colorization (C) and super-resolution (S). Row 0 is the baseline. Rows 1 and describe the results obtained using only the input and output predictors, respectively. Rows 3 and combines them with the absolute predictor. Row 5 denotes the combination of the input and output predictors.
CIFAR- 10 32x32
IMAGENET 32x32
IMAGENET 64
U
S
C
U
S
0
ABSOLUTE
3.02
1.25
2.87
3.83
2.17
3.74
3.55
2.02
3,49
1
RELATIVE(O)
2.99
0.99
2.82
3.80
2.08
3.71
3.53
1.63
3.41
2
RELATIVE(1)
1.21
2.85
2.16
3.73
1.99
3.46
3
ABSOLUTE + RELATIVE(O)
3.00
1.03
2.83
3.79
2.11
3.69
3.52
1.69
3.40
4
ABSOLUTE RELATIVE())
1.20
2.84
2.14
3.72
1.98
3,46
5
RELATIVE(I+O)
0.97
2.81
2.06
3.70
1.63
3,38
of Hg. Furthermore, we add skip connection (Ronneberger et 2015), between each intermediate layer to the final layer. This final layer corresponds to the hidden states h. For CIFAR-10, we set D 15, H1 64 and H2 512, and train with batch size of 32. For ImageNet, set D 20, H1 = 256 and H2 = 2048 and train with a batch size 128.
For image-to-imaae translation, we apply the operations to both and y to obtain and g, re- spectively. However, as is fully observed, no mask is applied in the convolutions to obtain For CIFAR-IO, set D 15, 11, 128 and H₂ 512, and train with a batch size of 32, For ImageNet, we set D 20, H1 256 and H2 = 2048 and train with batch size of 64. For colorization, we remove the colorization of the original image. For super-resolution we follow the approach applied in (Dahl et al. 2017), and reduce resolution of the image to 8. Evaluation is performed in terms negative log- log-likelihood measured in bits/dim
As for the prediction mechanisms, we compare the baseline that relies solely absolute predictors against the best setups found on CIFAR-I0 from our ablation study (Appendix[A) (A), which carried out on the validation set. For unconditional image generation, copy from the pixel to the left and top of the generated pixel. We report results, when only relative prediction is used and when both absolute and relative predictors are used. Here, the pruning threshold set to 20%, which also chosen based on the results the ablation test. For image-to to-mage translation, the same setup is used when only relative predictors that copy from the output is used. When we consider the input image as another source that can be copied from, we define a 3 x 3 grid around the position of the output pixel, and any corresponding pixel in the input image that falls within that grid can be selected as target to be copied from.
relative predictor that can
5.2 QUANTITATIVE RESULTS
reepotsss the results on the test set of CIFAR-10 and validation sets of ImageNet. Our baseline implementation of PixeICNN (row 0) achieves negative log-likelihood of 3.02 bits/dim. From rows and 2, we observe that both relative predictors yield improvements over the baseline. Overall, combining each relative form of prediction with the absolute predictor (rows 3 and 4) tends to work well for all tasks except for colorization, where combining the output relative predictor and absolute predictors tend yield lower results. Finally, combining the input and output relative predictors yields the optimal overall results (row 5). We do not illustrate results when all predictors are combined, the presence of the input predictor and the absolute predictor causes the latent predictor network (Ling et al.) 2016) to ignore the absolute predictor, and thereby, results are identical that in row
We acknowledge that extensive work has been conducted the tasks of super-resolution (edig et al. 2016) and colorization (Nazeri & Ng 2018), which are evaluated on different metrics, refraain from directly comparing model to the aforementioned work as our main motivation is to measure the effectiveness of the relative pixel prediction mechanism with respect copying pixels from the input.

---

Under review as a conference paper at ICLR 2020
Figure 3: 32x images generated by our model on CIFAR-IO. Columns correspond the input of the model, the sample obtained from the model, the sampled latent variables and the reference. Three colors are used for the latent variables: White absolute predictor; Blue - relative predictor from the left pixel and Green - relative predictor from the top pixel.
INPUT
SAMPLE
LATENT
ORIGINAL
S
U
5.3 LATENT VARIABLE ANALYSIS
For more in-depth understanding of the relative and absolute prediction choices, we provide samples and latent variable choices for CIFAR-10 10 validation set in Table Columns "Input", "Sample" and "Reference" denote the input observed by the model, the sample that it generates, and the expected result, respectively. In the column denoted as "latent", we illustrate the latent choices made by the model during the sampling process, where white pixels correspond to the absolute predictor, and blue and green pixels correspond the relative predictor that copies from the left and top pixels, respectively.
We observe that the majority of the images generated using the relative prediction mechanism from the left pixel, with top pixels only used when the left pixel is unavailable (c.g. left edge of the image) or vertically oriented objects (e.g. legs of the cat). However, there are some instances where images are predominantly predicted by copying from the top pixel. In simpler tasks, such as colorization, the model seems rarely use the absolute predictors applying only to generate the transitions between objects. In more advanced tasks, such super-resolution and unconditional image generation, relative predictors are often applied to generate continuous textures (e.g. sky and glass) whereas more complex objects such houses and trees are generated by combining both absolute and relative predictors.
5.4 IMPACT ON STATE- OF- ARE MODELS
While we show that relative pixel prediction can improve the quality of the predictions of the model presented in (van den Oord al. 2016a), state-of-the-art models can achieve significantly better results by incorporating multiple modeling innovations (Salimans et al. 2017] |armar et al. 2018 Chen et al. 2018). Thus, to investigate if the work proposed can be used to further augment state-of- the-art models. We build upon the model described in (Razavi al. 2019). which defines 30-layer neural network with transformer layers. We replace the discretized logistic mixture components with 32 components with relative pixel prediction that copies from the left and top of the desired pixel with, whose output is pruned with a 20% threshold. Finally, we also use outcome masking. Results illustrated in Table where
that our model fares favourably with state-of-the-
7

---

Under review as a conference paper at ICLR 2020
Table 2: Comparison between state-of-the- models in unconditional image generation on CIFAR-10.
Cells report log- seglikelihood scores measured in bits/dim.
CIFAR-
DRAW
GREGOR ET AL. 2015
4.10
GROW KINGMA DHARIWAL 2018
3.35
MINTNET SONG ET AL. 2019)
3.32
PIXELCNN (VAN DEN OORD AL.
2016A)
3.03
PIXELCN++ (SALIMANS ET AL.
2017
2.92
TRANSFORMER (PARMAR ET AL. 2018)
2.90
PIXEL-SNAIL (CHEN ET
2018
2.85
DELTA VAE RAZAVI AL.
2019
2.83
RELATIVE PIXELCNN (THIS WORK
2.82
SPARSE TRANSFORMER (CHILD ET AL. 2019)
2.80
models in this task. Improvements are more marginal (from 2.83 2.82) but supplements the upward trend from experiments using the vanilla PixeICNN.
RELATED WORK
Copy mechanisms have been widely used in sequence-to-seqqenee models (Sutskever et al. 2014) in order to copy tokens from the input sequence (Ling et al., 2016 Gu et al. (2016), previously generated tokens (Merity et al. 2016), or from structured data, such as databases (Yang et al. 2016 Ahn et al.) 2016). The basic copy mechanism recalls pointer networks (Vinyals et al. 2015 but extends by allowing the copied element to be modified. A similar approach can be found (Ling et al. (017). where copied tokens be modified with multiple operations before generation. Image compression techniques encode deltas between pixels rather than absolute intensities (Harrison 1952), and even
natural visual systems code deltas rather than absolute magnitudes Rao & Ballard (199)..
major force driving progress on auto-regressive image generation models is the advancements of self-attention mechanisms (Parmar al. 2018 Child et al. 2019), which allows better modeling of long range dependencies within images. However, improving generalization by incorporating better prediction mechinism has also been explored in the past. The usage of discretized logistic mixture likelihoods (Salimans et al., 2017). where the independence between consecutive pixel intensities is alleviated here, which leads be better generalization is an example of such exploits. Our work is another instance of such lines of research, where we incorporate mechanism that exploits the fact that consecutive pixels tend to share similar values.
Another major driver of progress in image generation is the trend of adversarial image genera- tion (Goodfellow et al. 2014 Isola et al., 2016, Karras al.) 2017 Brock et al.] [0188] Rivière et al.) 2019 Wang & Huan 2019). While our model is not directly applicable this set of models, as proposed work operates on explicit probability estimates of pixel distributions rather than implicit ones, the concept of relative pixel prediction is likely to be beneficial to these models.
7 CONCLUSION
We presented an alternative parameterization for autoregressive models for image generation, where pixels are generated relative to pre-existing pixels. This is achieved by learning two probability mass functions over what pixel to copy and how the copied pixel must be adjusted to obtain the new pixel. We show that generating new pixels as deltas from previously generated neighbouring pixels has better generalization properties compared to existing methods that generate pixels in absolute terms. Finally, we propose a mechanism that combines the multiple types of predictions into mixture model, which allows exact inference training and decoding time. Experimental results show that proposed models achieve improvements to their absolute counterparts, and can achieve competitive results compared state-of-the-an ahr models in unconditional image generation.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Sungjin Ahn, Heeyoul Choi, Tanel Pärnamaa, and Yoshua Bengio. A neural knowledge language model. CoRR, abs/1608.0053 2016.
Haider al Mahmood and Zainab Al-Rubaye. Lossless image compression based on predictive coding and bit plane slicing. International Journal of Computer Applications, 93:1-6, 05 2014. doi: 10.5120/16176-2665
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. CoRR, abs/1809. 2018. URL http://arxiv.org/
1809. 11096
XI Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. PixeISNAIL: An improved autore- gressive generative model. Proceedings of the 35th International Conference on Machine Learn- ing. volume 80 of Proceedings of Machine Learning Research, pp. 864-872, Stockholmsmssan. Stockholm Sweden, 10-15 Jul 2018 8. PMLR.
Zezhou Cheng, Qingxiong Yang, and Bin Sheng. Deep colorization. CoRR, 0tbs/1605.00075.. 2016. URL http://arxiv. wtt.://............../.. 00075
Rewon Child, Scott Gray, Alec Radford, and llya Sutskever. Generating long sequences with sparse transformers. URL https://openai.com https::/openat..om/bllgg http://ogg/sspssssaaanaooooeeee 2019.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. downsampled variant of imagenet as an alternative to the CIFAR datasets. CoRR, abs/ 008819 19, 2017.
Ryan Dahl, Mohammad Norouzi, and Jonathon Shlens. Pixel recursive super resolution. CoRR. 702.
abs/1702.00788, 2017. URL http:::/axxx...rrrr///// /ttp///.0//170000373 00783
Deng, Dong, R. Socher, L.-. Li, Li, and L. Fei-Feii ImageNet: Large-Scale Hierarchical Image Database. In CVPRO9, 2009.
Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances Neural Information Process- ing Systems 27, pp. 2680.. Curran Associates, Inc.. 2014. URL http: http:://aperr: nips pdf
cC/ hcc/paper/5t0 5423 gene rat www.eeeeaatiieeeeeeeeeeeeeeeeeee aew.atiiaavvvvvaeaeeeeeaaaaaaa
Karol Gregor, Ivo Danihelka, Alex Graves, and Daan Wierstra. DRAW: A recurrent neural network for image generation. CoRR, abs/1502.04623, 2015. URL http::tttt:/ii...////////////att httpp://aaxii...rg/ab/////// owtp:///ggg/bb///bb/bo/
04623
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. Incorporating copying mechanism in sequence-to-sequeece learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1631-1640. Association for Computa- tional Linguistics, 2016. doi: 11886653111P11 11P116111544 1154.
Sergio Guadarrama, Ryan Dahl, David Bieber, Mohammad Norouzi, Jonathon Shlens, and Kevin Murphy. Pixcolor: Pixel recursive colorization. CoRR, abs/1705.07200 2017. URL http:
7/arxiv. wwwp///g//baaaaaaee
abaaabaao10000 /1705. 07208
C. W. Harrison. Experiments with linear prediction in television. Bell System Technical Journal, 31: 764-783, 1952.
Barry G. Haskell and Arun N. Netravali. Digital Pictures: Representation, Compression, and Standards. Perseus Publishing, 2nd edition, 1997. ISBN 030644917X.
Satoshi lizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Let there be color!: Joint und-to-ent learning of global and local image priors for automatic image colorization with simultancous classification. ACM Trans. Graph., 10.1 00.11445/29782242955774 URL htl:http:////
July 2016. ISSN 0730-0301. doi: or. /1 21019999222229999288 29259774

---

Under review as a conference paper at ICLR 2020
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-ime -i-eaae translation with conditional adversarial networks. CoRR, abs/1611.07004, 2016. URL
abs/1611. 07004
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. CoRR, 0bb//171010099, 2017. URL http://aaxxi
org/aba/1710. 10196
Diederik Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions, 2018.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Christian Ledig, Lucas Theis, Ferene Huszar, Jose Caballero, Andrew P. Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single image super-resolution using generative adversarial network. CoRR, abs/ (bs//600..oe (69900992222 2016. URL httpt:tttp//.....tt///
abs 609. 04802
Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiský, Fumin Wang, and Andrew Senior. Latent predictor networks for code generation. In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp.
Association for Computational Linguistics, 2016.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. CoRR, abs/1705.04146, 2017. URL http://arxiv/ wrtp
04146
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. models. CoRR, abs//100900884,, 2016.
Pointer sentinel mixture
Kamyar Nazeri and Eric Ng. Image colorization with generative adversarial networks. CoRR, 2018 URL
hwtp:/ab//11 05400
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In ICMI. volume 80 of JMLR Workshop and Conference Proceedings, pp. 4052-4061. JML..org. 2018.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. CoRR, M0./1511.064433 2015. URL http:
// www.xii.wgg.aaaaa/aaaaaaaaaaaaaa
06434
Rajesh P. N. Rao and Dana H. Ballard. Predictive coding in the visual cortex: a functional in- terpretation of some extra-classical receptive-fiele ffeld effects. Nature Neuroscience, 2(1):79-87, 1999.
Ali Razavi, Aaron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with delta-VAEs In International Conference on Learning Representations, 2019. URL https
Morgane Rivière, Olivier Teytaud, Jérémy Rapin, Yann LeCun, and Camille Couprie. Inspirational adversarial image generation. CoRR, abs/1906. 11661, 2019. URL attp://arxiv.oog/315
1906.1166
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. CoRR, abs/1505.00 005977 2015. URL http::p//aaxx..oog///a//////////
04597
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. Pixelenn++: A pixelenn implementation with discretized logistic mixture likelihood and other modifications. ICLR, 2017.
Yang Song, Chenlin Meng, and Stefano Ermon. Mintnet: Building invertible neural networks with masked convolutions. CoRR, Abs/1997.07945, 2019. URLLtttp://aaatt //atxiiv///vv...////a www.ivooorrr.oo.. orq/abs.........oooo... abs/1901
07945
10

---

Under review as a conference paper at ICLR 2020
Ilya Sutskever, Oriol Vinyals, and Quoe V. Le. Sequence sequence learning with neural networks. CoRR, abs/1409.3215, 2014.
Seishi Takamura. Efficient Lossless Still Image Coding Based on the Autoregressive Model. PhD thesis, University of Tokyo, 1996.
Aaron van den Oord, Nal Kalchbrenner, Lasse Espeholt, koray kavukcuoglu, Oriol Vinyals, and Alex Graves. Conditional image generation with pixelcnn decoders. In D. D. Lee, M. Sugiyama, U. V. Luxburg. I. Guyon, and R. Garnett (eds.), Advances Neural Information Processing Systems 29, pp. 4790-4798. Curran Associates, Inc., 2016a.
Aäron den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. CoRR, abs//6601.0679, 2016b.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu, Neural discrete representation learning. CoRR, abs/1711.0092 2017.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 2692-2700. Curran Associates, Inc., 2015.
Hanchao Wang and Jun Huan. AGAN: towards automated design of generative adversarial networks. CoRR, abs/1906. 1080, 2019.
1100 11080
Zichao Yang, Phil Blunsom, Chris Dyer, and Wang Ling. Reference-aware language models. CoRR, Mos/1611.01122 2016.
Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. CoRR, abs/1603.08511, 2016. URL http:ttt//aaatt www.vv.rrr//bbeee.. org..bs/6603. 08511

---

Under review as a conference paper at ICLR 2020
Table 3: Ablation study over different predictors. Each cell reports the negative log-likelihoods in bits/dim on the CIFAR-10 10 for unconditional image generation (U), colorization (C) and super- resolution (S). Row 0 the baseline. Rows 1-3 denote results using only relative predictors from the left pixel, top pixel and both, respectively. Rows 3 and allow the usage of 2x2 and 3 grid centered around the predicted pixel. Rows 6-8 combine the absolute predictor with the relative one, and rows 9-11 use pruned distance list for the relative predictors. Rows 12-14 use only the relative predictor that copies from the input pixel in the same position as the predicted pixel, and in 2x2 2 and 3x3 grids around that position. Rows 15-17 denote the combinations between the absolute and relative predictor from the input. Finally, we combine all the aforementioned predictors in rows 18-19.
ABSOLUTE
3.02
1.23
2.86
RELATIVE(O))
LEFT PIXEL
3.00
0.96
2.84
TOP PIXEL
3.00
0.98
2.83
LEFT+TOP PIXEL
2.99
1.00
2.80
2x2
3.00
1.06
2.82
3x3
3.01
1.05
2.83
(3)+ (0)
3.01
1.22
2.85
(4)+
3.02
1.26
(5)+(0)
3.02
1.29
2.86
(3P)+(0)
3.00
1.01
2.80
10
(4))+
3.00
1.00
2.81
(5P)
3.00
1.00
2.81
RELATIVE(1)
12
CENTER PIXEL
1.20
2.91
13
2x2
1.19
2.85
14
3x3
1.19
2.84
15
(12)+(00)
1.21
2.86
16
(13)+(0))
1.18
2.86
17
(14) (0)
1.17
2.85
RELATIVE(L4O)
(3P) (14)
0.94
2.78
19
(3P) + (14) + (0)
0.98
2.79
MASKED RELATIVE(O)
20
(3PM)
2.98
0.95
2.79
MASKED RELATIVE(L+O)
(3PM) + (14)
0.93
2.78
22
(3PM) + (14) (0)
0.96
2.79
A ABLATION STUDY
To test the effect of different components described in our work, we start by performing an ablation test over different setups on CIF and select the setups that are most promising to applied to the larger datasets. This study is reported on Table 3 Columns U, and S report the negative log- likelihood scores bits/dim for unconditional image generation, colorization and super-resolution, respectively. Each numbered row
different set of predictors starting from our baseline implementation of PixelCNN (van den Oord et al. 2016a) in row 0.
Output Relative Prediction Rows and 2 show the results obtained by computing the adjustment score in Equation 7 with respect to the left top pixels, respectively. As the choice of which pixel to select the reference for the adjustment is predetermined, the copy mechanism is not required. Surprisingly, this simple change yields improvements in multiple datasets compared to absolute predictors. On average, adjusting from the left pixel yields more improvement compared to adjusting the top pixel, yet differences seem to be minor. Combining both predictors tends to improve the scores of the model, observed in row 3, with the exception of the colorization task. In row also observe that adding all pixels within an 2x2 grid apart from the generated pixel (top, left and diagonals) does not yield additional improvements. The same occurs if we extend this grid two
12

---

Under review as a conference paper at ICLR 2020
0.8
% relative
0.6
improvement
0.4
0.2
0.2
0.4
0.6
0.8
Figure 4: Effects of the pruning threshold when both absolute and relative predictors are used. The X-axis denotes the percentage of values that kept in the relative predictor's softmax.
3.4
absolute relative
3.3
3.2
absolute relative
3.1
2.9
0.25
0.5
0.75
1.25
1.5
1.75
2
Figure 5: Effects of changing the number of parameters in the model using the absolute (red line) and relative predictors (blue line). The r-axis denotes the fraction of the parameters used in the hidden layers, and the y-axis reports the log-likelihood obtained on the CIFAR-10 validation set for unconditional image generation.
positions, shown in row 5, We believe this is because neighbouring pixels tend share similar colors, so more additional pixels are unlikely to add any additional information than the left and top pixels.
Rows 6, and illustrate the results obtained by combining the absolute predictor from row 0 with each of the relative predictors in rows 4, and 5. Such setups cause the latent predictor networks to assign most probability mass to the absolute predictor. This suggests that the model is over-fitting to the absolute predictor, which has faster convergence rate due to its smaller softmax (256 vs. 511).
Thresholding Relative Prediction To effectively combine the absolute and relative predictors pruned the softmax of the adjustment mechanism to the most observed values in the training set (as discussed in (4). Rows 9, 10 and 11 report the results obtained when set the pruning threshold to 20%, keeping only the 20% most frequent adjustment values in the training set. In CIFAR-10, we obtain results that are close the best performing setups. Figure we study the results obtained using different threshold values in CIFAR-10. The red line represents the relative improvement obtained for unconditional image generation in the validation set using the model setup corresponding to row 9, and we can observe the highest improvement at 20%. We also report the percentage of pixels generated using the relative prediction mechanism when generating 100 samples with the blue line. We observe that while at lower thresholds the relative prediction mechanism is used approximately for the generation of 70% of images, when this value approaches 100%, this relative prediction is rarely used, explaining the lack of improvements without pruning.
Parameter Efficiency Figure Blilusttitls the log-likelite ligeliicl scores for unconditioned image gener- ation for different hyper-paramett for the hidden layer sizes, namely H1 and H2. We can observe that both absolute (red line) and relative (blue line) loss functions start under-performing as we increase the size of the model. Yet, the relative improvement
our loss function maintained. However, the relative predictor requires more parameters to start performing effectively, if we drop the number of parameters to half, the absolute predictor exhibits a superior performance. We believe that this is because the relative predictor requires more parameters to learn the harsher transitions between pixels. Finally, if we combine the absolute and pruned relative predictors, the model achieves lower overall ligelikelioood scores across different parametrizations.
13

---

Under review as a conference paper at ICLR 2020
I| Absoolute
Relative(i) Relative(o)
0.8
0.6
0.4
0.2
A+Ro
A+Ri
A+Ri+Ro
Figure 6: Predictor usage different setups for colorization CIFAR-10. Each bar denotes the percentage of usages after sampling once for each image in the validation set.
Input Relative Prediction For relative pixel prediction from the input, we tested models that copy from the center pixel, then in a 2x2 grid and 3x3 grid centered at the position of the pixel to be predicted. Results reported in rows 12, 13 and 14. Here, observe that large patch of pixels to copy from benefits the model. Interestingly. we did not encounter any problems combining the absolute and relative predictors when copying from the input these tasks. We believe this is because the transitions between the input and output pixels in these tasks is more trraight-fowaate whereas the transition between neighbouring pixels can contain harsh transitions (e.g. edges between objects), which take longer to learn. Results that combine the input relative predictor and absolute predictor are shown in rows 15, 16 and 17. Here, we observe that combining predictors yields in marginally better scores on average.
Predictor Combination Rows 18 and 19 show the improvements obtained by combining both input and output predictors, where we observe that optimal results exclude the usage of an absolute predictor. This suggests that the information that the two forms of relative prediction are complementary for the tasks that tested. That is, the output relative prediction learns the softer transitions between continuous surfaces, whereas input relative prediction learns the harsher transitions between objects. Figure 6 illustrates the percentage usages of the absolute predictor, input relative predictor and output relative predictor for the colorization task. We observe that the absolute predictor is applied considerably when used individually with either of the relative predictors (A+Ro and A+Ri), while rarely used with both predictors (A+Ri+Ro).
Outcome Masking Rows 20, and 22 show the improvements that can be obtained by masking invalid entries in the relative predictors. While improvements are marginal, we can observe that perplexities are consistently better using masking, as we are removing invalid outcomes that are certainly leading to losses in probability mass.
B SAMPLED IMAGES
Examples of sampled images are illustrated in Figure 7 Columns "Absolute" "Left Pixel Only" and "Absolute + Relative" refer to the models trained using setups 0, and in Table 3 Here, we can observe that the proposed models Interestingly, sharp transitions are also captured when all pixels are predicted relatively from the pixel the left (Column "Left Pixel").
capture the richness of the image distribution from the data.
14

---

Under review as a conference paper at ICLR 2020
Figure 7: 32x images generated by models trained on CIFAR-10 10 with using the absolute predictor (Column "Absolute"), a relative predictor that always copies from the left pixel (Column "Left Pixel Only") and the combination of the absolute predictor and relative predictor that copies from the top and left pixels (Column "Absolute+Relative).
ABSOLUTE
LEFT PIXEL ONLY
ABSOLUTE RELATIVE
15


Reviews:
[{"review_id": "rJgWhRKyjH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper bases its methodology on well known developments in image analysis/synthesis about similarity of pixel values in adjacent locations. Many techniques have been used for modelling this similarity, including predictive models, cliques and graphs. The paper uses a simple autoregressive model for generating pixel values based on the values of previously processed pixels, estimating the differences between these neighboring pixel values. \n\nThe method is implemented through copying the pixel values and adjusting the differences.  Three types of prediction, based on absolute, or relative values are examined, for image generation, colorization, super-resolution. The problems are significant, but the approach rather superficial. A small experimental study is presented, based on CIFAR-10 and downsampled ImageNet datsaets. Much more experiments, including quantitative and qualitative results are reuired, to validate the prospects of the method in different types of (complex) problems and contexts. Marginal improvements are observed in the presented results. Since image generation and image to image translation are targeted, comparison and/or combined use with Sota methods, i.e., GANs should be examined. \n\nMoreover, the paper presentation needs improvement; for example, symbols are undefined when used for the first time in the text (see eq. 3), etc.  \n\n   ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SyxKniRnYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper the authors present a new way to use autoregressive modeling to generate images pixel by pixel where each pixel is generated by modeling the difference between the current pixel value and  the preexistent ones. In order to achieve that, the authors propose a copy and adjustment mechanism that select an existing pixel, and then adjust its sub-pixel (channel values) to generate the new pixel. The proposed model is demonstrated with a suite of experiments in classic image generation benchmark. The authors also demonstrate the use of their technique in Image to Image translation.\nOverall, although the paper explain clearly the intuition and the motivation of the proposed technique, I think that the paper in its present state have low novelty, weak related work analysis review and insufficient experiments to support a publication at ICLR. \n\n\n\n**Novelty, contribution and related work**\nThe authors should highlight better their main contribution novelty of the proposed method compared to their baseline.\n\n\n**Result and conducted experiments**\nthe correctness of the proposed approach is not proved by the conducted experiment  in fact:\nThe experiments do not provide the details of the used architecture compared to your baseline. \nIn Table 1 you report the results using your technique on several computer vision tasks (generation, colorization and super-resolution) but you're not comparing with the SoA of each of these tasks.\nThe  results reported in Tables 1 and 2 are not convincing  when compared to existing approaches (using only CIFAR10 in Table2). \nThere are so many missing details specially to validate Image-To-image translation \nFigure 3 is confusing and  not clear  \n\n**Minor comments**\nIn  references section : (Kingma & Dhariwal, 2018) is not in a proper format (nips 2018)\nBad quality of illustrations and images \nBe coherent with the position of captions (figure 3)", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1xVFeinYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an approach for image generation that relies on an autoregressive model for the image pixels. These models are popularly used in image coding and compression settings, and have been used in generative models like PixelCNN. In contrast to this prior work, the proposed model is based on the selection of a previously available pixel and the modeling of the differences between the old pixel and the new one. The copy and adjustment models, i.e., eqs (3) and (5-6), are straightforward. Applications to image-to-image translation are also presented.\n\nI am rating the paper \"weak reject\" mostly due to the limited set of comparisons in experimental results. There is no qualitative comparison to other algorithms for two of the problems considered (colorization, super-resolution) and the comparison with other algorithms for unconditional image generation is limited to CIFAR-10; thus, the impact of this contribution is not clear. Furthermore there is no discussion of these comparison results - i.e., what the proposed algorithm contributes given that it's outperformed by the sparse transformer.\n\nIt is not clear at first what the authors mean by \"sub-pixel\", which appears to be one of the color/spectrum channels of a pixel of the image? Also not clear what \"outcome masking\" refers to. The explanation of the hidden states (g,h) used for each mechanism are not always clear or explicit. For example, can you write an equation for h_{i,c-1} which is more explicit than \"composing the history of generated sub-pixels\"? Can you define Ui when it is first used in (6)? What is the difference between the pixel state h_{r,C} and its values x_r?\n\nMinor comments\nThe second equation in Section 2.3 is missing =\nIn Section 5.1, it is not clear what is meant by \"discrediting\" the image.\nThe table in Fig. 3 could use full names for the problems instead of initials.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers rated this submission as a weak reject and there was no author response.
The AC recommends rejection.
Decision: Reject