Title: PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications
Authors: Tim Salimans
Andrej Karpathy
Xi Chen
Diederik P. Kingma
Venue: ICLR 2017

Abstract: PixelCNNs are a recently proposed class of powerful generative models with tractable likelihood. Here we discuss our implementation of PixelCNNs which we make available at https://github.com/openai/pixel-cnn. Our implementation contains a number of modifications to the original model that both simplify its structure and improve its performance. 1) We use a discretized logistic mixture likelihood on the pixels, rather than a 256-way softmax, which we find to speed up training. 2) We condition on whole pixels, rather than R/G/B sub-pixels, simplifying the model structure. 3) We use downsampling to efficiently capture structure at multiple resolutions. 4) We introduce additional short-cut connections to further speed up optimization. 5) We regularize the model using dropout. Finally, we present state-of-the-art log likelihood results on CIFAR-10 to demonstrate the usefulness of these modifications.

TL;DR: Adding discretized logistic mixture Likelihood and other modifications to PixelCNN improves performance.

Review discussion:
Did you try different ordering (ex. starting from the bottom right corner instead and going upward and leftward)? If so was the results similar?
We used the same ordering as the original PixelCNN and did not explore alternatives. My guess is that it would not matter much, but it's worth trying.
1. Do you have any experiments where the discretization is dropped?
2. Why the particular choice of logistic distributions? How do the properties compare to the more flexible family of Gaussian scale mixtures which have been used successfully in very similar work e.g. (Theis et al. 2012) http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0039857 Is it mainly/only about ease of evaluation?
1. Early on I experimented with continuous distributions and no discretization, but this didn't work very well. The main benefit of using discretization is to capture the relatively large probability mass of the data distribution at pixel values of 0 and 255: Continuous distributions cannot model this without effectively reserving a mixture component for all possible combinations of edge values.
2. Yes, ease of evaluation was the main motivation, since the logistic CDF is simply the sigmoid function. The number of mixture components seems to be the dominant factor in determining the distribution expressiveness. Since we saw virtually no effect from adding more than 5 components, we did not further explore methods for making the distribution more flexible. For other types of data, Gaussian scale mixtures might prove very useful though.
Perhaps this is easy to see, but can you provide a bit more detail to ensure us that after down- and upsampling the logistic distribution can still not access the pixel it is trying to predict?
The precise structure of the downsampling/upsampling steps can be found at https://github.com/openai/pixel-cnn/blob/master/pixel_cnn_pp/nn.py: We use strided convolution and strided deconvolution, so there is no information leakage. I double-checked this by computing the gradient of the output parameters at each spatial location with respect to all the pixels in the input image: The gradient of the output at location (i,j) is strictly zero for all pixels at locations (i,>=j) and (>i,:).
Summary:
This paper on autoregressive generative models explores various extensions of PixelCNNs. The proposed changes are to replace the softmax function with a logistic mixture model, to use dropout for regularization, to use downsampling to increase receptive field size, and the introduction of particular skip connections. The authors find that this allows the PixelCNN to outperform a PixelRNN on CIFAR-10, the previous state-of-the-art model. The authors further explore the performance of PixelCNNs with smaller receptive field sizes.
Review:
This is a useful contribution towards better tractable image models.
In particular, autoregressive models can be quite slow at test time, and the more efficient architectures described here should help with that.
My main criticism regards the severe neglect of related work. Mixture models have been used a lot in autoregressive image modeling, including for multivariate conditional densities and including downsampling to increase receptive field size, albeit in a different manner: Domke (2008), Hosseini et al. (2010), Theis et al. (2012), Uria et al. (2013), Theis et al. (2015). Note that the logistic distribution is a special case of the Gaussian scale mixture (West, 1978).
The main difference seems to be the integration of the density to model integers.
While this is clearly a good idea and the right way forward, the authors claim but do not support that not doing this has “proved to be a problem for earlier models based on continuous distributions”. Please elaborate, add a reference, or ideally report the performance achieved by PixelCNN++ without integration (and instead adding uniform noise to make the variables continuous).
60,000 images are not a lot in a high-dimensional space.
While I can see the usefulness of regularization for specialized content – and this can serve as a good example to demonstrate the usefulness of dropout – why not use “80 million tiny images” (superset of CIFAR-10) for natural images? Semi-supervised learning should be fairly trivial here (because the model’s likelihood is tractable), so this data could even be used in the class-conditional case.
It would be interesting to know how fast the different models are at test time (i.e., when generating images).
Thanks for the citation suggestions. I will add these, as well as a comparison of the proposed likelihood model to a continuous mixture likelihood. (I have done experiments using continuous mixtures before, but not yet with the final architecture presented in the paper)
I ran the model without discretization of the mixture distribution on the pixels. Using the same model settings as for the 2.92 bits per sub-pixel reported in the paper, it got to 3.11 bits per dim. (This is a variational lower bound, adding small uniform noise to the discrete pixels). The code can be found here: https://www.dropbox.com/sh/tcx5bg76xngmm8a/AADyvRZ0H98rcGU-7wULjvV0a?dl=1, and I will also include this result in the paper.
# Review
This paper proposes five modifications to improve PixelCNN, a generative model with tractable likelihood. The authors empirically showed the impact of each of their proposed modifications using a series of ablation experiments. They also reported a new state-of-the-art result on CIFAR-10.
Improving generative models, especially for images, is an active research area and this paper definitely contributes to it.
# Pros
The authors motivate each modification well they proposed. They also used ablation experiments to show each of them is important.
The authors use a discretized mixture of logistic distributions to model the conditional distribution of a sub-pixel instead of a 256-way softmax. This allows to have a lower output dimension and to be better suited at learning ordinal relationships between sub-pixel values. The authors also mentioned it speeded up training time (less computation) as well as the convergence during the optimization of the model (as shown in Fig.6).
The authors make an interesting remark about how the dependencies between the color channels of a pixel are likely to be relatively simple and do not require a deep network to model. This allows them to have a simplified architecture where you don't have to separate out all feature maps in 3 groups depending on whether or not they can see the R/G/B sub-pixel of the current location.
# Cons
It is not clear to me what the predictive distribution for the green channel (and the blue) looks like.
More precisely, how are the means of the mixture components linearly depending on the value of the red sub-pixel? I would have liked to see the equations for them.
# Minor Comments
In Fig.2 it is written "Sequence of 6 layers" but in the text (Section 2.4) it says 6 blocks of 5 ResNet layers. What is the remaining layer?
In Fig.2 what does the first "green square -> blue square" which isn't in the white rectangle represents?
Is there any reason why the mixture indicator is shared across all three channels?
Thanks for the review! I will add a bit more detail to the explanation of the mixture likelihood.
Apologies for the late submission of this review, and thank you for the author’s responses to earlier questions.
This submission proposes an improved implementation of the PixelCNN generative model. Most of the improvements are small and can be considered as specific technical details such as the use of dropout and skip connections, while others are slightly more substantial such as the use of a different likelihood model and multiscale analysis. The submission demonstrates state-of-the-art likelihood results on CIFAR-10.
My summary of the main contribution:
Autoregressive-type models - of which PixelCNN is an example - are a nice class of models as their likelihood can be evaluated in closed form. A main differentiator for this type of models is how the conditional likelihood of one pixel conditioned on its causal neighbourhood is modelled:
- In one line of work such as (Theis et al, 2012 MCGSM, Theis et al 2015 Spatial LSTM) the conditional distribution is modelled as a continuous density over real numbers. This approach has limitations: We know that in observed data pixel intensities are quantized to a discrete integer representation so a discrete distribution could give better likelihoods.
Furthermore these continuous distributions have a tail and assign some probability mass outside the valid range of pixel intensities, which may hurt the likelihood.
- In more recent work by van den Oord and colleagues the conditional likelihood is modelled as an arbitrary discrete distribution over the 256 possible values for pixel intensities. This does not suffer from the limitations of continuous likelihoods, but it also seems wasteful and is not very data efficient.
The authors propose something in the middle by keeping the discretized nature of the conditional likelihood, but restricting the discrete distribution to ones whose CDF that can be modeled as a linear combination of sigmoids. This approach makes sense to me, and is new in a way, but it doesn’t appear to be very revolutionary or significant to me.
The second somewhat significant modification is the use of downsampling and multiscale modelling (as opposed to dilated convolutions). The main motivation for the authors to do this is saving computation time while keeping the multiscale flexibility of the model. The authors also introduce shortcut connections to compensate for the potential loss of information as they perform downsampling. Again, I feel that this modification not particularly revolutionary. Multiscale image analysis with autoregressive generative models has been done for example in (Theis et al, 2012) and several other papers.
Overall I felt that this submission falls short on presenting substantially new ideas, and reads more like documentation for a particular implementation of an existing idea.
Thanks for you review! The ideas in our paper are incremental, but non-trivial, improvements upon existing work. We have deliberately been modest in the presentation of these ideas, rather than overclaiming paradigm shifting insights as already happens all too often in our field. The contributions we make are useful to many people, as evidenced by the SOTA results, the number of github stars, the papers building on our work (e.g. https://arxiv.org/pdf/1611.02731.pdf, https://arxiv.org/pdf/1612.08185v1.pdf), and the large amount of positive personal communication I have received. Taking this into account, I feel the paper contributes more to our field than most ICLR submissions. Could you please reconsider your rating in this light?
Thank you for the response, and for the revised submission detailing comparisons to continuous likelihoods.
After re-evaluating the submission, and all the responses, I have increased the score and reduced the confidence for my review.
Just to clarify, I do not want to dispute the value of the contribution itself, which is indeed evidenced by all the things you mention (github stars, citations), and high quality open source code is a great contribution, etc. My review was more about whether this should be one of the papers presented at the conference, especially given the unusually good crop of papers I have personally reviewed this year (it might have been a very lucky sample) which, in many cases, presented more out-of-the-box and original ideas. I do agree I probably have been too harsh in my initial judgment.
We uploaded a new revision incorporating some of the suggestions made by the reviewers. Thank you for your input!
Specifically, the revision incorporates additional references and an experimental comparison with continuous mixture models as suggested by reviewer 2, and a more detailed description of the likelihood model, including equations, as requested by reviewer 3.

Meta-review:
role: PC
data: comment:  The authors acknowledge that the ideas in the paper are incremental, but assert these are not-trivial improvements upon prior work on pixel CNNs. The reviewers tended to agree with this characterization. The paper presents SOTA pixel likelihood results on CIFAR-10. This work is also coupled with a high quality source code contribution, which also appears to have already been well received by the github community. Reviewer 1 made the point that in terms of raw novelty this work is probably a little below the bar for an oral presentation. A public reviewer rated this paper as a strong accept. Given the statistics, quality and originality of the other papers in my AC batch I recommend poster.

Decision: Accept (Poster)