Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: # Review
This paper proposes five modifications to improve PixelCNN, a generative model with tractable likelihood. The authors empirically showed the impact of each of their proposed modifications using a series of ablation experiments. They also reported a new state-of-the-art result on CIFAR-10.
Improving generative models, especially for images, is an active research area and this paper definitely contributes to it.


# Pros
The authors motivate each modification well they proposed. They also used ablation experiments to show each of them is important.

The authors use a discretized mixture of logistic distributions to model the conditional distribution of a sub-pixel instead of a 256-way softmax. This allows to have a lower output dimension and to be better suited at learning ordinal relationships between sub-pixel values. The authors also mentioned it speeded up training time (less computation) as well as the convergence during the optimization of the model (as shown in Fig.6).

The authors make an interesting remark about how the dependencies between the color channels of a pixel are likely to be relatively simple and do not require a deep network to model. This allows them to have a simplified architecture where you don't have to separate out all feature maps in 3 groups depending on whether or not they can see the R/G/B sub-pixel of the current location.


# Cons
It is not clear to me what the predictive distribution for the green channel (and the blue) looks like. More precisely, how are the means of the mixture components linearly depending on the value of the red sub-pixel? I would have liked to see the equations for them.


# Minor Comments
In Fig.2 it is written "Sequence of 6 layers" but in the text (Section 2.4) it says 6 blocks of 5 ResNet layers. What is the remaining layer?
In Fig.2 what does the first "green square -> blue square" which isn't in the white rectangle represents?
Is there any reason why the mixture indicator is shared across all three channels?
review_title: Great empirical work, insightful ablation experiments, code is available which is a nice contribution to the community

final_score: rating: 7: Good paper, accept
review_content: Summary:
This paper on autoregressive generative models explores various extensions of PixelCNNs. The proposed changes are to replace the softmax function with a logistic mixture model, to use dropout for regularization, to use downsampling to increase receptive field size, and the introduction of particular skip connections. The authors find that this allows the PixelCNN to outperform a PixelRNN on CIFAR-10, the previous state-of-the-art model. The authors further explore the performance of PixelCNNs with smaller receptive field sizes.

Review:
This is a useful contribution towards better tractable image models. In particular, autoregressive models can be quite slow at test time, and the more efficient architectures described here should help with that.

My main criticism regards the severe neglect of related work. Mixture models have been used a lot in autoregressive image modeling, including for multivariate conditional densities and including downsampling to increase receptive field size, albeit in a different manner: Domke (2008), Hosseini et al. (2010), Theis et al. (2012), Uria et al. (2013), Theis et al. (2015). Note that the logistic distribution is a special case of the Gaussian scale mixture (West, 1978).

The main difference seems to be the integration of the density to model integers. While this is clearly a good idea and the right way forward, the authors claim but do not support that not doing this has “proved to be a problem for earlier models based on continuous distributions”. Please elaborate, add a reference, or ideally report the performance achieved by PixelCNN++ without integration (and instead adding uniform noise to make the variables continuous).

60,000 images are not a lot in a high-dimensional space. While I can see the usefulness of regularization for specialized content – and this can serve as a good example to demonstrate the usefulness of dropout – why not use “80 million tiny images” (superset of CIFAR-10) for natural images? Semi-supervised learning should be fairly trivial here (because the model’s likelihood is tractable), so this data could even be used in the class-conditional case.

It would be interesting to know how fast the different models are at test time (i.e., when generating images).
review_title: Related work

final_score: rating: null
review_content: Perhaps this is easy to see, but can you provide a bit more detail to ensure us that after down- and upsampling the logistic distribution can still not access the pixel it is trying to predict?
review_title: Model validity after down- and upsampling

final_score: rating: 6: Marginally above acceptance threshold
review_content: Apologies for the late submission of this review, and thank you for the author’s responses to earlier questions.

This submission proposes an improved implementation of the PixelCNN generative model. Most of the improvements are small and can be considered as specific technical details such as the use of dropout and skip connections, while others are slightly more substantial such as the use of a different likelihood model and multiscale analysis. The submission demonstrates state-of-the-art likelihood results on CIFAR-10.

My summary of the main contribution:
Autoregressive-type models - of which PixelCNN is an example - are a nice class of models as their likelihood can be evaluated in closed form. A main differentiator for this type of models is how the conditional likelihood of one pixel conditioned on its causal neighbourhood is modelled:

- In one line of work such as (Theis et al, 2012 MCGSM, Theis et al 2015 Spatial LSTM) the conditional distribution is modelled as a continuous density over real numbers. This approach has limitations: We know that in observed data pixel intensities are quantized to a discrete integer representation so a discrete distribution could give better likelihoods. Furthermore these continuous distributions have a tail and assign some probability mass outside the valid range of pixel intensities, which may hurt the likelihood.
- In more recent work by van den Oord and colleagues the conditional likelihood is modelled as an arbitrary discrete distribution over the 256 possible values for pixel intensities. This does not suffer from the limitations of continuous likelihoods, but it also seems wasteful and is not very data efficient.

The authors propose something in the middle by keeping the discretized nature of the conditional likelihood, but restricting the discrete distribution to ones whose CDF that can be modeled as a linear combination of sigmoids. This approach makes sense to me, and is new in a way, but it doesn’t appear to be very revolutionary or significant to me.

The second somewhat significant modification is the use of downsampling and multiscale modelling (as opposed to dilated convolutions). The main motivation for the authors to do this is saving computation time while keeping the multiscale flexibility of the model. The authors also introduce shortcut connections to compensate for the potential loss of information as they perform downsampling. Again, I feel that this modification not particularly revolutionary. Multiscale image analysis with autoregressive generative models has been done for example in (Theis et al, 2012) and several other papers.

Overall I felt that this submission falls short on presenting substantially new ideas, and reads more like documentation for a particular implementation of an existing idea.
review_title: Review

final_score: rating: null
review_content: 1. Do you have any experiments where the discretization is dropped?
2. Why the particular choice of logistic distributions? How do the properties compare to the more flexible family of Gaussian scale mixtures which have been used successfully in very similar work e.g. (Theis et al. 2012) http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0039857 Is it mainly/only about ease of evaluation?
review_title: Comparison to non-discretized distributions

final_score: rating: null
review_content: Did you try different ordering (ex. starting from the bottom right corner instead and going upward and leftward)? If so was the results similar?
review_title: Traversal Ordering

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: It is refreshing that OpenAI has taken the time to resurrect classic heuristics like down-sampling and dropout into PixelCNN. Some sort of AR technique like PixelCNN probably holds the missing keys needed to eventually have decent originally-created images from CIFAR10 or other real-life data-sets. So any engineering streamlining, as in this paper, is welcome to the general public, especially when helping to avoid expensive clusters of GPUs, only DeepMind can afford. In this sense, OpenAI is fulfilling its mission and we are all very grateful! Thus the paper is a welcome addition and we hope it finds its way into what appears to be an experimental CS conference anyway. 

On a more conceptual level, our hope is that OpenAI, with so talented a team, will stop competing in these contrived contests to improve by basis points certain obscure log-likelihoods and instead focus on the bigger picture problems. Why for example, almost two years later, the class-conditional CIFAR10 samples, as on the left of Figure 4 in this paper (column 8 - class of horses), are still inferior to, say, the samples on Figure 4 of reference [2]? Forgive the pun, but aren't we beating a dead horse here? Yes, resolution and sharpness have improved, due to good engineering but nobody in the general public will take these samples seriously! Despite the claims put forward by some on the DeepMind team, PixelCNN is not a "fully-generative" neural net (as rigorously defined in section 3 of reference [1]), but merely a perturbative net, in the vain of the Boltzmann machine. After the Procrustean experience of lost decades on Boltzmann machines, the time perhaps has come to think more about the fundamentals and less about the heuristics?

[1] https://arxiv.org/pdf/1508.06585v5.pdf
[2] https://arxiv.org/pdf/1511.02841v3.pdf
review_title: Good experimental work, reinstating classic heuristics back into AR CNN.

Meta-review:  The authors acknowledge that the ideas in the paper are incremental, but assert these are not-trivial improvements upon prior work on pixel CNNs. The reviewers tended to agree with this characterization. The paper presents SOTA pixel likelihood results on CIFAR-10. This work is also coupled with a high quality source code contribution, which also appears to have already been well received by the github community. Reviewer 1 made the point that in terms of raw novelty this work is probably a little below the bar for an oral presentation. A public reviewer rated this paper as a strong accept. Given the statistics, quality and originality of the other papers in my AC batch I recommend poster.
Decision: Accept (Poster)