Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: "First, it allows us to assess whether auto-regressive models are able to match the GAN results of Reed et al. (2016a)." Does it, though? Because the resolution is so bad. And resolution limitations aren't addressed until the second-to-last paragraph of the paper. And Figure 9 only shows 3 results (picked randomly? Picked to be favorable to PixelCNN?). That hardly seems conclusive.

The segmentation masks and keypoints are pretty strong input constraints. It's hard to tell how much coherent object and scene detail is emerging because the resolution is so low. For example, the cows in figure 5 look like color blobs, basically. Any color blob that follows an exact cow segmentation mask will look cow-like.

The amount of variation is impressive, though.

How can we assess how much the model is "replaying" training data? Figure 8 tries to get at this, but I wonder how much each of the "red birds", for instance, is mostly copied from a particular training example.

I'm unsatisfied with the answers to the pre-review questions. You didn't answer my questions. The paper would benefit from concrete numbers on training time / epochs and testing time. You didn't say why you can't make high resolution comparisons. Yes, it's slower at test time. Is it prohibitively slow? Or is it slightly inconvenient? There really aren't that many comparisons in the paper, anyway, so it if takes an hour to generate a result that doesn't seem prohibitive. 

To be clear about my biases: I don't think PixelCNN is "the right way" to do deep image generation. Texture synthesis methods used these causal neighborhoods with some success, but only because there wasn't a clear way to do the optimization more globally (Kwatra et al, Texture Optimization for Example-based Synthesis being one of the first alternatives). It seems simply incorrect to make hard decisions about what pixel values should be in one part of the image before synthesizing another part of the image (Another texture synthesis strategy to help fight back against this strict causality was coarse-to-fine synthesis. And I do see some deep image synthesis methods exploring that). It seems much more correct to have a deeper network and let all output pixels be conditioned on all other pixels (this conditioning will implicitly emerge at intermediate parts of the network). That said, I could be totally wrong, and the advantages stated in the paper could outweigh the disadvantages. But this paper doesn't feel very honest about the disadvantages.

Overall, I think the results are somewhat tantalizing, especially the ability to generate diverse outputs. But the resolution is extremely low, especially compared to the richness of the inputs. The network gets a lot of hand holding from rich inputs (it does at least learn to obey them). 

The deep image synthesis literature is moving very quickly. The field needs to move on from "proof of concept" papers (the first to show a particular result is possible) to more thorough comparisons. This paper has an opportunity to be a more in depth comparison, but it's not very deep in that regard. There isn't really an apples to apples comparison between PixelCNN and GAN nor is there a conclusion statement about why that is impossible. There isn't any large scale comparison, either qualitative or quantified by user studies, about the quality of the results.
review_title: Interesting results, but comparisons seem lacking

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes an extension of PixelCNN method that can be conditioned on text and spatially-structured constraints (segmentation / keypoints). It is similar to Reed 2016a except the extension is built on top of PixelCNN instead of GAN. After reading the author's comment, I realized the argument is not that conditional PixelCNN is much better than conditional GAN. I think the authors can add more discussions about pros and cons of each model in the paper. I agree with the other reviewer that some analysis of training and generation time would be helpful. I understand it takes O(N) instead of O(1) for PixelCNN method to do sampling, but is that the main reason that the experiments can only be conducted in low resolution (32 x 32)? I also think since there are not quantitative comparisons, it makes more sense to show more visualizations than 3 examples. Overall, the generated results look reasonably good and have enough diversity. The color mistake is an issue where the author has provided some explanations in the comment. I would say the technical novelty is incremental since the extension is straightforward and similar to previous work. I lean toward accepting because it is very relevant to ICLR community and it provides a good opportunity for future investigation and comparison between different deep image synthesis methods. 




review_title: Conditional PixelCNN 

final_score: rating: null
review_content: I have basically the same questions as Reviewer2. This paper is all about pixelCNN vs GAN -- why the different resolutions? What are the actual train times, test times, and memory requirements for the two competing methods?
review_title: Resolution

final_score: rating: 7: Good paper, accept
review_content: This work focuses on conditional image synthesis in the autoregressive framework.  Based on PixelCNN, it trains models that condition on text as well as segmentation masks or keypoints.  Experiments show results for keypoint conditional synthesis on the CUB (birds) and MHP (human pose) dataset, and segmentation conditional synthesis on MS-COCO (objects).  This extension to keypoint/segment conditioning is the primary contribution over existing PixelCNN work.  Qualitative comparison is made to GAN approaches for synthesis.

Pros:
(1) The paper demonstrates additional capabilities for image generation in the autoregressive framework, suggesting that it can keep pace with the latest capabilities of GANs.
(2) Qualitative comparison in Figure 9 suggests that PixelCNN and GAN-based methods may make different kinds of mistakes, with PixelCNN being more robust against introducing artifacts.
(3) Some effort is put forth to establish quantitative evaluation in terms of log-likelihoods (Table 1).

Cons:
(1) Comparison to other work is difficult and limited to qualitative results.  The qualitative results can still be somewhat difficult to interpret.  I believe supplementary material or an appendix with many additional examples could partially alleviate this problem.
(2) The extension of PixelCNN to conditioning on additional data is fairly straightforward.  This is a solid engineering contribution, but not a surprising new concept.
review_title: review: worthwhile extension of PixelCNN capabilities

final_score: rating: null
review_content: As comparison to prior work must be done based on qualitative results, could you elaborate on the comparison with GAN (Reed 2016a) shown in Figure 9.  In particular, if viewed at the same resolution (eg GAN results downsampled to 32x32), are the GAN artifacts still apparent?  Can the PixelCNN scale to higher resolutions without artifacts?  Secondly, the PixelCNN results seem only weakly influenced by the text, while text appears to have a much stronger influence on the results in (Reed 2016a).  Is this a weakness with respect to the GAN approach?

Also, within Figure 9, I believe (Reed 2016b) should be (Reed 2016a) (label above images).
review_title: comparison with GANs

Meta-review: The paper extends PixelCNN to do text and location conditional image generation. The reviewers praise the diversity of the generated samples, which seems like the strongest result of the paper. On the other hand, they are concerned with their low resolution. The authors made an effort of showing a few high-resolution samples in the rebuttal, which indeed look better. Two reviewers mention that the work with respect to PixelCNN is very incremental, and the AC agrees. Overall, this paper is very borderline. While all reviewers became slightly more positive, none was particularly swayed. The paper will make a nice workshop contribution.
Decision: Invite to Workshop Track