Venue: ICLR 2018 conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: This paper extends the PixelCNN/RNN based (conditional) image generation approaches with self-attention mechanism. 

Pros:
- qualitatively the proposed method has good results in several tasks

Cons:
- writing needs to be improved
- lack of motivation
- not easy to follow technique details


The motivation part is missing. It seems to me that the paper simply try to combine the Transformer with PixelCNN/RNN based image generation without a clear explanation why this is needed. Why self-attention is so important for image generation? Why not just a deeper network with more parameters? Throughout the paper I cannot find a clear answer for this. Based on this I couldn't see a clear contribution. 

The paper is difficult to keep the track given the current flow. Each subsection of section 3 starts with technique details without explaining why we do this. Some sentences like "look pretty cool" is not academic. 

The experiments lack comparisons except the human evaluation, while the log-likelihood improvement is marginal. I am wondering how the human evaluation is conducted. Does it compare all the competing algorithms against the same sub-samples of the GT data? How many pairs have been compared for each algorithm?  Apart from this metric, I would like to see qualitative comparison between competing algorithms in the paper as well. Also other approaches e.g. SRGAN could be compared. 

I am also interested about the author's claim that the implementation error that influences the log-likelihood. Has this been fixed after the deadline?
review_title: Requires improvement

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary

This paper extends self-attention layers (Vaswani et al., 2017) from sequences to images and proposes to use the layers as part of PixelCNNs (van den Oord et al., 2016). The proposed model is evaluated in terms of visual appearance of samples and log-likelihoods. The authors find a small improvement in terms of log-likelihood over PixelCNNs and that super-resolved CelebA images are able to fool human observers significantly more often than PixelRNN based super-resolution (Dahl et al., 2017).

Review

Autoregressive models are of large interest to the ICLR community and exploring new architectures is a valuable contribution. Using self-attention in autoregressive models is an intriguing idea. It is a little bit disappointing that the added model complexity only yields a small improvement compared to the more straight-forward modifications of the PixelCNN++. I think the paper would benefit from a little bit more work, but I am open to adjusting my score based on feedback.

I find it somewhat surprising that the proposed model is only slightly better in terms of log-likelihood than a PixelRNN, but much better in terms of human evaluation – given that both models were optimized for log-likelihood. Was the setup used with Mechanical Turk exactly the same as the one used by Dahl et al.? These types of human evaluations can be extremely sensitive to changes in the setup, even the phrasing of the task can influence results. E.g., presenting images scaled differently can mask certain artifacts. In addition, the variance between subjects can be very high. Ideally, each method included in the comparison would be re-evaluated using the same set of observers. Please include error bars.

The CelebA super-resolution task furthermore seems fairly limited. Given the extreme downsampling of the input, the task becomes similar to simply generating any realistic image. A useful baseline would be the following method: Store the entire training set. For a given query image, look for the nearest neighbor in the downsampled space, then return the corresponding high-resolution image. This trivial method might not only perform well, it also highlights a flaw in the evaluation: Any method which returns stored high-resolution images – even if they don’t match the input – would perform at 50%. To fix this, the human observers should also receive the low-resolution image and be asked to identify the correct corresponding high-resolution image.

Using multiplicative operations to model images seems important. How does the self-attention mechanism relate to “gated” convolutions used in PixelCNNs? Could gated convolutions not also be considered a form of self-attention?

The presentation/text could use some work. Much of the text assumes that the reader is familiar with Vaswani et al. (2017) but could easily be made more self-contained by directly including the definitions used. E.g., the encoding of positions using sines and cosines or the multi-head attention model. I also felt too much of the architecture is described in prose and could be more efficiently and precisely conveyed in equations.

On page 7 the authors write “we believe our cherry-picked images for various classes to be of higher perceptual quality”. This is a meaningless result, not only because the images were cherry-picked. Generating realistic images is trivial - you just need to store the training images. Analyzing samples generated by a generative model (outside the context of an application) should therefore only be used for diagnostic purposes or to build intuitions but not to judge the quality of a model.

Please consider rephrasing the last sentence of the abstract. Generating images which “look pretty cool” should not be the goal of a serious machine learning paper or a respected machine learning conference.
review_title: Interesting idea, weak evaluation

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper the authors propose an autoregressive image generation model that incorporates a self-attention mechanism. The latter is inspired by the work of [Vaswani et al., 2016], which was proposed for sequences and is extended to 2D images in this work. The authors apply their model to super-resolution of face images, as well as image completion (aka inpainting) and generation, both unconditioned or conditioned on one of a small number of image classes from the CIFAR10 and ImageNet datasets. The authors evaluate their method in terms of visual quality of their generated images via an Amazon Mechanical Turk survey and quantitatively by reporting slightly improved log-likelihoods. 

While the paper is well written, the motivation for combining self-attention and autoregressive models remains unclear unfortunately, even more though as the reported quantitative improvement in terms of log-likelihood are only marginal. The technical exposition is at times difficult to follow with some design decisions of the network layout being quite ad hoc and not well motivated. Expressing the involved operations in mathematical terms would help comprehend some of the technical details and add to the reproducibility of the proposed model. 

Another concern is the experimental evaluation. While the reported log-likelihoods are only marginally better, the authors report a significant boost in how often humans are fooled by the generated images. While the image generation is conditioned on the low-resolution input, the workers in the Amazon Mechanical Turk study get to see the high-resolution images only. Of course, a human observer would pick the one image out of the two shown images which is more realistic although it might have nothing to do with the input image, which seems wrong. Instead, the workers should see the low-res input image and then have to decide which high-res image seems a better match or more likely.

Overall, the presented work looks quite promising and an interesting line of research. However, in its present form the manuscript doesn't seem quite ready for publication yet. Though, I would strongly encourage the authors to make the exposition more self-contained and accessible, in particular through rigorous mathematical terms, which would help comprehend the involved operations and help understand the proposed mechanism.

Additional comments:
- Abstract: "we also believe to look pretty cool". Please re-consider the wording here. Generating "pretty cool" images  should not be the goal of a scientific work.

review_title: Missing motivation and mathematical details

Meta-review: This paper had some quality and clarity issues and the lack of motivation for the approach was pointed out by multiple reviewers.  Just too far away from the acceptance threshold.
Decision: Reject