Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper focuses on few shot learning with autoregressive density estimation. Specifically, the paper improves PixelCNN with  1) neural attention, 2) meta learning techniques, and shows that the model achieve STOA few showt density estimation on the Omniglot dataset and demonstrate the few showt image generation on the Stanford Online Products dataset. 

The model is interesting, however, several details are not clear, which  makes it harder to repeat the model and the experimental results. For example, what is the reason to use the (key, value) pair to encode these support images, what does the "key" means and what is the difference between "keys" and "values"? In the experiments, the author did not explain the meaning of "nats/dim" and how to compute it. Another question is about the repetition of the experimental results. We know that PixelCNN is already a quite complicated model, it would be even harder to implement the proposed model. I wonder whether the author will release the official code to public to help the community?
review_title: Few shot learning with autoregressive density estimation

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper focuses on the density estimation when the amount of data available for training is low. The main idea is that a meta-learning model must be learnt, which learns to generate novel density distributions by learn to adapt a basic model on few new samples. The paper presents two independent method.

The first method is effectively a PixelCNN combined with an attention module. Specifically, the support set is convolved to generate two sets of feature maps, the so called "key" and the "value" feature maps. The key feature map is used from the model to compute the attention in particular regions in the support images to generate the pixels for the new "target" image. The value feature maps are used to copmpute the local encoding, which is used to generate the respective pixels for the new target image, taking into account also the attention values. The second method is simpler, and very similar to fine-tuning the basis network on the few new samples provided during training. Despite some interesting elements, the paper has problems.

First, the novelty is rather limited. The first method seems to be slightly more novel, although it is unclear whether the contribution by combining different models is significant. The second method is too similar to fine-tuning: although the authors claim that \mathcal{L}_inner can be any function that minimizes the total loss \mathcal{L}, in the end it is clear that the log-likelihood is used. How is this approach (much) different from standard fine-tuning, since the quantity P(x; \theta') is anyways unknown and cannot be "trained" to be maximized.

Besides the limited novelty, the submission leaves several parts unclear. First, why are the convolutional features of the support set in the first methods divided into "key" and "value" feature maps as in p_key=p[:, 0:P], p_value=p[:, P:2*P]? Is this division arbitrary, or is there a more basic reason? Also, is there any different between key and value? Why not use the same feature map for computing the attention and computing eq (7)?

Also, in the first model it is suggested that an additional feature can be having a 1-of-K channel for the supporting image label: the reason is that you might have multiple views of objects, and knowing which view contributes to the attention can help learning the density. However, this assumes that the views are ordered, namely that the recording stage has a very particular format. Isn't this a bit unrealistic, given the proposed setup anyways?

Regarding the second method, it is not clear why leaving this room for flexibility (by allowing L_inner to be any function) to the model is a good idea. Isn't this effectively opening the doors to massive overfitting? Besides, isn't the statement that the function \mathcal{L}_inner void? At the end of the day one can also claim the same for gradient descent: you don't need to have the true gradients of the true loss, as long as the objective function obtains gradually lower and lower values?

Last, it is unclear what is the connection between the first and the second model. Are these two independent models that solve the same problem? Or are they connected?

Regarding the evaluation of the models, the nature of the task makes the evaluation hard: for real data like images one cannot know the true distribution of particular support examples. Surrogate tasks are explored, first image flipping, then likelihood estimation of Omniglot characters, then image generation. Image flipping does not sound a very relevant task  to density estimation, given that the task is deterministic. Perhaps, what would make more sense would be to generate a new image given that the support set has images of a particular orientation, meaning that the model must learn how to learn densities from arbitrary rotations. Regarding Omniglot character generation, the surrogate task of computing likelihood of known samples gives a bit better, however, this is to be expected when combining a model without attention, with an attention module.

All in all, the paper has some interesting ideas. I encourage the authors to work more on their submission and think of a better evaluation and resubmit.


review_title: Official Reviewer 1

final_score: rating: 7: Good paper, accept
review_content: This paper considers the problem of one/few-shot density estimation, using metalearning techniques that have been applied to one/few-shot supervised learning. The application is an obvious target for research and some relevant citations are missing, e.g. "Towards a Neural Statistician" (Edwards et al., ICLR 2017). Nonetheless, I think the current paper seems interesting enough to merit publication.

The paper is well-produced, i.e. the overall writing, visuals, and narrative flow are good. It was easy to read the paper straight through while understanding both the technical details and more intuitive motivations.

I have some concerns about the architectures and experiments presented in the paper. For architectures: the attention-based model seems powerful but difficult to scale to problems with more inputs for conditioning, and the meta PixelCNN model is a standard PixelCNN trained with the MAML approach by Finn et al. For experiments: the ImageNet flipping task is clearly tailored to the strengths of the attention-based model, and the presentation of the general Omniglot results could be improved. The image flipping experiment is neat, but the attention-based model's strong performance is unsurprising. I think the results in Tables 1/2 should be merged into a single table. It would make it clear that the MAML-based and attention-based models achieve similar performance on this task.

Overall, I think the paper makes a nice contribution. The paper could be improved significantly, e.g., by showing how to scale the attention-based architecture to problems with more data or by designing an architecture specifically for use with MAML-based inference.
review_title: A solid paper

Meta-review: This paper incorporates attention in the PixelCNN model and shows how to use MAML to enable few-shot density estimation. The paper received mixed reviews (7,6,4). After rebuttal the first reviewer updated the score to accept. The AC shares the concern of novelty with the first reviewer. However, it is also not trivial to incorporate attention and MAML in PixelCNN, thus the AC decided to accept the paper. 
Decision: Accept (Poster)