Title: Towards a Neural Statistician
Authors: Harrison Edwards
Amos Storkey
Venue: ICLR 2017

Abstract: An efficient learner is one who reuses what they already know to tackle a new problem. For a machine learner, this means understanding the similarities amongst datasets. In order to do this, one must take seriously the idea of working with datasets, rather than datapoints, as the key objects to model. Towards this goal, we demonstrate an extension of a variational autoencoder that can learn a method for computing representations, or statistics, of datasets in an unsupervised fashion. The network is trained to produce statistics that encapsulate a generative model for each dataset. Hence the network enables efficient learning from new datasets for both unsupervised and supervised tasks. We show that we are able to learn statistics that can be used for: clustering datasets, transferring generative models to new datasets, selecting representative samples of datasets and classifying previously unseen classes. We refer to our model as a neural statistician, and by this we mean a neural network that can learn to compute summary statistics of datasets without supervision.

TL;DR: Learning representations of datasets with an extension of VAEs.

Review discussion:
I don't have any major clarification questions -- the exposition seems clear and well motivated.
But I am curious how well you can classify spatial MNIST digits from the summary statistics. Is it competitive with algorithms that are given the original image?
Pedantic question: in what sense can this network be called a "statistician"?  The tasks under consideration don't seem very representative of what statisticians do.
Thanks for your questions!
Classifying an MNIST digit from a sample of its coordinates is considerably more difficult than from an image because it is ambiguous: you don't know if there are no black pixels in a given region or whether they are simply missing from your sample by chance. Using a nearest neighbor classifier on the context variables gives a classification accuracy of around 85%. This is much better than nearest neighbors on raw images, but a representation from a VAE trained on MNIST images, for example, would likely fare better.
We could likely do well if we used some labels for the digits to inform our representations, that is modeling the joint distribution over datasets and dataset labels p(D, y). It may be interesting to investigate such a supervised or semi-supervised approach for 3d shapes given as point clouds in the future. There is some related work on classifying datasets that we mention in the paper, such as the support measure machine, which it would be interesting to compare against in future work.
Regarding your second question, in this context by 'statistician' we mean someone who chooses and computes statistics of datasets. We agree that this covers only a small part of what human statisticians do!
Thanks for the interesting paper, can you clarify more precisely what the "one-shot" task for generative models is? If it is essentially a transfer type task (assign high probability to unseen but valid regions of the space), then why not evaluate the log-probability of these inputs?
Thank you for reading the paper and your question.
The task is really the same for all experiments: produce a summary of each dataset as an approximate posterior over the context variables. We probe these summaries by looking at synthesized samples, and using the representations to do one-shot classification. The log-probability of a new dataset under the model would be hard to interpret given the extensive data augmentation we have used.
The authors introduce a variant of the variational autoencoder (VAE) that models dataset-level latent variables. The idea is clearly motivated and well described. In my mind the greatest contribution of this paper is the movement beyond the relatively simple graphical model structure of the traditional VAEs and the introduction of more interesting structures to the deep learning community.
Comments:
- It's not clear to me why this should be called a "statistician". Learning an approximate posterior over summary statistics is not the only imaginable way to summarize a dataset with a neural network. One could consider a maximum likelihood approach, etc. In general it felt like the paper could be more clear, if it avoided coining new terms like "statistic network" and stuck to the more accurate "approximate posterior".
- The experiments are nice, and I appreciate the response to my question regarding "one shot generation". I still think that language needs to be clarified, specifically at the end of page 6. My understanding of Figure 5 is the following: Take an input set, compute the approximate posterior over the context vector, then generate from the forward model given samples from the approximate posterior. I would like clarification on the following:
(a) Are the data point dependent vectors z generated from the forward model or taken from the approximate posterior?
(b) I agree that the samples are of high-quality, but that is not a quantified statement. The advantage of VAEs over GANs is that we have natural ways of computing log-probabilities. To that end, one "proper" way of computing the "one shot generation" performance is to report log p(x | c) (where c is sampled from the approximate posterior) or log p(x) for held-out datasets. I suspect that log probability performance of these networks relative to a vanilla VAE without the context latent variable will be impressive. I still don't see a reason not to include that.
Thanks for your review and useful comments.
"- It's not clear to me why this should be called a "statistician". Learning an approximate posterior over summary statistics is not the only imaginable way to summarize a dataset with a neural network. One could consider a maximum likelihood approach, etc. In general it felt like the paper could be more clear, if it avoided coining new terms like "statistic network" and stuck to the more accurate "approximate posterior."
Please see our reply to AnonReviewer1. We think the point about statistician is fair, but that statistic network is reasonable. The statistic network is an inference network, the output of which is a parameterization of an approximate posterior over the context latent variable. The use of the word statistic is to remind the reader that the input to this kind of inference network is a dataset and that it's output is invariant to the order of its inputs.
"(a) Are the data point dependent vectors z generated from the forward model or taken from the approximate posterior?"
When generating samples we first take the dataset, pass it through the statistic network to get an approximate posterior over c. We then set c to be the mean of this posterior, and sample from p(x | c). This means that the datapoint encoders q(z | x,c) are not used (which is I think what you are asking). This process is detailed in pseudo-code in the appendix (algorithm 2) and referred to at the beginning of section 5 (Experimental results). Based on your feedback we have made it clearer in the first paragraph of section 5 that this is how we obtain samples in the experiments.
"(b) I agree that the samples are of high-quality, but that is not a quantified statement. The advantage of VAEs over GANs is that we have natural ways of computing log-probabilities."
Our quantitative evidence for our approach consists in our few-shot classification results, complemented by qualitative investigations of the representations learned. Because of some of the choices we have made with regard to data augmentation, it is difficult to make fair comparison in terms of likelihoods. We take your point that it would have been better to have this comparison, and will make this a priority in future investigations of this area.
Sorry for the late review -- I've been having technical problems with OpenReview which prevented me from posting.
This paper presents a method for learning to predict things from sets of data points. The method is a hierarchical version of the VAE, where the top layer consists of an abstract context unit that summarizes a dataset. Experiments show that the method is able to "learn to learn" by acquiring the ability to learn distributions from small numbers of examples.
Overall, this paper is a nice addition to the literature on one- or few-shot learning. The method is conceptually simple and elegant, and seems to perform well. Compared to other recent papers on one-shot learning, the proposed method is simpler, and is based on unsupervised representation learning. The paper is clearly written and a pleasure to read.
The name of the paper is overly grandiose relative to what was done; the proposed method doesn’t seem to have much in common with a statistician, unless one means by that "someone who thinks up statistics".
The experiments are well chosen, and the few-shot learning results seem pretty solid given the simplicity of the method.
The spatial MNIST dataset is interesting and might make a good toy benchmark. The inputs in Figure 4 seem pretty dense, though; shouldn’t the method be able to recognize the distribution with fewer samples?  (Nitpick: the red points in Figure 4 don’t seem to correspond to meaningful points as was claimed in the text.)
Will the authors release the code?
Thank you for your review!
Firstly regarding your comments on 'neural statistician'. We accept your criticism (shared by AnonReviewer3) that 'statistician' is overbroad. We do defend the use of the term statistic network, as this appropriately captures the fact that the inferential network is exchangeable across the elements in the set - i.e. it does not depend on the order that elements are presented.
However extending that to the title "towards a neural statistician was perhaps overenthusiastic, though done with no desire to mislead or slight real statisticians!
On the other hand since the paper has been on arxiv for some time now, and is referred to by name by several other submissions to ICLR, we feel it would be a bit awkward at this point to change the title. I have added a line to the abstract that I hope clarifies the scope of what we mean by neural statistician.
Regarding spatial mnist, we agree that it would make a fun benchmark for future work in this area.
Regarding the number of samples used in the experiment.
It is true that the model can learn useful representations with around 25 samples per digit, I choose the number of samples such that I could nearly always recognize a digit from its samples. If we do use the data as a benchmark in future work I will explore the sample size more carefully.
As to whether the model selects meaningful points: that may have been confirmation bias on my part. I have weakened the language to just say that 'sensible subsets' are often selected.
Finally regarding the code we plan to release it upon acceptance.
This paper proposes a hierarchical generative model where the lower level consists of points within datasets and the higher level models unordered sets of datasets.  The basic idea is to use a "double" variational bound where a higher level latent variable describes datasets and a lower level latent variable describes individual examples.
Hierarchical modeling is an important and high impact problem, and I think that it's under-explored in the Deep Learning literature.
Pros:
-The few-shot learning results look good, but I'm not an expert in this area.
-The idea of using a "double" variational bound in a hierarchical generative model is well presented and seems widely applicable.
Questions:
-When training the statistic network, are minibatches (i.e. subsets of the examples) used?
-If not, does using minibatches actually give you an unbiased estimator of the full gradient (if you had used all examples)?
For example, what if the statistic network wants to pull out if *any* example from the dataset has a certain feature and treat that as the characterization.  This seems to fit the graphical model on the right side of figure 1.  If your statistic network is trained on minibatches, it won't be able to learn this characterization, because a given minibatch will be missing some of the examples from the dataset.  Using minibatches (as opposed to using all examples in the dataset) to train the statistic network seems like it would limit the expressive power of the model.
Suggestions:
-Hierarchical forecasting (electricity / sales) could be an interesting and practical use case for this type of model.
Firstly thank you for taking the time to read our paper and write your review.
-When training the statistic network, are minibatches (i.e. subsets of the examples) used?
In our experiments we typically consider the datasets to be small and use the entire dataset, but if you had large datasets then something like  minibatches would have to be used.
does using minibatches actually give you an unbiased estimator of the full gradient (if you had used all examples)?
You raise a good point, because of the nonlinearity in the statistic network, there can be no guarantee of an unbiased gradient.
As I understand your next question, you are asking about whether this would pose a problem if your datasets are characterized by rare examples in the datasets. I think in this kind of case the statistic network will struggle to learn useful representations if only presented with small numbers of examples. The problem is that it will become computationally infeasible to pass large datasets through the network for every parameter update. One direction towards addressing these kinds of problems is to use small minibatches but to have some persistent state between them.
For instance the statistic network could use a running mean that is updated everytime we pass a minibatch from the same dataset. This is similar to truncated backprogagation through time that is used to handle long sequences in recurrent networks.
Thank you for your suggestion regarding hierarchical forecasting we will look into whether our approach could give some advantage in this area.
"As I understand your next question, you are asking about whether this would pose a problem if your datasets are characterized by rare examples in the datasets. I think in this kind of case the statistic network will struggle to learn useful representations if only presented with small numbers of examples. The problem is that it will become computationally infeasible to pass large datasets through the network for every parameter update. One direction towards addressing these kinds of problems is to use small minibatches but to have some persistent state between them.
For instance the statistic network could use a running mean that is updated everytime we pass a minibatch from the same dataset. This is similar to truncated backprogagation through time that is used to handle long sequences in recurrent networks."
I don't see how having persistent states addresses this issue.  As I see it, the reason why your estimator is biased is because you're optimizing a term including a log of a sum of non-negative values (over a function of examples in the dataset, in the loss L_D in equations 9 and 10).
If your q(c|D) needs to be determined by a single example in the dataset, then I think you'll have an issue where minibatches where it hasn't happened to appear yet will have incorrect gradients.  This will be the case even if you have a persistent state between minibatches.

Meta-review:
role: PC
data: comment: This is an interesting paper that adds nicely to the literature on VAEs and one-shot generalisation. This will be of interest to the community and will contribute positively to the conference.

Decision: Accept (Poster)