Title: Fast Adaptation in Generative Models with Generative Matching Networks
Authors: Sergey Bartunov
Dmitry P. Vetrov
Venue: ICLR 2017
Keywords: Deep learning
Unsupervised Learning

Abstract: Despite recent advances, the remaining bottlenecks in deep generative models are necessity of extensive training and difficulties with generalization from small number of training examples.
Both problems may be addressed by conditional generative models that are trained to adapt the generative distribution to additional input data.
So far this idea was explored only under certain limitations such as restricting the input data to be a single object or multiple objects representing the same concept.  
In this work we develop a new class of deep generative model called generative matching networks which is inspired by the recently proposed matching networks for one-shot learning in discriminative tasks and the ideas from meta-learning.
By conditioning on the additional input dataset, generative matching networks may instantly learn new concepts that were not available during the training but conform to a similar generative process, without explicit limitations on the number of additional input objects or the number of concepts they represent. 
Our experiments on the Omniglot dataset demonstrate that generative matching networks can significantly improve predictive performance on the fly as more additional data is available to the model and also adapt the latent space which is beneficial in the context of feature extraction.

TL;DR: A nonparametric conditional generative model with fast small-shot adaptation

Review discussion:
Can you expand on the relationship between your approach and the matching networks of Vinyals et al. (2016)?  You seem to be building on that work, but it's only discussed briefly in the introduction.
Thank you for the question.
That is totally correct, as we mention in the paper our approach is inspired by the matching networks architecture which was originally proposed for supervised learning problems.
The matching procedure described in section 3.1 is mostly similar to the one used in discriminative matching networks.
However, there is a key difference in that since no label information is available to the network we interpolate between conditioning data not in a label space, but rather in a prototype space which is defined by the model itself and is also learned together with a matching (or feature, if you like) space.
This is a good point, our paper would benefit from highlighting these similarities and differences which we will address in future revisions.
Could you clarify what the averaging method is doing in Figure 3 and Table 1? Is this simply averaging the latent representations of the dataset as opposed to using a soft attention mechanism + LSTM?
Yes, that's correct. In the models which are denoted as "average" the matching procedure is replaced by averaging prototype vectors and hence the attention distribution is uniform.
We have just uploaded a new revision of the paper where we tried to make this more explicit.
The paper explores a VAE architecture and training procedure that allows to generate new samples of a concept based on several exemplars that are shown to the model. The proposed architecture processes the set of exemplars with a recurrent neural network and aggregation procedure similar to the one used in Matching Networks. The resulting "summary" is used to condition a generative model (a VAE) that produces new samples of the same kind as the exemplars shown. The proposed aggregation and conditioning procedure are better suited to sets of exemplars that come from several classes than simple averaging.
Perhaps surprisingly the model generalizes from generation conditioned on samples from 2 classes to generation conditioned on samples from 4 classes.
The experiments are conducted on the OMNIGLOT dataset and are quite convincing. An explicit comparison to previous works is lacking, but this is explained in the appendices, and a comparison to architectures similar to previous work is presented.
Thank you for your review.
We agree that explicit comparison to the neural statistician model (also submitted to ICLR) would make the comparison more direct.
We also would like to mention that before submission we contacted authors of the paper and discussed with Harrison Edwards the issues of estimating conditional log-likelihoods. At that moment, noone of us came up with a reliable solution so we had to resort to closest “tractable analog” as a baseline that could allow us to demonstrate the benefits of our model.
This paper proposes an interesting idea for rapidly adapting generative models in the low data regime. The idea is to use similar techniques that are used in one-shot learning, specifically ideas from matching networks. To that end, the authors propose the generative matching networks model, which is effectively a variational auto-encoder that can be conditioned on an input dataset. Given a query point, the model matches the query point to points in the conditioning set using an attention model in an embedding space (this is similar to matching networks). The results on the Omniglot dataset show that this method is successfully able to rapidly adapt to new input distributions given few examples.
I think that the method is very interesting, however the major issue for me with this paper is a lack of clarity. I outline more details below, but overall I found the paper somewhat difficult to follow. There are a lot of details that I feel are scattered throughout, and I did not get a sense after reading this paper that I would be able to implement the method and replicate the results. My suggestion is to consolidate the major implementation details into a single section, and be explicit about the functional form of the different embedding functions and their variants.
I was a bit disappointed to see that weak supervision in the form of labels had to be used. How does the method perform in a completely unsupervised setting? This could be an interesting baseline.
There is a lack of definition of the different functions. Some basic insight into the functional forms of f, g, \phi, sim and R would be nice.
Otherwise it is very unclear to me what’s going on.
Section 3.2: “only state of the recurrent controller was used for matching”, my reading of this section (after several passes) is that the pseudo-input is used in the place of a regular input. Is this correct?
Otherwise, this sentence/section needs more clarification. I noticed upon further reading in section 4.2 that there are two versions of the model: one in which a pseudo input is used, and one in which a pseudo input is not used (the conditional version). What is the difference in functional form between these?
That is, how do the formulas for the embeddings f and g change between these settings?
“since the result was fully contrastive we did not apply any further binarization” what does it mean for a result to be fully contrastive?
For clarity, the figures and table refer to the number of shots, but this is never defined. I assume this is T here. This should be made consistent.
Figure 2: why is the value of T only 9 in this case? What does it mean for it to be 0? It is stated earlier that T should go up to 20 (I assume #shot corresponds to T). It also looks like the results continue to improve with an increased number of steps, I would like to see the results for 5 and maybe 6 steps as well. Presumably there will come a point where you get diminishing returns.
Table 1: is the VAE a fair baseline? You mention that Ctest affects Pd() in the evaluation. The fact that the VAE does not have an associated Ctest implies that the two models are being evaluated with a different metric. Can the authors clarify this? It’s important that the comparison is apples-to-apples.
MNIST is much more common than Omniglot for evaluating generative models. Would it be possible to perform similar experiments on this dataset? That way it can be compared with many more models.
Further, why are the negative log-likelihood values monotonically decreasing in the number of shots?
That is, is there ever a case where increasing the number of shots can hurt things? What happens at T=30? 40?
As a minor grammatical issue, the paper is missing determiners in several sentences. At one point, the model is referred to as “she” instead of “it”. “On figure 3” should be changed to “in figure 3” in the experiments section.
First, we would like to thank you for a thoughtful review of our work.
We definitely agree that bringing more clarity would improve the paper, so we revised the text based on the provided feedback.
We would like to address the issues raised in your review one by one.
>There are a lot of details that I feel are scattered throughout, and I did not get a sense after reading this paper that I would be able to implement the method and replicate the results. My suggestion is to consolidate the major implementation details into a single section, and be explicit about the functional form of the different embedding functions and their variants.
>There is a lack of definition of the different functions. Some basic insight into the functional forms of f, g, \phi, sim and R would be nice.
Otherwise it is very unclear to me what’s going on.
By no means we wanted to hide any implementation details.
Hence, we released the code to reproduce our results on Github, but of course a paper should also be self-contained.
Section 3 contains an outline of the model while implementation details and concrete functional forms of functions f, g and psi as well as missing information is provided in Appendix A which we now refer to in the beginning and the end of the section.
>I was a bit disappointed to see that weak supervision in the form of labels had to be used. How does the method perform in a completely unsupervised setting? This could be an interesting baseline.
We agree that completely unsupervised approach for training would be much more appealing.
The most straightforward way of doing that is mentioned in the beginning of section 3.3 which is maximization of marginal likelihood of the whole dataset.
It is infeasible on modern widely-available hardware as it would require an enormous amount of memory and computations even for a relatively small dataset such as Omniglot.
However, it is likely that it would lead to even better results with our model as it would have to perform more active discrimination to match relevant objects and and the same time more relevant objects would be available.
Hence, models such as the neural statistician (Amos & Storkey, 2016) and sequential generative models (Rezende et al, 2016) which are closest to our model have to put certain limitations on the datasets they are modelling during the training.
For example, forming a dataset with examples of the same class which is used in (Amos & Storkey, 2016) is maybe even stronger form of supervision which we certainly relax in generative matching networks thus making a step towards less supervision.
>Section 3.2: “only state of the recurrent controller was used for matching”, my reading of this section (after several passes) is that the pseudo-input is used in the place of a regular input. Is this correct?
Otherwise, this sentence/section needs more clarification. I noticed upon further reading in section 4.2 that there are two versions of the model: one in which a pseudo input is used, and one in which a pseudo input is not used (the conditional version). What is the difference in functional form between these?
That is, how do the formulas for the embeddings f and g change between these settings?
We made this place clear in the paper.
There is no functional difference betweens these two versions of the model.
The one that uses a pseudo-input simply sums over T+1 objects in eq. (6) where the last additional object is pseudo-input. It is not necessary to actually have an actual input object, so instead we just model activations of functions f, g and \psi for that pseudo-input as trainable parameters (see Figure 1).
>“since the result was fully contrastive we did not apply any further binarization” what does it mean for a result to be fully contrastive?
This means that the result was already binary, i.e. pixel values were either 0 or 255.
> For clarity, the figures and table refer to the number of shots, but this is never defined. I assume this is T here. This should be made consistent.
We changed the notation, it is now referred to as number of conditional objects.
So it is not exactly T, but rather T-1.
> Figure 2: why is the value of T only 9 in this case? What does it mean for it to be 0? It is stated earlier that T should go up to 20 (I assume #shot corresponds to T).
We performed this experiment with T=10 to qualitative assess dependence on the number of attention steps without spending much time to train large number of models.
> It also looks like the results continue to improve with an increased number of steps, I would like to see the results for 5 and maybe 6 steps as well. Presumably there will come a point where you get diminishing returns.
Yes, you are right. We actually tried larger number of steps but they lead to insignificant improvements, so we decided that more parameter configurations would rather mess up the plot.
One can see that even between “steps=4, prior steps=4” and “steps=4, prior steps=1” there is very little difference on the figure.
> Table 1: is the VAE a fair baseline?
Yes, we think that VAE is a fair baseline in the sense that as all other models in our comparison it defines a marginal distribution over observed objects.
Of course it has less available information than other models as it cannot be conditioned on an additional dataset, but this is exactly why it makes sense to compare against it.
For example, there could be a situation when generative matching networks perform worse than just VAE when no or just a few conditioning objects are available which certainly would be undesirable.
> MNIST is much more common than Omniglot for evaluating generative models. Would it be possible to perform similar experiments on this dataset? That way it can be compared with many more models.
Of course, MNIST is a much more standard dataset, but comparing to Omniglot it is unsuitable for evaluating few-shot learning systems.
In particular, it contains just 10 classes which are shared between train and test parts, so it is unclear what experimental protocol would demonstrate an ability to adapt to unseen classes in such a setting.
The lack of diversity implies that t is not beneficial for the model to learn how to learn on the fly because it is easier to “memorize” class information directly into weights.
Please note that we still evaluate our model on MNIST in Appendix B.
> Further, why are the negative log-likelihood values monotonically decreasing in the number of shots?
That is, is there ever a case where increasing the number of shots can hurt things?
Assuming that our model can efficiently re-use conditioning examples belonging to the same class as the new object thus discriminating between classes, conditional likelihood of a new object x_t may decrease relative to conditional likelihoods of a previous object x_{t-1} only if x_t belongs to a class that was not represented so far in the dataset.
So it is definitely possible to construct a dataset that conditional log-likelihoods would be non-monotone, e.g. the following sequence of objects belonging to classes A and B: A B A A B.
Our data-generating process consists of uniform selection of C classes, followed by uniform sampling of objects corresponding to the selected classes.
Due to uniformity of the sampling processes, the probability that x_t belongs to a new class decreases with t and thus in expectation conditional likelihoods p(x_t | x_{<t}) must increase.
> What happens at T=30? 40?
Unfortunately, our computational resources don’t allow us to train models on larger datasets than T=20.
However, the model should benefit from larger number of relevant conditional objects.
This paper presents a meta-learning algorithm which learns to learn generative models from a small set of examples. It’s similar in structure to the matching networks of Vinyals et al. (2016), and is trained in a meta-learning framework where the inputs correspond to datasets. Results are shown on Omniglot in terms of log-likelihoods and in terms of generated samples.
The proposed idea seems reasonable, but I’m struggling to understand various aspects of the paper. The exposition is hard to follow, partly because existing methods are described using terminology fairly different from that of the original authors. Most importantly, I can’t tell which aspects are meant to be novel, since there are only a few sentences devoted to matching networks, even though this work builds closely upon them. (I brought this up in my Reviewer Question, and the paper has not been revised to make this clearer.)
I’m also confused about the meta-learning setup. One natural formulation for meta-learning of generative models would be that the inputs consist of small datasets X, and the task is to predict the distribution from which X was sampled.
But this would imply a uniform weighting of data points, which is different from the proposed method. Based on 3.1, it seems like one additionally has some sort of query q, but it’s not clear what this represents.
In terms of experimental validation, there aren’t any comparisons against prior work. This seems necessary, since several other methods have already been proposed which are similar in spirit.
Thank you for your review and the patience with which you assessed our paper.
The paper in it’s pre-review form definitely had problems with clarity in certain parts.
We have rewritten them based on the useful feedback provided by reviewers.
Besides that, we would like to address other issues mentioned in your review.
> Most importantly, I can’t tell which aspects are meant to be novel, since there are only a few sentences devoted to matching networks, even though this work builds closely upon them. (I brought this up in my Reviewer Question, and the paper has not been revised to make this clearer.)
The originally proposed matching networks were designed for supervised learning problems. In order to e.g. classify an image the model interpolated between conditioning objects in the one-hot label space.
In contrast, we operate in an unsupervised learning scenario where label information is not available.
Hence, generative matching networks interpolate in a prototype space Psi which is itself defined by the model.
Moreover, in contrast to discriminative (original) matching networks, our model employs the matching procedure in more than just one part of the network. We use it in the prior p(z | X), the conditional likelihood p(x | z, X) and the approximate posterior q(z | x, X).
> I’m also confused about the meta-learning setup. One natural formulation for meta-learning of generative models would be that the inputs consist of small datasets X, and the task is to predict the distribution from which X was sampled.
But this would imply a uniform weighting of data points, which is different from the proposed method.
Your intuition is correct, we implicitly aim to model the distribution of a sampled small dataset.
However, the weights you are mentioning are different from the similarity weights used in equations (5) and (6). The similarity weights depend on a query (be it value of the latent variable z or an observation x), hence they are non-uniform.
However, when sampling from a data-depending prior p(z | X), in average all objects from X should be used with the same frequency in order to maximize the objective.
> Based on 3.1, it seems like one additionally has some sort of query q, but it’s not clear what this represents.
Since the matching procedure is used in all parts of our model, namely prior, likelihood and variational approximation where different information is used to match conditioning data, we decided to first describe the generic matching procedure that operates with an abstract query and then specify the particular form of a query for each part.
Thus, the example of what may be a query is provided in the second paragraph of section 3.1 where we introduce the matching procedure and in the next section we describe the details.
You may also refer to the post-review revision of the paper where we, hopefully, made this more clear.
> In terms of experimental validation, there aren’t any comparisons against prior work. This seems necessary, since several other methods have already been proposed which are similar in spirit.
To the best of our knowledge, the existing class of conditional generative models that explicitly learn fast adaptation mechanisms instead of relying on computationally hard (and thus arguably fast) inference currently consists of two models: sequential generative models (Rezende et al, 2016) and the neural statistician (Amos & Storkey, 2016).
As we mention in section 4.2, sequential generative models unfortunately are reported to overfit in the one-shot generation scenario when using the canonical train/test split of Omniglot.
We also explain that does not seem to be feasible to evaluate the full neural statistican model as it is described in the original paper due to intractable predictive density (see appendix D for the details).
Please note, the authors also didn’t report loglikelihood evaluation in their paper and we have contacted with them to discuss the evaluation and at the moment we could not find the reliable way of estimating the predictive density.
Hence, we implemented the similar model that does not have a global “context” variable causing the intractability, but otherwise uses the way of summarizing the conditioning data and adaptation.
We would like to thank the reviewers for their efforts!
We received many useful comments that helped us to improve the text.
Using the provided feedback, we have edited the paper and uploaded a new version which, hopefully, addresses the issues raised in reviews.
We have also sent our response to all three reviews and would be glad to continue the discussion.

Meta-review:
role: PC
data: comment: This work extends variational autoencoders to adapt to a new dataset containing a small number of examples. While this work is promising, two of the reviewers had serious concerns about clarity. A new version of the paper has been submitted, however I still find it too hard to follow and would find it hard to accurately describe what was done having read the main body of the paper.

Decision: Reject