Title: Prototypical Networks for Few-shot Learning
Authors: Jake Snell
Kevin Swersky
Richard Zemel
Venue: ICLR 2017
Keywords: Deep learning
Transfer Learning

Abstract: A recent approach to few-shot classification called matching networks has demonstrated the benefits of coupling metric learning with a training procedure that mimics test. This approach relies on a complicated fine-tuning procedure and an attention scheme that forms a distribution over all points in the support set, scaling poorly with its size. We propose a more streamlined approach, prototypical networks, that learns a metric space in which few-shot classification can be performed by computing Euclidean distances to prototype representations of each class, rather than individual points. Our method is competitive with state-of-the-art one-shot classification approaches while being much simpler and more scalable with the size of the support set. We empirically demonstrate the performance of our approach on the Omniglot and mini-ImageNet datasets. We further demonstrate that a similar idea can be used for zero-shot learning, where each class is described by a set of attributes, and achieve state-of-the-art results on the Caltech UCSD bird dataset.

TL;DR: We learn a metric space in which few-shot classification can be performed by computing Euclidean distances to a single prototype representative of each class.

Review discussion:
There is another paper reporting zero-shot classification on CUB that appears to use the same protocol and get a better (state-of-the-art) result, using text features instead of attributes:
https://arxiv.org/pdf/1605.05395v1.pdf
Also, in the cited paper "Evaluation of Output Embeddings for Fine-Grained Image Classification", the best result is actually 50.1% using GoogLeNet features instead of AlexNet. Do you know how the proposed model performs with GoogLeNet features, or another more modern architecture such as an open-sourced VGG model or Inception model? Was there a reason for using AlexNet? Since the baselines in table 3 are not state of the art, it is not clear why AlexNet features would be used instead of something more modern.
Hi,
Thanks for your question. In light of the linked paper by Reed et al. 2016, we have updated the zero-shot CUB section of our paper with new results. We trained our model on GoogLeNet extracted features, as the DA-SJE/DS-SJE approach does. We continue to use the 312-dimensional continuous attribute vectors. We achieve accuracy of 54.6%, compared to DA-SJE which gets 50.9% on top of attribute vectors.
Cheers,
Jake
*** Paper Summary ***
This paper simplify matching network by considering only a single prototype per class which is obtained as the average of the embedding of the training class samples. Empirical comparisons with matching networks are reported.
*** Review ***
The paper reads well and clearly motivate the work. This work of learning metric learning propose to simplify an earlier work (matching network) which is a great objective.
However, I am not sure it achieve better results than matching networks. The space of learning embeddings to optimize nearest neighbor classification has been explored before, but the idea of averaging the propotypes is interesting (as a non-linear extension of Mensink et al 2013). I would suggest to improve the discussion of related work and to consolidate the results section to help distinguish between the methods you outperform and the one you do not.
The related work section can be extended to include work on learning distance metric to optimize a nearest neighbor classification, see Weinberger et al, 2005 and subsequent work. Extensions to perform the same task with neural networks can be found in Min et al, 09 that purse a goal very close to yours.
Regarding approaches pursuing similar goals with a different learning objective, you cite siamese network with pairwise supervision. The learning to rank (for websearch) litterature with triplet supervision or global ranking losses is also highly relevant, ie. one example "the query" defines the class and the embedding space need to be such that positive/relevant document are closer to the query than the others. I would suggest to start with Chris Burges 2010 tutorial. One learning class
I am not sure the reported results correctly reflect the state of the art for all tasks. The results are positive on Omniglot but I feel that you should also report the better results of matching networks on miniImageNet with fine tuning and full contextual embeddings. It can be considered misleading not to report it. On Cub 200, I thought that the state-of-the-art was 50.1%, when using features from GoogLeNet (Akata et al 2015), could you comment on this?
Overall, paper could greatly be improved, both in the discussion of related work and with a less partial reporting of prior empirical results.
*** References ***
Large Margin Nearest Neighbors. Weinberger et al, 2005
From RankNet to LambdaRank to LambdaMART: An Overview, Chris J.C. Burges, June 23, 2010
A Deep Non-linear Feature Mapping for Large-Margin kNN Classification, Min et al, 09
Thank you very much for your review.
Regarding the CUB result, we ran experiments where we replaced the AlexNet features with GoogLeNet features and obtained a result of 54.6% accuracy (please see Table 3 of our paper revision dated Dec 6, 2016). We believe this truly represents the state of the art at this point in time.
Regarding Omniglot and miniImagenet, we originally reported the non-FCE/fine-tuned matching networks results because we wanted to keep the comparison limited to the basic models themselves.
However, we agree that we should be thorough in our reporting and will include these results as well.
Regarding the relevant literature, both our model and LMNNs are based on Neighborhood Component Analysis and we believe that our model is more similar to NCA than to LMNN, as we do not use a margin-based loss and do not pre-select a set of target neighbors. We believe that our model is closest in spirit to nonlinear NCA (from follow-up work by Salakhutdinov and Hinton, 2007). We are certainly happy to include your provided references from the metric learning literature as they are indeed relevant. We will also include a reference to the learning to rank literature as it relates to siamese networks, although we feel that this criticism applies to the current one-shot learning literature as opposed to just our paper.
This paper proposes an improved version of matching networks, with better scalability properties with respect to the support set of a few-shot classifier.
Instead of considering each support point individually, they learn an embedding function that aggregates over items of each class within the support set (eq. 1). This is combined with episodic few-shot training with randomly-sampled partitions of the training set classes, so that the training and testing scenarios match closely.
Although the idea is quite straightforward, and there are a great many prior works on zero-shot and few-shot learning, the proposed technique is novel to my knowledge, and achieves state-of-the-art results on several  benchmark datasets. One addition that I think would improve the paper is a clearer description of the training algorithm (perhaps pseudocode). In its current form the paper a bit vague about this.
Thanks for taking the time to review our paper.
Including pseudocode is a valuable suggestion -- we will update the paper to include this.
The paper is an extension of the matching networks by Vinyals et al. in NIPS2016.
Instead of using all the examples in the support set during test, the method represents each class by the mean of its learned embeddings. The training procedure and experimental setting are very similar to the original matching networks. I am not completely sure about its advantages over the original matching networks. It seems to me when dealing with 1-shot case, these two methods are identical since there is only one example seen in this class, so the mean of the embedding is the embedding itself.
When dealing with 5-shot case, original matching networks compute the weighted average of all examples, but it is at most 5x cost. The experimental results reported for prototypical nets are only slightly better than matching networks. I  think it is a simple, straightforward,  novel extension, but I am not fully convinced its advantages.
We thank Reviewer 2 for reviewing our paper. This review’s main criticism is that our approach is too similar to the matching networks model proposed by Vinyals et al. 2016. We will update the related work section of our paper to clarify the relationship between prototypical networks and matching nets.
In the meantime, we would like to take this opportunity to elaborate upon some of the benefits of our approach relative to matching networks: better computational efficiency, a simpler form for the classifier, and a straightforward extension to the zero-shot setting.
First and foremost, prototypical networks are computationally more efficient than matching networks. If there are K classes, each with N support examples, then computing distances for a query point will take O(K) time for prototypical networks vs. O(KN) for matching networks. This is indeed a 5x speedup for the 5-shot scenario, which though not enormous is also not insignificant. Even slightly larger scales such as N=10 or N=100 would lead to useful performance gains. We see favorable applications of our approach to tasks such as document tagging where it is important to quickly label new documents without needing to compute distances to multiple support examples per tag.
Second is the simpler expression of our classifier (see Section 3.2 of our paper) compared to the non-parametric form of matching networks. Not only does this require less storage (O(K) vs. O(KN)) but it also sets the stage for future work that explores the prediction of alternate classifiers beyond the linear ones we investigate here.
Finally, our approach affords a straightforward extension to the zero-shot setting that matching networks do not. The matching networks approach is fundamentally about embedding support examples whereas prototypical networks focuses on learning an embedded representation for each class. For the few-shot scenario we chose our representation to be the mean of embedded points, while for the zero-shot scenario we utilized the embedded metadata. Such a generalization of the matching networks approach to zero-shot classification does not exist.

Meta-review:
role: PC
data: comment: The program committee appreciates the authors' response to concerns raised in the reviews. Unfortunately, reviews are not leaning sufficiently towards acceptance. Reviewers have concerns about the relationships of this work to existing work in literature (both in terms of a discussion to clarify the novelty, and in terms of more complete empirical comparisons). Authors are strongly encouraged to incorporate reviewer feedback in future iterations of the work.

Decision: Reject