Stochastic Prototype Embeddings
Authors: Tyler R. Scott
Karl Ridgeway
Michael C. Mozer
Venue:  (2020)
Keywords: deep embeddings
stochastic embeddings
probabilistic embeddings
deep metric learning
few-shot learning

Abstract: Supervised deep-embedding methods project inputs of a domain to a representational space in which same-class instances lie near one another and different-class instances lie far apart. We propose a probabilistic method that treats embeddings as random variables. Extending a state-of-the-art deterministic method, Prototypical Networks (Snell et al., 2017), our approach supposes the existence of a class prototype around which class instances are Gaussian distributed. The prototype posterior is a product distribution over labeled instances, and query instances are classified by marginalizing relative prototype proximity over embedding uncertainty. We describe an efficient sampler for approximate inference that allows us to train the model at roughly the same space and time cost as its deterministic sibling. Incorporating uncertainty improves performance on few-shot learning and gracefully handles label noise and out-of-distribution inputs. Compared to the state-of-the-art stochastic method, Hedged Instance Embeddings (Oh et al., 2019), we achieve superior large- and open-set classification accuracy. Our method also aligns class-discriminating features with the axes of the embedding space, yielding an interpretable, disentangled representation.

Under review as a conference paper at ICLR 2020
STOCHASTIC PROTOTYPE EMBEDDINGS
Anonymous authors
Paper under double-blind review
ABSTRACT
Supervised deep-embedding methods project inputs of a domain representa- tional space in which same-class instances lie near one another and different-class instances lie far apart. We propose probabilistic method that treats embeddings as random variables. Extending a state-of-the-arr deterministic method, Prototypi- cal Networks (Snell et al. 2017), our approach supposes the existence of a class prototype around which class instances are Gaussian distributed. The prototype posterior a product distribution over labeled instances, and query instances are classified by marginalizing relative prototype proximity over embedding uncer- tainty. We describe an efficient sampler for approximate inference that allows us train the model at roughly the same space and time cost as its deterministic sibling. Incorporating uncertainty improves performance on few-shot learning and gracefully handles label noise and -uttrf-diitribution inputs. Compared the state-of-the-ar stochastic method, Hedged Instance Embeddings (Oh et al.) (201)), we achieve superior large- and open-set classification accuracy. Our method also aligns dlas--disrimiiating features with the axes of the embedding space, yielding interpretable, disentangled representation.
INTRODUCTION
Supervised deep-embedding methods map instances from an input space to a latent embedding space in which same-label pairs are near and different-label pairs are far. The embedding thus captures semantic relationships without discarding inter-class structure. In contrast, consider standard neural network classifier with a softmax output layer trained with a cross-entropy loss. Although its penultimate layer might be treated as an embedding, the classifier's training objective attempts to orthogonalize all classes and thereby eliminate any information about inter-class structure. Supervised embedding methods are critical for large- and open-set classification tasks, and are popular for fhw-shot and lifelong learning tasks.
Nearly all previous methods for deep embeddings are deterministic: an instance projects to point in the embedding space. Deterministic embeddings fail to capture uncertainty due either to out-of- distribution inputs (e.g., data corruption) or label ambiguity (e.g., overlapping classes). Representing uncertainty is important for many reasons, including robust classification and decision making, informing downstream models, interpreting representations, and detecting out-o--distribution samples. In this article, we propose a method for discovering stochastic embeddings, where each embedded instance is random variable whose distribution reflects the uncertainty in the embedding space.
Our proposed method, the Stochastic Prototype Embedding (SPE), is an extension of the Prototypical Network (PN) (Snell al. (017). As in the PN, our SPE assumes each class can be characterized by prototype in the embedding space and an instance is classified based on its proximity to a prototype. In the case of the SPE, the embeddings and prototypes are Gaussian random variables. each class instance is assumed to be Gaussian perturbation of the prototype, and a query instance classified by marginalizing over the embedding uncertainty. Our main contribution is to show that SPE outperforms the only other fully-formulated method for stochastic supervised embeddings. the Hedged Instance Embedding (HIB) (Oh et al., 2019), on a superset of the complete battery of experiments used to justify HIB. SPE is also more computation efficient to train than HIB, with complexity comparable to that of the PN, and has no hand-tuned parameters. We also demonstrate that embedding distributions intriguing emergent property of SPE: that it attains more interpretable representations by disentangling dass-discriminaaive features.
related to label uncertainty and input ambiguity. Finally, we explore an

---

Under review as a conference paper at ICLR 2020
2 RELATED WORK
Supervised embedding methods are popular in the few-shot learning literature (Koch et al. 2015
Vinyals et al. 2016 Snell et al. 2017; Triantafillou et al., 2017, Finn et al. 2017 Edwards and Storkey| [017 Scott et al. 2018 Ridgeway and Mozer 2018 Mishra et al. 2018) where the goal is to classity query instances based on one small number of labeled exemplars of novel classes. These methods operate by embedding the queries and exemplars using pre-trained network, and classifying each query according to its proximity the exemplars. Embedding methods are also critical in open-set recognition domains such as face recognition and person re-identification (Chopra
al. 2005 Li al. 2014 Yi al. 2014 Zheng et al. 2015 Schroff et al. 2015 Liu al. 2015
Ustinova and Lempitsky|| 2 Song et al. 2016 Wang et al. 2017).
Loss functions used to obtain embeddings can be characterized according to the number of instances required to specify a loss. To describe these losses, we will use the notation za for an embedding
of class a. Pairwise losses attempt to minimize within-class distances, Iz. =z,ll. and maximize between-class distances. ||z,, - ا (Chopra et al. 20055 Hadsell et al. [006 Yi et al.
2014). Tripler
losses attempt to ensure within-class instances are e closer than between-clas instances, | - - < ||2, = |l (Schroff et al. 2015 Song et al. 2016 Wang et al. 2017). Quadrupler losses attempt to ensure every within-class par is closer than every between- cass pair, =/= - ==,'1 < <|==" - =a|| (Ustinova and Lempitsky| (2016). Finally, cluster-based losses attempt to use all instances of class (Rippel et 2016 Fort 2017] SSng et al.) |217]] Snell et al., 2017 Ridgeway and Mozer 2018). In particular, the Prototypical Network (Snell et al.] 2017) computes the mean of set of instances of class, _,. and ensures that additional instances of that class, za. satisfy a proximity constraint such
as ||=-- 2. || æ Cluster-based methods represent state-of-the-ar over, in particular, pairwise and triplet losses, as one might expect given the chronology of publications.
Probabilistic emhedding methods have recently appeared. Allen et al. (209)) extend PNs via Bayesian nonparametrics to treat each prototype as a mixture distribution, though they do not allow uncertainty in the embedding space, the critical element of our work. |Vilnis and McCallum (0018) propose method for learning density-based word embeddings, but it is insupervised. Deep Variational Transfer (Belhaj et al., 2018) a generative form of the discriminative model we propose, which requires modeling input distributions; this work tackled the somewhat different problem of covariate shift. The method closest to ours is an extension of PNs that uses a Mahalanobis distance instead of Euclidean distance to assess similarity (Fori. 2017). Although this method lacks probabilistic semantics, it has similarity with SPE (comparison in Appendix G).
Two prior methods have been proposed for discovering stochastic embeddings in supervised setting, i... for few-shot and open-set recognition. The Hedged Instance Embedding (HIB) (Oh et al. 2019) utilizes probabilistic alternative to the contrastive loss and is trained using variational approximation to the information bottleneck principle. HIB is critically dependent on a constant, B, that determines characteristics of an information bottleneck (i.e., how much of the input entropy is retained in the embedding). Choosing this constant is a matter of art. The Network (OPBN) (Karaletsos al., 2016) a generative model that learns a joint distribution over inputs and stucle-provide triplct constraints. The OPBN was not tested on few-shot and open-set recognition because it requires extensions to be applied to classification tasks. In the deterministic setting, |Scott et al. (2018) argue that cluster- based methods outperform pairwise and triplet methods; thus, we have reason to expect that stochastic setting, a cluster- based method like the one we propose in this article, SPE, will outperform pairwise (HIB) and triplet (OPBN) methods.
saicle-Prioritized Belief
3
THE MODEL
The SPE assumes that the latent representation, 2, is a Gaussian RV conditioned on the input, æ::
p( m=) = N2 (2;12,0011
(1)
with mean, Mg. and variance, 2² computed by a deep neural network, similar to a Variational Autoencoder (Kingma and Welling 2014). The classification, y. in turn conditioned on z. with p(y)z) x) taking the same form as in the original PN (Snell et al. (017)). to be described shortly. Given an input, a class prediction is made by marginalizing over the embedding uncertainty:
p(yla:) = p(ul (Mulz)p(z() az)2
(2)
2

---

Under review as a conference paper at ICLR 2020
y
class
P2
Mx
My
(a)
dlass
(b)
P3
Figure 1: (a) Illustration of the stochastic prototype embedding. The model learns mapping from input space, æ, to embedding space, 2, which same-class instances are near and different-class instances are far. Embeddings are represented Gaussian random variables. Prototypes, noted as + symbols in the embedding, are formed via a sonfidence-weightee average of the embeddings of instances known belong to class (support instances). Prototype uncertainty is depicted with the dotted ovals. Given the prototypes. a prediction of class y is made for a query instance by marginalizing softmax prediction over the embedding space. (b) Depiction of intersection sampler.
Informally, Figure D depicts the relationship between the input, latent, and class representations, for formal depiction of SPE, see the chain graph in Appendix FF We train the SPE using the standard paradigm for supervised embedding methods, via sequence of episodes. each with instances of n classes. We split the m n instances into n support examples, defining set S, and (m - k) query examples. The support instances for each class Se E S, are used to determine the class prototype, Pe. and the query instances are evaluated to predict class label (Equation 2
3.1 FORMING CLASS PROTOTYPES
In the SPE, each class y has an associated prototype in the embedding space, Py: and each instance i of class y projects to an embedding, 2i, in the neighborhood of py such that:
p(z:)p,) o(py.o.
(3)
With independent support instances and uniform class priors, Bayes rule obtains the posterior of prototype Py for support set Sy:
p(py)Sy) = Ilis p(zi
(4)
T.,s, p(z, p)dpp
This normalized product of Gaussians is itself Gaussian:
((pylSy)) = N I) with o = (EES,, o=??) and Myy = (Eres, o= pes) (5)
where 62 o² o² and denotes the Hadamard product and {14 of) the outputs of the embedding procedure for input æ. Essentially, the prototype is a confidence-weightedd average of the support instances. This formulation has clear advantage over the deterministic PN, which is premised on an unweighted average, because it de- te-eempaasiees noisy support instances.
3.2 PREDICTION AND APPROXIMATE INFERENCE
Given query embedding, z, and uniform class priors, the class posterior is:
ply|z, S) x N2; 1 I)
(6)
where o oi + of, and H,y and o are the mean and diagonal covariance of P. (Equation
5)
Expressing this posterior in terms of the query æ (i.c., combining Equations 1 and 6
N(2: T dz. plula, S) No2 Mas 1 E.N(z; T2.)
The class distribution is equivalent that produced by the deterministic PN as o² - 0 when o = o for all class pairs (y. y'). However, in the general case, the integral has no closed form solution; thus, we must sample to approximate p x S). both for training and evaluation. We employ samplers, which we refer to naive and intersection.
3

---

Under review as a conference paper at ICLR 2020
3.2.1 NAÎVE SAMPLING
direct approach to approximating the class posterior is express Equation as an expectation. Ez~p(z(a) |p[v|z, S)}, and to replace the expectation with the average over a set of samples. We utilize the reparameterization trick of Kingma and Welling (2014) train the model. Although this Monte Carlo method is the simplest approach, it is sample-ineficient during training, and when the number of samples is reduced, model performance impacted.
3.2.2 INTERSECTION SAMPLING
In Equation 7 the product of Gaussian densities in the numerator can be rewritten:
N
Zz U2: O N (z:
My. T) ) N (z: Pay: Iz N (Hei My. (o o))
where o
=0 + o 2) and May o o (o 2 pz o o My). Substituting Equation
into Equation 7
p(ylx, N My (o) + Iz)) E E2-N(un Σ N(z: 0.1
(9)
By approximating the expectation with samples from N (May, 1) we obtain an elegant impor- tance sampler that focuses the intersection of the input distribution and given class distribution, as illustrated in Figure Dp. During training with a cross-entrops loss, we need only sample for the known (target) class y. As will demonstrate, this method is more robust and significantly more sample efficient than the naïve sampler, requiring only a single sample to train effectively.
4 EXPERIMENTAL RESULTS
We report on three sets of experiments. In Section 4.1 we demonstrate, using a synthetic data set, that SPE infers the generative structure of a domain, disentangles dlas--disriminating features, and provides meaningful estimates of label uncertainty and input noise. In Section 4.2 we show that SPE obtains state-of-the-ant results on few-shot learning via a comparison to its deterministic sibling, PN, the previous state-of-the-arr method. We evaluate in the few-shot learning literature, Omniglot (Lake et al., 2015). In Section 4.3] we show that SPE obtains state-of-the ofee----t-htrrtr-- art results large-set classification via comparison the only other fully developed stochastic method for supervised embeddings, HIB (Oh et al.) (019).. We evaluate on the only data set that Oh al.) (2019) used to explore HIB, multi-digit variant of MNIST. For details regarding network architectures and hyperparameters, see Appendix A and for simulation details, including the choice of initialization for of. see Appendix B
standard data set used to compare methods
4.1 SYNTHETIC COLOR- COT.R-ORINNAAIOON DATA SET
The data consists of 64 64 pixel images of 'L shapes, with four classes that distinguished by orientation, color, or both (Figure 24). Instances are sampled from class-conditional isotropic Gaussian distribution in the generative space. (The isotropy of these qualitatively different dimensions comes from the fact that both can be mapped as directional quantities.) Because classes overlap on both color and orientation dimensions, elicited embeddings should indicate increased uncertainty near class boundaries. Full details of the synthetic data set can be found in Appendix A.2
We trained a two-dimensional, intersection samppiinnnn SPE on samples from this domain, using two instances per class to form prototypes. Classification accuracy of held-out samples is approximately 86%. Accounting for class overlap, Bayes optimal classifier has an accuracy of approximately 87%. For visualization, Figure 2 presents a 5 X 5 array of examples with the class centroids in the corners and the other examples obtained by linear interpolation in the generative space. The resulting embeddings are presented in Figure 2 Although the correspondence between Figures 2 and 2 seems trivial (mirror one set along the horizontal axis to obtain the other set), remember that the input space is 64 x 64 dimensional and the latent space is 2 dimensional. The network has captured the structure of the domain by disentangling the two factors of variation. Further, the embedding variance encodes label ambiguity; instances halfway between classes dimension have maximal
4

---

Under review as a conference paper at ICLR 2020
Class 2
Class
(a)
(b)
(c)
Class 4
Class
Figure 2: (a) Samples from the four classes in our synthetic data set. In each plot, class centroids are circled, along with samples spanning +2 standard deviations in both orientation and color. A sample's transparency is set according to its conditionn likelihood. Both dimensions can be coded directional variables. The class centroids on each dimension are 90" apart with standard deviation of 30%. (b) set of examples, with the four class centroids located in the corners and other examples obtained by linear interpolation in the generative space. (c) The 2D stochastic prototype embedding for the examples in (b). The shape is plotted at the mean of p( la:). and the outlines of the ovals represent equiprobability contours at 0.4 standard deviations.
variance along that dimension. Label ambiguity is one type of uncertainty. An equally important source of uncertainty comes from noisy or out-of-distribttion (OOD) inputs. We examined OOD inputs generated in two different ways. In the left panel of Figure 3 we show the consequence of adding pixel hue noise to the four class centroids. Only one of these centroids is shown along the abscissa, but all four are used to make the graph, with many samples per noise level. The grey and black bars in the graph indicate variance on the horizontal and vertical dimensions the embedding space, respectively. As pixel hue noise increases, uncertainty in color grows but uncertainty in orientation does not. In the right panel of Figure 3 we show the consequence of shortening the leg-length of the shape. Shortening the legs removes cues that can be used both for determining color and orientation. As result, the uncertainty grows on both dimensions.
4.2 OMNIGLOT
The Omniglot data set contains images of labeled, handwritten characters from diverse alphabets. Omniglot is one of the standard data sets for comparing methods in the few-shot learning literature.
The data set contains 1623 unique characters, each with 20 instances. Following (nell et al. (2017), each grayscale image is resized from 32 32 28 28, and we augment the original classes with all 90" rotations, resulting in 6492 total classes. We train PNs and SPEs episodically, where training episode contains 60 randomly sampled classes and query instances per class.
To compare the relative effectiveness of naive and intersection samplers, we train the SPE on Omniglot varying both the sampler and the number of samples drawn per training query, denoted by We evaluate in 1-shor 20-class setting, where shot refers to the number of support examples used to compute each prototype.
[ test classification accuracy as the number of samples drawn per training trial (8) increases. As previously stated, the intersection-sampliin more sample efficient, to the point that the intersection sampler with s = 1 outperforms the na've is consistent across simulations;
SPE is far
sampler with 81. We have verified that the pattern in Figure i consequently, we present only intersection entereetton-sammpinng SPE results in the remainder of the article, and
Mean Embordinn Standard Dovlation
Moan Embording Standard Dovlation
Figure 3: Synthetic data set: uncertainty on the two embed- ding dimensions as becomes more diffi- cult to discern the hue (left) and orientation (right).
Ascitoooll Rential
10.2-
2.30
0.33
0.35
0.31

---

Under review as a conference paper at ICLR 2020
Tagt clssifiiationn accursy
0.51
Test classification accuracy function of number of training samples per query instance for naive-sampling and intersection-samplinn 2D SPE on a 1-shot, 20-class Omniglot task. Performance is a mean over 5 replications of running the model, showing +1 standard error of the mean.
Figure 4:
0.50
0.49
Intersection sampler Naive sampler
0.48
0.47
0.46
0.45
3
27
81
Number of samples drawn (6)
all SPEs are trained with single sample (s = 1) per query. This choice causes the SPE to be on par with the PN in time and space requirements, even though using more samples may boost classification accuracy, as suggested by the trend in Figure
Figure 5 is visualization of a 2D embedding learned by the intersection-sampping SPE on Omniglot. All classes shown in the figure were held-out during training. Omniglot characters clearly vary along more than two dimensions, the synthetic data set. However, we can still interpret the axes of the embedding. The horizontal axis appears to represent character complexity, with single-stroke characters on the left and many-stroke characters the right. The vertical axis appears encode the aspect ratio of the characters, with horizontally extended characters on the bottom and vertically extended characters on the top.
2D SPE cannot learn fully-disentanglee representation
did with
Figure 6 compares the PN and SPE with 2D embeddings Omniglot test classes. Each bar is the mean accuracy across four conditions: 1-shot/5-class, 5-shot/5-class, 1-shot/20-class, and 5-shot/20- class. The first pair of bars perform the standard comparison in which the (1 or 5 instance) support set used to obtain an embedding for each class, prototypes are formed, and query instances are classified. SPE is reliably better than the PN. Because the Omniglot data carefully curated, the instances have little noise and therefore offer little opportunity to leverage SPE's assessment of uncertainty. Consequently, we corrupted instances by masking out rectangular regions of the input, as proposed by Oh et al. (2019). (See Appendix D for details.) The second and third sets of bars in Figure P correspond the situations where the support and query instances are corrupted, respectively. SPE's advantage over PN increases significantly when the support instances are corrupted due to the fact that SPE's confidence- weighted prototypes (Equation 5) discount noisier support examples. Although the SPE is still superior when only the query is corrupted, the benefit is small. We also compared PN and SPE using a 64D embedding, but with high dimensional embeddings, both methods are near ceiling this data set, resulting in comparable performance between the two methods. (See Appendix D for additional results, broken down by condition.)
To emphasize, SPE outperforms the PN, arguably the leading few-shot learning method, especially when inputs are corrupted, at essentially the same computational cost for training. And by providing an estimate of uncertainty associated with embedded instances, the SPE offers the possibility of detecting OOD samples and informing downstream systems that operate on the embedding.
Two-dimensional em- bedding learned by the SPE the Omniglot test set. Each square thumbnail image in the figure is randomly-sample instance from one of 125 randomly-sample classes and the location of the im- age represents the location of class prototype. The images have gray bounding box for visualiza- tion purposes only.
Figure 5:
Aspect Ratio of Character
test
Character Complexity

---

Under review as a conference paper at ICLR 2020
(a) Omniglot Unseen Classes
(b) N- N-Digi: MNIST Seen Classes
[c) Digit MNIST Unseen Classes
70
90
90
80
555
70
50
45
60
60
35
40
40
" Supp. Clean Clean
Supp.
Supp
Clean Supp.
Cornupt Supp.
Clean Supp. CorruptQuery
Supp. Clean
Supp. Clean
Supp.
Figure 6: (a) Comparison of few-shot accuracy on Omniglot test classes for the PN Snell et al. 2017) and our SPE. (b) Comparison of test accuracy on seen classes for and 3-digit MNIST for HIB (Oh et al. 2019) and our SPE. (c) Same as (b) except for unseen classes. In (a)-(c), error bars reflect + standard error of the mean, corrected to remove cross-condition variance (Masson and Loftus
2003
70
40
90
BG
60
70
40
90
50
80
60
20
76
46
96
86
26
76
66
BG
26
75
45
55
85
05
75
45
35
55
B5
D5
53
33
83
23
73
53
33
633
23
48
58
38
06
28
48
58
28
72
42
92
52
32
22
72
92
52
32
22
79
99
19
09
29
79
49
99
19
09
24
74
97
64
47
27
27
71
91
51
01
91
01
21
21
Figure 7: Two-dine two-dimensional embedding learned by the SPE on the 2-digit MNIST test set. A class is specified by two-digit number. In both figures, the location of the class corresponds to the mean of the prototype in the test set using 140 support instances. The digits surrounded by a black border are classes that were not seen during training. In the left and right figures, the prototypes are colored according to the first and second digit of the class, respectively.
4.3 N-DIGIT MNIST
The digigt MNIST data set was proposed to evaluate HIB at 2019); it formed by horizontal concatenation of N MNIST digit images. The resulting images are 28 28N, To compare with HIB, we study 2- and 3-digit MNIST, and use a network architecture identical to that Oh al. (2019) Oh et al. (2019) split the data into training set (with 70% of the total classes), seen test set, and an unseen test set. For 2-digit MNIST, the seen test set has the same 70 of 100 classes as the training and the unseen test set has the remaining 30 classes. For 3-digit MNIST, the training set has 700 classes, the seen and unseen test sets each have a sample of 100 of the 700 seen or 300 unseen classes, respectively. We use the same train and test data splits as Oh et al. (011)), but further divide the training split to include a validation set for early stopping.
views of the 2D embedding learned by the SPE the 2-digit MNIST test set. Each number is a class label; for example, 71, located in the lower left of the embedding, is the class in which the first of the two MNIST digits a and the second is The location of label in the space corresponds to the mean of its prototype. In the left plot, each class is colored according to the first digit. The right plot the same embedding, but each prototype is colored according to the second digit. The SPE learns an incredibly robust factorial representation in which the horizontal dimension represents the first digit of a class and the vertical dimension represents the second digit. A black bounding box indicates the unseen test classes, classes not presented during training. Impressively, the unseen test classes are embedded in exactly the positions where they belong, indicating that the SPE can discover relationships among classes that allow it to generalize classes it has never during training. Furthermore, the embedding has captured inter-class similarity structure by placing visually similar digits close to another. For example, both the vertical and horizontal bands,
Figure shows
7

---

Under review as a conference paper at ICLR 2020
nines (teal) and fours (purple) are adjacent, and fives (brown) and threes (red) are adjacent. HIB discovers a clean decomposition along one dimension (Oh et al. 2019), but the second dimension is somewhat more entangled, suggesting that the SPE learns more robust representation. Additionally, embeddings for the unseen class are not presented for HIB. The ability to sensibly embed novel classes is essential for any model that will be used for open-set recognition few-shot learning. As we show in Appendix F PNs do not obtain a clean compositional structure.
Figure bp. compare N N-digit MNIST test accuracy on seen and unseen classes, respectively! bar is the mean test accuracy across the Cartesian product of conditions specified by the number of MNIST digits in each image, N {2. 3). and the dimensionality of the embedding, D E (2.3). As in the Omniglot simulation, we varied whether support and query instances were clean or corrupted. The SPE outperforms HIB in all six comparisons. In the 24 individual conditions, SPE is worse on only 7. As in the Omniglot simulation, SPE shines best when support instances may be corrupted. (Appendix A..3 provides tabular results by condition, not only for HIB and SPE, but also their deterministic counterparts, contrastive loss and PN. Because the deterministic methods perform consistently worse than the stochastic methods, we omit the deterministic methods from the figure.)
! Each
Whereas SPE is a discriminative model with specified classification procedure, Oh et al. (2019) had the freedom to design one. They use all available data- roughly 140 examples per class and perform leave-one-out -ncarest-neighbor classification. To be consistent with our episodic test procedure, the SPE uses only 50 support instances per class to form prototypes. It is particularly impressive that the SPE, based single stored prototype and approximately 1/3 the labeled data, outperforms a nonparametric method that is able to model arbitrary distributions in latent space.
5 DISCUSSION AND CONCLUSIONS
We proposed the Stochastic Prototype Embedding (SPE) as a method for obtaining supervised embeddings which encode class-label uncertainty in their distributions. Such methods are useful for large- and open-set classification as well as few-shot learning. particularly for domains with ambiguous inputs or label noise. We compared SPE to the only fully-developed alternative method, the Hedged Instance Embedding (HIB), on the complete battery of tasks used to evaluate HIB. On these large-set classification tasks, SPE consistently outperforms HIB. Beyond its performance gains, SPE has no hand tuned parameters, whereas HIB has constant 3 that determines characteristics of an information bottleneck (i.e., how much of the input entropy is retained in the embedding). Although one could simply set 3 0, doing would encourage the net to perform like a softmax classifier and discard all information about inter-class similarity. Such similarities are essential in order to generalize unseen classes (e.g., Figure
SPE, which is a nondeterministic extension of the Prototypical Net (PN), matches or outperforms the PN on few-shot learning. Because the SPE extends the PN, it seems unlikely to fare worse; but because it can handle uncertainty in both the query and support set, it can fare better, particularly when the embedding space is low dimensional and the support instances may be corrupted. Extensions have been proposed to the PN compatible with ours (e.g., Allen et al. 2019); combining methods may potentially attain even stronger few-shot learning performance under uncertainty.
We proposed and evaluated an intersection sampler to train the SPE, which makes the SPE as time and space efficient for training as the deterministic PN, and more efficient for training than HIB, which relies on about 8 samples per item. (Our evaluation method for SPE presently involves drawing 200 samples from the naive sampler, though this conservative decision was arbitrary and not tuned.)
An unanticipated virtue of SPE is its ability to obtain interpretable, disentangled representations (Figures 2 5 7 Because uncertainty is encoded in a diagonal covariance matrix, any classification ambiguity maps to uncertainty in the value of individual features of the embedding. Thus, class- discriminating feature dimensions must align with the principle axes of the embedding space. In contrast to traditional unsupervised disentangling methods, which aim to discover the underlying generative factors of a domain, the SPE obtains a supervised analog in which the underlying class- discriminative factors represented explicitly. This representation facilitates generalization novel unseen classes and is therefore valuable for few-shot and lifelong learning paradigms.
HIB results are from Oh et al. (2019). We thank the authors for providing us results unseen classes, which were included in their publication.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Allen, K. R., Shelhamer, E., Shin, H., and Tenenbaum, J. B. (2019). Infinite Mixture Prototypes for Few-Shot Learning. arXiv e-prints 1902.04552 cs.LG.
Belhaj, M., Protopapas, P., and Pan, W. (2018). Deep Variational Transfer: Transfer Learning through Semi-supervised Deep Generative Models. arXiv e-prints 1812.03123 cs.LG.
Chopra, S., Hadsell, R., and LeCun, Y. (2005). Learning a Similarity Metric Discriminatively, with Application to Face Verification. In JEEE Conference on Computer Vision and Pattern Recognition.
Edwards, H. and Storkey, A. (2017). Towards a Neural Statistician. In International Conference Learning Representations.
Finn, C., Abbeel, P., and Levine, S. (2017). Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1 1126-1135.
Fort, S. (2017). Gaussian Prototypical Networks for Few-Shot Learning on Omniglot. In Second Workshop on Bayesian Deep Learning (NIPS 2017).
Hadsell, R., Chopra, S., and LeCun, Y. (2006). Dimensionality Reduction by Learning an Invariant Mapping. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 2, pages 1735-1742.
Karaletsos, T., Belongie, S., and Rätsch, G. (2016). Bayesian Representation Learning with Oracle Constraints. In International Conference on Learning Representations.
Kingma, D. P. and Welling, M. (2014). Auto-Encoding Variational Bayes. In International Conference on Learning Representations.
Koch, G., Zemel, R., and Salakhutdinov, R. (2015). Siamese Neural Networks for One-Shot Image Recognition. In ICMI. Deep Learning Workshop, volume 2.
Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. (2015). Human-Level Concept Learning through Probabilistic Program Induction. Science, 350(6266))332-13333
Li, W., Zhao, R., Xiao, T., and Wang, X. (2014). DeepRelD: Deep Filter Pairing Neural Network for Person Re-identification In IEEE Conference on Computer Vision and Pattern Recognition.
Liu, J., Deng, Y., Bai, T., Wei, Z., and Huang, C. (2015). Targeting Ultimate Accuracy: Face Recognition via Deep Embedding. arXiv e-prints 1506.07310 CV.
Masson, M. E. J. and Loftus, G. R. (2003). Using Confidence Intervals for Graphically Based Data Interpretation. In Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale, volume 57, pages 203-220.
Mishra, N., Rohaninejad, M.. Chen, X., and Abbeel, P. (2018). A Simple Neural Attentive Meta- Learner. In International Conference on Learning Representations.
Oh, S. J., Gallagher, A. C., Murphy, K. P., Schroff, F., Pan, J., and Roth, J. (2019). Modeling Uncertainty with Hedged Instance Embeddings. In International Conference on Learning Repre- sentations.
Ridgeway, K. and Mozer, M. C. (2018). Learning Deep Disentangled Embeddings With the F-Statistic Loss. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Biancha N., and Garnett, R., editors, Advances in Neural Information Processing Systems 31, pages 185-194, Curran Associates, Inc.
Rippel, O., Paluri, M., Dollar, P., and Bourdev, I.. (2016). Metric Learning with Adaptive Density Discrimination. In International Conference on Learning Representations.
Schroff. F., Kalenichenko, D., and Philbin, J. (2015). FaceNet: A Unified Embedding for Face Recognition and Clustering. In IEFE Conference Computer Vision and Pattern Recognition.

---

Under review as a conference paper at ICLR 2020
Scott, T., Ridgeway, K., and Mozer, M. C. (2018). Adapted Deep Embeddings: A Synthesis of Methods for k-Shot Inductive Transfer Learning. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R., editors, Advances in Neural Information Processing Systems 31, pages 76-85. Curran Associates, Inc.
Snell, J., Swersky, K., and Zemel, R. (2017). Prototypical Networks for Few-shot Learning. In Advances in Neural Information Processing Systems 31, pages 4077-4087.
Song. O., Jegelka, S., Rathod, V., and Murphy, (2017). Deep Metric Learning via Facility Location. In IEEE Conference Computer Vision and Pattern Recognition, pages 2206-2214 14.
Song, H. O., Xiang, Y., Jegelka, S., and Savarese, S. (2016). Deep Metrie Learning via Lifted Structured Feature Embedding. In IEEE Conference on Computer Vision and Pattern Recognition.
Triantafillou, E., Zemel, R., and Urtasun, R. (2017). Few-Shot Learning Through an Information Retrieval Lens. In Advances in Neural Information Processing Systems 31, pages 2255-2265.
Ustinova, and Lempitsky, (2016). Learning Deep Embeddings with Histogram Loss. In Advances in Neural Information Processing Systems 30, pages 4170-4178.
Vilnis, L. and McCallum, A. (2018). Word Representations via Gaussian Embedding. In International Conference on Learning Representations.
Vinyals, O., Blundell, C., Lillicrap, T., Kavukcuoglu, K., and Wierstra, D. (2016). Matching Networks for One Shot Learning. In Advances in Neural Information Processing Systems 30, pages 3630-3638.
Wang, J., Zhou, F., Wen, S., Liu, X., and Lin, Y. (2017). Deep Metric Learning with Angular Loss. In IEEE International Conference on Computer Vision, pages 2612-2620.
Yi, D., Lei, Z., Liao, S., and Li, S. Z. (2014). Deep Metric Learning for Person Re-identification In International Conference Pattern Recognition, pages 34-39.
Zheng, L., Shen, L., Tian, L., Wang, S., Wang. J., and Tian, Q. (2015). Scalable Person Re- identification: A Benchmark. In IEEE International Conference on Computer Vision.
10

---

Under review as a conference paper at ICLR 2020
A NETWORK ARCHITECTURES AND HYPERPARAMETERS
A.1 OMNIGLOT
For all Omniglot experiments, the network consisted of four convolutional blocks. The first three blocks had convolutional layer with 64 filters, a 3 x 3 kernel, zero- soro-padding of length 1, and stride of 1, followed by a batch normalization layer, ReL.U activation, and 2 max max-pooling. The fourth and final block had convolutional layer with 2d filters, a 3 x 3 kernel, zero-padding of length 1, and a stride of 1, followed by 2 x max-poon max-pooling, where d represents the dimensionality of the embedding space. The flattened output of the network is vector of length 2d, where the first d elements were considered the mean of the Gaussian distribution and the remaining d elements were the diagonal covariance entries. The weights were initialized using He initialization and the biases with the following uniform distribution: U(
VTan
Via
All Omniglot models were trained with an initial learning rate of 0.001 which was cut in half every 50 epochs. The models were stopped early using a patience parameter when performance on the validation set no longer increased.
A.2 SYNTHETIC DATA
The images in the synthetic data set are 64 x 64 pixels in size. For orientation, we chose class centers at 90" and 180", with a standard deviation of 30". For color, we manipulated the hue and kept value and saturation constant. Like orientation, hue is circular quantity. If hue ranges from O to 360 degrees, we chose color class centers and standard deviation in the same way as orientation. Additionally, we add noise to minority (15%) of the images used to train the model. For these, add Gaussian noise to the hue of each pixel inside the shape. The standard deviation of the hue noise was chosen uniformly between 18 and 54%. We also added noise to the leg lengths of the I. shapes. The leg length was chosen uniformly between 10% and 98% of its original length. See Figure (3 for some examples.
The network followed an architecture similar to the one we used for Omniglot, except that we added two additional blocks of convolution, batch normalization, ReLU, and max-pooling because the images are larger. We used 2 instances per class to form prototypes and 8 samples per query instance during training. We used a learning rate 0.0001 and the models were stopped early using patience parameter when performance on the validation set no longer increased.
A.3 N-DIGIT MNIST
For all N -dgigi MNIST experiments, we constructed architecture which we believe to be identical to that used for HIB MNIST experiments, based on code provided by the authors (Oh et al. 20T9). The network consisted of two convolutional blocks followed by two fully-conneeted layers. The convolutional blocks each contained a convolutional layer, followed by an ReLU activation, and x max-pooling. The first convolutional layer had 6 filters, 5 x 5 kernel, zero-padding of length and a stride of 1. The second convolutional layer was identical to the first, but had 16 filters instead of 6. The output of the second convolutional block was flattened, passed through fully- fully-connected layer with 120 units, an ReLU activation, and a final fully-connected layer with 2d units, where d represents the dimensionality of the embedding space. Like the Omniglot architectures, the first d entries in the output vector are treated as the mean and the remaining d elements as the diagonal covariance entries. The weights were initialized using a Xavier uniform initialization and biases were initialized to zero
The PN and SPE are trained episodically with all performance results in the main article measured as the mean over 1000 random test episodes. All N N-digit MNIST models were trained with an initial learning rate of 0.001 which was cut in half every 50 epochs. The models were stopped early using a patience parameter when performance on the validation set no longer increased. For 2-digit MNIST, each episode in training, validation, and seen-class testing contained all 70 classes and 50 support instances per class. For testing of unseen classes, each episode contained all 30 classes. For 3-digit MNIST, each episode contained 100 classes and either 20 support instances per class for training/validation or 50 support instances per class for seen- and unseen-class testing.
11

---

Under review as a conference paper at ICLR 2020
B SIMULATION DETAILS
For all SPE models,
a. softplus (7),
where y is trainable parameter. We initialize using the following prescription:
= 1ho
where |S] the number of support examples per episode during training and is the dimensionality of the embedding. We chose this prescription for two reasons: (1) the number of support examples increases, the variance of the prototype distribution approaches zero, so scaling linearly by |S] tends to provide stronger training signal early on, and (2) the amount of noise in the projection of an embedding should scale with the dimensionality of the embedding space as to maintain unit-volume. All models used "o 0.01.
The variance of each dimension i, 22 is guaranteed to be non-negative by using a softplus transfer function.
Whether trained with the naïve or intersection sampler, we evaluate model performance using the na've sampler with 200 samples. This approach ensures that comparing the quality of models based only on the method by which they were trained.
C SPE VARIANTS
We assumed only diagonal covariance matrices in this work. Switching to a full covariance matrix would require matrix inversion, which ordinarily infeasible, but because one purpose of deep embeddings is visualization, there may be interesting cases involving 2D embeddings where the cost of inversion trivial. However, using diagonal covariance matrix causes class-de discriminating features to be aligned with the axes of the latent space, as we argued in the main article, and this alignment is a virtue for interpretation.
12

---

Under review as a conference paper at ICLR 2020
D TABULAR RESULTS
D.1 OMNIGLOT
Table 1: Test classification accuracy (%) on Omniglot with both 2D and 64D embedding for weenn-supprriclla--queyy corupt-supporiclea--nueyy and cleanssupportorrup--ueey
PN is our implementation of Prototypical Networks (Snell et al. 2017). SPE is our model. SPE is trained with intersection sampling (1 sample per trial). Reported accuracy for each experimental configuration is the mean over 1000 random test episodes.
2D CLEAN SUPPORT, CLEAN QUERY
1-SHOT, 5-CLASS
5-SHOT, 5-CLASS
1-SHOT, 20-CLASS
5-SHOT, 20-CLASS
MEAN
PN
75.7
82.6
45.0
55.9
64.8
SPE
76.9
82.3
49.7
55.3
66.1
2D CORRUPT SUPPORT. CLEAN QUERY
1-SHOT, 5-CLASS
5-SHOT, 5-CLASS
1-SHOT, 20-CLASS
5-SHOT, 20-CLASS
MEAN
50.0
65.9
23.6
31.7
42.8
SPE
50.7
73.9
25.6
41.6
48.0
2D CLEAN SUPPORT, CORRUPT QUERY
1-SHOT, 5-CLASS
5-SHOT, 5-CLASS
1-SHOT, 20-CLASS
5-SHOT, 20-CLASS
MEAN
PN
48.9
52.3
21.7
25.6
37.1
SPE
47.8
52.3
22.8
26.8
37.4
64D CLEAN SUPPORT, CLEAN QUERY
1-SHOT, 5-CLASS
5-SHOT, 5-CLASS
1-SHOT, 20-CLASS
5-SHOT, 20-CLASS
MEAN
PN
98.5
99.6
94.9
98.6
97.9
SPE
98.5
99.5
94.9
98.6
97.9
64D CORRUPT SUPPORI, CLEAN QUERY
1-SHOT, 5-CLASS
5-SHOT, 5-CLASS
1-SHOT, 20-CLASS
5-SHOT, 20-CLASS
MEAN
85.0
98.7
68.5
95.6
87.0
SPE
85.7
98.8
69.3
95.7
87.4
64D CLEAN SUPPORT, CORRUPT QUERY
1-SHOT, 5-CLASS
5-SHOT, 5-CLASS
1-SHOT, 20-CLASS
5-SHOT, 20-CLASS
MEAN
PN
80.9
84.9
66.7
74.7
76.8
SPE
80.3
84.8
66.3
74.7
76.5
13

---

Under review as a conference paper at ICLR 2020
D.2 N-DIGIT MNIST
Table 2: Test classification accuracy (%) on and 3-digit MNIST for seaaa-spppo/clln-qqqeeeee
corrupt-support/clenn- -eeeeeeaeannnnn.nee. and dean-suppotteorrupqqqueyy N: number of digits in each image; D: dimensionality of the embedding. Contrastive and HIB results from mommOhen et al. (2019). PN is our implementation of Prototypical Networks (Snell et al. 2017). SPE is our model. SPE is trained with intersection sampling (I sample per trial). Reported accuracy for PN and SPE for each experimental configuration the mean over 1000 random episodes.
CLEAN SUPPORT, CLEAN QUERY
SEEN TEST CLASSES
UNSEEN TEST CLASSES
N=2
N=3
N=2
N=3
D=2
D=3
D=2
D=3
MEAN
D=2
D=3
D=2
D=3
MEAN
CONTRASTIVE
88.2
95.0
65.8
87.3
84.1
85.5
84.8
59.0
85.5
78.7
HIB
87.9
95.2
65.0
87.3
83.9
87.3
91.0
64.4
88.2
82.7
PN
91.1
95.0
65.8
90.6
85.6
82.0
89.5
64.3
89.1
81.2
SPE
93.0
94.2
80.2
89.0
89.1
90.0
80.2
88.2
86.9
CORRUPT SUPPORT, CLEAN QUERY
SEEN TEST CLASSES
UNSEEN TEST CLASSES
2=2
N=3
N=2
N=3
D=2
D=3
D=2
D=3
MEAN
D=2
D=3
D=2
D=3
MEAN
CONTRASTIVE
76.2
92.2
49.5
77.6
73.9
76.5
73.3
42.6
73.2
66.4
HTB
81.6
94.3
54.0
81.2
77.8
80.8
86.7
53.9
81.2
75.7
PN
72.7
93.3
44.6
82.7
73.3
70.9
42.9
79.6
69.9
SPE
92.4
93.8
76.7
87.8
87.7
88.8
86.3
75.4
86.3
84.2
CLEAN SUPPORT, CORRUPT QUERY
SEEN TEST CLASSES
UNSEEN TEST CLASSES
N=2
N=3
N=2
N=3
D=2
D=3
D=2
D=3
MEAN
D=2
D=3
D=2
D=3
MEAN
CONTRASTIVE
51.6
29.3
44.7
42.3
46.3
44.8
26.2
42.0
39.8
HIB
49.9
57.8
31.8
49.9
47.4
53.5
57.0
32.1
50.2
48.2
PN
53.1
61.1
33.8
56.4
51.1
51.1
57.9
33.0
54.8
49.2
SPE
53.7
58.2
40.2
48.1
50.
56.3
56.5
39.3
46.6
49.7
14

---

Under review as a conference paper at ICLR 2020
E
CORRUPTION PROCEDURE
Figure 8: Examples occluded 2-digit se- quences. Occlusion is based on random rectan- gles that black out portions each digit.
3
3 3
0 4
2
b
The algorithm for applying corruption identical the scheme used in Oh et al. (2019). A random rectangular-size occlusion of black pixels was determined by first sampling a patch width, Tez, and patch height, Ly from uniform distribution, Lx, By (0, 28), and then sampling the top-left corner coordinates, TL ~ U(O,, 28 L.), Ty ~ (1, 28 Lg). This resulted in an occlusion of area Lg Ly. Note that if La 0 or Ly 0, the image left unoccluded. Figure shows examples of occluded 2-digit images.
For Omniglot, we only trained/validated on corrupted imagery if the support or corrupted query set. When testing on clean support and clean query, the training and validation sets were left unoceluded. When testing on corrupted imagery, the training and validation sets corrupted each character independently with probability of 0.2.
contained corrupted
The training and validation sets for digit MNIST corrupted each digit of each image independently with probability of 0.2, regardless of test imagery. This matched Oh et al. (2019).
During testing on both data sets, we considered both clean and corrupt support sets, as well clean and corrupt query sets. A clean set was one in which all digits/characters were unoccluded. A corrupt set occluded each digit/character each image according to the procedure described above.
15

---

Under review as a conference paper at ICLR 2020
F VISUALIZATIONS OF EMBEDDINGS DISCOVERED BY PROTOTYPICAL NETWORKS
66
12
4
Figure 9: Two-dimensional embedding learned by the PN on the Omniglot test set. Each square thumbnail image the figure is randomly-sampies instance from of 125 randomly-samples test classes and the location of the image represents the location of the class prototype. The images have a gray bounding box for visualization purposes only.
76
77
76
71
92
75
71
92
75
97
94
96
97
94
93
96
90
91
48
45
91
48
43
45
49
49
41
12
1G
41
10
17
14
30
17
14
30
52
36
19
18
52
36
11
57
5
58
50
11
57
84
53
50
51
55
51
55
29
24
2:1
26
29
24
21
60
67
60
64
61
00
61
00
9 5
06
05
06
01
09
01
04
Figure 10: Two-dimensional embedding learned by the PN on the 2-digit MNIST test set. A class is specified by two-digit number. In both figures, the location of the class corresponds to the mean of the prototype in the set using 140 support instances. The digits surrounded by a black border are classes that were not seen during training. In the left and right figures, the prototypes are colored according to the first and second digit of the class, respectively.
16

---

Under review as a conference paper at ICLR 2020
G GRAPHICAL MODEL UNDERLYING SPE
Figure [1 presents SPE as a chain graph with both directed and undirected edges. The left column of the Figure depicts the relationship between the support set and the prototypes; the plates are instantiated for each class, and each support example within a class, s. The right column depicts the relationship between query instance and its class label.
The notation in the Figure is consistent with the notation in the main article, except that we have split the latent representation into bottom up embedding, 2, and a top-down embedding z'. The two representations are constrained by the corruption relationship mentioned in the main article:
z ~ (z, of or equivalently, z ~ >=((2", oi
The z nodes are included for clarity, but they are deterministic functions of their parents:
zsc - Pe and 2 Py
y
E
E
Sc
Z
Zg
Sc
xsc
xq
Figure 11: Chain graph representing SPE
The model proposed by Fort (2117) shares with SPE the notion that prototypes confidence- weighted average of support instances (Equation B). and the embedding procedure produces a scaling matrix for computing Mahalanobis distance, similar to the Gaussian covariance matrix of SPE. However, Fort (2017) does not treat the embedding as stochastic, for to do so, would need to marginalize over the uncertainty in the embedding to predict class label. This marginalization is the core of probabilistic model and is the critical component of SPE. Fort (2017) also obtain best results with spherical scaling matrix for the Mahalanobis distance, whereas a critical property of SPE is that the uncertainty varies on each dimension of the latent space; our disentangling and uncertainty results all hinge on using more flexible diagonal covariance matrix. And And (2017 7) omits the de noise term which turns out to be critical both to obtain a well-formed probabilistic model (Figure [1 and for the model to work in practice.
17


Reviews:
[{"review_id": "Syx5X7zatB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes stochastic prototype embeddings (SPE) for few-shot learning. The method is an extension of Prototypical Networks (PN, [1]) with Gaussian embeddings. The idea is to take representation uncertainty into account when classifying objects which makes the model more robust to input and label noise. The authors propose an efficient sampling algorithm to train SPE which outperforms naive Monte Carlo sampling. They conduct a range of experiments on few-shot learning tasks on a synthetic dataset, Omniglot and N-digit MNIST and compare to Prototypical Networks and previous stochastic embedding state-of-the-art method HIB [3]; SPE was shown to outperform prior work in most settings. The authors also plot interpretable disentangled representations learned in 2D embedding space. \n\nHowever, in my opinion there are a few weaknesses in the paper in its current state: (1) the clarity of sections 3.1-3.2 could be significantly improved as currently the proposed probabilistic model framework is confusing and not well-defined; (2) the experimental results are provided only for embedding spaces of dimensionality 2-3 which significantly hinders performance of Prototypical Networks compared to having a much higher dimensional embedding space, so it would help to see comparisons of SPE and PN using high-dimensional embeddings; (3) the central idea and proposed model seem to be very close to those of [2] which is mentioned in the related work, so, please, list differences with this prior work in the updated version.\nFor these reasons, I recommend a weak reject for the paper. I explain these issues in more detail below.\n\n(1) SPE model assumes that each embedding z is Gaussian-distributed p(z|x) = N(z; mu_x, sigma^2_x I) where parameters of the distribution mu_x and sigma_x are outputted by an embedding neural network. In equation (3) the author define each class prototype as rho_y = z_i + eps where z_i are instances of class y and eps is noise ~ N(0, sigma^2_eps I), so p(rho_y | z_i) = N(rho_y | z_i, sigma^2_eps I) for every z_i from class y. This definition is confusing to me since rho_y is redefined for each z_i, so it is not clear what the generative process for rho_y is. In the Introduction section it is mentioned “...each class instance is assumed to be a Gaussian perturbation of the prototype” which suggests z_i = rho_y + eps for every z_i, so graphical model would be rho_y -> z_i instead of z_i -> rho_y implied in section 3.1. Please, clarify in the rebuttal which the graphical model is implied in SPE. Is it x_i -> z_i -> rho_y? How is then the distribution of rho_y defined given all z_i from class y?\nFurther, in the equation (4) the likelihood of rho_y given all x_i from class y is proportional to product of p(rho_y | x_i) so the expression was factorized over the condition which is the reflection of the authors’ assumption of consistency of the prototype. p(rho_y | x_1, … x_n) could be also written as proportional to the product of p(x_i | rho) times p(rho) using Bayes rule. It would help if authors could explain the differences between the two possible expressions for p(rho_y | x_1 … x_n) and why the former is chosen.\nAlso, in equation (6) the right-hand-side is p(z | y, S) = p(z | rho_y, S) = N(z | mu_y, \\hat{sigma}^2_y I): why is the variance is \\hat{sigma}^2_y (so extra sigma^2_eps added to sigma^2_y) instead of just sigma^2_y defined in equation (5)? The noise term with sigma^2_eps seems to be already included in sigma^2_y.\n\n(2) In all presented experiments, the dimensionality of the embedding is 2-3 instead of high dimensional embedding spaces in the original Prototypical Network paper which causes a significant drop in performance: e.g., on Omniglot 1-shot 5-class classification, the accuracy dropped from 98.8% to 75.7%. SPE outperformed PN, but an important aspect of the PN model, embedding space dimensionality, was changed from the original in a way that hindered the performance very much, so SPE’s advantage is unclear. It is mentioned in section 4.2 that “We also compared PN and SPE using a 64D embedding, but with high dimensional embeddings, both methods are near ceiling on this data set, resulting in comparable performance between the two methods. (See Appendix D for additional results, broken down by condition.)” but the results on 64D embeddings are not added and discussed in the appendix. If the performance on high dimensional embeddings in the standard setting is similar for SPE and PC, it would help to see the 64D embedding experiment with corrupted support and query data, because if SPE outperformed PN on the corrupted data setting, it would be a fair advantage. It is interesting to see what the 2D embedding space looks like in the visualizations (Figures 2, 5, 7), and SPE is learning interpretable disentangles representations. However, for better classification performance it makes sense to use higher dimensional embedding space to allow learning more expressive representations.\nFor Figures 2c, 5 and 7, it would be helpful to see how the embedding space of PN looks like and compare it visually to the one of SPE, e.g. whether PN has disentangled representations. It seems like we could get a similar embedding space to, for example, Figure 2c and the label uncertainty would come from the roughly equal distance of object representation to different class prototypes. Please, add respective embedding visualizations for PN and explain what the advantages of SPE learned representations over PN’s representations are.\n\n(3) The prior work of [2] referenced in the paper seems to have a very similar method (judging from Figure 2 and equations 5-6 which compute parameters of the prototype distribution, [2]). Please, list the differences with this prior work and provide experimental comparison if possible.\n\n\nOther questions:\n1. In Figure 2c, please plot and highlight the learned distribution of the class prototypes p(rho | S).\n2. Section 4.1: did each object in the training set have only one label or multiple labels (since classes are overlapping)?\n3. Experiments: please add HIB method to synthetic data and Omniglot experiments if possible. It would also be interesting to compare to PN on miniImageNet (section 3.2 of [1]). Another interesting experiment which could show advantage of SPE could be out-of-distribution data detection (e.g., comparing likelihoods p(z | y) for in-distribution representations z and out-of-distribution z).\n\n\n[1] Snell, Jake, Kevin Swersky, and Richard Zemel. \"Prototypical networks for few-shot learning.\" Advances in Neural Information Processing Systems. 2017.\n[2] Fort, Stanislav. \"Gaussian prototypical networks for few-shot learning on omniglot.\" arXiv preprint arXiv:1708.02735 (2017).\n[3] Oh, Seong Joon, et al. \"Modeling uncertainty with hedged instance embedding.\" arXiv preprint arXiv:1810.00319 (2018).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1xNTO8ptS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors propose Stochastic Prototype Embeddings, as a probabilistic extension of Prototype Networks (snell et al, 2017) for few-shot classification. Authors claim their method leads to better few-shot results in the case of corrupted data, and support their claims with a fair number of results over N-Mnist, with 'Hedge Instance Embeddings.\n\n\nI think overall is a good paper as it contains interesting new ideas and represents thorough work, including their empirical validation. However, I believe it is not strong enough for me to recommend acceptance at this point. I hope the authors will address my concerns\n\n1)The derivation of their probabilistic method is not satisfactorily explained from a statistical perspective. Why do they appeal to a product distribution? the kind of formulae the authors obtain (e.g. equation 4) resembles the usual ones for posterior distributions over gaussians. Instead of talking about a 'product distribution' it would be much better if the authors appealed to statistical principles so that their choices reveal themselves as sensible or natural. Additionally, their \"intersection sampling\" seems as rebranding of usual importance sampling. I hope the authors will comment more on the originality of their approach, and if not original, downplay the contribution of the sampler.\n\n2)Although results are solid, but when going through the results section I got the impressions the authors were not clear about what were their ultimate intentions, what they wanted to prove. Key results are presented in Figure 6 but they led me with the following questions that I hope the authors will be able to better respond. Is the main preoccupation about few(or zero)-shot learning? then, is HIB the proper baseline in figure 6 and 7? My concern comes from the fact that (to my understanding) HIB isn't stated in the context of few shot learning so it seems authors are defeating a straw man.  uthors may compare with a more naive baseline for uncertainty modeling. Alternatively, authors may compare with HIB in non-few shot regimes. In any case, I hope the authors will make this point clear.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgLg5vH5H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "By extending prototypical networks, this paper proposes a probabilistic model, i.e., stochastic prototype embedding, that treats embeddings as random variables. The model is very straightforward and easy to understand. The authors make a few assumptions to simplify the problem. For example, the distance between every instance $z_i$ of class $y$ and the class embedding $\\rho_y$ follows a Gaussian distribution, and the a softmax prediction for a query embedding also follows a Gaussian distribution. Combining these, the authors give the class posterior for the query. Since the class posterior involves an integral, the paper employs a naive samplying and an intersection sampling. The intersection sampling seems a bit interesting in the sense that the sampler focuses on the intersection of the input distribution and the class distribution and it is more sample-efficient.\n\nIn general, I think this probabilistic approach is very natural and simple, which is definitely one of the advantages. Compared to the deterministic approach, i.e., prototypical networks, one biggest advantage I can think of is that it will be more robust while dealing with noisy training set (many outliers exist). However, such probabilistic formulation does not seem to be new in terms of visual recognition, although it may be new in few-shot learning.\n\nI also have a few concerns. One of my biggest concerns is the experiment. The experiments conducted in the paper are toy-ish. I can see that the proposed method indeed shows some gains over the prototypical networks on some toy tasks, but one can not really tell whether this approach really works for more realistic settings. For few-shot learning, I believe the authors should try to run experiments on Mini-ImageNet or CUB. A good experimental example to follow is [A Closer Look at Few-shot Classification, ICLR 2019]. It should be easy to run your model in their settings, since they have oper-sourced code.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The consensus of reviewers is that this paper is not acceptable in present form, and the AC concurs.
Decision: Reject