Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: By extending prototypical networks, this paper proposes a probabilistic model, i.e., stochastic prototype embedding, that treats embeddings as random variables. The model is very straightforward and easy to understand. The authors make a few assumptions to simplify the problem. For example, the distance between every instance $z_i$ of class $y$ and the class embedding $\rho_y$ follows a Gaussian distribution, and the a softmax prediction for a query embedding also follows a Gaussian distribution. Combining these, the authors give the class posterior for the query. Since the class posterior involves an integral, the paper employs a naive samplying and an intersection sampling. The intersection sampling seems a bit interesting in the sense that the sampler focuses on the intersection of the input distribution and the class distribution and it is more sample-efficient.

In general, I think this probabilistic approach is very natural and simple, which is definitely one of the advantages. Compared to the deterministic approach, i.e., prototypical networks, one biggest advantage I can think of is that it will be more robust while dealing with noisy training set (many outliers exist). However, such probabilistic formulation does not seem to be new in terms of visual recognition, although it may be new in few-shot learning.

I also have a few concerns. One of my biggest concerns is the experiment. The experiments conducted in the paper are toy-ish. I can see that the proposed method indeed shows some gains over the prototypical networks on some toy tasks, but one can not really tell whether this approach really works for more realistic settings. For few-shot learning, I believe the authors should try to run experiments on Mini-ImageNet or CUB. A good experimental example to follow is [A Closer Look at Few-shot Classification, ICLR 2019]. It should be easy to run your model in their settings, since they have oper-sourced code.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Authors propose Stochastic Prototype Embeddings, as a probabilistic extension of Prototype Networks (snell et al, 2017) for few-shot classification. Authors claim their method leads to better few-shot results in the case of corrupted data, and support their claims with a fair number of results over N-Mnist, with 'Hedge Instance Embeddings.


I think overall is a good paper as it contains interesting new ideas and represents thorough work, including their empirical validation. However, I believe it is not strong enough for me to recommend acceptance at this point. I hope the authors will address my concerns

1)The derivation of their probabilistic method is not satisfactorily explained from a statistical perspective. Why do they appeal to a product distribution? the kind of formulae the authors obtain (e.g. equation 4) resembles the usual ones for posterior distributions over gaussians. Instead of talking about a 'product distribution' it would be much better if the authors appealed to statistical principles so that their choices reveal themselves as sensible or natural. Additionally, their "intersection sampling" seems as rebranding of usual importance sampling. I hope the authors will comment more on the originality of their approach, and if not original, downplay the contribution of the sampler.

2)Although results are solid, but when going through the results section I got the impressions the authors were not clear about what were their ultimate intentions, what they wanted to prove. Key results are presented in Figure 6 but they led me with the following questions that I hope the authors will be able to better respond. Is the main preoccupation about few(or zero)-shot learning? then, is HIB the proper baseline in figure 6 and 7? My concern comes from the fact that (to my understanding) HIB isn't stated in the context of few shot learning so it seems authors are defeating a straw man.  uthors may compare with a more naive baseline for uncertainty modeling. Alternatively, authors may compare with HIB in non-few shot regimes. In any case, I hope the authors will make this point clear.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper proposes stochastic prototype embeddings (SPE) for few-shot learning. The method is an extension of Prototypical Networks (PN, [1]) with Gaussian embeddings. The idea is to take representation uncertainty into account when classifying objects which makes the model more robust to input and label noise. The authors propose an efficient sampling algorithm to train SPE which outperforms naive Monte Carlo sampling. They conduct a range of experiments on few-shot learning tasks on a synthetic dataset, Omniglot and N-digit MNIST and compare to Prototypical Networks and previous stochastic embedding state-of-the-art method HIB [3]; SPE was shown to outperform prior work in most settings. The authors also plot interpretable disentangled representations learned in 2D embedding space. 

However, in my opinion there are a few weaknesses in the paper in its current state: (1) the clarity of sections 3.1-3.2 could be significantly improved as currently the proposed probabilistic model framework is confusing and not well-defined; (2) the experimental results are provided only for embedding spaces of dimensionality 2-3 which significantly hinders performance of Prototypical Networks compared to having a much higher dimensional embedding space, so it would help to see comparisons of SPE and PN using high-dimensional embeddings; (3) the central idea and proposed model seem to be very close to those of [2] which is mentioned in the related work, so, please, list differences with this prior work in the updated version.
For these reasons, I recommend a weak reject for the paper. I explain these issues in more detail below.

(1) SPE model assumes that each embedding z is Gaussian-distributed p(z|x) = N(z; mu_x, sigma^2_x I) where parameters of the distribution mu_x and sigma_x are outputted by an embedding neural network. In equation (3) the author define each class prototype as rho_y = z_i + eps where z_i are instances of class y and eps is noise ~ N(0, sigma^2_eps I), so p(rho_y | z_i) = N(rho_y | z_i, sigma^2_eps I) for every z_i from class y. This definition is confusing to me since rho_y is redefined for each z_i, so it is not clear what the generative process for rho_y is. In the Introduction section it is mentioned “...each class instance is assumed to be a Gaussian perturbation of the prototype” which suggests z_i = rho_y + eps for every z_i, so graphical model would be rho_y -> z_i instead of z_i -> rho_y implied in section 3.1. Please, clarify in the rebuttal which the graphical model is implied in SPE. Is it x_i -> z_i -> rho_y? How is then the distribution of rho_y defined given all z_i from class y?
Further, in the equation (4) the likelihood of rho_y given all x_i from class y is proportional to product of p(rho_y | x_i) so the expression was factorized over the condition which is the reflection of the authors’ assumption of consistency of the prototype. p(rho_y | x_1, … x_n) could be also written as proportional to the product of p(x_i | rho) times p(rho) using Bayes rule. It would help if authors could explain the differences between the two possible expressions for p(rho_y | x_1 … x_n) and why the former is chosen.
Also, in equation (6) the right-hand-side is p(z | y, S) = p(z | rho_y, S) = N(z | mu_y, \hat{sigma}^2_y I): why is the variance is \hat{sigma}^2_y (so extra sigma^2_eps added to sigma^2_y) instead of just sigma^2_y defined in equation (5)? The noise term with sigma^2_eps seems to be already included in sigma^2_y.

(2) In all presented experiments, the dimensionality of the embedding is 2-3 instead of high dimensional embedding spaces in the original Prototypical Network paper which causes a significant drop in performance: e.g., on Omniglot 1-shot 5-class classification, the accuracy dropped from 98.8% to 75.7%. SPE outperformed PN, but an important aspect of the PN model, embedding space dimensionality, was changed from the original in a way that hindered the performance very much, so SPE’s advantage is unclear. It is mentioned in section 4.2 that “We also compared PN and SPE using a 64D embedding, but with high dimensional embeddings, both methods are near ceiling on this data set, resulting in comparable performance between the two methods. (See Appendix D for additional results, broken down by condition.)” but the results on 64D embeddings are not added and discussed in the appendix. If the performance on high dimensional embeddings in the standard setting is similar for SPE and PC, it would help to see the 64D embedding experiment with corrupted support and query data, because if SPE outperformed PN on the corrupted data setting, it would be a fair advantage. It is interesting to see what the 2D embedding space looks like in the visualizations (Figures 2, 5, 7), and SPE is learning interpretable disentangles representations. However, for better classification performance it makes sense to use higher dimensional embedding space to allow learning more expressive representations.
For Figures 2c, 5 and 7, it would be helpful to see how the embedding space of PN looks like and compare it visually to the one of SPE, e.g. whether PN has disentangled representations. It seems like we could get a similar embedding space to, for example, Figure 2c and the label uncertainty would come from the roughly equal distance of object representation to different class prototypes. Please, add respective embedding visualizations for PN and explain what the advantages of SPE learned representations over PN’s representations are.

(3) The prior work of [2] referenced in the paper seems to have a very similar method (judging from Figure 2 and equations 5-6 which compute parameters of the prototype distribution, [2]). Please, list the differences with this prior work and provide experimental comparison if possible.


Other questions:
1. In Figure 2c, please plot and highlight the learned distribution of the class prototypes p(rho | S).
2. Section 4.1: did each object in the training set have only one label or multiple labels (since classes are overlapping)?
3. Experiments: please add HIB method to synthetic data and Omniglot experiments if possible. It would also be interesting to compare to PN on miniImageNet (section 3.2 of [1]). Another interesting experiment which could show advantage of SPE could be out-of-distribution data detection (e.g., comparing likelihoods p(z | y) for in-distribution representations z and out-of-distribution z).


[1] Snell, Jake, Kevin Swersky, and Richard Zemel. "Prototypical networks for few-shot learning." Advances in Neural Information Processing Systems. 2017.
[2] Fort, Stanislav. "Gaussian prototypical networks for few-shot learning on omniglot." arXiv preprint arXiv:1708.02735 (2017).
[3] Oh, Seong Joon, et al. "Modeling uncertainty with hedged instance embedding." arXiv preprint arXiv:1810.00319 (2018).
review_title: Official Blind Review #2

Meta-review: The consensus of reviewers is that this paper is not acceptable in present form, and the AC concurs.
Decision: Reject