Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper suggests a method for semi-supervised few-shot learning. It is based on prototypical network, but in addition to the supervised loss a regularisation term that encourages unlabelled sample to be closer to the prototypes. This regularisation term is adapted from Haeusser et al. (2017) and is encouraging a random walk on a graph (where nodes are prototypes and unlabelled samples) that begins at a certain prototype to end at the same prototype. 

The SS-FSL is indeed interesting and worth exploring. I also find the random walk on a graph of samples appealing and can potentially give insights on the structure of the embedding space. However, I think the paper is lacking in innovation, it is Haeusser et al. (2017) method applied to SS-FSL. The only difference is the usage of prototypes instead of labeled samples, this difference is also only relevant for 5-shot and not for 1-shot (where the prototype is the labeled sample) so there is only a single experiment (5-shot mini-imagenet) that is truly testing PRWN. Also, the choice of using prototypes (unlike Haeusser et al.) is neither justified theoretically nor empirically. Additionally, it is strange that the reported SS-FSL results are lower than standard FSL SOTA, e.g MetaOptNet (Lee et. al, 2019). 


Update after rebuttal:
After reviewing the authors' response I'm changing my rating from reject to weak-reject  

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper develops a random walk-based method on top of prototypical networks to address the semi-supervised few-shot learning, i.e. when each classification task can access only a few-shot labeled data but many unlabeled data. This paper defines a specific random walk: it walks from a prototype to an unlabeled sample, then walks for several steps between unlabeled samples, and then walk back to some prototype. They then propose two loss terms: one aims to maximize the probability of returning to the same prototype, and the other tries to make each unlabeled data having equal probability to be visited during the random walk. These two terms are added to the original prototypical network training loss and the resulted training procedure involves both the few-shot labeled data and the unlabeled data. Experiments on mini-ImageNet show that the proposed method outperforms the other baselines.

Intuitively, the main idea makes some sense to me. The writing is clear to understand the main idea but can be improved in various places. My major concerns are the motivation and analysis of the proposed random walk and the training algorithm, whose details cannot be found in the paper. The paper uses a lot of spaces to review the existing works and problem settings, but only spends one page (Section 2.2) on the description of their contribution, and the description lacks any in-depth discussion about “why it is designed in this way but not others” or “how this objective helps the few-shot learning”: it simply lists the procedures without convincing explanation.

Detailed comments:

1. It is called “random walks” but is different from the random walks used in classical methods in many key points and thus might not have all the existing properties of random walks. For example, it is not clear whether the negative of the square of Euclidean distance is a valid similarity for random walks. It is also not clear why the random walk probability matrix is computed by softmax instead of normalized Laplacian. Moreover, it fixes the starting nodes and end nodes, and also fixes the propagation steps: this cannot give any guarantee of any convergence (or asymptotic properties, mixing rate, etc) as the original random walks hold. This could be a serious problem because the random walks can stop at any non-stationary state after tau steps, which cannot be used for training. 

2. It is not clear why the two additional loss terms (L_walker and L_visit) can help to improve the prototype training. The claimed motivation of minimizing these two terms cannot be soundly related to the final few-shot classification goal, though each of them independently makes some sense if applied to a different goal. Why does maximizing the probability of a prototype transit to itself give new information of improving few-shot classification? If prototypes are involved in the random walk nodes, why do we need to walk back to itself through many unlabeled nodes rather than simply walk to itself (which has the largest probability)? Why do we need to visit as many unlabeled data as possible? Isn’t the case that we hope to only visit the unlabeled data mostly related to the prototype and its class and rule out other unrelated ones since they introduce noises?

Here is a counterexample, which is a trivial solution that can minimize the two terms but does make any sense here: it is easy to define a model that produces the same prototype for different classes (so L_walker is minimized), and achieve equal distance between all the unlabeled samples (so L_visit is minimized), but this does not help the prototype training at all. 

3. Another primary flaw here is that the training objective and the computational graph during training always change with the random walks, since the trajectory of the random walks keeps changing due to the randomness. Hence, the trained model does not have any guarantee and the convergence in the training is not the real convergence (it is not well-defined in the first place). If the random walks guarantee to converge to some stationary distribution, it might be possible to treat the varying computational graph as a stochastic one which has been analyzed in previous works. Unfortunately, due to the fixed number of steps and other issues mentioned above, this cannot hold. In Figure 2(a), we can see that even the largest tau they have tried cannot make the landing probability converge (not talking about the stationary distribution over all the nodes), which verifies my worries above.

4. Is the final performance sensitive to the hyperparameters lambda and tau? How did you choose their values in the experiments? 

5. The methodology behind most existing semi-supervised few-shot learning methods, as also indicated in this paper’s introduction, is to apply or extend existing semi-supervised learning techniques on top of existing few-shot learning models. In fact, it is quite straightforward to do so and test all the semi-supervised learning techniques on few-shot learning tasks. Hence, I think it is necessary to compare to baselines that directly use, for example, consistency loss from mean teacher method and progressive training on pseudo labeling, or other SSL techniques mentioned in this paper: https://papers.nips.cc/paper/7585-realistic-evaluation-of-deep-semi-supervised-learning-algorithms. 

-------------

Update:

Thanks for the authors' reply! After reading the rebuttal, I still think there are several main issues needed to be clarified or needing further study. Hence, I will keep my rating.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This submission introduces a version of the random-walk regularizer introduced in Haeusser et al. 2017, in the setting of a Prototypical Network for semi-supervised few-shot learning. The authors show that using this regularizer, SOTA results can be obtained, notably on the popular miniImageNet. They also show that the random-walk can be used to successfully tackle the case with distractors.

I'm a bit on the fence for this submission. I could be convinced to accept, due to the impressive effectiveness of the method despite its simplicity. However, the submission is not without faults:

1. There are no results reported on tieredImageNet, despite it being used in the original paper on semi-supervised few-shot learning (Ren et al. 2018).

2. The proposed methods remains a fairly simple extension of the regularizer of Haeusser et al. 2017.

If the authors could address point 1 and add results on tieredImageNet, I would be willing to increase my rating.

Very small note: in the 4th section paragraph, is it possible that p(x_i|p_j) = \Gamma_{i,j}^{(p\rightarrow x)}, should be p(x_i|p_j) = \Gamma_{j,i}^{(p\rightarrow x)}?
review_title: Official Blind Review #2

Meta-review: This paper proposed a semi-supervised few-shot learning method, on top of Prototypical Networks, wherein a regularization term that involves a random walk from a prototype to unlabeled samples and back to the same prototype.  SotA results were obtained in several experiments by using this method.  All reviewers agreed that the novelty of the paper is not such high compared with Haeusser et al. (2017) and the analysis and the experiments could be improved.
Decision: Reject