Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: Paper Summary:

This paper extends TapNet, a few short learning approach, to the setting of semi-supervised few shot learning. For that, it borrows the idea of soft-labeling from Ren et al 2018, a semi-supervised few shot learning algorithm. Experiments over mini and tiered imagenet compare the approach with alternatives.

Review Summary:

The paper could be clearer. The main idea of the paper and its goal/motivation is not concisely given in the intro and abstract. The paper refers to a lot of terminology from TapNet without defining them which makes it difficult to understand. I am not an expert in the field and I have a hard time judging whether the empirical improvement are exceptional but I would advocate against accepting the paper based on clarity alone at this point. Moreover the contribution -- adding soft labeling to TapNet -- seems modest.

Detailed Review:

The abstract should not be a sketch of the proposed algorithm. It should highlight: what is the problem, what is the key idea of what you propose to address it (one sentence), why is it original and better than prior work, what is the foreseen impact of this work. 

The terminology specific to your problem and family of model need to be defined: what is the clustering space? which clustering? what is the classification space? what is a per class network? None of these concepts are defined.

In section 2.2, please rephrase "Nulling of the errors is essentially finding a projection matrix M such that epsilonM = 0 for all n. The resulting M is a matrix whose columns span the task-adaptive classification space" as "We identify M such that (i)  epsilonM = 0 for all n and (ii)  M columns span the task-adaptive classification space". The text make it sounds like your are finding the solution of (i) which happened to also verify (ii) which is not true. For instance, M=0 satistifies (i) but not (ii).
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary
========
This paper tackles the more realistic variant of few-shot classification where a large portion of the available data is unlabeled, both at meta-training time in order to meta-learn a learner capable of fast adaptation, as well as meta-test time for adapting said learner to each individual held-out task. 

Their approach is based on TapNet, a model which constructs a task-specific projection based on the support set of each episode, and then classifies each query according to its distance to each class prototype in the projected space. The projection is computed so that the class prototypes of the episode (averaged support examples) are well-aligned with a set of meta-learned references. Intuitively, those references learn to be far from each other, so that aligning the prototypes with them leads to a space where the episode’s classes are well separated, allowing for easier classification between them.

They then extend this model to incorporate unlabeled examples by performing task projection as follows: 1) the projection is computed so as to align the initial prototypes (computed only using labeled examples) to the meta-learned references. 2) In that projected space, each unlabeled example is assigned a predicted label based on its proximity to the projected prototypes. 3) Then, back at the original space, those predicted labels of the unlabeled examples are used to refine the class prototypes (weighted average as in Ren et al). 4) The projected space so that the *refined* prototypes are best aligned with the meta-learned references. 5) Possibly repeat 2-4 (at meta-test time).

Experimentally, they outperform recent approaches to semi-supervied few-shot classification on standard benchmarks, though not by far. Perhaps more interestingly, their performance degrades less than that of Ren et al as the distractor ratio increases, and they show that their method benefits from additional steps of task adaptation, whereas that of Ren et al reaches its performance limits after the first step of soft clustering.

High-level comments
==================
A) Ablation: An interesting ablation would be: instead of going back and forth between the embedding space and the projected space, the task adaptation happens only in the initially-computed projection space (i.e. the one computed based on the labeled data only). This would amount to: computing the projection space, and then performing a few steps of soft clustering, similar to Ren et al. in that space. This would help determine how beneficial it is to re-compute that projection space according to the current ‘best guess’ of where the class prototypes lie at each iteration. The way I see it, it is an empirical question whether the initially computed projection space already sufficiently separates the classes or not. I assume this would also lead to a more computationally efficient solution?

B) Handling distractors: In the case of distractors, they use an additional centroid (and a corresponding additional reference vector) for the purpose of ‘capturing’ the unlabeled examples that don’t belong to candidate classes. I find the initialization of this strange: this additional centroid is computed as the mean of all unlabeled examples, and the initial projection construction is influenced by a term that matches this centroid to a corresponding reference. This would mean, however, that even the unlabeled examples which do indeed belong to one of the candidate classes end up far from those classes in the projected space, in order to be close to the designated extra reference in that space. We know that this is not ideal, since we assume that some unlabeled examples do belong to the same classes as the labeled ones. Is there a way to quantify how severely this affects the quality of this initial projection? I would also be curious about the meta-learned location of the extra reference. Does it end up being roughly in the center of the references corresponding to the labeled classes?

C) Inference-only baselines. Ren et al. experimented with inference-only baselines: meta-learning happens only using the labeled subset, and the proposed clustering approach only takes place at meta-test time. In this case this would amount to meta-training a standard TapNet and then performing the proposed refinement only in test episodes. This is interesting as it allows to understand the importance of learning an embedding end-to-end for being more appropriate for unlabeled example refinement. It is not obvious that this is required, so I would be curious to see these results too. (This differs from the reported TapNet baseline in that at meta-test time it would make use of the proposed semi-supervised refinement).

Clarity / quality of presentation:
============================
D) A lot of emphasis is placed on the ability of the proposed method to control the degree of task conditioning. I would like to emphasize that this is not something that previous methods lack. Ren et al.’s approach could also perform multiple steps of clustering for example. Whether or not this is beneficial is an empirical question, but I wouldn’t say that the proposed method does something special to “control” how much adaptation is performed.

E) It would have been useful to have a separate section or subsection that explains TapNet, since this is the model that the proposed method is built upon. Instead, the authors blend the explanation of TapNet in the same section as their method which makes it hard to understand the approach and to separate the contribution of this method from TapNet.

G) The length of the paper is over the recommended pages, and I did feel that a few parts were too lengthy or repetitive (e.g. the last paragraph of the introduction described the model in detail. I felt that it would have been more appropriate to give the higher level intuition there and leave the detailed description for the next section). 

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper presents a semi-supervised few-shot learning method that combines the TapNet architecture with the soft-clustering approach in the original semi-supervised few-shot learning method of Ren et al. 2018. Results are reported on mini-ImageNet and tiered-ImageNet, demonstrating superior results to Ren et al. 2018 and some more recent work.

Though the results are strong, I'm personally leaning towards rejecting this submission. The main reason is that the contribution is an arguably simple combination of 2 methods (TapNet + Ren et al. 2018). Specifically, I notice that the difference between the performance of the TapNet and Prototypical Networks baselines (6%, 7.5% for 1 and 5 shot on miniImageNet, and 5% and 3% for 1 and 5 shot on tieredImageNet) is roughly also found in the difference between TAC and Soft- k-Means (Ren et al. 2018). This suggests that most of the benefits found in TAC can simply be explained by the adoption of the TapNet architecture. And I consider this proposed extension to semi-supervised few-shot learning of TapNet to be pretty straightforward.

I don't really expect the authors can improve on the above point in a revision, so I doubt I'll be convinced to change my score. However, if the authors believe that I've missed a subtlety of their method that is less technically straightforward than my analysis suggests, of course I'd like to know.
review_title: Official Blind Review #2

Meta-review: This paper proposes an approach to semi-supervised few-shot learning. In a discussion after the rebuttal phase, the reviewers were somewhat split on this paper, appreciating the advantages of the algorithm such as increased robustness to distractors and the ability to adapt with additional iterations, but were concerned that the contributions over Ren et al were not significant. Overall, the contributions of this paper don't quite warrant publication at ICLR.
Decision: Reject