Semi-Supervised Few-Shot Learning with a Controlled Degree of Task-Adaptive Conditioning
Authors: Sung Whan Yoon
Jun Seo
Jaekyun Moon
Venue:  (2020)
Keywords: few-shot learning
meta-learning
semi-supervised learning
task-adaptive clustering
task-adaptive projection space

Abstract: Few-shot learning aims to handle previously unseen tasks using only a small amount of new training data. In preparing (or meta-training) a few-shot learner, however, massive labeled data are necessary. In the real world, unfortunately, labeled data are expensive and/or scarce. In this work, we propose a few-shot learner that can work well under the semi-supervised setting where a large portion of training data is unlabeled. Our method employs explicit task-conditioning in which unlabeled sample clustering for the current task takes place in a new projection space different from the embedding feature space. The conditioned clustering space is linearly constructed so as to quickly close the gap between the class centroids for the current task and the independent per-class reference vectors meta-trained across tasks. In a more general setting, our method introduces a concept of controlling the degree of task-conditioning for meta-learning: the amount of task-conditioning varies with the number of repetitive updates for the clustering space. During each update, the soft labels of the unlabeled samples estimated in the conditioned clustering space are used to update the class averages in the original embedded space, which in turn are used to reconstruct the clustering space. Extensive simulation results based on the miniImageNet and tieredImageNet datasets show state-of-the-art semi-supervised few-shot classification performance of the proposed method. Simulation results also indicate that the proposed task-adaptive clustering shows graceful degradation with a growing number of distractor samples, i.e., unlabeled samples coming from outside the candidate classes.

Under review as a conference paper at ICLR 2020
SEMI-SUPERVISED FEW-SHOT LEARNING WITH A CONTROLLED DEGREE OF TASK-ADAPTIVE CONDI- TIONING
Anonymous authors Paper under double-blind review
ABSTRACT
Few-shot learning aims to handle previously unseen tasks using only small amount of new training data. In preparing (or meta-training) few-shot learner, however, massive labeled data are necessary. In the real world, unfortunately, labeled data are expensive and/or scarce. In this work, we propose a few-shot learner that can work well under the sumi-supervised setting where large portion of training data is unlabeled. Our method employs explicit task-coniiioning in which unlabeled sample clustering for the current task takes place in new projection space different from the embedding feature space. The conditioned clustering space is linearly constructed so as to quickly close the gap between the class centroids for the current task and the independent per-class reference vectors meta-trained across tasks. In a more general setting, our method introduces a concept of controlling the degree of task conditiniiio for meta- veta-learning: the amount of task condiiiiiii varies with the number of repetitive updates for the clustering space. Extensive simulation results based on the minilmageNee and migredl datasets show state-of-the. art semi- semi-superrs supervised few-shot classification performance of the proposed method. Simulation results also indicate that the proposed task-adaptive clustering shows graceful degradation with growing number of distractor samples, i.e., unlabeled samples coming from outside the candidate classes.
INTRODUCTION
Deep neural networks rely critically massive training with large amounts of annotated data. When training data is scarce and/or unlabeled, which is typically the case in practice, current machine learning algorithms often struggle, failing to optimize deep neural networks. On the other hand, humans can quickly learn new concepts even from noisy and limited experiences. Training machines to learn new tasks rapidly from small amount of labeled data samples continues to be daunting challenge in realizing human-like adaptability in machines.
In tackling related problem in computer vision, many researchers have worked on few-shot learning algorithms, which aim to generalize models for classifying novel classes using only a small number of example images. popular way of developing few-shot learners is to apply ample initial learning or meta-training in episodic form (Vinyals et al. 2016). where the learner is exposed to a large number of widely varying tasks one by one, each time with just few labeled samples. Built upon the strategy of episodic training and the concept of meta-learnine or learning to learn, successful few-shot learners strive achieve both proper inductive bias and task-specific adaptability.
Some methods rely on meta-training the base model without explicit task deppndent conditioning at few-shot- based evaluation time (Koch al. 2015) Vinyals et al.) 2016, Snell et al., 2017]] Mishra et al. 2018, (Sung et al. 2018 Allen et al., 2019). Of these, Prototypical Networks of (Snell et al. 2017) train single embedder such that its per-class averages of the features act as prototypes for representing given tasks. Despite its simplicity, this method has consistently produced relatively strong few-shot learning results. Some other algorithms try to build good inductive bias with no fine-tuning at evaluation time, while also employing additional networks on top of the base feature extractor. Temporal convolutional networks with soft attention (Mishra et al., 2018) and relation modules (Sung et al. 2018) are well-known examples. On the other side of the spectrum, there

---

Under review as a conference paper at ICLR 2020
algorithms rely rely heavily task-specific adaptation via explicit learning of model parameters
deployment (Finn et al. [017]] Ravi & Larochelle (017). These approaches first build a meta-trained initialization model and then execute number of model parameter updates given new task. There
also exist few-shot learners that employ explicit task-condiiioning without direct model parameter
updates at evaluation time (Oreshkin al. 2018 [ al. 2019 Liu et al. 2019 Yoon et al. (219)) For Transductive Propagation Networks (TPNs) of (Liu et al. 2019). a graph construction module trained across tasks by repetitively forming an episode-specific graph. On the task-dependent graph, which shows relationship between embedded samples, labels from support samples propagate to query samples for predicting query labels. In (Oreshkin et al.] 2018). conditionn is done through scaling and shifts of feature vectors at individual layers of the embedding network. More recently, explicit task-conditioning
(Yoon et al., 2019) takes the form of task aask-adaetive projection networks (TapNets), where new classification space is linearly constructed by quickly zeroing the gap between the embedded output of the current task and the independent reference vectors learned across tasks during meta- veta-traiinigg In yet another type of task-adaptation (Li et al. 2019), model learns to generate the parameters of a task-specific matching module which processes and compares embedded support and query samples.
While recent advances in few-shot learning algorithms have enabled steady performance improve- ments, they invariably rely on heavy meta-training with diverse and massive labeled sample images, which is not practical. Semi-supervised few-shot learning algorithms tackle this challenge by allow- ing a learner to be trained using largely unlabeled data sets. The goal of semi-supervised few-shot learning is to acquire generalization capability by effectively utilizing abundantly available unlabeled data. In an early attempt (Ren et al. 2018), using the base Prototypical Networks, semi-supervised clustering is proposed for leveraging unlabeled examples. There, the per-class prototypes, the class centroids of the embedded features, are first obtained only with labeled samples, after which clusters are formed using unlabeled samples in the embedded space around these prototypes. Based on the distances of each unlabeled sample to the prototypes, soft labeling is done. The original prototypes then refined by the soft labels, and in turn used for classifying the queries. The key idea of the approach is that the embedding space is meta- meta-trained for both clustering of unlabeled samples to adjust the prototypes and classifying the query samples around them. In a recent work of (Sun et al. 2019), a model learns to self-train a few-shot learner using unlabeled samples. There, unlabeled samples with high confidence scores are cherry-picked and the self-training weights of the selected unlabeled samples are computed by a meta-trained module, Then, the learner is self-trained through fine-tuning using the selected samples and computed weights.
We also focus on semi-supervised few-shot learning but our approach offers unique lask-conditioning algorithms to utilize unlabeled samples that can work well under crucial label-deficient settings. While we utilize TapNets of (Yoon et al. 2019) as our base architecture, we propose a novel way of constructing an alternative task-adaptive clustering (TAC) space where the embedded features of unlabeled samples are further projected and clustered. In our TAC method, the projection space where clustering occurs is reconstructed repetitively and is in general different from the final classification space. This repetitive projection, given new episode, also allows controlling the degree of task- conditioning during the evaluation phase. The gain we achieve using our method goes well beyond what is possible with a simple application of baseline TapNets.
Extensive evaluation based on partially labeled minilmageNet and rieredImageNet datasets show that our TAC algorithm achieves best accuracies with considerable margins, with or without the unlabeled distractor samples. We also suggest a more realistic and challenging test environment with a growing portion of distractor samples in every episode. In our test setup, the number of unlabeled samples per class is not uniform as well, which is more reflective of real world settings. Our TAC algorithm exhibits graceful degradation with a growing portion of distractor samples when compared with the simpler Prototypical Networks (Ren et al. 2018).
2

---

Under review as a conference paper at ICLR 2020
2
CLUSTERING FOR SEMI-SUPERVISED FEW-SHOT
LEARNING
2.1 PROBLEM DEFINITION FOR SEMI-SUPERVISED FEW-SHOT LEARNING
Episodic training, often employed in meta-training for few-shot learners, feeds the model with one episode at time, with each episode composed of support set with a few labeled samples and a query set with samples to be classified (Vinyals et al.) 2016). For N-way, K-shot learning, N classes are first chosen from a training set, and then a support set is formed by taking K labeled samples per chosen class, The query set is also constructed from the samples of these N classes, disjointly with the support set. For every episode, the model parameters are updated by the loss incurred during prediction of the labels for queries. The meta-trained few-shot learners evaluated unseen classes from the test set, which is disjoint with the train set.
For supi-super few-shot learning, follow the settings of (Ren al. 2018 Each episode is constructed by support set S, query set Q, and unlabeled set I, which contains inputs without labels: U {x1 Xx J. For N-way, K-shot semi-supervised few-shot classification, the support and query sets of every episode are composed of images chosen from the N classes while the unlabeled set can contain images coming from irrelevant classes outside the N candidate classes (distractors). The unlabeled set Ul is also utilized for few-shot classification together with the support set S for classifying queries Q
2.2 PRELIMINARIES ON TAPNET
TapNet of Yoon et al. 2019) a few-shot learning algorithm employing explicit task-conditioning by the task-adaptive projection. TapNet consists of an embedding network Sa. the per-class reference vector set {Φ,,) N. and the task-adaptive projection space M. The embedder fe and the reference vectors {Ø, 1N are meta-trained across varying tasks, and not updated during evaluation. Otherwise, the task-adaptive projection space M which works as the classification space, computed anew for every episode as form of task-conditioning. With some arbitrary initial labeling on { relabeling not necessary throughout episodic training, as linear projection solutions always exist irrespective of particular reference vector labeling. When task constructed by support set S and query set @ is given, the projection space M is computed based on the support set as follows. Using the embedding network fo and support set S, the per-class network output averages Cn obtained from the labeled samples support set S. Then the projection space constructed to align c., with the matching reference vector &n, while distancing disaligning it from all other non-matching references Pitn- To achieve this, a modified reference vector is first formed as Eiyn Φ1. As the inner product between C, and oh, is effectively maximized in
Φn Φ - N the process, the negative sign in front of the non- non-matching reference vectors would tend to disalign them from C,, The actual alignment is done by nulling every error vector e,, between the power- /./ |c,, Nulling of the errors essentially finding
normalized oh, and C,, i.e. C, / projection matrix M such that e., M = 0 for all Barring the trivial case of M O. the resulting projection space M a matrix whose columns span the task-adaptive classification space. The final classification is done by measuring the distance between the projected queries and the projected reference vectors.
2.3 ADAPTIVEE CLUSTERING
Our task adaktidae clustering is also based on the embedding network fe and the per-class reference vector set (&..), JM, both of which are meta-trained across episodes and fixed during evaluation. With the base learnable parts, task coss-condiiioning of clustering space is done via linear projection to the null space of classification errors. For each episode, the projection space is initialized using only the labeled support samples as done in the work of TapNet, and it hosts task-adaptive clustering. Also, the clustering space is iteratively improved utilizing unlabeled samples. M will simply be referred to as projection space, and M(z) will represent projection of vector z in M.
Now the TAC procedures based repetitive reconstruction of the clustering space using unlabeled samples are described as follows: First, project the embedded features of the unlabeled samples into the space M found above and estimate the soft labels (label probabilities) of the unlabeled
3

---

Under review as a conference paper at ICLR 2020
references {
Start with initialized {ca)] and M:
soft labels
fo (R)
recompute class av erages embedding space
unlabeled
clustering in M
reconstruct M
embedding features
[c,H]'
{c,
M
M
to next iteration
Figure 1: Iterative process for task-adaptive clustering
samples by clustering them around the projected per-class averages (see Figure D. Recompute class averages in the original embedded space using the soft labels just estimated. Using the recomputed average with the references, reconstruct M. In this sense, the projection space can be seen to be refined through an iterative process. The number of iterations in constructing projection space may be different between meta-training time and final testing time. More iterations would generally mean more aggressive adaptation to a given task. While a strong task-adaptation is desirable at deployment. it may result in excessive lask-conditioning during meta-training, possibly hampering proper buildup of inductive-bias
Soft labels for unlabeled samples estimated by the probability that unlabeled sample belongs to each class Pj,n exp { {-d(M (R,)), M(cn)))/E, exp (d(((fff()) M(c))). With these probabilities, the centroids is refined by soft assignments of the embedded unlabeled samples, as done in (Ren al. 2018):
Kc,, E, ()
K K+E, Pjnn
Finally, map the embedded features of the query samples and the reference vectors into the projection space, and adjust the parameters of the embedder and reference vectors using softmax based on Euclidean distances between projected query and reference vector pairs. Repeat this process until all episodes are exhausted.
The final few-shot classification procedure follows the same steps, up to the point where the mapped query is classified based on its distances to references in the final projection space. During model update, the Euclidean distances d(...) between the projected query samples and references are used to compute the softmax function
exp { d d(M((fo(En)) (())
Σ, exp (),
The corresponding cross entropy loss function, averaged over all queries and classes, is used to update the embedder fe and references { 1N.
2.4 HANDLING DISTRACTORS WITH TASK-ADAPTIVE CLUSTRRING
We also need to handle unlabeled distractor samples coming from non-candidate classes. To this end, we make use of an additional centroid and reference vector represent the distractor samples. The approach is similar to that taken in (Ren et al. 2018), 8), but here the additional pair of centroid and reference has a direct impact on the way projection space is constructed.
The above TAC procedure is modified as follows in handling distractor samples. We create an additional reference vector N Also, an initial centroid cx+1 for distractor samples obtained by averaging all unlabeled samples in a given episode. When constructing projection space M, utilize the additional eиror vector between CN+1 and d We also compute the probability Pj,N+1 that unlabeled sample x, is distractor. For recomputation of class averages, new centroid for distractors new is obtained by only considering soft labels p1.
+1
E, EjPhN+1fo(8))
N-11
ESPiN4i
(3)
4

---

Under review as a conference paper at ICLR 2020
At final classification time, projection space is recomputed with only N original pairs of centroids and references, i.e,, error vector between new NN+1 and ØN+1 is not considered.
3 RELATED WORK
Semi-supervised learning is the learning method utilizing unlabeled examples in addition to labeled samples, and there exist many known approaches (Zhu 2005 Chapelle et al., 2010). Among the various known semi-supervised learning methods, the self-training scheme of (Yarowsky) 1995] Rosenberg al., 2005) is closely related to our method. In the self-training approach, the model is initially trained with labeled samples only, and prediction on unlabeled samples is made with this model. The prediction results are then utilized as labels of the unlabeled samples, and used for additional training of the model. The self- training approach outperforms the other semi- supervised learning methods when labeled data is scarce (Yarowsky 1995 Triguero al. 2015 Oliver et al. 2018), and also works well for training deep neural networks by using soft pseudo-labell for unlabeled samples (Lee 2013). Our proposed task-adaptive clustering utilizes the prediction results for the unlabeled samples as soft labels in refining classification space, and this process also can be viewed as self-training with soft labels.
Semi-supervised few-shot k-means algorithm based on Prototypical Networks was suggested. the prototype for each class is generated with the labeled few shots of images first. Label predictions for the unlabeled samples are made using the prototypes, and in turn the results are used as soft labels for updating the class prototypes. Transductive Propagation Networks (TPNs) of (Liu et al., 2019) a few-shot learner utilizing graph construction to propagate label information. TPN also exhibits semi-supervised few-shot capability by propagating label information from labeled samples to unlabeled samples, and then to query samples. Our TAC pursues semi-supervised few-shot learning by updating class averages using the predicted soft labels for unlabeled samples, similar to Prototypical Networks based soft k-means. However, in TAC, label prediction is done in the projected space, not in the embedding space itself. We observe that unlabeled samples are better separated in the projected classification space than the embedding space, resulting in improved label prediction. Moreover, since the classification space itself can also be refined using the refined class averages, label prediction can be enhanced through the iterative process of constructing classification space. In the recent work of (Sun et al., 2019), semi-supervised few-shot learning with self-training is proposed. In that work, for each task, unlabeled samples are utilized to fine-tune the parameters of the model. For this purpose, confident unlabeled samples are first picked and a meta-trained soft- www.wwing- network computes the soft weights of the selected unlabeled samples. The computed weights are used to upweight beneficial examples and depress the less beneficial samples. This approach closely related to the few-shot learning methods with task-specific fine-tuning (Finn et al. 20177; Ravi & Larochelle 2017). Rather than adjusting model parameters, our TAC methods utilize unlabeled samples to update the alternative projection space.
learning was first proposed in (Ren et al. 2018). In that work, the soft
the soft k-means algorithm.
TapNets of (Yoon et al, 2019) is a metric-based few-shot learning algorithm employing explicit fask-conditioning via construction of task- adsk-idattive projection space. In TapNets, the projection space is constructed using the support set samples and the reference vectors, and classification of query samples is done there. The present task-adaptive clustering scheme also relies on an alternative classification space obtained from the support set and reference vectors, but the clustering of unlabeled samples (and subsequent soft label estimation) and final classification of query samples are carried out in different spaces. The TAC space is first generated only with the support set and the references as done in TapNets, but the space itself gets improved iteratively by updating the class averages utilizing unlabeled samples. Moreover, in the TAC algorithm with a special handling of distractor samples from irrelevant classes, an additional cluster and reference pair is introduced to represent the distractor samples. The clustering space is generated using the additional cluster and reference together with the class averages and existing references, while the classification space is constructed without using the additional cluster or reference. Overall, TAC uses TapNets base architecture, but a unique way of controlling the degree of condiitoii between the meta-training phase and the evaluation phase provides substantial gain beyond the TapNet baseline.
5

---

Under review as a conference paper at ICLR 2020
4 EXPERIMENT RESULTS
We evaluated the proposed semi-supervised few-shot learning algorithms with two benchmark datasets widely used to evaluate few-shot learning algorithms: 1) manelmaeeNet suggested in (Vinyals et al.. 2016) with the split introduced in (Ravi & Larochelle, 2017) and 2) Iagged suggested in (Ren al. 2018), with the classes in test set highly distinct from those in training set.
4.1 EPISODE COMPOSITION FOR SEMI-SUPERVISED FEW-SHOT LEARNING
In our semi-supervised few-shot learning experiment, an additional dataset split is applied for each dataset as done in (Ren et al. 2018). We only use a small portion of label information from the dataset by dividing the samples in each class into disjoint labeled set and unlabeled set. For rigredlmageNet, 10% of samples per class constitute the labeled set while 90% of samples remain unlabeled. Meanwhile, for minilmageNet, 40% of images used labeled set, and remaining 60% of samples are used as an unlabeled set. The ratio of labeled samples for each dataset is based on the ratio used in (Ren et al. 2018).
For given dataset and a split between labeled and unlabeled sets, episodes are created as follows. For meta-training, generation of episodes starts with sampling N target classes from the meta-training set. Then, K support set samples are selected from the labeled split of each target class. On the other hand, u unlabeled images are chosen randomly from the unlabeled split of each class and form the unlabeled set. q query samples per class are selected from the labeled split of each target class, disjointly with the support set. When the distractor samples are considered, Na different classes are additionally sampled from the meta-training set, and d unlabeled images are sampled from the unlabeled split of each class. Consequently, the unlabeled set consists of N x u samples from N candidate classes for classification and Na d images from the N& distractor classes. The test episodes are constructed in the same way as the training episodes but using the meta-lest set. Note that both the number of classes N and the number of distractor classes Na may be different between test episodes and training episodes.
4.2 EXPERIMENTAL SETTINGS
For the miniImageNet experiment, we use an embedding network named CONV4, which is widely used in prior work of (Vinyals et al. 2016 Snell et al., 2017). For migred experiments, however, we added 2 average pooling layer on top of this embedding network.
We evaluated the proposed method with semi remi-supervised 1-shot and 5-shot accuracies. For both 1-shot and 5-shot, we evaluate the algorithm with and without distractor samples. For the unlabeled set, we use 5 unlabeled samples per class for training and 20 unlabeled samples per class in evaluation. When the distractor samples are considered, the number of distractor class N₂ is the same as the number of target classes N in an episode and the number of distractor samples per class d is the same as the number of unlabeled samples per class u, for both training and evaluation.
Iterative task-adaptive clustering is employed in TAC experiments. We use just 1 iteration (single clustering) in meta- training and more iterations in evaluation generally. We believe that repetitive projection and clustering in episodic training sometimes result in excessive cank-conditioning, which could hinder effective meta-learning of the model across tasks (i.e., effective buildup of inductive bias). The number of iterations in evaluation is chosen based on validation accuracy.
4.3 RESULTS
In Table semi-supervised few-shot classification accuracies on minilmageNet are presented. For the evaluation, we use classification accuracy averaged across 10 random splits between labeled and unlabeled samples. Accuracy of each split is computed with 3.0 x 10 episodes with 15 queries per class. For experiments without distractor, we can observe that our task-adaptive clustering achieves the best accuracy for 5-shot. In 1-shot measurements, our TAC shows the best accuracy while TAC distractor aware projection (TACdap) shows an accuracy level slightly below that of TAC. With the distractor classes considered (1 or 5-shot w/D), our TACdap shows the best accuracy among the known methods. In Table [ experimental results on tieredImageNet are displayed. Again, without distractor samples, task-adaptive clustering yields the best results. For experiments with distractor
6

---

Under review as a conference paper at ICLR 2020
samples, TAC achieves the best 1-shot accuracy while TACdap shows the highest accuracy in the 5- shot case Notice that TAC or TACdap generally provides substantial gains over the TapNet baseline. For both PN and TapNet, baseline results are obtained using only the labeled samples. TapNet + Semi-Supervised Inference is a baseline which is meta-learned by supervised learning as TapNet and evaluated by task adaptive clustering utilizing the unlabeled samples. While this inference-only baseline shows the same level of accuracy with TAC in 5-shot minilmageNet classification, generally our TAC or TACdap shows meaningful gains.
Table Semi-supervised few-shot classification accuracies for 5-way mindmageNett
Methods
1-shot
5-shot
1-shot w/D
5-shot w/D
PN baseline (Ren et al.
2018
43.61 %
59.08%
43.61 %
59.08%
Soft k-Means (Ren al.
2018
50.09%
64.59%
48. 18.70%
63.55%
Soft k-Means + Cluster Ren et al.
2018
49.03%
63.08%
48.86%
61.27%
Masked Soft k-Means Ren ct al. 2018)
50.41%
64.39%
49.04%
62.96%
TPN (Liu et al. 2019)
52.78%
66.42%
50.43%
64.95%
MetaGAN RN (Zhang et al. 2018
50.35%
64.43%
IMP (Allen et al. (2199)
49.2%
64.7%
TapNet baseline
49.58%
66.86%
49.58%
66.86%
TapNet + Semi- Srri-Supervised Inference
53.19%
69.53%
50.53%
66.93%
TAC (Ours)
55.50%
69.21%
50.95%
66.39%
TACdap (Ours)
52.60%
69.05%
51.56%
67.75%
"Due space limitation, 95% confidence intervals of the reported accuracies given in Appendix. t The result from (Alien al. (219)) based on the setting with unlabeled samples evaluation.
Table Semi-supervised few-shot classification accuracies for 5-way lieredImageNett
Methods
1-shot
5-shot
1-shot w/D
5-shot w/D
PN baseline (Ren et al. 2018
46.52%
66. 15%
46.52%
65%
Soft k-Means (Ren al.
2018)
51.52%
70.25%
49.88%
68.32%
Soft k-Means Cluster Ren et al.
2018)
51.85%
69.42%
51.36%
67. 67.56%
Masked Soft k-Means
(Ren et al. 2018)
52.39%
69. 88%
51.38%
69.08%
TPN (Liu et al. 2019)
55.74%
71.01 %
53.45%
69.93%
TapNet baseline
51.84%
69, 14%
51.84%
69.14%
TapNet + Semi-Supprvised Inference
5.5 66%
71.54%
51.84%
69.05%
TAC (Ours)
58.46%
72.05% 71.37%
54.80%
69.35%
TACdap (Ours)
56.06%
53.67%
70.72%
*95% confidence intervals are again given in Appendix.
4.4 ANALYSIS OF PROJECTION SPACE
ISNE plot. For 5-way, 1 1-shot rieredImageNet experiments without distractor, Figure illustrates the embedding space and the iteratively reconstructed TAC space with t-SNE. For better illustration, we visualize each space with 80 unlabeled samples per class. In the t-SNE plot, we display the unlabeled samples correctly clustered each class with colored dots, while the incorrectly clustered samples are marked as gray dots. The number marked beside each cluster indicates the number of correctly-clustered unlabeled samples for each class. We can see that the unlabeled samples result in better cluster separation in the TAC space than the embedding space, and the number of incorrectly clustered samples decreases the TAC space is updated repeatedly.
Performance improvement via iterative task-adaptive clustering. In Figure 3 classification accuracies are plotted versus the number of iterations. For 5-way 1-shot minilmage classification,
We do not include evaluation results of (Sun et al. 2019) because their base embedder is pretrained on fully-laheled training sets of miniltieredlmageNee datasets; we feel that direct comparison would not be fair at this point.
7

---

Under review as a conference paper at ICLR 2020
59
Embedding space
TAC space (iteration 1)
TAC space (iteration 2)
TAC space (iteration 3)
Figure 2: tSNE visualization of iterative improvements of clustering space
compare the proposed TAC with Soft k-Means of (Ren et al. 2018). We can that the accuracy of TAC steadily increases as the number of iterations grows, before saturating; on the other hand, the accuracy of Soft k-Means of (Ren al. 2018) fluctuates with the number of iterations. For the mageNNt classification, comparison is made against Masked Soft k-Means instead [among methods of (Ren et 2018), Soft k-Means and Masked Soft k-Means show the best performance for minilmageNet and IieredImageNe classification, respectively]. Similarly, there is no gain from the iteration for Masked Soft k-Means, while significant gain seen with the iteration for TAC. Note that more iteration here provides deeper level of condicinn.....ing. The results of Figure 3 suggest that TAC iterations up to would be beneficial.
5-way 1-shot minilmageeeet
5-way 1-shot tieredImageNet case
TAC
TAC Maskedd MeMans
Acurary (%6
Soft R-Meams
ocurary (%)
52
55
50
53
51
3
Number of iterations
Number of iterations
Figure 3: Semi-supervised few-shot classification accuracies versus number of iterations
4.5 SEMI-SUPERVISED FEW-SHOT LEARNING WITH REALISTIC UNLABELED SET
Experimental settings and episode composition for semi-supervised few-shot learning were first suggested in (Ren et al. 2018). However, we found that the episode composition prior work may not be realistic. In (Ren al. 2018), the unlabeled of each episode consists of N target class unlabeled samples and Na x d distractor samples, where N = Na and = d. In this setting, the unlabeled set contains exactly the same number of unlabeled samples for each class. Also, when distractor samples are considered, they are selected only from the N₂ classes, and the unlabeled set has the same number target class samples and distractor class samples. To construct the unlabeled set in this structured manner, we would need label information of the unlabeled split of the dataset, which not realistie. Here we propose more realistic episode composition for semi-supervised few-shot learning. We construct the unlabeled set in an uneven manner, with some randomness relative to the evenly composed unlabeled set. We select candidate-class unlabeled samples randomly but non-uniformly from the unlabeled splits of N candidate classes. As for distractor samples, we sample them randomly from the unlabeled split of whole classes except the candidate classes. We do not fix the number of samples per each class, nor the number of distractor classes. We only determine the total number of candidate-class unlabeled samples and distractor samples. Furthermore, it is more natural for unlabeled set include more distractor samples than candidate-class samples.
8

---

Under review as a conference paper at ICLR 2020
5-way, 1-shot Classification
5-way, 5-shol Classification
IAClap (Oum)
TAGGup (Duns
52
So0 A. I-Sesss
Soft J-
51,06
Sto. Xeenne Cluster
68.07
I-Mreus
Chusce
50.27
66.83
49.57
66.86
heseline
Accuracy (96
47.03
Accuracy
62.50
45.33
44.99
60.71
61.64
43.97
beseline
43.61
bespline
59.00
58.65
57.80
0.5
0.6
0.7
0.8
0.5
0.6
0.7
0.8
Ratio of distractor samples in each episode
Ratio of distractor samples each episode
Figure 4: Semi-supervised few-shot classification accuracies of the proposed TAC distractor-aware projection and with a growing fraction of distractor samples in each episode
We perform experiments with the episodes constructed by this more realistic unlabeled compo- sition. In the experiment, we use 50 unlabeled samples per episode in training, and 200 unlabeled samples per episode in evaluation. It is the same as total number unlaheled samples in the experiment in (Ren et al., 2018). We compare k-Means and Soft k-Means + Cluster semi-supervised few-shot learner of (Ren et al. 2018). The 1 1-shot and 5-shot accuracies with varying ratios of distractor samples shown in Figure The accuracy degrades as the ratio of distractor samples increases. We can observe that the 1-shot and 5-shot accuracies of Soft k-Means degrade from 48. 70% and 63.55% to 47.44..4 47.43% and 62.50% when unevenly composed episodes are applied. Also, we can observe that the 5-shot accuracy of Soft k-Means and Soft k-Means Cluster drops below the supervised baseline when 80% of unlabeled samples are distractor samples while our TACdap shows the same level of accuracy as the supervised baseline.
TAC distractor-awarr projection with the Soft
5 CONCLUSION
We proposed semi-supervised few-shot learning method utilizing task-adaptive clustering, which performs clustering in a new projection space for providing a controlled amount of task-conditioning. The TAC space is first constructed to align the class averages with the meta-trained reference vectors, and then gets iteratively refined with the aid of the unlabeled samples clustered the TAC space. The TAC-based algorithms shows ofte-oof-theart semi-supervised few-shot classification accuracies the minilmageNeet and tiereddmageeet datasets, with and without distractor samples. The proposed TAC algorithms also show graceful degradation with a growing portion of distractor samples the unlabeled sample set.
REFERENCES
Kelsey Allen, Evan Shelhamer, Hanul Shin, and Joshua Tenenbaum. Infinite mixture prototypes for few-shot learning. In Proceedings of the 36th International Conference Machine Learning (ICML), volume PMLR 97, pp. 232-241, 2019.
Olivier Chapelle, Bernhard Schlkopf, and Alexander Zien. Semi-supervised learning. 2010.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), vo. 97, pp. 1126-1135, 2017.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In /CML Deep Learning Workshop, volume 2, 2015.

---

Under review as a conference paper at ICLR 2020
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning, ICML, volume 3. pp. 2013.
Huaiyu Li, Weiming Dong, Xing Mei, Chongyang Ma, Feiyue Huang, and Bao-Gang Hu. Lgm- net: Learning to generate matching networks for few-shot learning. In Proceedings of the 36th International Conference on Machine Learning (ICML), volume PMLR 97, pp. 3825-3834, 2019.
Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, Eunho Yang, Sungju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. In Interna- tional Conference on Learning Representations (ICLR), 2019, URL httpttt/o/peeeeeeeet wtr.morrmmm../moom SyVuRiC5K7
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In International Conference on Learning Representations (ICLR), 2018. URL
net forum? BiB.mmmmAABBDDBUUAAA ZMA
Avital Oliver, Augustus Odena, Colin Raffel, Ekin Cubuk, and lan Goodfellow. Realistic
evaluation of semi-supervised learning algorithms. 2018.
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive
metric for improved few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 721-731. 2018.
Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning. In International Conference on Learning Representations (ICLR), 2017.
Mengye Ren, Sachin Ravi, Eleni Triantafillou, Jake Snell, Kevin Swersky, Josh B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. Meta-learning for semi-supervised few-shot clas- sification. In International Conference on Learning Representations (ICIR), 2018. URL
/ettps://openrevieee www.eeenreeeee.teeeooororrrrrenn het.:/oorrummiooHHssss
Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. Semi-supervised self-training of object detection models. WACV/MOTIO,, 2005.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems (NIPS), pp. 4080-4090, 2017.
Qianru Sun, Xinzhe Li, Yaoyao Liu, Shibao Zheng, Tat-Seng Chua, and Bernt Schiele. Learning to self-train for supersuper few-shot classification. arXiv preprint arXiv: /906. 00562, 2019.
Flood Sung, Yongxin Yang. Li Zhang. Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 1199-1208, 2018
Isaac Triguero, Salvador García, and Francisco Herrera. Self-labeled techniques for semi-supervised learning: taxonomy, software and empirical study. Knowledge and Information systems, 42(2): 245-284, 2015.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Advances in Neural Information Processing Systems (NIPS), pp. 3630-3638, 2016.
David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd annual meeting of the association for computational linguistics, 1995.
Sung Whan Yoon, Jun Sco, and Jackyun Moon. Tapnet: Neural network augmented task-adaptive projection for few-shot learning. In Proceedings of the 36th International Conference on Machine Learning (ICML), volume PMLR 97, pp. 7115-7123, 2019.
Ruixiang Zhang. Tong Che, Zoubin Ghaharamani, Yoshua Bengio, and Yangqin Song. Metagan: An adversarial approach to few-shot learning. In Advances in neural information processing systems (NeurIPS), pp. 2365-2374, 2018.
Xiaojin Jerry Zhu. Semi-supervised learning literature survey. Technical report, University of Wisconsin-Maddisn Department of Computer Sciences, 2005.
10

---

Under review as a conference paper at ICLR 2020
A HYPERPARAMETER SETTINGS
Tables and show the hyperparameter settings of our experiments in the main paper. We utilize Adam optimizer with an initial learning rate of 10-3 for all experiments. We cut the learning rate by a factor of 10 once after training 2.5 x 10 episodes or twice after training 2.5 10* and 2.75 x 10 episodes. For regularization, the 12 weight decay with optimized decay rate is also applied in all experiments. In the meta-training for all cases, 10-way training is adopted. The number of queries per class 12 for 1-shot training, and 8 for 5-shot training. Some 1-shot models adopt higher-shot training. For both minilmageNeet and tieredImageNet experiments, TAC 1-shot and TAC I-shot w/D models adopt 5-shot training with queries.
Table 3: Hyperparameter settings for minilmageNet experiments
Model
Training shot
N
Ng
Ir decay step
12 decay rate
Niter
TAC 1-shot
5
10
25000 & 27500
3e-4
5
TACdap 1-shot
1
10
12
25000 & 27500
3e-4
3
TAC 5-shot
5
10
25000 27500
3e-4
2
TACdap 5-shot
5
10
8
25000 & 27500
3e-4
2
TAC 1-shot w/D
5
10
8
25000
3e-4
3
TACdap I-shot w/D
1
10
12
25000
27500
3e-4
TAC 5-shot w/D case
5
10
25000
3e-4
1
TACdap 5 s-oho w/D
5
10
25000 27500
3e-4
Table 4: Hyperparameter settings for tieredImageNet experiments
Model
Training shot
N
Ng
Ir decay step
12 decay rate
Nier
TAC 1-shot
10
8
25000 & 27500
3e-5
5
TACdap 1-shot
10
12
25000 & 27500
3e-5
4
TAC 5-shot
10
8
25000 & 27500
3e-5
3
TACdap 5-shot
5
10
8
25000 & 27500
3e-5
2
TAC 1-shot w/D
5
10
8
25000 & 27500
3e-5
2
TACdap 1-shot w/D
10
12
25000 & 27500
le-5
3
TAC 5-shot w/D
10
8
25000 & 27500
3e-5
1
TACdap 5-shot w/D
10
8
25000 & 27500
3e-5
B CONFIDENCE INTERVALS
Table 5 and show the classification accuracies of our method with 95% confidence intervals for minilmageNet and fiered/mageNet datastets, respectively.
C ABLATION STUDY
C.1 NUMBER OF CLUSTERING ITERATIONS IN EVALUATION
The number of iterations for clustering the unlabeled sample is optimized for each experiment. Figures and show minilmageNeet and fieredImageNet classification accuracies on validation set with varying numbers iterations. We fixed the iteration number to during episodic meta-training. Iteration I means that clustering of unlabeled samples is done once in TAC space. We measured the classification accuracies with iteration numbers ranging from to 5. For every case, we marked the number of iterations yielding the best accuracy with solid dot. The iteration number used in the main paper is set to be the best choice observed in this validation process. Note that the aggressive iteration is advantagcous for 1 1-shot cases (3 to 5 iterations result in considerable gains). On the other hand, the iterative process is less beneficial to 5-shot or distractor sample cases (w/D). Also, in general TAC seems benefit somewhat more from iterations than TACdap.
11

---

Under review as a conference paper at ICLR 2020
Table 5: Semi-supervised few-shot classification accuracies for 5-way minilmaeeeet
Methods
1-shot
5-shot
1-shot w/D)
5-shot w/D
PN baseline
43.61+0..7%%
59.08-0..22%
43.61%
59.08-+0.222%
Soft k-Means
50.09=+0.5%
64.59=0.28%
48.70++0.32%
63.55++0.28%
Soft k-Means Cluster
49.03=+0.24%
63.08=0018%%
48. 0..3%
61.27=+0.24%
Masked Soft k-Means
50.41=10.31 %
64.39=0.24%
49.04:10.31%
02.96:10.149
TPN
52.78=+0.27%
66.42:#0.21%
50.43:t0.84%
64.95:10.73%
MetaGAN + RN
50.35%
64.43:10.27%
IMP
49.2.10.7%
64.7:10.7%
TapNet baseline
49.58=1.200%
66.86=0.16%
49.58=+0.00%
66.86-10.16%
TAC (Ours)
55.50 -.250
69.21 0.19%
50.95++0.23%
66.39-+0.199%
TACdap (Ours)
0..50%
69.05-0.144%
51.56+
67.75-+00.66%
Table 6: Semi-supervised few-shot classification accuracies for 5-way tieredImageNet
Methods
1-shot
5-shot
1-shot w/l)
5-shot w/l)
PN baseline
46.52:10.52% 51.52:+0.36%
66.15:0.22%%
46.52:10.52%
66. 16:10.202
Soft k-Means
70.25=0.31%
49.88++0.529
68.32:+0.2299
Soft k-Means Cluster
51.85=+0.25%
69.42:10.17%
51.36:+0.31%
67.56+10. 10%
Masked Soft k-Means
52.39:10.44%
0..2%%
51.38:10.38%
69.08%
TPN
55.74=10.29%
71.01=0..3%%
53.45++0.33%
69.93:10.80%
TapNet baseline
51.84:10.169
69. 1..13%
51,84-0..1 166
69. 10.10.0..1%
TAC (Ours)
58.46-10.19%
0.15%
69.35=0.11% 54.80+0.200%
TACdap (Ours)
56.06=10.189
71.37=0.12%
53.67
70.72+10.13%
C.2 NUMBER OF UNLABELED SAMPLES IN EVALUATION
Figure 7 shows minilmageNet classification accuracies of the proposed methods with an increasing number of unlabeled samples in evaluation. All models are trained and tested with single iteration, and trained with 5 unlabeled samples per class. We can see that the test accuracy gradually increases as the number of unlabeled samples grows. Note that when the number of unlabeled samples low, the accuracy of TACdap is always higher than the accuracy of TAC. Moreover, for 1 1-shot cases, the accuracies of TACdap with unlabeled sample is fairly close to the accuracies with many unlabeled samples.
C.3 CLASSIFICATION ACCURACY AND CROSS ENTROPY VERSUS NUMBER OF ITERATIONS.
In Figure 8 classification accuracies and cross entropy are plotted versus the number of iterations for 5-way, -shot cases of minilmageNet and liered experiments without distractor. We consider two cases for clustering at evaluation time: projection space and in embedding space. The final classification is done in the adsk-adaptive projection space. Cross entropy L,, is obtained the number of iterations grows, classification accuracy improves
from the label probabilities. and cross entropy decreases for both cases, i.e., the superviid classification performance is enhanced and the soft labels become more reliable. The initial gap between the two cases actually comes from the clustering in projected space rather than embedding space. Also, as iteration grows from to 5, the gap is actually seen to gradually increase: from 0.90% to 1.19% and from 1.27% to 1.65% for minilmageNet and IieredImageNet, respectively. The results clearly show that clustering (and subsequent soft-labeling) is much better off done in the projection space than in the original embedding space for leveraging unlabeled samples.
12

---

Under review as a conference paper at ICLR 2020
16
Accunag (83
Acamng (67 89.3
56
80.2
Accuncy
30.5
Accumcy
54
2
Number of itemtoons
Number of deetioos
Numberof of dtntions
Number of Itrrtions
(a) TAC 1-shot
(h) TAC: 5-shot
TAC 1-shot w/D
(d) TAC 5-shot w/D
Acguncy (8)
Aceumcy (8)
$1.5
Acmmay
52.5
68.5
64
Numberot of itemioons
Number of stemtoons
Numberot Stettonn
Number of iterations
(e) TACdap 1-shot
(f) TACdap 5-shot
(g) TACdap I-shot w/D
(h) TACdap 5-shot w/D
Figure minllmageNet classification accuracies vs iteration number
55
66
Aocing (N)
(80
51.8
63
51.8
51.4
Number of iterations
Number of Etertioon
Number iteratons
Number of Seratoos
(a) TAC 1-shot
(h) TAC 5-shot
TAC 1-shot w/D
(d) TAC 5-shot w/D
Accuncy 18g
Ancungy
Accuncy
$9.5
Accumay
11.5
91
30.5
49
Number tbemtons
Number
(c) TACdap 1-shot
(f) TACdap 5-shot
(g) TACdap 1-shot w/D (h) TACdap 5-shot w/D
Figure 6: lieredImageNet classification accuracies vs iteration number
C.4 CHOICE OF CLUSTERING SPACE IN META-TRAINING AND IN EVALUATION
In the main paper, we showed that clustering in the TAC space results in better classification accuracies than clustering the embedding space. A question may arise: Is clustering TAC space still beneficial even when clustering is rather done in the embedding space during meta-training? In other words, how would the learner fare the episode training condition deviates from the actual few- shot evaluation condition in terms of clustering space. Note that the principle established with regards to episodic training is that the training condition must match the evaluation condition. Interestingly, for the TAC methods, clustering in the embedding space during meta- training did not affect final performance even as evaluation is based on clustering in TAC space. See the comparison results in Fi Clustering in the embedding space in fact gave similar (or slightly better at low iteration numbers) results compared to clustering in projection space, when it comes to meta-training, As for better clustering evaluation time, TAC space the better choice than original embedding space.
C.5 NUMBER OF CLUSTERING ITERATIONS IN
The effect of varying the iteration number in meta-training is also investigated. We wish to understand whether adaptation through iterative projection and clustering in meta-training is productive not. It may be possible that strong adaptation for every episode during meta-training may hinder the process building a proper inductive bias in the model. For the results obtained in the main manuscript, adopted iteration of clustering in TAC space. Here present measured
13

---

Under review as a conference paper at ICLR 2020
Semi-supeene supervised 5-way 1-shot
Semi supervised 5-way 1-shot w/ Distractors
TACdap
2
10
15
20
25
2
10
15
20
25
Unlabelldd Samples per Class
Number
Somples per
Sumi--speen sumii-spprvisee 5-way S-shot
Semi-supern supri-speeeiiedd 5-way 5-shot Distractors
70
70
B.C dap
IAC 1ACddap
Accurey
Actuanc'
6B
68
66
86
63
Number of Unlabeled Samples per Class
Numbero of Unlabeled Samples per Class
Figure 7: minilmageNet classification accuracy versus number of unlabeled samples
5- way, 1-shot minimmageNett case
5-way, 1-shot IageNNet case
clustering in proj. space
clusering pmj. space
3.5
clustecing in emb. space
clustering in emh. space
Accuracy
Accuracy
54
2.5
-e- clustaring proj. space
cluustetiingg in pruj. spece clustering in emb. space
clustaring cmb. space
1.5
2
Number of iterations
Number of iterations
Number of iterations
Number of iterations
Figure 8: Classification accuracy and cross entropy number of iterations
tieredImageNet classification accuracies for TAC and TACdap methods with varying number of iterations meta-training.
See Figure 10 For TAC methods, a more aggressive adaptation through an increasing number of iterations during meta-training does not result in significant gains. Note that iteration here means that clustering is done in the embedding space.
For TACdap, on the other hand, the number of iterations grows, performance is substantially degraded. For 1-shot experiments, accuracy is consistently degraded as the number of iterations grows to more than 1. Also for 5-shot experiments, when we adopt iterative clustering in TAC space, classification accuracy decreases steadily with iteration. Thus, for TACdap, multiple rounds of clustering during meta-training are actually harmful.
Overall, during meta-training, iterative projection followed by subsequent clustering is not beneficial for our TAC- PAC-based methods. We believe that multiple rounds projection/clustering at train time may excessive consiinndiiin,,ng which in turn would prevent the machine from developing healthy level of inductive bias for meta-lcarning.
14

---

Under review as a conference paper at ICLR 2020
58
57
A Acuuaac (%)
56
55
TAC evelustion embedded space meta-tricling ambeected spaca evalustion, embedded space neta-briinnn space evalustio, space traiiing
-d-enieecled space evalustion. TAC space in beli-briiiing
53
Number of iterations
Figure Classification accuracy on lieredImageNet vs space where clusters form: TAC space versus embedded space, in meta- meta-training and in evaluation
56
50
Accurscy (56)
33.5
54.3
54
50
53
Number of Iterations nete-training
Number Lterations meta-braniing
(a) TAC 1-shot
(b) TACdap 1-shot
72.5
72.5
Acoursty (50)
Accurary
21.5
28.3
70.5
70.5
70
70
Number of iterations in mebs trining
Number itontiins in mota training
(c) TAC 5-shot
(d) TACdap 5-shot
Figure 10: tieredImageNet classification accuracy versus number of iterations in meta-training
15


Reviews:
[{"review_id": "SJltUIRoYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a semi-supervised few-shot learning method that combines the TapNet architecture with the soft-clustering approach in the original semi-supervised few-shot learning method of Ren et al. 2018. Results are reported on mini-ImageNet and tiered-ImageNet, demonstrating superior results to Ren et al. 2018 and some more recent work.\n\nThough the results are strong, I'm personally leaning towards rejecting this submission. The main reason is that the contribution is an arguably simple combination of 2 methods (TapNet + Ren et al. 2018). Specifically, I notice that the difference between the performance of the TapNet and Prototypical Networks baselines (6%, 7.5% for 1 and 5 shot on miniImageNet, and 5% and 3% for 1 and 5 shot on tieredImageNet) is roughly also found in the difference between TAC and Soft- k-Means (Ren et al. 2018). This suggests that most of the benefits found in TAC can simply be explained by the adoption of the TapNet architecture. And I consider this proposed extension to semi-supervised few-shot learning of TapNet to be pretty straightforward.\n\nI don't really expect the authors can improve on the above point in a revision, so I doubt I'll be convinced to change my score. However, if the authors believe that I've missed a subtlety of their method that is less technically straightforward than my analysis suggests, of course I'd like to know.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Bkxb6KTpYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n========\nThis paper tackles the more realistic variant of few-shot classification where a large portion of the available data is unlabeled, both at meta-training time in order to meta-learn a learner capable of fast adaptation, as well as meta-test time for adapting said learner to each individual held-out task. \n\nTheir approach is based on TapNet, a model which constructs a task-specific projection based on the support set of each episode, and then classifies each query according to its distance to each class prototype in the projected space. The projection is computed so that the class prototypes of the episode (averaged support examples) are well-aligned with a set of meta-learned references. Intuitively, those references learn to be far from each other, so that aligning the prototypes with them leads to a space where the episode’s classes are well separated, allowing for easier classification between them.\n\nThey then extend this model to incorporate unlabeled examples by performing task projection as follows: 1) the projection is computed so as to align the initial prototypes (computed only using labeled examples) to the meta-learned references. 2) In that projected space, each unlabeled example is assigned a predicted label based on its proximity to the projected prototypes. 3) Then, back at the original space, those predicted labels of the unlabeled examples are used to refine the class prototypes (weighted average as in Ren et al). 4) The projected space so that the *refined* prototypes are best aligned with the meta-learned references. 5) Possibly repeat 2-4 (at meta-test time).\n\nExperimentally, they outperform recent approaches to semi-supervied few-shot classification on standard benchmarks, though not by far. Perhaps more interestingly, their performance degrades less than that of Ren et al as the distractor ratio increases, and they show that their method benefits from additional steps of task adaptation, whereas that of Ren et al reaches its performance limits after the first step of soft clustering.\n\nHigh-level comments\n==================\nA) Ablation: An interesting ablation would be: instead of going back and forth between the embedding space and the projected space, the task adaptation happens only in the initially-computed projection space (i.e. the one computed based on the labeled data only). This would amount to: computing the projection space, and then performing a few steps of soft clustering, similar to Ren et al. in that space. This would help determine how beneficial it is to re-compute that projection space according to the current ‘best guess’ of where the class prototypes lie at each iteration. The way I see it, it is an empirical question whether the initially computed projection space already sufficiently separates the classes or not. I assume this would also lead to a more computationally efficient solution?\n\nB) Handling distractors: In the case of distractors, they use an additional centroid (and a corresponding additional reference vector) for the purpose of ‘capturing’ the unlabeled examples that don’t belong to candidate classes. I find the initialization of this strange: this additional centroid is computed as the mean of all unlabeled examples, and the initial projection construction is influenced by a term that matches this centroid to a corresponding reference. This would mean, however, that even the unlabeled examples which do indeed belong to one of the candidate classes end up far from those classes in the projected space, in order to be close to the designated extra reference in that space. We know that this is not ideal, since we assume that some unlabeled examples do belong to the same classes as the labeled ones. Is there a way to quantify how severely this affects the quality of this initial projection? I would also be curious about the meta-learned location of the extra reference. Does it end up being roughly in the center of the references corresponding to the labeled classes?\n\nC) Inference-only baselines. Ren et al. experimented with inference-only baselines: meta-learning happens only using the labeled subset, and the proposed clustering approach only takes place at meta-test time. In this case this would amount to meta-training a standard TapNet and then performing the proposed refinement only in test episodes. This is interesting as it allows to understand the importance of learning an embedding end-to-end for being more appropriate for unlabeled example refinement. It is not obvious that this is required, so I would be curious to see these results too. (This differs from the reported TapNet baseline in that at meta-test time it would make use of the proposed semi-supervised refinement).\n\nClarity / quality of presentation:\n============================\nD) A lot of emphasis is placed on the ability of the proposed method to control the degree of task conditioning. I would like to emphasize that this is not something that previous methods lack. Ren et al.’s approach could also perform multiple steps of clustering for example. Whether or not this is beneficial is an empirical question, but I wouldn’t say that the proposed method does something special to “control” how much adaptation is performed.\n\nE) It would have been useful to have a separate section or subsection that explains TapNet, since this is the model that the proposed method is built upon. Instead, the authors blend the explanation of TapNet in the same section as their method which makes it hard to understand the approach and to separate the contribution of this method from TapNet.\n\nG) The length of the paper is over the recommended pages, and I did feel that a few parts were too lengthy or repetitive (e.g. the last paragraph of the introduction described the model in detail. I felt that it would have been more appropriate to give the higher level intuition there and leave the detailed description for the next section). \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BkgW8i5RKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper Summary:\n\nThis paper extends TapNet, a few short learning approach, to the setting of semi-supervised few shot learning. For that, it borrows the idea of soft-labeling from Ren et al 2018, a semi-supervised few shot learning algorithm. Experiments over mini and tiered imagenet compare the approach with alternatives.\n\nReview Summary:\n\nThe paper could be clearer. The main idea of the paper and its goal/motivation is not concisely given in the intro and abstract. The paper refers to a lot of terminology from TapNet without defining them which makes it difficult to understand. I am not an expert in the field and I have a hard time judging whether the empirical improvement are exceptional but I would advocate against accepting the paper based on clarity alone at this point. Moreover the contribution -- adding soft labeling to TapNet -- seems modest.\n\nDetailed Review:\n\nThe abstract should not be a sketch of the proposed algorithm. It should highlight: what is the problem, what is the key idea of what you propose to address it (one sentence), why is it original and better than prior work, what is the foreseen impact of this work. \n\nThe terminology specific to your problem and family of model need to be defined: what is the clustering space? which clustering? what is the classification space? what is a per class network? None of these concepts are defined.\n\nIn section 2.2, please rephrase \"Nulling of the errors is essentially finding a projection matrix M such that epsilonM = 0 for all n. The resulting M is a matrix whose columns span the task-adaptive classification space\" as \"We identify M such that (i)  epsilonM = 0 for all n and (ii)  M columns span the task-adaptive classification space\". The text make it sounds like your are finding the solution of (i) which happened to also verify (ii) which is not true. For instance, M=0 satistifies (i) but not (ii).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach to semi-supervised few-shot learning. In a discussion after the rebuttal phase, the reviewers were somewhat split on this paper, appreciating the advantages of the algorithm such as increased robustness to distractors and the ability to adapt with additional iterations, but were concerned that the contributions over Ren et al were not significant. Overall, the contributions of this paper don't quite warrant publication at ICLR.
Decision: Reject