An Embarrassingly Simple Approach to Semi-Supervised Few-Shot Learning
Authors: Xiu-Shen Wei
He-Yang Xu
Faen Zhang
Yuxin Peng
Wei Zhou
Venue: NeurIPS 2022 Accept (2022)
Keywords: Semi-Supervised Few-Shot Learning
Negative Learning
Few-Shot Learning

Abstract: Semi-supervised few-shot learning consists in training a classifier to adapt to new tasks with limited labeled data and a fixed quantity of unlabeled data. Many sophisticated methods have been developed to address the challenges this problem comprises. In this paper, we propose a simple but quite effective approach to predict accurate negative pseudo-labels of unlabeled data from an indirect learning perspective, and then augment the extremely label-constrained support set in few-shot classification tasks. Our approach can be implemented in just few lines of code by only using off-the-shelf operations, yet it is able to outperform state-of-the-art methods on four benchmark datasets.

An Embarrassingly Simple Approach to Semi-Supervised Few-Shot Learning
Xiu-Shen Wei He-Yang Xu Faen Zhang Yuxin Peng"; Wei Zhou"
School of Computer Science and Engineering, Nanjing University of Science and Technology State Key Laboratory of Integrated Services Networks, Xidian University Qingdaa Alnnovation Technology Group Co., Ltd Wangxuan Institute Computer Technology, Peking University
SCICC Alpha (Beijing) Private Equity
Abstract
Semi-supervised few-shot learning consists in training a classifier to adapt to new tasks with limited labeled data and fixed quantity of unlabeled data. Many sophisticated methods have been developed to address the challenges this problem comprises. In this paper, we propose a simple but quite effective approach predict accurate negarive pseudo-labels of unlabeled data from an indirect learning perspective, and then augment the extremely shot classification tasks. Our approach can be implemented in just few lines of code by only using off- off-the-shelf operations, yet it is able to outperform state-of-the-art methods on four benchmark datasets.
constraind support set in few-
1 Introduction
Deep learning 16 allows computational models that are composed of multiple processing layers to learn representations of data with multiple levels of abstraction, which has already demonstrated its powerful capabilities in many computer vision tasks, e.g., object recognition [7]. fine-grained classification [39], object detection [18), etc. However, deep learning based models always require large amounts of supervised data for good generalization performance. Few-Shot Learning (FSL) [37), as an important technique
alleviate label dependence, has received great attention recent years. It has formed several learning paradigms including metric-basee methods 22
33 45
optimization-
based methods [4] 25 28
and cransfe--earming based methods
24
More recently, it is intriguing to observe that there has been extensive research in FSL on exploring how to utilize unlabeled data to improve model performance under few-shot supervisions, which is Semi-Supervised Few-Shot Learning (SSFSL) (9 [5] of SSFSL. is predict unlabeled data with pseudo-labels by carefully devising tailored strategies, and then augment the extremely small support set of labeled data in few-shot classification, e.g., 15 36). In this paper, we follow this fashion and propose a simple but quite effective approach to SSFSL, i.e., Method of SUvcesSIve exClusions (MUSIC), cf. Figure ||
19 23
36 44 The most popular fashion
As you imagine, in such lubel-constraieed tasks, e.g., 1-shot classification, it would be difficult to learn a good classifier, and thus cannot obtain sufficiently accurate pseudo-labels. Therefore, we
"Corresponding author, X.-S Wei and H.-Y. Xu with Key Lab of Intelligent Perception and Systems for nigh-Diiensiioal Information Ministry of Education, and Jiangsu Key Lab Image and Video Understanding for Social Security, Nanjing University of Science and Technology. This work was supported by National Key R&D Program of China (2021 YFA 1001 100), National Natural Science Foundation of China under Grant (62272231, 61925201, 62132001, U21 221520,5), Natural Science Foundation of Jiangsu Province of China under Grant (BK20210340) the Fundamental Research Funds for the Central Universities (30920041 111, NI2022028), CAAI-Huawar MindSpore Open Fund, and Boijing Academy of Artificial Intelligence.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

0:05
COS
2
611
Neualive loming
?
6:05
Squaive looning
?
Pretiin loming
6885
6.95
6.95
CLU
CXN
CLU CON
CLS CON
8.50
Updine
2.5
Updane
8.55
Uptane
Iter. #1
Iter: R2
Iter: R3
Figure 1: Pipelliee of our MUSIC approach, where take 3-way few-shot classification for exam- plc. Specifically, the green data points marked with question mark are unlabeld data in SSFSL, "CLS" represents the classes of data, and "CON" stands for the predicted confidence of the corresponding pseudo-labels. In each iteration, we highlight the selected (negative/positive) pseudo-label with the red color, and exclude that class in the following iteration. Note that, the unlabeled data at the bottom not returned with the final positive label due to our reject option strategy (5 = 0.2), cf. Eqn. (4)
think about the problem in turn, and realize the process of pseudo-labeling in SSFSL as series of successive exclusion operations. In concretely, since it is hard to annotate which class the unlabeled data belongs to, in turn, it should be relatively easy predict which class it does not belong to based on the lowest confidence prediction score. Thus, if we treat the predicted pscudo-labels in the previous traditional way as labeling positive labels, our exclusion operation is to assign negative pseudo-labels to unlabeled data. In the following, we can use the negative learning paradigm [IO] to update the classifier parameters and continue the negative pseudo-labeling process by excluding the predicted negative label in the previous iteration, until all negative pseudo-labels are obtained. Moreover, it is apparent to find that when all negative labels of unlabeled data are sequentially excluded and labeled, their positive pseudo-labels are also obtained. We can thus eventually augment the small support set with positive pseudo-labels, and fully utilize the auxiliary information from both labeled base-class data and unlabeled novel-class data in SSFSL. Also, in MUSIC, to further improve few-shot classification accuracy, we equip minimum-entree loss into our successive exclusion operations for enhancing the predicted confidence of both positive and negative labels.
In summary, the main contributions of this work are as follows:
We propose simple but effective approach, i.e., MUSIC, to deal with semi-supervised few-shot classification tasks. To our best knowledge, MUSIC the first approach to leverage negative learning straightforward way to provide pseudo- with as much confidence as possible in such extremely cobbl-constrained scenarios.
We can implement the proposed approach using only off-the-shelf deep learning compu- tational operations, and can be implemented in just few lines of code. Besides, also provide the default value recommendations of hyper-parametent validate its strong practicality and generalization ability via various SSFSI. tasks.
our MUSIC, and further
We conduct comprehensive experiments on four few-shot benchmark datasets, i.e., mini- ImageNet, tieredlmageNet. CIFAR-FS and CUB, for demonstrating our superiority over Llate-of-the-at FSL and SSFSL methods. Moreover, series of ablation studies and discus- sions are performed to explore working mechanism of each component in our approach.
Related Work
aims explore the possibility of endowing learning systems the ability of rapid learning for novel categories from a few examples. In the literature, few-shot learning methods can be roughly separated into two groups: 1) Meta-learning based methods and 2) Transfer-learning based methods.
Few-shot learning The research of few-shot learning 4
29
33
45
Regarding meta-learning based methods, aka "learning-to-leant" there two popular learning
paradigms, i.e., metric-based methods 22 33 45 and optimization-bae methods 4 25 28
More specifically, Prototypical Networks 29 as a classical metric-based method was considered
Because the probability of selecting a class that does not belong to the correct label is high, the risk of providing incorrect information in doing is low, especially for SSFSL.

---

to generate an embedding in which data points cluster around a single prototype representation for each class. DeepEMD [45 proposed adopt the Earth Mover's Distance as a metric to compute a structural distance hetween dense image representations to determine image relevance for few-shot learning. For optimization- fast gradient direction to rapidly learn the classifier for novel classes. In [2]. it reformulated the parameter update into an LSTM and achieved this via a meta-learner.
methods, MAMI. 4 learned an optimization method to follow the
Regarding transfer-learning based methods, they are expected to leverage techniques to pre-train a model on the large amount of data from the base classes, without using the episode training strategy. The pre-tra pre-trained model is then utilized to recognize novel classes of few-shot classification. In concretely, (24) proposed to directly set the final layer weights from novel training examples during few-shot learning as weight imprinting process. In [3), the authors investigated and shown such transfer-learning based methods can achieve competitive performance as meta-learning methods.
Semi-supervised few-shot learning Semi-Supervised Learning (SSL) is an approach to machine learning that combines a small amount of labeled data with a large amount of unlabeled data during training 16 [6]. In the era of deep learning, SSL generally utilizes unlabeled data from the following perspectives, e.g., considering consistency regularization [4], employing moving average strategy B0], applying adversarial perturbation regularization
22 etc.
In recent years, the use of unlabeled data to improve the accuracy of few-shot learning has received 44 which leads to the family of Semi-Supervised Few-
increasing attention [9 15
19
23
36
Shot Learning (SSFSL.) methods. However, directly applying SSL. methods to few-shot supervised scenarios usually causes inferior results due to the extreme small number of labeled data, e.g., 1- shot. More specifically, to deal with the challenging SSFSI,, Ren al. (26) extended Prototypical
Networks [29] to use unlabeled samples when producing prototypes. TPN [9] was developed to propagate labels from labeled data to unlabeled data by learning a graph that exploits the manifold structure of the data. Recently, state-of-the-art SSFSL methods, e.g., [9] 15 36] were proposed to predict unlabeled data by pseudo-labeling and further augment the constrinnd support set in few-shot classification. Distant from previous work, to our best knowledge,
the first to explore leveraging complementary labels (i.e., negative learning) to pseudo-label unlabeled data in SSFSL.
Negative learning As indirect learning method for training CNNs, Negative Learning (NL) I0) was proposed as a novel learning paradigm w.r.t. typical supervised learning (aka Positive Learning, PL). More specifically,, PL indicates that "input image belongs to this label", while NL means "input image does not belong to this complementary label". Compared to collecting ordinary labels in PL, it would be less laborious for collecting complementary labels in NL 10. Therefore, NL can not only be easily combined with ordinary classification (5] [O]. but also assist various vision applications, e.g. [12 dealing with noisy labels by applying NL, 35 using unreliable pixels for semantic segmentation with NL, etc. In this paper, we attempt to leverage NL to augment the few-shot labeled set by predicting negative pseudo-labels from unlabeled data, and thus obtain more accurate pseudo labels to assist classifier modeling under cabet-constrained scenarios.
3 Methodology
3.1 Problem Formulation
Definition In Semi-Supervised Few-Shot Learning (SSFSL), have large-scale dataset Daase containing many-shot labeled data from each base class in Clase. and a small-scale dataset D Dnovel consisting of few-shot labeled data as a support S from the category set Crowel, as well certain number of unlabeled data UI acquired also from Crovel. Note that, Dnovet is disjoint from Dhase for generalization test. The task of SSFSL is to learn a robust classifier f-; based on both and l for making predictions on new queries Q from Dnovel, where Drase is utilized as auxiliary data.
Setting Regarding the basic semi-supervised few-shot classification setting, generally faces the W-way-K-shee problem, where only K labeled data from S and U unlabeled data from 2 per class are available to learn an N-way classifier. In this setting, queries Q are treated independently of each other, and are not observed in U. It is referred to inductive inference.
3

---

For another important setting in SSFSL, i.e., transductive inference, the query set Q is observed also during training and joint with U.
3.2 MUSIC: A Simple Method of vUccesSIve exClusions for SSFSL
The basic idea of our MUSIC is to augment the few-shot labeled set (the support set) S by predicting "negative" (i.e., "saying not belonging to") pseudo-labels to unlabeled data 4, particularly for such label-constrained scenarios.
Given an image I, we can obtain its representation by training a deep network F(:; Q) based on auxiliary data Drase:
x F); E R"
(1)
where O is the parameter of the network. After that, F(;; θ) is treated as general feature embedding function for other images and @ is also fixed []. Then, considering the task of c-class classification, the aforementioned classifier f-; 8) maps the input space to a c-dimensional score space as
p soft (x;; (x;0), 0)) R
where is indeed the predicted probability score belonging to the c-dimensional simplex A softmax(-) is the softmax normalization, and @ the parameter. In SSFSL, 0 is randomly initialized and fine-tuned only by NK labeled data in S by the cross-entropy loss:
C(f.y)= = Σ yv log Pk
(3)
where y E IRC a one-hot vector denoted as the ground-truth label w.r.t x, and ys and pk the k-th element in y and p. respectively.
To augment the limited labeled data in S, then propose predict unlabeled images (e.g., Th) in U with pseudo-labels from an indirect learning perspective, i.e., excluding negative labels. In concretely, regarding conventional classification task, the ground-truth yı = 1 represents that its data x belongs to class k; which can be also termed as positive learning. In contrast, we hereby denote another one-hot vector y E R as the counterpart to be the complementary label [TO, T2). where Tn = means that x does not belong to class k aka negative learning. Due to the quite limited labeled data in few-shot learning scenarios, the classifier f(:;0) is inaccurate to assign correct positive labels to I". On the contrary, however, it could be relatively easy and accurate to give such a negative pseudo-label to describe that 1" is not from class lo by assigning yı 1. Therefore, we realize such an idea of "exclusion" by obtaining the most confident negative pseudo-label based on the class having the lowest probability score. The process formulated as:
if arg min(p") and pk 8 otherwise
TK
rejection
where p" represents the prediction probability w.r.t. T", and 5 is reject option ensure that there is sufficiently strong confidence to assign pseudo-labels. While if all ph are larger than 8, no negative pseudo-labels are returned for I" in this iteration.
Thus, after obtaining samples and negative pseudo-label pairs (I","") f(; can be updated by
c(fr")= = Σ v log(1l (lg((-pp)). p
(5)
In the next iteration, exclude the k-th class, i.e., the negative pseudo-label in the previous iteration, from the remaining candidate classes. After that, the updated classifier is employed to give the probability score P*, R- of I", without considering class k. The similar pseudo-labeling process is conducted in successive exclusion manner until all negative pseudo-labels are predicted according to Eqn. (4). or no negative pseudo-labels are able to be predicted with a strong confidence.
Finally, in the last iteration, for those samples in l4 whose negative labels are all labeled, their positive pseudo-labels are naturally available. We can further update the classifier by following Eqn. (3) based on the final positive labels. Then, the updated classifier f(; () is ready for predicting Q as evaluation.
Moreover, further improve the probability confidence and then promote pseudo-labeling, we propose to equip a minimum-entropy loss (MinEnt) upon p" by optimizing the following objective:
C(f. p²)
Σ
pk log Pg....
(6)
k

---

Algorithm Pseudo-code of the proposed MUSIC
classifier, cf. Egn. (2) of the paper
reject option to select the negative label, of, Eqp. of the paper
c: the nunber of classes
Position: 1ist record the label which has selected negative label iteration 0: anboddings of the support and unlabeled set which have bean extracted by the pre-trainad CM model ([8|-L, 01-81
begin:
logits f(8) aupport: logita (4,
loss - (algga((oogite targets) 4 CrossBntzopy
while Troe:
negative logits and negative label (8)
neg_logits, neg_label jet_aeg_sumleer(toiiti. f,
if
the condition to stop the itarations
Eqn. of paper 1oss + legg neg_labell (inlSatzopyylee__oogtttt
HagCroasRntropy
Egn. Mininu--nttropy
pos_logits, pos_label + get_ joo_eennles(Posstiiil loss CElaasa (ponsst las_label) Kiniintropyypoo_
and
It could sharp the distribution of p" and discriminate the confidence of both positive and negative labels. Algorithm 3.2 provides the pseudo-code of our MUSIC.
Experiments
4.1 Datasets and Empirical Settings
We conduct experiments on four widely-used few-shot learning benchmark datasets for general object recognition and fine-grained classification, including minilmageNet (25), tieredImageNee [26]. CIFAR-FS (2) and CUB (34). Specifically, minilmageNer consists of 100 classes with 600 samples of 84 x 84 resolution per class, which are selected from ILSVRC-2012 (27). tieredImageNet is larger subset from ILSVRC-2012 with 608 classes in a man-made hierarchical structure, where its samples also of 84 x 84 image resolution. CIFAR-FS is variant of CIFAR-101 100 [13] with low resolution, which has 100 classes and each of them has 600 samples of 32 x 32 size. Regarding CUB, it is a fine-grained classification dataset of 200 different bird species with 11.788 images in total.
For fair comparisons, we obey the protocol of data splits in 9 15 36 train the feature embedding function and conduct experiments for evaluations in SSFSL. We choose the commonly used ResNet-
12 as the backbone network, and the network configurations are followed 1 15 36). For pre-training, we just follow the same way of [38] to pre-train the network, but do not use any pseudo labels during pre-training. For optimization, Stochastic Gradient Descent (SGD) with momentum of 0.9 and weight decay of 5 10-4 4 is adopted the optimizer to train the feature extractor from scratch. The initial learning rate is 0. and decayed as 6 x 10 3 1.2 x 10 3 and 2.4 x 10 after 60, 70 and 80 epochs, by following (38]. Regarding the hyper-parameters in MUSIC, the reject option Eqn. (4) is set to and the tradc-off parameter over Eqn. (6) is set to as default for all experiments and iterations, which shows its practicality and non-tricky. During evaluation, the last layer of pre-trained model is replaced by an ngnormaalizttion layer and a c-dimensional fully connected layer as the classifier. We also use SGD for optimization. Our MUSIC and all baselines are evaluated over 600 episodes with 15 test samples in each class. All experiments are conducted by MindSpore with a GeForce RTX 3060 GPU.
4.2
Main Results
We report the empirical results the following four setups. All results are the average accuracy and the corresponding 95% confidence interval over the 600 episodes are also conducted.
Basic semi-supervised few-shot setup We compare our MUSIC with state-of-the-art methods in the literature in Table [] As shown, our simple approach outperforms the competing methods of both generic few-shot learning and semi-supervised few-shot learning by a large margin across different shot tasks over all the datasets. Beyond that, we also report the results of solely using
5

---

Table 1: Comparisons of 5-way few-shot classification with the basic semi-supervisee few-shot setup. These results performed with 30/50 unlabeled samples for -oays---to/-wwa---aaa -wat-5-shot, respectively. The light blue blocks represent that these methods are tested in the inductive setup, and light yellow blocks are tested in basic aami-suprrvised setup. The highest accuracy is marked red, and the second highest accuracy is in blue.
minilmageNer 5-shot
tieredlmageNer 1-shot
CIFAR-FS
CUB
METHOD
BACKBONE
1-shot
5-shot
I-shot
5-shot
1-shot
5-shot
MatchingNet 33
CONV
43.56
55.31
MAML
CONV
48.70
63.11
51.67
70.30
58,90
71.50
54.73
75.75
ProtoNet
4 CONV
49.42
68.20
53.31
72.69
55.50
72.00
50.46
76.39
LEO 28
WRN-28-10
61.76
77.59
66.33
81.44
CAN 8
ResNet-12 12
63.85
79.44
69.89
84.23
DeepEMD 45
ResNet-12
65.91
82.41
71.16
86.03
74.58
86.92
75.65
88.69
FEAT 43
ResNet-12
66.78
82.05
70.80
84.79
73.27
85.77
RENet
ResNet-12 67.60
82.58
71.61
85.28
74.51
86.60
82.85
91.32
FRN 40
ResNet-12
66.45
82.83
72.06
86.89
83.55
92.92
COSOC
ResNet-12
69.28
85.16
73.57
87.57
SetFeet
ResNet-12
68.32
82.71
73.63
87.59
79.60
90.48
MCL 20
ResNet-12
69.31
85.11
73.62
86.29
85.63
93.18
STL DC
ResNet-12
67.83
85.45
73.82
89.00
84.01
94.02
TPN [9
CONV
52.78
66.42
55.74
71.01
TransMatch
44
WRN-28-10
60.02
79.30
72.19
82.12
LST 117
ResNet-12
70,01
78.70
77.70
85,20
EPNet
ResNet-12
70.50
80.20
75.90
82.11
ICI 36
ResNet-12
69.66
80.11
84.01
89.00
76.51
84.32
89.58
92.48
iLPC [S
ResNet-12
70.99
85.04
89.63
78.57
85.84
90.11
PLCM 9
ResNet-12
72.06
83.71
84.78
90.11
77.62
86.13
Ours
ResNet-12
74.96
85.99
85.40
90.79
78.96
87.25
90.76
93.27
Ours (only neg)
ResNet-12
73.86
85.11
84.91
90.29
78.26
86.53
89.91
92,46
Ours (only pos)
ResNet-12
74.44
85.86
85.33
90.62
78.81
87.11
90.27
93.11
Table 2: Comparisons of 5-way few-shot classification with the transductive setup. The highest accuracy is marked in red, the second highest accuracy in blue.
minilmageNet
fieredlmageNNr 5-shot
CIFAR-FS 1-shot
CUB 1-shot
METHOD
BACKBONE
1-shot
5-shot
1-shot
5-shot
5-shot
TPN 19
CONV
55.51
69.86
59.91
73.30
EPNet 23
ResNet-12 12
66.50
81.06
76.53
87.32
ICI 36
ResNet-12
66.80
79.26
80.79
87.92
73.97
84.13
88.06
92.53
iLPC 15
ResNet-12
69.79
79.82
83.49
89.48
77.14
85.23
89.00
92.74
PLCM 9
ResNet- 12
70.92
82.74
82.61
89.47
Ours
ResNet-12
72.01
83.49
83,57
89,81
77.56
85,49
89.40
92.91
Ours (only neg)
ResNet-12
71.46
83.04
83.20
89,33
77.26
85.10
88.73
92.51
Ours (only pos)
ResNet-12
71.83
83.31
83.44
89.57
77.42
85.33
89.31
92.78
the pseudo-labeled negative positive samples generated by MUSIC, which is denoted by "Ours (only neg)" or "Ours (only pos)" in that table. It is apparent to observe that even only using negative lebull..abeling. MUSIC can still be superior other existing FSL methods. Moreover, compared with the results of only using positive pseudo-labeling the results of only using negative are worse. reveals that accurate positive labels still provide more information than negative labels [10]
Transductive semi-supervised few-shot setup In the transductive setup, it is available to access the query data in the inference stage. We also perform experiments in such setup and report results in Table As seen, our approach can still achieve the optimal accuracy on all the four datasets, which justifies the effectiveness of our MUSIC. Regarding the comparisons between (only) using negative and positive pscudo-labels, has similar observations those in Table D

---

Table 3: Comparisons of 5-way few-shot classifica-
tion with the distractive semi-supervise setup.
minilmageNet NieredImageNeet
METHOD
1-shot
5-shot
1-shot
5-shot
MS k-Means
26
49.00
63.00
51.40
69.10
TPN 19
50.40
64.90
53.50
69.90
TPN with MTL.
19
61.30
72.40
71.50
82.70
LST 17
64.10
77.40
73.50
83.40
LST
Curt 11)
EPNet
64.70
76.80
72.20
82.10
i ofuutabbeeedd somplie
Figure 2:
Comparison results of varied unlabeled samples on minilmageNet, The number in the brack- of the legend represents K K-shot in SSFSL.
PLCM 99
68.50
80.20
79.10
87.80
Ours
68.62 80.67
79.69
88.50
Table 4: Comparisons 5-way few-shot classifi- cation with different orders of negative and positive pseudo-labeling w.r.t. unlabeled data.
Table 5: Comparisons of 5-way few-shot classifi- cation without with the minimum entropp loss (Mi- nEnt), cf. Eqn. (6
minilmageNel
CUB
minilmageNer
CUB
SETTINGS
SETTINGS
I-shot
5-shot
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
> pos ->
74.77
85,43
90.47
92.59
Ours (w/o MinEnt)
74.73
85.74
90.45
92.98
pos + neg
74.54
85.09
90.29
92.24
Ours (w/ MinEnt)
74.96
85.99
90.76
93.27
Distractive semi-supervised few-shot setup In real applications, it might not be realistic to collect a clean unlabeled set without mixing any data of other classes. To further validate the robustness of MUSIC, we conduct experiments with the distractive setup, i.e., the unlabeled set contains distractive classes which are excluded in the support set. In that case, positive pseudo-labels are more prone to error, while negative pseudo-labels have much lower risk of error. Table 3 presents the comparison results and shows that our approach can perform as the best solution in all distractive semi-supervised few-shot classification tasks.
Variety-unlabeles semi-supervised few-shot setup order to analyze the performance in the semi-
case of different unlabeled samples, we perform our MUSIC under the supervised setup and compare with state-of-the-aare egg., ICI (36, LST (17) and PLCM [9]. As shown in Figure 2 our approach significantly outperforms over these methods in different K-shot tasks of SSFSL. It further validates the effectiveness and generalization ability of our MUSIC.
Ablation Studies and Discussions
We hereby analyze and discuss our MUSIC approach by answering the following questions based on ablation studies on two datasets, i.e., minilmageNet and CUB.
Will negative pseudo-labels be easier to predict under SSFSI. than positive ones? As assumed previously, in such an extremely label-constrained scenario, e.g., 1-shot learning, might be hard to learn an accurate classifier for correctly predicting positive pseudo-labels. In this sub-section, we conduct ablation studies by alternatively performing negative and positive lseudo-labeling to verify this assumption. In Table different settings denote different orders of negative and positive pseudo-labeling SSFSL. For example, "neg -> pos - represents that we firstly obtain negative pseudo-labels by our MUSIC (without using the final positive labels) and update the model, and then we obtain positive pseudo-label and and update model, and so on. Regarding the iteration time, is relevant to the number of K in the K- K-way classification. In concretely, for 5-way classification, our MUSIC returns the most confident negative pscudo-label it for the next iteration. Thus, after four times of "neg pos", all negative pseudo-labelings finished and the results can be reported. Similarly. "pos - neg + ... means that we get the positive pseudo-labels first, followed by the negative ones. As the results shown in Table 4 obtaining negative pseudo-labele first obviously achieves better results than positive first, which
the current iteration and excludes
that
'The method of positive entropy and obtains the positive lseudo-label by the highest logits above certain threshold (e.g., 0.7).
here is baseline solution, which trains a classifier with cross-
7

---

shows that labeling negative pseudo-labels first can lay better foundation for model training, and further answers the question in this sub-section as "YES".
Is the minimum-entropp loss effective? In MUSIC, further improve the probability confi- dence and then promote pseudo-labeling we equip the minimum-entron test its effectiveness and report the results in Table 5 It can be found that training with MinEnt (i.e. the proposed MUSIC) brings 0.2~0.3% improvements over training without MinEnt in SSFSL.
loss (MinEnt). We here
Is the reject option ò effective? We hereby verify the effectiveness and necessity of the reject option ō in MUSIC. The 8 in our ap- proach acts as safeguard to ensure that the obtained negative pseudo-labels are con- fident as possible. We present the results in Table [ and can observe that MUSIC with å achieves significantly better few-shot classifi- cation accuracy than MUSIC without 5. Additionally, even without 8, our approach still perform well, i.e., the results being comparable or even superior to the results of state-of-the-ane
Table 6: Comparisons of 5-way sew-shot classification without / with the reject option 6, cf. Eqn.
minilmageNet
CUB
SETTINGS
1-shot
5-shot
1-shot
5-shot
Ours (w/o 8)
74.04
85.31
90.44
92.87
Ours (w/ 8)
74.96
85,99
90.76
93.27
What is the effect of iteration manner in our MUSIC? As aforementioned, our approach works as a successive exclusion manner until all negative pseudo-labels are predicted, and even- tually obtaining positive pseudo-labels, labels. As pseudo-labeling conducting, it interesting to investigate how the performance changes as the iteration progresses. We report the correspond- ing results in Figure As shown, on each task of these two datasets, our approach all shows relatively stable growth trend, i.e. 0.5~2% improvements over the previous iteration.
Test ascurasy
15
roarageter DUB (Saho)
75
What is the performance of pseudo-labeling in MUSIC? In this sub-section, we explicitly investigate the error rates of both negative and positive pseudo-labels predicted by our ap- proach. We take 5-way-5-shot classification on minilmageNer and CUB as examples, and first present the useudo-labeling error rates of negative labels in T Since the task is 5-way
teration
Figure 3: Our results of 5-way few-shot
classification with iteration increasing.
prediction, there are totally four iterations of negative pseudo-labeling in MUSIC reported in that table. Except for error rates, we also detailedly report the number of wrong labeled samples in each iteration, as well as the total number of labeled samples. Note that, in the third and forth iterations of negative pseudo-labeling, the total number of labeled samples are less than the number of unlabeled data (i.e., 250), which due to the reject option in MUSIC. That is to say, those samples cannot be lecule-lbbeled with strongly high confidence. Mcanwhile, we also see that, as the pseudo-labeling progresses, the error rates slowly increase, but the final error rate of negative labeling is still no higher than 6.7%. It demonstrates the effectiveness of our approach from a straightforward view.
On the other side, Table X compares the positive pseudo-labeling error rates, and also reports the proportion of labeled samples in the total number of unlabeled samples. Regarding ICI [36 and iLPC (15), although they designed tailored strategies to ensure the correctness of pseudo-labels, e.g., instance credibility inference [36] and label cleaning [15,, these methods still have high pseudo- labeling error rates (over 25%) Compared with them, our approach has significantly low error rates, i.e., about 10%. Meanwhile, we also note that our MUSIC only predicts about 80% of the unlabeled data, which can be regarded to be relatively conservative. However, it reveals that our
Table 7: Pseudo-labeling error rates of negative labels of each iteration in 5-way-5-shot classification.
ITERATION
2
3
4
minilmageNet
0.43% (1. (1.5/)
1.17% (2.9/250)
2.93% (7.3/249)
4. 4.15% (8. (8.2/198) 6.63% (10/152)
CUB
0.69% (1.3/195)
1.83% (3.6/195)
4.47% (8.5/190)
8

---

Table 8: Pseudo-labes labeling error rates and proportion of positive labels in 5-way-5-shot classification.
METRIC
DATASET
ICI 36
iLPC [5
Ours (w/o 6)
Ours
minilaageeet
24.72% (61.8/250)
23.92% (59.8/250)
14.92% (37.3/250)
9.09% (18.0/198)
ERROR RATE
CUB
29.03% (56.6/195)
26. 10% (50.9/195)
20.00% (39.0/195) 11.91 % (18.1/152)
minilmageNet
100%
100%
100%
79.20%
PROPORTION
CUB
100%
100%
100%
77.95%
(a) ICI 36
(b) iLPC 15
(c) Our MUSIC
Figure 4: t-SNE 2 visualization samples for the 5-way 5-shot 50-unlaheled task. Different colors denote different classes. For ICT and iL,C, the circled data points represent the selected samples. For our MUSIC, the circled are the samples associated with high confidence of positive pscudo-labels in the final iteration.
approach still has large room for performance improvement. Moreover, Table 8 also shows that, even our approach removes the reject option strategy, its error rates are still lower than those of state-of-the-arts.
Additionally, we visualize the positive pseudo-labels with high confidence by t-SNE 32 in Figure Compared with these methods, can obviously find that the positive samples with high confidence predicted by our MUSIC are both more centralized and distinct. This also explains the satisfactory performance of our approach when using the positive pseudo-labels and using the positive alone (cf. Table T and Table from the qualitative perspective.
Are the pseudo-labels of MU- SIC a balanced distribution? In this sub-section, interested in investigating what kind of data distribution the pseudo-labeled samples are to further analyze how ap- proach works well. As shown in Table 9 we present the averaged number of both negative and positive pseudo-labeled samples in all 600 episodes of 5-way-5-shot classification tasks on mini- ImageNet. It is apparent to see that the pscudo-labeled samples present a very clearly balanced distribution, which aids in the modeling of classifiers across different classes in SSFSL.
Table 9:
The averaged number of negative (positive) peeudo-labelen samples not selooelnging (belonging) to diffcrent classes in our MUSIC.
still
CLASS INDEX
3
5
# of neg.
49.85
49.74
49.93
50.19
50.30
# of pos. pseudo-labeling
40.13
41.13
40.21
39.89
39.99
Conclusion
In this paper, dealt with semi-supervised few-shot classification by proposing simple but effective approach, termed as MUSIC. Our MUSIC worked in a successive exclusion manner to predict negative pseudo-labels with much confidence as possible in the extremely label-constrained tasks. After that, models can be updated by leveraging negative learning based on the obtained negative pseudo-labels, and continued negative pseudo-labeling until all negative labels were returned. Finally, combined with the incidental positive lseudo-labels, we augmented the small support set of labeled data for evaluation in SSFSI. In experiments, comprehensive empirical studies validated the effectiveness of MUSIC and revealed its working mechanism. In the future, we would like to investigate the theoretical analyses about our MUSIC in terms of its convergence and estimation error bound, as well as how performing on traditional semi-supervised learning tasks.

---

References
[1] Arman Afrasiyabi, Hugo Larochelle, Jean-François Lalonde, and Christian Gagné. Matching feature sets for few-shot image classification. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2022.
[2] Luca Bertinetto, Joao F. Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. In Proc. Int. Conf. Learn. Representations. 2019.
[3] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Wang, and Jia-Bin Huang. A closer look
at few-shot classification. In Proc. Int. Conf. Learn. Representations, 2019.
[4] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta-
tion of deep networks. In Proc. Int. Conf. Mach. Learn., pages 1 1126-1135, 2017.
[5] Yi Gao and Min-Ling Zhang. Discriminative complementary-labe learning with weighted loss. In Proc. Int. Conf. Mach. Learn.. pages 3587-3597, 2021.
[6] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In
Advances in Neural Inf. Prucess. Syst., pages 529-536, 2005.
[7] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. IEEE Conf. Comp. Vis. Pull. Recogn., pages 770-778, 2016.
[8] Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Cross attention network for few-shot classification. In Advances Neural Inf. Process. Syst. pages 4005-4016, 2019.
[9] Kai Huang, Jie Geng, Wen Jiang, Xinyang Deng, and Zhe Xu. Pseudo-loss confidence metric for semi-supervised few-shot learning. In Proc. IEEE Int. Conf. Comp. Vis., pages 8671-8680, 2021.
[10] Takashi Ishida, Gang Niu, Weihua Hu, and Masashi Sugiyama. Learning from complementary
labels. In Advances in Neural Inf. Process. Syst., pages 5644-5654, 2017
[11] Dahyun Kang, Heeseung Kwon, Juhong Min, and Minsu Cho. Relational embedding for few-shot classification. In Proc. IEFE Int. Conf. Comp. Vis., pages 8822-8833, 2021.
[12] Youngdong Kim, Junho Yim, Juseung Yun, and Junmo Kim. NLNL: Negative learning for
noisy labels. In Proc. IEEE Int. Conf. Comp. Vis., pages 101-110, 2019.
[13] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
[14] Samuli Laine and Timo Aila. Temporal ensembling for semisupervised learning. In Proc. Int.
Conf. Learn. Representations, 2017.
[15] Michalis Lazarou, Tania Stathaki, and Yannis Avrithis. Iterative label cleaning for transductive and semi-supervised few-shot learning. In Proc. IEEE Int. Conf. Comp. Vis., pages 8751-8760,
2021.
[16] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521:436-444, 2015, [17] Xinzhe Li, Qianru Sun, Yaoyao Liu, Shibao Zheng, Qin Zhou, Tat-Seng Chua, and Bernt Schiele. Learning to self-train for semi-supervised few-shot classification. In Advances in Neural Inf. Process. Syst.. pages 10276-10286, 2019.
[18] Li Liu, Wanli Ouyang, Xiaogang Wang, Paul Fieguth, Jie Chen, Xinwang Liu, and Matti Pietikäinen. Deep learning for generic object detection: A survey. Int. J. Comput. Vision, 128:261-318, 2020.
[19] Yanbin Liu, Juho Lee, Minseop Park, Sachoon Kim, Eunho Yang, Sung Ju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. In Proc. Int. Conf. Learn. Representations, 2019.
[20] Yang Liu, Weifeng Zhang, Chao Xiang, Tu Zheng, Deng Cai, and Xiaofei He. Learning to affiliate: Mutual centralized learning for few-shot classification. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2022.
[21] Xu Luo, Longhui Wei, Liangjian Wen, Jinrong Yang, Lingxi Xie, Zenglin Xu, and Qi Tian. Rectifying the shortcut learning of background for few-shot learning. In Advances in Neural Inf. Process. Syst., pages 13073-13000 13085, 2021.
[22] Takeru Miyato, Andrew M. Dai, and Ian Goodfellow. Adversarial training methods for semi- supervised text classification. In Proc. Int. Conf. Learn. Representations. 2017.
[23] Rodríguez Pau, Issam Laradji, Alexandre Drouin, and Alexandre Lacoste. Embedding propa-
gation: Smoother manifold for few-shot classification. In Proc. Eur. Conf. Comp. Vis., pages 121-138, 2020.
[24] Hang Qi, Matthew Brown, and David G. Lowe. Low-shot learning with imprinted weights. In
Proc. IEEE Conf. Comp. Vis. Patt. Recogn. pages 5822-5830, 2018.
[25] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In Proc. Ini.
Conf. Learn. Representations, 2017.
10

---

[26]
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenen- baum, Hugo Larochelle, and Richard S. Zemel. Meta-learning for semi-supervised few-shot classification. In Proc. Int. Conf. Learn. Representations, 2018.
[27] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. Int. J. Comput. Vision, 115(3):211222 11-222, 2015.
(28) Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. Proc. Int. Conf. Learn. Representations, 2019.
[29] Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Inf. Process. Syst., pages 4077-4087, 2017.
[30] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Inf. 2017.
Process. Syst., pages
[31] Yonglong Tian, Yue Wang. Dilip Krishnan, Joshua B. Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: good embedding is all you need? In Proc. Eur. Conf. Comp. Vis., pages 266-282, 2020.
[32] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Mach. Learn. Res., 9:2579-2605, 2008
[33] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Advances in Neural Inf. Process. Syst., pages 3637-3645, 2016.
[34] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech-UCSS
hirds-200-201 dataset. Technical report, California Institute of Technology,
2011.
[35] Yuchao Wang, Haochen Wang, Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Le. Sumi-super semantic segmentation using unreliable pseudo-labels. In Proc. IEFE Conf. Comp. Vis. Patt. Recogn., 2022.
[36] Yikai Wang, Chengming Xu, Chen Liu, Li Zhang, and Yanwei Fu. Instance credibility inference for few-shot learning. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 12836-12845, 2020.
[37] Yaqing Wang. Quanming Yao, James T. Kwok, and Lionel M. Ni. Generalizing from a few examples: A survey few-shot learning. ACM Comput. Surveys, 53(63):1-34, 2021. [38] Yikai Wang. Li Zhang, Yuan Yao. and Yanwei Fu. How trust unlabeled data instance credibility inference for few-shot learning. IEEE Trans. Pattern Anal. Mach. Intell., DOI:
10.1 10.11099TPMMM 2022022.1120 .30861 1886140,220 2021.
[39] Xiu-Shen Wei, Yi-Zhe Song, Oisin Mac Aodha, Jianxin Wu, Yuxin Peng, Jinhui Tang, Jian Yang, and Serge Belongie. Fine-graines image analysis with deep learning: A survey, IEEE Trans. Pattern Anal. Mach. Intell., DOI: 10. 109/TPAMM 113612648,, 2021.
[40] Davis Wertheimer, Luming Tang, and Bharath Hariharan. Few-shot classification with feature map reconstruction networks. Proc. IEEE Conf. Comp. Vis. Part. Recogn., pages 8012-8021,
2021.
[41] Jiangtao Xie, Fei Long, Jiaming Lv, Qilong Wang, and Peihua Li. Joint distribution matters: Deep brownian distance covariance for fhw-shot classification. In Proc. IEFE Conf. Comp. Vis. Patt. Recogn., 2022.
[42] Han-Jia Ye. Hexiang Hu, and De-Chuan Zhan. Learning adaptive classifiers synthesis for generalized few-shot learning. Int. J. Comput. Vision. 1990 1959309999,,.0111111111111911 2021.
[43] Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn.. pages 8808-8817, 2020.
[44] Zhongjie Yu, Lin Chen, Zhongwei Cheng, and Jiebo Luo. TransMatch; transfer-learning scheme for semi- supervuper few-shot learning. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 1886412864. 2020.
[45] Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. DeepEMD: Few-shot image classification with differentiable earth mover's 's distance and structured classifiers. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn, pages 12203-1221,, 2020.
[46] Zhi-Hua Zhou and Ming Li. Semi-supervised learning by disagreement. Knowl, and Inf. Syst.,
24(3):415-44 2010.
11

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes] See Section 5 and we treat it as the future work.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or a URL)? [Yes] See Section 4.1
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] We have run experiments multiple times and report the averaged results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
12


Reviews:
[{"review_id": "jjGSW4yO_z_", "reviewer": "Reviewer_46zy", "summary": "This submission proposes a simple yet effective learning method for Semi-Supervised Few-Shot Learning (SSFSL) called MUSIC. Compared to previous methods, the authors propose to learn negative labels first and then focus on positive label learning. The underlying logic is that under few-shot learning scenario, it is easier to exclude negative predicted labels than select positive labels. The experiments show that the simple method achieves state-of-the-art performance on four benchmark datasets. ", "questions": "Overall I have little confusion for the methodology and technical details. There are two suggestions:\n\n1) In the ablation study, the authors only investigate whether the reject option $\\delta$ is effective or not. It would be better to further study what would be the optimal value of $\\delta$ and whether this hyper parameter is agnostic to different datasets' distribution.\n\n2) In Table 4, the authors conduct an interesting experiment for the order of negative and positive pseudo labels learning. I am curious what is the optimal number of iterations for the neg -> pos -> neg ... in the MUSIC method, or it is dependent on different datasets?", "limitations": "I am basically satisfied with the submission in terms of methodology, motivation and experiments. I provided some suggestions in the above section (\"questions\") which also contain some constructive suggestions. ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The authors propose a straightforward learning method for SSFSL. The motivation is clear and technical details are illustrated with sufficient details.\n\n+ The authors conduct experiments on four benchmark datasets and achieves state-of-the-art performance.\n\n+ The authors further dive into different aspects of the MUSIC method and provide ablation study, which is much appreciated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_i89ciFLXVq", "reviewer": "Reviewer_bRfN", "summary": "This work proposes a novel negative pseudo-labeling algorithm to tackle semi-supervised few-shot learning. The key insight is negative labels are easier to predict, therefore, pseudo labels on unlabeled samples can be better predicted by iteratively predicting negative labels until all the negative ones are excluded. Extensive experiments have been conducted on four few-shot learning benchmark and show better performance than SOTA. ", "questions": "I would suggest the authors to address my concerns regarding the marginal improvement and hard negative classes, as listed in the weakness. \n\nPost-rebuttal \nThe authors addressed all of my concerns. Therefore, I would recommend a Weak Accept.", "limitations": "I cannot find the limitations and potential negative societal impact in this paper. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. The idea of generating pseudo-labels by gradually rejecting negative labels is novel and interesting. \n2. The experiments are quite extensive, including results on four public benchmarks and many analysis. \n3. The paper is written well and easy to follow.\n\nWeakness\n\n1. Although some negative labels may be indeed easier to be predicted than the positive one, there still exist hard negative classes that are equally hard to be recognized. Those hard negative classes are in fact the most important information for learning a good classifier.  I am missing how this work can handle this case. \n\n2. Although the method is simple and novel, the achieved improvement over SOTA is marginal (less than 1%) in most of cases, see Table 1.\n\nPost-rebuttal \n\nMy concerns about the weakness have been addressed.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BFq-3JfeWUk", "reviewer": "Reviewer_nug7", "summary": "This work applies negative learning to the problem of semi-supervised few-shot learning. It uses negative pseudo-label to gradually get rid of unlikely predictions. It also minimizes the entropy of the predicted probability of unlabeled target domain data to promote pseudo-labeling.", "questions": "1. The entropy-loss in (6) seems to share a similar spirit with pseudo-labeling (also known as soft pseudo-labeling in other places). What if you apply only (6) and do not use (5) at all? \n\n2. The author of the baseline work [38] also uses pseudo labels, and they rank pseudo labels in a sophisticated way. When you say pretraining the network as in [38], do you mean you use their method to pre-train the model, or do you mean you pre-train it in the same way but do not use pseudo labels at all during pre-training? ", "limitations": "The author needs to compare the baseline work [38] clearly to show a more convincing result.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nIn general, the paper is easy to follow and highlights the key point clearly.\n\nWeakness: \n1. The major problem of this work lies in the experiment section. The author mentioned that this work pretraining the network as in previous work [38], so it should be the baseline of this work. However, the table does not include that work [38] at all. Furthermore, comparing Table 2 in the baseline work [38] and Table 1&2 in this work, we can find that the baseline work [38] actually performs better in some settings. \n\n2. While negative learning is the key point of this work, as shown in Table 1&2, the improvement from using only positive pseudo-label to using both negative and positive pseudo-labels seems trivial. Since pseudo-label itself is a well-known idea, the novelty will be very limited if the negative pseudo-labels do not really make a difference.\n\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1bpoBdoXfwh", "reviewer": "Reviewer_QttX", "summary": "The paper proses a simple approach to SSFSL that employs negative label prediction when producing pseudo-labels for unlabelled examples. The model consists of a standard ResNet12 architecture that is pre-trained on the base data following an L2-normalizing single-layer classifier that is fine-tuned based on the support data. The fine-tuning process first updates parameters based on standard cross-entropy loss of the support examples. It then iteratively removes negative pseudo-labels from the unlabelled examples but using a modified cross-entropy negative label predictor loss, and finally, performs updates based on the positive pseudo-labels of the unlabelled set for examples which all negative labels have been identified. Various experiments and ablations studies are reported that demonstrate the efficacy of the approach.", "questions": "The choice to decouple the training of the feature extractor from the few-shot learning algorithm itself is an interesting one. It is often seen that end-to-end training of the extractor through episodic procedures where the few-shot updates are also applied results in better performance. More specifically, the updates shown in the PyTorch code block could be applied directly to f and F together where the inputs are just the raw images. Was this something the authors explored? If so, what was the outcome? If not, why not?", "limitations": "The authors have adequately addressed technical limitations of the work (although further studies on empirical biases based on data domain would have been interesting to see). However, there is no discussion of potential negative societal impact of the work; in fairness to the authors, this is an algorithmic work and the societal impacts can be speculative at time; but they could benefit from a short discussion of what their method, as an effective SSFSL classifier, can enable in applied industrial settings.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Paper is overall very well-written.\n- Algorithmic choices are well-motivated, backed up by both good intuition and supportive ablation studies.\n- Method is simple to understand on the first go but also very empirically powerful as demonstrated by series of experiments.\n- Negative labelling is a very interesting insight and can prove consequential specifically in the domain of SSFSL which is very applicable to applied settings where labelling can be expensive but lots of unlabelled data is available.\n\nWeakness:\n- There are some typos, such as \"detailedly\" (261), \"can performs\" (208), and most important \"logits = f.forward(x)\" and\n\"loss = F.nll_loss(F.log_softmax(logits), labels)\" in the algorithm blocks where \"x\" should be \"S\" and \"labels\" should be \"targets\" to my understanding of the procedure\n- The algorithm block is in PyTorch which I personally appreciate but can be difficult to navigate if read doesn't have existing PyTorch proficiency; I believe a pseudo-code algorithm block would be more appropriate with the PyTorch code moved to the supplementary material. I recognize that this was done to reinforce the fact that the method can be implemented in a few-lines of code.\n\nMiddleground:\n- The algorithm is simple and effective; but as a result doesn't contain very significant technical novelty and contribution. That being said, the authors have embraced its simplicity in the language of the paper throughout which addresses this potential problem.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept