FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling
Authors: Bowen Zhang
Yidong Wang
Wenxin Hou
Hao Wu
Jindong Wang
Manabu Okumura
Takahiro Shinozaki
Venue: NeurIPS 2021 Poster (2021)
Keywords: semi-supervised learning
pseudo labeling
consistency regularization

Abstract: The recently proposed FixMatch achieved state-of-the-art results on most semi-supervised learning (SSL) benchmarks. However, like other modern SSL algorithms, FixMatch uses a pre-defined constant threshold for all classes to select unlabeled data that contribute to the training, thus failing to consider different learning status and learning difficulties of different classes. To address this issue, we propose Curriculum Pseudo Labeling (CPL), a curriculum learning approach to leverage unlabeled data according to the model's learning status. The core of CPL is to flexibly adjust thresholds for different classes at each time step to let pass informative unlabeled data and their pseudo labels. CPL does not introduce additional parameters or computations (forward or backward propagation). We apply CPL to FixMatch and call our improved algorithm FlexMatch. FlexMatch achieves state-of-the-art performance on a variety of SSL benchmarks, with especially strong performances when the labeled data are extremely limited or when the task is challenging. For example, FlexMatch achieves 13.96% and 18.96% error rate reduction over FixMatch on CIFAR-100 and STL-10 datasets respectively, when there are only 4 labels per class. CPL also significantly boosts the convergence speed, e.g., FlexMatch can use only 1/5 training time of FixMatch to achieve even better performance. Furthermore, we show that CPL can be easily adapted to other SSL algorithms and remarkably improve their performances. We open-source our code at https://github.com/TorchSSL/TorchSSL.

FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling
Bowen Zhang Tokyo Institute of Technology bowen abem. titech. ac jp
Yidong Wang" Tokyo Institute of Technology wang y. caum. titech. ac. jp
Wenxin Hou
Hao Wu Tokyo Institute of Technology wu. wu.h. jmm titech.ac. ac. jp
Jindong Wang Microsoft Research Asia indwang@miccosoff com
Microsoft wenxinhooummicrroooft.. com
Manabu Okumura
Takahiro Shinozaki!
Tokyo Institute of Technology okuêpi. titech.ec. titech.ac.jp
Tokyo Institute of Technology shinot@ict. e. titech. ac.
Abstract
The recently proposed FixMatch achieved state-of-the-art results on most semi- supervised learning (SSL) benchmarks. However, like other modern SSL algo- rithms, FixMatch uses pre-defined constant threshold for all classes to select unlabeled data that contribute to the training, thus failing to consider different learning status and learning difficulties of different classes. To address this issue, we propose Curriculum Pseudo Labeling (CPL.), curriculum learning approach to leverage unlabeled data according to the model's 's learning status. The core of CPL is flexibly adjust thresholds for different classes at each time step let pass informative unlabeled data and their pseudo labels. CPL does not introduce additional parameters or computations (forward backward propagation). We apply CPL to FixMatch and call our improved algorithm FlexMatch. FlexMatch achieves state-of- of-th-------rrr art performance on variety of SSL benchmarks, with espe- cially strong performances when the labeled data are extremely limited or when the task challenging. For example, FlexMatch achieves 13.96% and 18.96% еrror rate reduction over FixMatch tively, when there are only 4 labels per class. CPI. also significantly boosts the convergence speed, e.g., FlexMatch can use only 1/5 training time of FixMatch achieve even better performance. Furthermore, we show that CPL can be easily adapted to other SSL algorithmss and remarkably improve their performances. We open-source our code at httpps: //tgt//gttttt/t/// com/TorchsSLLTorccsSS
CIFAR-100 and STL-100 datasets respec-
Introduction
Semi-supervised learning (SSL) has attracted increasing attention in recent years due to its superiority in leveraging a large amount of unlabeled data. This is particularly advantageous when the labeled data are limited in quantity or laborious to obtain. Consistency regularization B and pseudo labeling 4 g are two powerful techniques for utilizing unlabeled data and have been widely used in The recently proposed FixMatch [14] achieves competitive results
modern SSL algorithms [9
"Equal contribution.
i "oorrsppnniigg author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

by combining these techniques with weak and strong data augmentations and using cross-entropy loss the consistency regularization criterion.
However, drawback of FixMatch and other popular SSL algorithms such as Pseudo-L Labeling 4 and Unsupervised Data Augmentation (UDA) [TT] is that they rely a fixed threshold to compute the unsupervised loss, using only unlabeled data whose prediction confidence is above the threshold. While this strategy can make sure that only high-quality unlabeled data contribute the model training, it ignores a considerable amount of other unlabeled data, especially at the early stage of the training process, where only few unlabeled data have their prediction confidence above the threshold. Moreover, modern SSL algorithms handle all classes equally without considering their different learning difficulties.
To address these issues, we propose Curriculum Pseudo Labeling (CPL., curriculum learning 15 strategy to take into account the learning status of each class for semi-supervised learning. CPI. substitutes the pre-defined thresholds with flexible thresholds that are dynamically adjusted for each class according to the current learning status. Notably, this process does not introduce any additional parameter (hyper-parameter or trainable parameter) or extra computation (forward or back propagation). We apply this curriculum learning strategy directly FixMatch and call the improved algorithm FlexMatch.
While the training speed remains efficient that of FixMatch, FlexMatch converges significantly faster and achieves state-of stute-of-the-ae aftthh-art performances on most SSL image classification benchmarks. The benefit of introducing CPL is particularly remarkable when the labels are scarce when the task challenging. For instance, on the STL-110 dataset, FlexMatch achieves relative performance improvement over FixMatch by 18.96%, 16. 16.11% and 7. 7.68% when the label amount is 400, 2500, and 10000 respectively. Moreover, CPL further shows its superiority by boosting the convergence speed with CPL, FlexMatch takes less than 1/5 training time of FixMatch reach its final accuracy. Adapting CPL to other modern SSL algorithms also leads to improvements in accuracy and convergence speed.
To sum up, this paper makes the following three contributions:
We propose Curriculum Pseudo Labeling (CPL), curriculum learning approach of dynami- cally leveraging unlabeled data for SSL. It is almost cost-free and can be easily integrated to other SSL methods.
CPI. significantly boosts the accuracy and convergence performance of several popular SSL algorithms common benchmarks. Specifically, FlexMatch, the integration of FixMatch and CPL, achieves state-of-the-arl results.
We open-source TorchSSL, a unified PyTorch-basee based semi-supervised learning codebase for the fair study of SSL algorithms. TorchSSL includes implementations of popular SSL algorithms and their corresponding training strategies, and is easy to use or customize.
2 Background
Consistency regularization follows the continuity assumption of SSL (2). The most basic consis- tency loss SSL., such as in IT Model (9) Mean Teacher 10 and MixMatch [2], is the f-2 loss:
aB
1p (v)))(u))) - P.. (ylw(())))
where B is the batch size of labeled data, l is the ratio of unlabeled data labeled data, is stochastic data augmentation function (thus the two terms in [ are different). uy denotes a piece of unlabeled data, and Pm. represents the output probability of the model. With the introduction pseudo labeling techniques [5] 77 the consistency regularization is converted to an entropy minimization process [16], which is more suitable for the classification task. The improved consistency loss with pseudo labeling can be represented as:
(maxx(... > T) H (pm (wo)) Pm (v | (4b)) (ub)))

---

Step 1
Step 2
Step 1-1
Estimateed learning effects each class
Step
Class
Adjust flexible thresholds
samples reaching
Class 2
Class C
Figure 1: Illustration of Curriculum Pseudo Label (CPL). The estimated learning effects of each class are decided by the number unlabeled data samples falling into this class and above the fixed threshold. They are then used to adjust the flexible thresholds to let pass the optimal unlabeled data. Note that the estimated learning effects do not always grow - they may also decrease if the predictions of the unlabeled data fall into other classes in later iterations.
where H is tross-entropy. т is the pre-defined threshold and pm (g/ (w)))) is the pseudo label that can either be 'hard' one-hot label [4 14 or a sharpened 'soft' one [.]. The intention of using threshold is to mask out noisy unlabeled data that have low prediction confidence.
FixMatch utilizes such consistency regularization with strong augmentation achieve competitive performance. For unlabeled data, FixMatch first uses weak augmentation to generate artificial labels. These labels then used the target of strongly-augmented data. The unsupervised loss term in FixMatch thereby has the form:
1 (max(pm (y)w(()))) F) (m.
where Q2 is a strong augmentation function instead of weak augmentation w.
Of the aforementioned works, the pre-defined threshold (T) is constant. We believe this can be improved because the data of some classes may be inherently more difficult to learn than others. Curriculum learning [1] is a learning strategy where learning samples are gradually introduced according to the model's learning process. In such way, the model always optimally challenged. This technique is widely employed in deep learning research 117
3 FlexMatch
3.1 Curriculum Pseudo Labeling
While current SSL algorithms render pseudo labels of only high-confidence unlabeled data cut off by a pre-defined threshold, CPL renders the pseudo labels to different classes and at different time steps. Such process is realized by adjusting the thresholds according to the model's learning status of each class.
However, it is non-trivial to dynamically determine the thresholds according to the learning status. The most ideal approach would calculating evaluation accuracies for each class and use them to scale the threshold, as:
Te(c) = (c)
(4)
where Te(c) is the flexible threshold for class at time step and (c(c) is the corresponding evaluation accuracy. this way. lower accuracy that indicates less satisfactory learning status of the class will lead to lower threshold that encourages more samples of this class to be learned. Since we cannot use the evaluation set the model learning process, one may have to separate an extra validation set from the training set for such accuracy evaluations. However, this practice show two fatal problems: First, such labeled validation set separated from the training set is expensive under SSL scenario as the labeled data are already scarce. Second, to dynamically adjust the thresholds in the training process, accuracy evaluations must be done continually each time step I, which will considerably slow down the training speed.

---

In this work, we propose Curriculum Pseudo Labeling (CPL) for semi-supervised learning. Our CPL uses an alternative way estimate the learning status, which does not introduce additional inference processes, nor needs an extra validation set. As believed in [1]. high threshold that filters out noisy pseudo labels and leaves only high-quality ones can considerably reduce the confirmation bias (22) Therefore, our key assumption is that when the threshold is high, the learning effect of a class can be reflected by the number of samples whose predictions fall into this class and above the threshold. Namely, the class with fewer samples having their prediction confidence reach the threshold is considered to have a greater learning difficulty or a worse learning status, formulated as:
(c))) Σ I (ma un)) > T) 1 [arg max(pm.c(ylun)) =c).
where (c(c) reflects the learning effect of class c time step (y.((y)u,)) is the model's prediction for unlabeled data un at time step (, and N is the total number of unlabeled data. When the unlabeled dataset is balanced (i.e., the number of unlabeled data belonging to different classes are equal or close), larger (r(c) indicates better estimated learning effect. By applying the following normalization to
(c) to make its range between 0 to 1, it can then be used to scale the fixed threshold T:
oc(c)
Be(c) max Oz
(6)
Te(c) = Be(c) .T.
One characteristic of such normalization approach is that the best-learned class has its 3e(c) equal to 1, causing its flexible threshold equal to т. This desirable. For classes that are hard learn, the thresholds are lowered down, encouraging more training samples in these classes to be learned. This also improves the data utilization ratio. As learning proceeds, the threshold of a well-learned class is raised higher to selectively pick up higher-quality samples. Eventually, when all classes have reached reliable accuracies, the thresholds will all approach т. Note that the thresholds do not always grow, may also decrease if the unlabeled data is classified into different class in later iterations. This new threshold used for calculating the unsupervised loss in FlexMatch, which can be formulated as:
1 B.
Cu,t
1 (max(1)) Te(arg max(4)))) H Pm (y)
where 9b = Pm (y Lol (w.)). The flexible thresholds are updated at each iteration. Finally, we can formulate the loss in FlexMatch the weighted combination (by X) of supervised and unsupervised loss:
L. = C. + ALu,t,
where Cg is the supervised loss on labeled data:
C. 1 B
Σ
H Pm (ylw(())))
(10)
Note that the cost of introducing CPL is almost free. Practically, every time the prediction confidence of an unlabeled data u, is above the fixed threshold r, the data, and its predicted class are marked and will be used for calculating (c(c) at the next time step. Such marking actions time the consistency loss is computed. Therefore, FlexMatch does not introduce additional forward propagation processes for evaluating the model's learning status, nor new parameters.
bonus actions each
3.2 Threshold warm-up
We noticed in our experiments that at the early stage of the training, the model may blindly predict most unlabeled samples into certain class depending the parameter initialization(iee. more likely to have confirmation bias). Hence, the estimated learning status may not be reliable at this stage. Therefore, we introduce a warm-up process by rewriting the denominator in Eq. (6) as:
(c(c)
Be(c)=
(11)
max
max Ct. N - -oo

---

Algorithm 1 FlexMatch algorithm.
Input: X' {(rm 3m) E (I....,M) M)}, U {un n E (I,...,,)) {M labeled data and N unlabeled data.
2: in n (I,...,N) [lnitialize predictions of all unlabeled data indicating unused.) )
3: while not reach the maximum iteration do
4: for c
do
5: (c)) (re)=Em=] ( c) (Compute estimated learning effect.)
6: if max (cc) >m== ( =-1) then
7:
Calculate ((c) using Eq. (Threshold warms up when unused data dominate.) |
8:
else
9: Calculate ((c) using Eq. 6 (Compute normalized estimated learning effect.)
end if
11: Calculate T(c) using Eq. 7 (Determine the flexible threshold for class c.)
12:
end for
13:
for b
to BB do if Pm (ug)) > T then
14:
15:
in arg max 9b (Update the prediction of unlabeled data uo.
16:
end if end for
17:
18:
Compute the loss via Eq. (8.
10p and 9
19:
end while
20:
Return: Model parameters.
where the term N - -ee (c) can be regarded the number of unlabeled data that have not been used. This ensures that at the beginning of the training, all estimated learning effects gradually rise from O until the number of unused unlabeled data is no longer predominant. The duration of such period depends on the unlabeled data amount (ref. N in Eq. (II) and the learning difficulty (ref. the growing speed of (c) in Eq. (TT)) of the dataset. In practice, such a warm-up process is very easy to implement as we can add an extra class to denote the unused unlabeled data. Thus calculating the denominator of Eq.
simply converted finding the maximum among c + classes.
3.3 Non-linear mapping function
The flexible threshold in Eq. (7 is determined by the normalized estimated learning effects via linear mapping. However, it may not be the most suitable mapping in the real training process, where the increase or decrease of (c(c) may make big jumps in the early phase where the predictions of the model are still unstable; and only make small fluctuations after the class is well-learned in the mid and late training stage. Therefore, it is preferable the flexible thresholds can be more sensitive when (c) is large and vice versa.
We propose non-linear mapping function to enable the thresholds have non-linear increasing curve when (c(c ranges uniformly from to 1, as formulated below:
(i()) M(B((c))
(12)
where M(-) is a non- linear mapping function. It is clear that Eq. can be seen as a special case by setting M to the identity function. The mapping function M should be monotonically increasing and have a maximum no larger than 1/r (otherwise the flexible threshold can be larger than 1 and filter out all samples). To avoid introducing additional byper-parameters (e.g. lower limits of the flexible thresholds), we consider the mapping function to have a range from 0 to so that the flexible thresholds range from 0 to
monotone increasing convex function lets the thresholds grow slowly when 3x(c) is small, and become more sensitive as (c(c) gets larger. Hence, intuitively choose convex function with the 2-r for our experiments. We also conduct an ablation study
mhove-mentioned properties M(x) = =r
to compare among mapping functions with different convexity and concavity Sec. 4.4 The full algorithm of FlexMatch is shown in Algorithm
5

---

Table 1: Error rates on DIFAR-10/100 SVHN, and STL-10 datasets. The 'Flex' prefix denotes applying CPL to the algorithm, and 'PL' an abbreviation of Pseudo-Labeling. STL-10 dataset does not have label information for unlabeled data, thus its fully-supervised result is unavailable.
Dzraset
CIPAR-10
CIFAR-100
STL-10
SVHN
Label Amouni
40
250
4000
400
2500
10000
40
230
1000
40
1000
PL.
74.6..0
46.49.223
15.05.015
37,45.0
57. 57.744025
36.55
14.68.008
55.45.20
3264467
64.61.50
9,40,003
Plex-PL
13
41.14
14.75a011
85.72
56.12
2515
13.42:211
52,06:19
32.05
63.21
12,05
UDA
10462335
5.16
4.29.000
46.39
27.73,031
22.49.023
57.4 57.42
9,72g115
6.646617
5.12
188
5.4
5.02
4.24
45.17
27.08
21.91.4.00
29.53:2.10
9.03.446
6.10
3.4
2.02
FiaMauch
747aro
4.86
2.00
46.42-012
28.03:1016
22
35,07
6.25
2.
1,962000
FlexMarch
4.97
u298
4.19
39.94-142
26.49.000
21.90
29.15
8,23:a3
5.77
8.19:331
6.72
Fully-Supervised
4 624 on5
19.30= 1:4
2.13ge80
4 Experiments
We evaluate FlexMatch and other CPL-enabled algorithms on common SSL datasets: CIFAR- 10/100 (23), SVHN (24). STL-10 (25] and ImageNet (26), and extensively investigate the perfor- mance under various labeled data amounts. We mainly compare our method with Pseudo-Labeling 4. UDA [I] and FixMatch [14), since they all involve pre-defined threshold. The results of other popular SSL algorithms are in the appendix. We also add a fully-supervised experiment for each dataset to better understand the results of SSL algorithms. Note that previously suggested [27] fully-supervised comparisons use only the labeled set for training, whose purpose is to manifest the improvement brought by the introduction of unlabeled data. With the development of modern SSL algorithms, however, semi-supervised approaches are achieving competitive performance with supervised ones, even better performance due to the strength of consistency regularization. There- fore, our fully-sueervised comparisons are conducted with all data labeled, and apply weak data augmentations following Eq. (TO). We re-implement all baselines using our PyTorch 28 codebase: TorchSSL, which is introduced in the appendix.
For fair comparison, optimizer for all experiments is standard stochastic gradient descent (SGD) with a momentum of 0.9 initial learning rate of 0.03 with cosine learning rate decay 7sk ), where 770 is the initial learning rate, k is the current training step schedule [3 as 1) no cos( I&K
the same Syper-parametery following FixMatch [4]. Concretely, the
29
30
For all datasets, we
and K is the total training step that is set to 220 We also perform an exponential moving average with the momentum of 0.999. The batch size of labeled data is 64 except for ImageNet. 1L is set to be 1 for Pscudo-Label and 7 for UDA. FixMatch, and FlexMatch. is set to 0.8 for UDA and 0.95 for Pseudo Label, FixMatch, and FlexMatch. These setups follow the original papers. The strong augmentation function used in our experiments is RandAugment (32] We use ResNet-50 33 for the
ImageNet experiment and Wide ResNet (WRN) [4] and its variant 35 for other datasets. Detailed hyper-parameters are listed in the appendix.
We adopt two evaluation metrics: (I) the median error rate of the last 20 checkpoints following [2] 14, and (2) the best error rate in all checkpoints. We argue that the median approach is not suitable when the convergence speeds of the algorithms show significant differences the large number of redundant iterations may result in over-fitting for the fast-converge algorithms. Therefore, we report the best error rates for all algorithms, while the results of the median approach are also provided in the appendix, showing that our FlexMatch still achieves the best performance. We run each task three times using distinct random seeds to obtain the error bars.
4.1
Main results
The classification error rates on CIFAR-10/10N STL and SVHN datasets are in Table 1 and the results on ImageNet are in Sec. 4.2 Note that the SVHN dataset used in our experiment also includes the extra set that contains 31 additional samples. Results demonstrate that Flex achieves the state-of-the-a performance on most of the benchmark datasets except for SVHN where Flex-UDA (i.e., UDA with CPL) and UDA have the lowest error rate the 40-label split and the 1000-label split, respectively. We also provide the detailed precision, recall, Fl, and AUC results in the appendix. Our CPL (FlexMatch) has the following advantages:
CPL achieves better performance on tasks with extremely limited labeled data. Our Fltxh significantly outperforms other methods when the amount of labels is extremely small. For instance,

---

0.6
1.
FizMaich
0.6
0.5
Flastfatch
10
FlesMaich
0.0
5c 2004 4DOK BOCK
1000k
200c 100k cODE BOOK
1000c
40k
80k
120K
Jock
2008
45k
50%
120k
150K
200K
(a) Loss
Top-1 acc.
(c) FixMatch: 56.4%
(d) FlexMatch: 94.3%
Figure 3: Convergence analysis of FixMatch and FlexMatch. (a) and (b) depict the loss and top- 1 1-accuracy on CIFAR-100 with 400 labels. Evaluations are done every 5K iterations. (c) and (d) demonstrate the class-wise accuracy within the first 200K iterations on CIFAR-10 dataset. The numbers in legend correspond to the ten classes in the dataset.
on the CIFAR-100 100 dataset with 400 labels (i.e., only 4 label samples per class), FlexMatch achieves an average error rate of 39.94%, which significantly outperforms FixMatch (46.42%)
CPL improves the performance of existing SSL algorithms. Other than FixMatch, CPL. can also improve the performance of other existing SSL. algorithms such Pseudo-Labeling and UDA. For instance, the error rate is reduced from 37.4% to 29.53% for UDA the STL-10 40-label split after introducing CPL (refer to Flex-UDA in T.. These results further prove the effectiveness of CPL in better leveraging unlabeled data. Figure shows the average running time of a single iteration with or without adding our CPL, it clear that while improving the performance of existing SSL algorithms, our CPL does not introduce additional computational burden.
0.3
Psoudb-Labll
Meth
Figure 2: Average running
time of iteration on sin- gle GeForce RTX 3090 GPU.
CPL achieves better performance on complicated tasks. The STL-10 dataset contains unlabeled data from a similar but broader distribution of images than its labeled set. The existence of new types of objects in the unlabeled dataset makes STL-10 10 a more challenging and realistic task. FlexMatch achieves greater performance improvement under such challenging situation. The error rate on STL-10 with only 40 labels is 29.15% which is relatively 18.96% better than FixMatch (35.97%) Similar strong improvements are also observed on CIFAR-100 dataset, which has as many as one hundred classes.
We also analyze the reason why FlexMatch performs less favorably on SVHN. This is probably because SVHN is relatively simple (i.e., to classify digits) yet unbalanced dataset. The class-wise imbalance leads to the classes with fewer samples never have their estimated learning effects close to 1 according to Eq. ( even when they are already well-learned. Such low thresholds allow noisy pscudo-labeled samples to be trusted and learned throughout the training process, which is also reflected by the loss descent curve where the low-threshold classes have major fluctuations. FixMatch, on the other hand, fixes its threshold at 0.95 to filter out noisy samples. Such a fixed high threshold is not preferable with respect both accuracies of hard-to-learn classes and overall convergence speed as explained earlier, but since SVHN is an easy task, the model can easily learn the task and make high-confidence predictions, setting high-fixed threshold thus becomes less problematic and has its advantages overweighed.
4.2 Results on ImageNet
We also verify the effectiveness of CPL on ImageNet-IK 226 which is much more realistic and complicated dataset. We randomly choose the same 100K labeled data (i.e., 100 labels per class), which used for
Table 2: Error rate results on ImageNet after 220 iterations.
is less than 8% of the total labels. The ImageNet can be found in the appendix, where the two algorithms share the same hyyeer-prameter after running 220 iterations in T This result indicates that when the task is complicated, despite the class imbalance issue (the number
Method
Top-1
Top-5
FixMatch Flex
43.66
21.80
We show the error rate comparison
42.02
19.49
7

---

w/b oee mhpp
My Wartt-ipp
4.9
Erro Rate
Erro Rate
9.8
CIFAR
4.7
5.0
0.95
Mapping
(a) Threshold r
(b) Mapping function
(c) Threshold warm-up
Figure 4: Ablation study of FlexMatch.
of images within each class ranges from 732 to 1300), CPL can still bring improvements. Note that this result does represent the best performance each algorithm as the model cannot fully converge after 2 iterations, and due to the computational resource limitation, we did not further tune the prree-pprameters to obtain the best results on ImageNet.
4.3 Convergence speed acceleration
Another strong advantage of FlexMatch is superior convergence speed. Figure (aa and (bb shows the comparison between FlexMatch and FixMatch with respect to the loss and laceuracy on CIFAR-I00 400-label split. The loss of FlexMatch decreases much faster and smoother than FixMatch, demonstrating its superior convergence speed. The major fluctuations of the loss in FixMatch may due to the pre-defined threshold that lets pass most unlabeled data belonging to certain classes, whereas with CPL larger batch of unlabeled data containing samples from various classes enables the gradient to more directly head toward the global optimum. As a result, with only 50K iterations, FlexMatch has already surpassed the final results of FixMatch. After 800K iterations, however, we observe a further decrease loss and accuracy. This is likely due to over-fitting, which also occurs in Fixhatch after 900K iterations. Thus, believe it is not fair of the last few checkpoints for evaluating algorithms with different convergence speeds.
the median results
We further compare the class-wise accuracy of FixMatch and FlexMatch CIFAR-10 in their early training stages. As shown in Figure 3(c) and 3(d) at iteration 200K, FixMatch only hits an overall accuracy of 56.35% as half of the classes are still learned unsatisfactorily, whereas Flex Match has already achieved an overall accuracy of 94.29% which even higher than the final accuracy reached by FixMatch after 1M iterations. It is manifest that the introduction of CPL successfully encourages the model to proactively learn those difficult classes thereby improving the overall learning effect.
4.4 Ablation study
We conduct experiments to evaluate three components of FlexMatch: the upper limit of thresholds т, mapping functions Mxr), and threshold warm-up.
Threshold upper bound. We investigate 5 different values and different mapping functions CIFAR-10 10 dataset with 40 labels. As shown in Figure [(a) the optimal choice of r is around 0.95, either increasing decreasing this value results in performance decay. Note that in Flex Match, tuning does not only affect the upper limit of the threshold but also the estimated learning effects because they are determined by the number of samples that fall above T.
Mapping function. We explore three different mapping functions in Figure 4(b) (1) concave: M(x) = In( 1)/ In 2, (2) linear: M(z) x, and (3) convex: .Mxx) z x) We see that the convex function shows the best performance and the concave function shows the worst. Although tweaking the degree of convexity may probably lead to further improvement, we do not make further investigation in this paper. It is noteworthy that all these functions have their outputs grow from to 1 when the inputs go from 0 to 1. One may also design function with different range, for instance, from 0.5 In this case, is equivalent to setting a lower limit to the flexible threshold so that even at the beginning of the training, only samples with prediction confidence higher than this limit will contribute the unsupervised loss. We do not include such a lower limit FlexMatch since will introduce new paremeearameter. However, we did find that setting a lower limit at 0.5 can slightly

---

improve the performance. A possible reason is that the lower threshold prevents noisy training caused by incorrect pseudo labels at the early stage [36].
Threshold warm-up. We analyze the performance of threshold warm-up on both CIFAR-10 (40 labels) and CIFAR-100 (400 labels) datasets. As shown in Figure 4(c) threshold warm-up can bring about 0.2% absolute improvement on CIFAR- 10 and about 1 % on CIFAR- 100. At the beginning of the training without the threshold warm-up, the flexible thresholds may go through heavy fluctuations because the denominator in Eq.(6) (6) is small. In the meantime, there will always be some classes whose flexible thresholds reach or approach т, thereby filtering out most unlabeled data the batch. The threshold warm-up solves this issue by gradually raising the thresholds of all classes from zero it creates a learning boom at the early training stage where most of the unlabeled data can be utilized.
Comparison with class balancing objectives. CPL the effect of balancing across classes the number of unlabeled samples used compute pseudo-labeling loss in each batch. Similar effect can be achieved by making the marginal class distribution close to a uniform distribution for each batch. We conduct such a comparative experiment by directly adding an additional objective to FixMatch: La = -Ee 9e log(ge/ Pe) [22] where Pe. is the mean predicted probability of class e accross all samples in the batch, and q is a uniform distribution: 9e = 1/C. The error rate of adding such an objective is 7.16% on the CIFAR-10 10 40-label split (compared with FixMatch 4.97%=10.06) While this approach requires instances of each class within each batch to be balanced to make sense, CPL does not have such constraint. It is more flexible and involves less human intervention to adjust thresholds than adjusting model's 's predictions.
and FlexMatch
Related Work
Pseudo-Labeling [4] is pioneer SSL method that uses hard artificial labels converted from model predictions. A confidence-based strategy was used in [6] along with pseudo labeling so that the unlabeled data are used only when the predictions thresholding also presents in recently proposed UDA [I] and FixMatch [4] with the difference being that UDA used sharpened 'soft' pseudo labels with a temperature whereas Fixmatch adopted one-hot 'hard' labels. The success of UDA and FixMatch, however, relies heavily on the usage of strong data augmentations improve the consistency regularization. ReMixMatch [3 also leveraged such strong augmentations.
sufficiently confident. Such confidence-based
The combination of curriculum learning and semi-supervised learning is popular in recent years [37 39. For multi-model image classification task, [37] optimized the learning process of unlabeled images by judging their reliability and discriminability. In (38. the easy image-level properties learned first and then used to facilitate segmentation via constrained CNNs. Curriculum learning is also used alleviate distiiibbittnoionn problems by picking up in-distribution samples from unlabeled data according to the out-of-distributin scores [39.
Several researches have investigated on dynamic threshold in related fields such sentiment anal- ysis 44) and semantic segmentation 41 In [40]. the threshold was gradually reduced to make high-quality data selected into labeled data set in the early stage and large-quantity in the later stage.
An extra classifier is added to automate the threshold to deal with domain inconsistency in [41]. [2]
introduced curriculum learning to self- self-training with steadily increasing threshold and achieved near state-of-the-arr results.
Conclusion and Future Work
In this paper, we introduce Curriculum Pseudo Labeling (CPL), a curriculum learning approach of leveraging unlabeled data for SSL. CPL dramatically improves the performance and convergence speed of SSL algorithms that involve thresholds while being extremely simple and almost cost-free. FlexMatch, our improved algorithm of FixMatch, achieves state-of-the-art performance on a variety of SSL benchmarks. In future work, we would like to improve our method under the long-tail scenario where the unlabeled data belonging to each class are extremely unbalanced.

---

Broader Impact
CPL fills the gap that no modern SSL algorithm considers the inherent learning difficulties of different classes during the training, and shows that by doing the convergence speed and final accuracy can both be improved. We hope that CPL can attract more future attention to explore the effectiveness of utilizing unlabeled data according to the model's ' learning status as well as the per-class learning difficulty.
Funding Disclosure
Funding in direct support of this work: computing resource granted by Tokyo Institute of Technology and Microsoft Research Asia. This work was partially supported by Toray Science Foundation.
References
[1] Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. In NeurlPS, pages 3365-3373, 2014.
[2] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. ICLR, 2017.
[3] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic trans- formations and perturbations for deep semi-supervised learning. In NewrlPS, pages 1171-1179, 2016.
[4] Mong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning. ICML, volume 3, 2013.
[5] Geoffrey J McLachlan. Iterative reclassification procedure for constructing an asymptotically op- timal rule of allocation in discriminant analysis. Journal of the American Statisticul Association, 70(350):355366) 1975.
[6] Chuck Rosenberg. Martial Hebert, and Henry Schneiderman. Semi-supervised self-training of object detection models. 2005.
[7] Henry Scudder. Probability of error of some adaptive pattern-recognition machines. IEEE Transactions on Information Theory, 11(3):363-31 -37111 1965.
[8] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In CVPR, pages 10687-10698. 2020.
[9] Antti Rasmus, Harri Valpola, Mikko Honkala, Mathias Berglund, and Tapani Raiko. Semi- supervised learning with ladder networks. In NeurIPS, pages 3546-3554, 2015.
[10] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. NeurIPS, pages 1195- 1204, 2017.
[11] Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmen- tation for consistency training. NeurlPS, 33, 2020.
[12] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. NewrlPS, page 5050-5060, 2019.
[13] David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: semi-supervised learning with distribution matching and augmentation anchoring. In ICLR, 2019.
[14] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. NeurIPS, 33, 2020.
10

---

[15] Yoshua Bengio, Jéröme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML, pages 41-48, 2009.
[16] Yves Grandvalet, Yoshua Bengio, et al. Semi-supervised learning by entropy minimization. In CAP, pages 281-296, 2005.
[17] Anastasia Pentina, Viktoriaa Sharmanska, and Christoph H Lampert. Curriculum learning of multiple tasks. In CVPR, pages 5492-5500, 2015.
[18] Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander Hauptmann. Self-paced curriculum learning. In AAAI, volume 29, 2015.
[19] Alex Graves, Mare G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Auto- mated curriculum learning for neural networks. In ICML, pages 1311-1320. PMLR, 2017.
[20] Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep networks. ICML, pages 2535-2544, PMLR, 2019.
[21] Daphna Weinshall, Gad Cohen, and Dan Amir. Curriculum learning by transfer learning: Theory and experiments with deep networks. In ICML, pages 5238-5246. PMLR, 2018.
[22] Eric Arazo, Diego Ortego, Paul Albert, Noel O' O'Connor and Kevin McGuinness. Pseudo- labeling and confirmation bias in deep semi-supervised learning. In IJCNN, pages 1-8, 2020.
[23] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[24] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[25] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In AISTAYS, pages 215-222 2011.
[26] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255. leee, 2009.
[27] Avital Oliver, Augustus Odena, Colin Raffel, Ekin D Cubuk, and Ian J Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In NewrlPS, pages 3239-3250, 2018.
[28] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, performeffe deep learning library. NeurIPS, 32:8026-800 8037, 2019.
[29] Iya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In ICML, pages 1 1139-1147. PMLR, 2013.
[30] Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Ussr computational mathematics and mathematical physics, 4(5):1-17, 1964.
[31] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. ICLR, 2016.
[32] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702-703, 2020.
[33] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[34] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
[35] Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Time-consistent self-supervision for semi- supervised learning. In ICML, pages 11523-11533. PMLR, 2020.

---

[36] Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. arXiv preprint arXiv:2101.06322, 2021.
[37] Chen Gong, Dacheng Tao, Stephen J Maybank, Wei Liu, Guoliang Kang, and Jie Yang. Multi- modal curriculum learning for semi- -upervied image classification. IEEE Transactions on Image Processing, 25(7):3249-333) -2660, 2016.
[38] Hoel Kervadec, Jose Dolz, Éric Granger, and Ismail Ben Ayed. Curriculum semi-supervised segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 568-576. Springer, 2019.
[39] Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa. Multi-task curriculum framework for open-set semi-supervised learning. In ECCV, pages 438-454. Springer, 2020.
[40] Yue Han, Yuhong Liu, and Zhigang Jin. Sentiment analysis via semi-supervised learning: model based on dynamic threshold and multi-classifiers. Neural Computing and Applications, 32(9):5117-5129, 2020.
[41] Zhedong Zheng and Yi Yang. Rectifying pseudo label learning via uncertainty estimation for domain adaptive semantic segmentation. IJCV, 106-11100011000 2021.
[42] Paola Revisiting pseudo-labelingg for semi-supervised learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6912-6920, 2021.
Fuwen Tan, Yanjun Qi, and Vicente Ordonez. Curriculum labeling:
12


Reviews:
[{"review_id": "zeX5FWVQFRn", "reviewer": "Reviewer_Pkje", "summary": "This paper introduces a curriculum learning strategy for semi-supervised learning. Their motivation is that most existing semi-supervised learning utilizes a fixed threshold to compute pseudo-labeling loss, but the fixed threshold should be adjusted according to the state of the model. Then, they propose to adjust the threshold based on the prediction of the model and design a class-specific confidence threshold. Their threshold is designed so that the class with a larger number of confident samples has a higher value of the threshold. Thus, the classes hard to learn will have a small threshold. Their way of designing the threshold is computationally efficient and will not cause much training time increase. \n\n\nThe experimental results show that their method is very effective in challenging situations such as when the number of labeled samples is limited or when the target dataset is challenging. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n1. Their method is simple and easy to implement. It should be important in real applications. \n2. The idea behind the method is reasonable. \n3. Their method is computationally efficient but provides great gains in challenging semi-supervised learning settings. \n4. This paper is very easy to follow. \n5. Experiments support the effectiveness of their method. \n\nWeakness \n1. The idea of introducing curriculum learning for semi-supervised learning is not very novel as done in [A].\n2. Given 1, their experiments lack comparison to semi-supervised learning methods based on curriculum learning. There should be other ways of designing curriculum such as gradually increasing the confidence threshold. This paper lacks empirical and intuitive comparison to this kind of method, which makes it hard to judge the novelty of the proposed method. \n3. Related to 2, the analysis of results is not enough to fully understand the method. From the intuition of the method, the expected results can be that this method improves accuracy on hard-to-classify classes. Did you observe the results?\n4. The proposed threshold design will balance the number of unlabeled samples used to compute pseudo-labeling loss per class. If we have the objective to balance it such as entropy of marginal class distribution for unlabeled samples, do we have the same effect as the proposed method? \n\nIn summary, this paper provides a simple and well-motivated technique to improve semi-supervised learning based on pseudo-labeling. However, it lacks comparison to existing methods with curriculum learning. If they can provide simple baselines with some curriculum learning, the readers would understand the advantage of the proposed method better. \n\n\n[A] Curriculum Labeling: Revisiting Pseudo-Labeling for Semi-Supervised Learning, AAAI 2021\n\n\n---post rebuttal update---\nThe rebuttal addressed my concerns. Despite its simplicity, the proposed method is able to boost the performance of semi-supervised learning even for hard classes. I would like to raise my rating to 7. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "They lack a section to discuss the potential social impact of their research. I do not think the research includes a particular content, which can bring negative impact, but they need to mention it somewhere in the paper. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "jrgkVIDno4u", "reviewer": "Reviewer_Lmyo", "summary": "This work extends the popular FixMatch framework with dynamic, class-specific thresholds. Specifically, a multiplier is designed to scale the fixed threshold for each class, which is computed based on the proportion of pseudo-labeled instances with confidence values passing the threshold—the motivation is to lower the threshold for classes that have fewer pseudo-labeled instances. The effectiveness of the proposed approach, named FlexMatch, is demonstrated on CIFAR, STL-10 and ImageNet where clear improvements over FixMatch are observed.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strength**\n\n- The design of dynamic thresholds is simple and seems easy-to-implement.\n\n- The improvements brought by the dynamic thresholds are clear on CIFAR, STL-10 and ImageNet.\n\n**Weakness**\n\n- My biggest concern is with the novelty of this work, which seems incremental. Though the improvements look good on three datasets, the method's generality is questionable, as reflected in the poor performance on SVHN. Lines 224-230 have associated the failure on SVHN to imbalanced classes, but more concrete analysis in this regard is missing. More insights should be provided to explain and understand why the performance on SVHN is worse than FixMatch.\n\n- What's special about TorchSSL? Does it solve any particular engineering challenges that existing code fails to cope with? This section should be moved to the appendix as it doesn't provide any new insight.\n\n---\nPost-rebuttal update:\n\nI was initially concerned with the generality of FlexMatch due to its relatively lower performance on the simple SVHN dataset, and the paper lacking sufficient discussions in regards to how FlexMatch should be applied in practice and under which circumstances FlexMatch might perform well/poorly. These discussions are essential for practitioners since FlexMatch's underlying mechanism is hand-crafted and could possibly overfit the curated datasets. The authors' responses have addressed this concern and I suggest the authors add these discussions to the paper.\n\nGiven that the proposed approach is simple and yet effective, which is also appreciated by other reviewers, I change my rating to clear accept.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have discussed the limitations in Sec.4.1. There is no obvious negative impact associated to this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "4aCYomKaOuH", "reviewer": "Reviewer_w5Zg", "summary": "The authors proposed a simple and straightforward method to address an obvious but critical and long-lasting issue in semi-supervised learning. They showed a number of analyses and demonstrated superior performance over state-of-the-art methods (mostly) consistently on five datasets. The proposed method is simple and easy to use without extra computation needed. Most importantly, I do see the possibilities of adapting this method widely across semi-supervised learning or self-training methods that use pseudo-labeling. I would recommend acceptance at this time. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n### Strengths: Describe the strengths of the work. Typical criteria include: soundness of the claims (theoretical grounding, empirical evaluation), significance and novelty of the contribution, and relevance to the NeurIPS community.\n\n- Paper is well-written through entire paper and super easy to follow\n- Proposed method is extremely simple without adding extra computation \n- Plan to public release TorchSSL, a PyTorch library for semi-supervised learning\n- Providing two versions of evaluation metrics for fair comparisons with the existing method and argue that choosing the best error rates might be more suitable for future research because the convergence speed is also important and won't be able to be reflected in the median error rate of the last 20 checkpoints. \n- thorough experiments with 5 datasets and analysis on convergence speed\n- analyze the failed cases and identify the potential limitation of the proposed method (L224 - L230)\n\n### Weaknesses: Explain the limitations of this work along the same axes as above.\n- Interestingly, I don't think I can find any obvious weaknesses. This is a well-written paper.\n\n### Additional feedback, comments, suggestions for improvement and questions for the authors:\n- L202: \"When the amount\" --> \"when the amount\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did analyze the failed cases and identify the potential limitation of the proposed method (L224 - L230). ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {"ethical_concerns": "As with any other papers for semi-supervised learning, we might have a risk of introducing biases, because we are leveraging a smaller number of samples that are labeled. These samples could already be biased. However, all recent semi-supervised learning methods are striving to address this issue. "}}, {"review_id": "2X9BFvgvll", "reviewer": "Reviewer_Qc4v", "summary": "The paper proposes to incorporate ideas from curriculum learning into semi-supervised learning. Semi-supervised learning consists on training models with only a subset of labeled samples, and a typically larger set of unlabeled samples. The paper proposes to improve on ideas from pseudo-labeling whereas fixed thresholds are defined to decide the pseudo-labels for unlabeled samples as the training process goes. In this approach, the idea is to modulate the thresholds used during the training process as a function of time (iterations) and the individual classes being classified. The motivation is that some classes might be harder to learn. In order to deal with initial conditions, the paper proposes to use a warming up strategy whereby an additional class (undecided) is considered for samples that have not reached a confidence to be assigned to any class. The proposed method in combination with other methods based on consistency regularization such as FixMatch lead to improved accuracies.", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ Curriculum learning is well motivated in terms of the presented task and seems a good strategy to pace the thresholds used in semi-supervised learning.\n+ TorchSSL is proposed as a library re-implementing prior algorithms in a unified codebase. This could potentially provide a resource for further development in this area.\n+ The proposed method is general enough that it can be combined with others and in experimental results higher accuracies are reported after applying the proposed method.\n\nWeaknesses:\n+ There has been already a prior proposal to use Curriculum-based pacing for choosing the thresholds for Semi-supervised learning for pseudo-labeling: Curriculum Labeling: Revisiting Pseudo-Labeling for Semi-Supervised Learning. AAAI 2021. February 2021. The idea of changing thresholds for deciding on the unlabeled samples hence has already explored before. Without direct comparisons or discussions it diminishes the contributions of the present work. However I do acknowledge some differences: the current work seems to consider per-category adaptation of the pacing on the curriculum, whereas this other work considers a single threshold. However whether this difference is important as claimed in the present work, it seems that it would be something that needs validation.\n+ There were some technical details that could only be resolved after reading the supplementary material, something important such as what is or are the neural network architectures that are being trained for obtaining the results on Table 1. \n+ More importantly there are some inconsistencies reported on the number for Imagenet-1k where FixMatch is reported as having a top-5 error of 19.55% but the prior work both the original FixMatch paper, and the above referenced AAAI 2021 paper report 10.87% for FixMatch. The conditions seem the same 10% of data annotated (= 100k samples annotated). On the surface the presented method would be impressive as the reported FixMatch error is reported as 19.55% and it goes down to 13.96% -- however only if we accept a version of FixMatch that starts at a much worse level of performance this might be true. Section 4.2 is really light on the details for this part and from the Supplementary material it doesn't seem like the settings were fundamentally different than in these two prior works.\n+ The authors cite [27] but don't seem to follow some of the recommendations in that work regarding the brittleness of many methods under unlabeled samples that include categories that are not part of the target set. Other recent works have tested also under this scenario but I don't seem this type of experiment in the current work. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't believe I saw much in the way of limitations or any discussion about this part.", "review_text": "", "extra_scores": {"ethical_concerns": "Not particularly.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes Curriculum Pseudo Labeling (CPL), an extension to semi-supervised learning algorithms that uses an estimate of how well the model has learned each class to set per-class thresholds for promoting unlabeled data to pseudolabeled data. This conceptually simple idea has a significant impact in practice. It is also simple to implement and does not significantly add to the computational cost of the base SSL methods. (It actually can speed up convergence significantly.) Further, the combination of CPL and FixMatch (called FlexMatch) sets new state-of-the-art scores on several SSL benchmarks. Finally, the paper includes the release of Torch SSL, a codebase for the study of SSL.

The reviewers generally liked that the method was conceptually simple, effective, and easy to combine with many existing methods. Three of the four reviewers clearly favored acceptance, for the above strengths. Some of the concerns they had initially were addressed during the discussion period, and the authors are encouraged to include these clarifications in the final version of the paper. One reviewer had serious concerns about the experimental setup. The results reported in the paper for baselines differ from some previously reported in the literature. The authors attribute this to both a difference in the number of labeled examples (which is fair to vary in SSL experiments) and differences in hyperparameters. The authors are encouraged to both clarify whether the hyperparameters were tuned to be optimal for each baseline and also to add comparisons on the same number of labeled examples as previously used in the literature (perhaps in an appendix) for a more detailed comparison.
Decision: Accept (Poster)