SoftMatch: Addressing the Quantity-Quality Tradeoff in Semi-supervised Learning
Authors: Hao Chen
Ran Tao
Yue Fan
Yidong Wang
Jindong Wang
Bernt Schiele
Xing Xie
Bhiksha Raj
Marios Savvides
Venue: ICLR 2023 poster (2023)
Keywords: Semi-Supervised Learning
Semi-Supervised Classification

Abstract: The critical challenge of Semi-Supervised Learning (SSL) is how to effectively leverage the limited labeled data and massive unlabeled data to improve the model's generalization performance. In this paper, we first revisit the popular pseudo-labeling methods via a unified sample weighting formulation and demonstrate the inherent quantity-quality trade-off problem of pseudo-labeling with thresholding, which may prohibit learning. To this end, we propose SoftMatch to overcome the trade-off by maintaining both high quantity and high quality of pseudo-labels during training, effectively exploiting the unlabeled data. We derive a truncated Gaussian function to weight samples based on their confidence, which can be viewed as a soft version of the confidence threshold. We further enhance the utilization of weakly-learned classes by proposing a uniform alignment approach. In experiments, SoftMatch shows substantial improvements across a wide variety of benchmarks, including image, text, and imbalanced classification.

Published as a conference paper at ICLR 2023
SOFTMATCH: ADDRESSING THE QUANTITY-QUALITY TRADE-OFF IN SEMI- SUPERVISED LEARNING
Hao Chen', Ran Tao Yue Fan Yidong Wang
Jindong Wang Bernt Schiele?, Xing Xie Bhiksha Raj-.1 Marios Savvides
"Carnegie Mellon University, "Max Planck Institute for Informatics, Saarland Informatics Campus, Miirooooft Research Asia, 'Mohamed bin Zayed University of Al
ABSTRACT
The critical challenge of Semi-Supervised Learning (SSL) is how to effectively leverage the limited labeled data and massive unlabeled data to improve the model's generalization performance. In this paper, we first revisit the popular pseudo-labeling methods via unified sample weighting formulation and demon- strate the inherent quantity-qualinn trade-off problem of pseudo-labeling with thresholding, which may prohibit learning. To this end, we propose SoftMatch to overcome the trade-off by maintaining both high quantity and high quality of pseudo-labels during training, effectively exploiting the unlabeled data. We derive a truncated Gaussian function to weight samples based on their confidence, which can be viewed as a soft version of the confidence threshold. We further enhance the utilization of weakly-learned classes by proposing a uniform alignment ap- proach. In experiments, SoftMatch shows substantial improvements across wide variety of benchmarks, including image, text, and imbalanced classification.
INTRODUCTION
Sumi-Supervised Learning (SSL), concerned with learning from a few labeled data and a large amount of unlabeled data, has shown great potential in practical applications for significantly re-
duced requirements on laborious annotations (Fan et al. 2021: Xie et al. 2020; Sohn et al. 2020 Pham et al. 2021 Zhang et al. 1021 Xu et 20.11 a Chen et al. 2022 Oliver et al. 2018). The main challenge of SSI. lies how to effectively exploit the information of unlabeled data to improve
the model's generalization performance [Chapelle et al. (2066). Among the efforts, pseudo-labeling (Lee et al. 2013 Arazo et al., 2020) with confidence thresholding (Xie et al. 22020) Sohn et al, 2020 Xu al. 2b2 Zhang et al. 2021) is highly-successie and widely-adopted.
The core idea of Chreshold-based pseudo-labeling (Xie et al., 2020 Sohn et al., 2020: Xu et al, 2021b Zhang et al., 2021) is to train the model with pseudo-label whose prediction confidence is above a hard threshold, with the others being simply ignored. However, such a mechanism inher- ently exhibits the quantity-quality trade-off. which undermines the learning process. On the one hand, a high confidence threshold as exploited in FixMatch (Sohn et al. (020)) ensures the quality of the pscudo-labels. However, it discards considerable number of unconfident yet correct pseudo- labels. As an example shown in Fig. [(a) around 71 % correct pseudo-labels are excluded from the training. On the other hand, dynamically growing threshold (Xu et al. 2021b Berthelot al., 2021], or class-wise threshold (Zhang et al.) (021)) encourages the utilization of more pseudo-labeld but inevitably fully enrolls erroneous pseudo-labels that may mislead training. As an example shown by Flex (Zhang et al.) 2021) in Fig. T(a) about 16% of the utilized pseudo-labels are in- correct. In summary, the quantity-qualit trade-off with a confidence threshold limits the unlabeled data utilization, which may hinder the model's generalization performance.
In this work, we formally define the quantity and quality of pseudo-labels in SSL and summarize the inherent trade-off present in previous methods from perspective of unified sample weighting for-
Equal Contribution: haoe3 www.dnnwnwnnnu...mmmm cww.edu. taoranl @emuu.daa Correspondence to: jindong, wang microsoft.com com, andrew edu.

---

Published as a conference paper at ICLR 2023
Naanber of Sompges
Woong
Sulilank
All
Pialana
Scmples
Quaatty of al
1.S--
Quality of Puraloo Momaito-LLsable Lalteds
Pishlzte
Piphlacc
Fipolandi
Confidencee
(a) Confi. Dist.
(b) Quantity
(c) Quality
(d) Decision Boundary
Figure I: Illustration on Two-Moon Dataset with only 4 labeled samples (triangle purple/pink points) with others as unlabeled samples in training a 3-layer MLP classifier. Training detail is in Appendix. (a) Confidence distribution, including all predictions and wrong predictions. The red line denotes the correct percentage of samples used by SoftMatch. The part of the line above seatter points denotes the correct percentage for FixMatch (blue) and Flex.atch (green). (b) Quantity of pseudo-labels; (c) Quality of pscudo-labels; (d) Decision boundary. SoftMatch exploits almost all samples during training with lowest error rate and best decision boundary.
mulation. We first identify the fundamental reason behind the quantity-qualit trade-off is the lack of sophisticated assumption imposed by the weighting function on the distribution of pseudo-labels. Especially, confidence thresholding can be regarded cording to samples' confidence, which assumes pscudo-labels with confidence above the threshold equally correct while others wrong. Based the analysis, we propose SoftMatch over- come the trade-off by maintaining high quantity and high quality of pseudo-labels during training. truncated Gaussian function derived from our assumption on the marginal distribution to fit the confidence distribution, which assigns lower weights to possibly correct pseudo-labels according to the deviation of their confidence from the mean of Gaussian. The parameters of the Gaussian function are estimated using the historical predictions from the model during training. Furthermore, we propose Uniform Alignment to resolve the imbalance issue in pseudo-labels, resulting from dif- ferent learning difficulties of different classes. It further consolidates the quantity of pseudo-labed labels while maintaining their quality. On the two-moon example, as shown in Fig. [[c]aand Fig. T(b) Soft- Match achieves distinctively better accuracy of pscudo-labels while retaining consistently higher utilization ratio of them during training, therefore, leading to better-learned decision boundary shown in Fig. [(d) We demonstrate that SoftMatch achieves a range of image and text classification tasks. We further validate the robustness of SoftMatch against long-tailed distribution by evaluating imbalanced classification tasks.
step function assigning binary weights ac-
saae-off-hee-ne
wide
Our contributions can be summarized as:
We demonstrate the importance of the unified weighting function by formally defining the quantity and quality of pseudo-labels, and the trade-off between them. We identify that the inherent trade-off in previous methods mainly stems from the lack of careful design on the distribution of pseudo-labels, which is imposed directly by the weighting function.
We propose SoftMatch effectively leverage the unconfident yet correct pseudo-labels, fitting truncated Gaussian function the distribution of confidence, which overcomes the tra We further propose Uniform Alignment resolve the imbalance issue of pseudo- labels while maintaining their high quantity and quality.
We demonstrate that SoftMatch outperforms previous methods various image and text evaluation settings. We also empirically verify the importance of maintaining the high accuracy of lseudo-labels while pursuing better unlabeled data utilization in SSL..
REVISIT QUANTITY- QUALIIYY TRADE-OFF OF SSL
In this section, we formulate the quantity and quality of pseudo- labels from a unified sample weight- ing perspective, by demonstrating the connection between sample weighting function and the quan- tity/quality of pseudo-labels. SoftMatch is naturally inspired by revisiting the inherent limitation in quantity-qualing trade-off of the existing methods.
2

---

Published as a conference paper at ICLR 2023
2.1 PROBLEM STATEMENT
We first formulate the framework of SSL in a C-class classification problem. Denote the labeled and unlabeled datasets D (x y!) Nu and Du {xx); No
Ti= respectively, where x/,x² x R is the d-dimensional labeled and unlabeled training sample, and y! is the one-hot ground-truth label for labeled data. We use Nr and No to represent the number of training samples in D and Du, respectively. Let p(y|x) E Rº denote the model's prediction. During training, given a batch of labeled data and unlabeled data, the model is optimized using a joint objective L = L, + Lu, where Lg is the supervised objective of the cross-entropy loss (H) on the Bi,-izzd labeled batch:
Br H(y, p(ylx))).
L.
(1)
BL
For the unsupervised loss, most existing methods with pscudo-labeling (Lee al. 2013 Arazo al., 2020 Xic et al.) 2020) Sohnn et al. 2020 Xu et al., 2021b Zhang et al., (022]) exploit a confidence thresholding mechanism to mask out the unconfident and possibly incorrect pseudo-labels from training. In this paper, we take step further and present a unified formulation of the confidence thresholding scheme (and other schemes) from the sample weighting perspective. Specifically, we formulate the unsupervised loss Lu as the weighted cross-entrope between the model's prediction of the strongly-augmentte data 2)x ) and pseudo-labels from the weakly-augment data wxx"):
By Lu BU Σ
(2)
where p is the abbreviation of p(y (x")), and fo is the one-hot pseudo-label argmax(p): A(p) is the sample weighting function with range [0, Amax]; and By is the batch size for unlabeled data.
2.2 QUANTITY- QUALITY TRADE- OFF FROM SAMPLE WEIGHTING PERSPECTIVE
In this section, we demonstrate the importance of the unified weighting function A(p), by showing its different instantiations previous methods and its essential connection with model predictions. We start by formulating the quantity and quality of pseudo-labed
Definition 2.1 (Quantity of pseudo-labels). The quantity (P) of pseudo-labels enrolled in training is defined as the expectation of the sample weight X(p) over the unlabeled data:
(()) = ED. [Xp)] E [0, Amax).
Definition 2.2 (Quality of pseudo labels). The quality g(p) the expectation of the weighted 0/1 eлror of pseudo-labele assuming the label y" is given for x" for only theoretical analysis purpose:
No
A(p.) Exp y²)] E [0,
g())
Σ
I(;: = yt) Σ No A(p,)
(4)
where A(p) >Xp))/ >>))/>(pp>>>)))))p)> is the probability mass function (PMF) of p being close y
Based on the definitions of quality and quantity, we present the quantity-quality trade-off' of SSL..
Definition 2.3 (The quantity-quality trade-off). Due to the implicit assumptions of PMF X(p) on the marginal distribution of model predictions, the lack of sophisticated design it usually results in trade-off in quantity and quality when one of them increases, the other must decrease. Ideally, well-defined X(p) should reflect the true distribution and lead to both high quantity and quality.
Despite its importance, X(p) has hardly been defined explicitly properly in previous methods. In this paper, we first summarize A(p). A(p). (p), and g(p) of relevant methods, as shown in Table I with the detailed derivation present in Appendix A. For example, naive pseudo-labeling (Lee et al. 2013) and loss weight ramp-up scheme (Samuli & Timo 2017| Tarvainen & Valpola 2017 Berthelot al. 2019b al exploit the fixed sample weight to fully enroll all laeudo-labels into
trainin. is equivalent to set X Amax and Ā = 1/Ny. regardless of p. which means each pseudo-
label is assumed equally correct. We can verify the quantity of pseudo-labels is maximized to Xmax-
3

---

Published as a conference paper at ICLR 2023
Table I: Summary of different sample weighting function A(p), probability density function X(p) of p. quantity f(p) and quality g(p) of pseudo-labels used in previous methods and SoftMatch.
Scheme
Pscudo-Label
FixMatch
SoftMatch
Manaso 1.00.
if max(p) 2 T, otherwise.
Amax exp (
pl (p)
if max(p) Me:
A(p)
Amax
Amax:
otherwise.
expl - Coaatiyy do tall
max(p) < Me
X(p)
1/Nc
[1/Ng.
if max(p) 2 otherwise.
Ny +
espl- (meer inmde
[0.0,
max(p) 2 He
Nu expl- (maat tm
S(p)
Amaoe
Amory //o
Auaxe 2 + Newss/Mi En expl (maxip.i-/o01
a(p)
ENe 1(p
A 1(p
201
1(p y2))222
Σ
No-M
I(p:
yt) exp(
1/2 No6)
"t
High Quantity Low Quality
Low Quantity High Quality
High Quantity High Quality
Note
However, maximizing quantity also fully involves the erroneous pseudo-labels, resulting in deficient quality, especially in early training. This failure trade-off is due to the implicit uniform assumption on PMF X(p) that is far from the realistic situation.
In confidence thresholding (Arazo et al. 2020; Sohn et al. 2020 Xie et al. 2020), we can view the sample weights being computed from a step function with confidence max(p) as the input and pre-defined threshold r as the breakpoint. sets A(p) to Amax when the confidence is above T and otherwise 0. Denoting No = EN
No (max(p) 2 T) as the total number of samples whose predicted confidence are above the threshold, À is set a uniform PMF with a total mass of Ni within fixed range [r, 1]. This is equal to constrain the unlabeled data as D7, {xx: mxx(((((())""" 2 r), with others simply being discarded. We can derive the quantity and the quality shown in Table A trade-off exists between the quality and quantity of pseudo-labels in confidence thresholding controlled by т. On the one hand, while high threshold ensures quality, it limits the quantity of enrolled samples. On the other hand, a low threshold sacrifices quality by fully involving more but possibly erroneous pseudo-labels in training. The trade-off still results from the over-simplification of the PMF from actual cases. Adaptive confidence thresholding (Zhang et al., [002T] Xu et al.] 2021b) adopts the dynamic and class-wise threshold, which alleviates the trade-off by evolving the (class-wise) threshold during learning. They impose further relaxation on the assumption of distribution, but the uniform nature of the assumed PMF remains unchanged.
While some methods indeed consider the definition of A(p) (Ren et al., 2020 Hu et al. 2021 Kim et al. 20022), interestingly, they all neglect the assumption induced on the PMF The lack of sophisticated modeling of x(P) usually leads to a quantity qualtity-quality trade-off in the unsupervised loss of SSL,, which motivates us to propose SoftMatch to overcome this challenge.
SOFTMATCH
3.1 GAUSSIAN FUNCTION FOR SAMPLE WEIGHTING
Inherently different from previous methods, we generally assume the underlying PMF A(p) of marginal distribution follows a dynamic and truncated Gaussian distribution of mean μr and vari- ance at t-th training iteration. We choose Gaussian for its maximum entropy property and empir- ically verified better generalization. Note that this is equivalent to treat the deviation of confidence max(p) from the mean μ4 of Gaussian as proxy measure of the correctness of the model's pre- diction, where samples with higher confidence are less prone to be erroneous than that with lower confidence, consistent to the observation as shown in Fig. ((a) To this end, we can derive X(p) as:
Amax exp
(maxx(p)
if max(p) < Pr: otherwise.
X(p) =
20
(5)
Amaxx
which is also truncated Gaussian function within the range [0, Amax), on the confidence max(p).
4

---

Published as a conference paper at ICLR 2023
However, the underlying true Gaussian parameters H4 and de are still unknown. Although we set the parameters to fixed values as in FixMatch (Sohn et al. 2020) or linearly interpolate them within some pre-defined range in Ramp-up Tarvainen & Valpola 2017), this might again over- simplify the PMF assumption as discussed before. Recall that the PMF X()) is defined over max(p), instead fit the truncated Gaussian function directly the confidence distribution for better generalization. Specifically, we can estimate and o² from the historical predictions of the model. At t-th iteration, we compute the empirical mean and the variance as:
1
ib EB, [max(p)]
max (p:),
Bu
(6)
By
1
o. VarBe [max(p)] = Bu
(max(p:) - (ia))
We then aggregate the batch statistics for more stable estimation, using Exponential Moving Av- erage (EMA) with a momentum m over previous batches:
jur mj² + (1 m))jb.
BU
o; =mm + (1 m)
0.
Bu
where we use unbiased variance for EMA and initialize Ωo as &. and oj 1.0. The estimated mean A and variance o? are plugged back into Eq. (5) to compute sample weights.
Estimating the Gaussian parameters adaptively from the confidence distribution during training not only improves the generalization but also better resolves the quantity-qualin trade-off. We can verify this by computing the quantity and quality of pseudo-labels as shown in Table The derived
quantity f(p) is bounded by X (1 + exp( (d- )), Amax]. indicating SoftMatch guarantees at least Amax/2 of quantity during training. As the model learns better and becomes more confident, ie., f44 increases and 6, decreases, the lower tail of the quantity becomes much tighter. While quantity maintains high, the quality of pseudo-labels also improves. As the tail of the Gaussian exponentially grows tighter during training, the erroncous pscudo-labels where the model is highly unconfident assigned with lower weights, and those whose confidence are around fú are more efficiently utilized. The truncated Gaussian weighting function generally behaves as a soft and adaptive version of confidence thresholding, thus term the proposed method as SoftMatch.
20,
3.2
UNIFORM ALIGNMENT FOR FAIR QUANTITY
As different classes exhibit different learning difficulties, generated pseudo-labels can have poten- tially imbalanced distribution, which may limit the generalization of the PMF assumption (Olieer et al. 2018 Zhang et al.) 2021). To overcome this problem, we propose Uniform Alignment (UA), encouraging more uniform pseudo-labell of different classes. Specifically, we define the distribution in pseudo-labels as the expectation of the model predictions on unlabeled data: ED, [p(y|x")], Dur- ing training, estimated E [p(y[x") using the EMA of batch predictions on unlabeled data.
We use the ratio between a uniform distribution UCC) RC and E E [p(y[x") [x") to normalize the each prediction p unlabeled data and use the normalized probability to calculate the per-sample loss weight. We formulate the UA operation as:
u(C)
UA(p) Normalize
p Eng
(8)
where the Normalize(-) E.), ensuring the normalized probability sums to 1.0. With UA plugged in, the final sample weighting function in SoftMatch becomes:
Amax exp (= (maxx(UA(p))--a
-A)))) if max(UUA()) < Ar.
A(p)
287
Amax:
otherwise.
When computing the sample weights, UA encourages larger weights to be assigned to less- cess-predicted leeuds-labels and smaller weights mrreepreddcted pseudo-labels, alleviating the imbalance issue.
5

---

Published as a conference paper at ICLR 2023
Table Top-1 error rate (%) on CIFAR-IO, CIFAR-100, STL-10, and SVHN of 3 different random seeds. Numbers with * are taken from the original papers. The best number is in bold.
Dataset
CIFAR-
CITAR-100
SVIN
STL- 10
Label
250
2,500
10,000
1,000
1,000
PaeudoLabel
14.6
46.49:220
15.08
87.45:a0s
57.74
36.55
64,61
9.40
74.68.11
32064
MeanTeacher
70.09
37.46=130
8.10=61
81.11=144
45.17s1.0
31.75+823
36.09=308
3.27=000
71.72+145
33.90=137
MixMatch
36. 19
13.63=0.9
6.66=025
67.5 67.59
39. 39.76=n44
27. 27.78=829
30.60=a39
3.60
54.93
21.70
ReMixMatch
9,88-103
6.30
4.84
42.75=105
26.03:1015
20.02
24.06:9.15
5.16a931
32.12:624
6.74
UDA
10.62+375
5.16200
4.29=007
46.39t1..9
27.73=+021
22.49x825
5.12+427
1.89:a0
37.42=844
6.64+a.77
FixMatch
7.47
0.86
=02s
46.422:04
28.03 16
22. 12.20
3.8
1. 1
35.97
6.252033
Influence
5.05
4.35 _ooe
2.63,a237
FlexMatch
2,97
4.98 on
4.19000
39. 19
26.492n20
21.90
8.19
6. 6.72a030
29. 15
5.77
Softaachh
4.91=
37.10
26.66.3015
22.03
2.33
2,01a0m
21.42:344
5.73
An essential difference between UA and Distribution Alignment (DA) (Berthelot al. 2019a) pro- posed earlier lies in the computation of unsupervised loss. The normalization operation makes the predicted probability biased towards the less-predicted classes. In DA, this might not be issue, as the normalized prediction is used as soft targel in the cross-entropy loss. However, with pseudo- labeling, more erroneous pseudo-labels are probably created after normalization, which damages the quality. UA avoids this issue by exploiting original predictions to compute pseudo-labels and normalized predictions to compute sample weights, maintaining both the quantity and quality of pseudo-labels in SoftMatch. The complete training algorithm is shown in Appendix A.2
4 EXPERIMENTS
While most SSL literature performs evaluation on image tasks, we extensively evaluate SoftMatch on various datasets including image and text datasets with classic and long-tailed settings. Moreover, We provide ablation study and qualitative comparison to analyze the effectiveness of SoftMatch. [
4.1 CLASSIC IMAGE CLASSIFICATION
Setup. For the classic image classification setting, we evaluate CIFAR- (Krizhessky al. 2009]. SVHN(Netzer et al. 201 STL-10 (Coates et al., 2011 and ImageNet (Deng et al. 2009). with various numbers of labeled data, where class distribution of the labeled data is balanced. We use the WRN-28-2 (Zagoruyko & Komodakis 2016) for CIFAR-10 10 and SVHN, WRN-28-8 for CIFAR- 100, WRN-37-2 (Zhou et [ (2200) for STL-10, and ResNet-50 (He et al. 2016) for ImageNet. For all experiments, we use SGD optimizer with a momentum of 0.9, where the initial learning rate 'oo set 0.03. We adopt the cosine learning rate annealing scheme a total training step of 220 The labeled batch size Bt is set to 64 and the unlabeled batch size By set to times of B2 for all datasets. We set m to 0.999 and divide the estimated variance à, by 4 for 20 of the Gaussian function. We record the EMA of model parameters for evaluation with momentum 0.999. Each experiment run with three random seeds on labeled data, where we report the top-1 error rate. More details on the hyper-parameters are shown in Appendix AA.3II
adjust the learning rate with
SoftMatch obtains the state-of-the-art results on almost all settings in Table and except CIFAR-100 with 2,500 and 10,000 labels and SVHN with 1,000 labels, where the results of SoftMatch are comparable to previous methods. Notably, FlcxMatch exhibits a performance drop compared to FixMatch on SVHN, since it enrolls too many erroneous pscudo-labels at the begin- ning of the training that prohibits learning afterward. In contrast, SoftMatch surpasses FixMatch by 1.48% SVHN with 40 labels, demonstrating its superiority for better utilization of the pseudo- labels. On more realistic datasets, CIFAR-100 with 400 labels, STL-10 with 40 labels, and ImageNet with 10% labels, SoftMatch exceeds Fltxh by margin of 7.73%, 2.84%, and 1.33%, respec- tively. SoftMatch shows the comparable results to Faex on CIFAR- 100 with 2,500 and 10,000 labels, whereas ReMixMatch (Berthelot al. 2019a) demonstrates the best results due to the Mixup (hhang et al. 2017) and Rotation loss.
Results.
and Table
All experiments in Section Section and Section are conducted TorchSSL Zhang et al. 2021 and Section are
conducted with USB (Wang et al. 2022b) since it only supports NLP tasks back then. More recent results of Motchacch are included in USB details.
along its updates, refer pps / www.h..tbhh.o....o.gooo nom/ hwtp//hhhhhhhh//SStMMetto

---

Published as a conference paper at ICLR 2023
Table 3: Top1 error rate (%) on ImageNet. The best number is in bold.
Table 4: Topl error rate (%) on CIFAR-IO-LT and CIFAR-100-LT of
5 different random seeds. The best number is in bold.
Dataset
CIFAR-IO-LT
CIFAR-100-1
# Label
100k
400k
Imbalance 7
50
100
150
20
50
100
FixMatch
43.66
32.28
FixMatch
18.46#430
25.11#120
29.62=0a8
50.42=a76
$7.89
62.40=0.s
FlexMatch
41.85
31.31
FlexMatch
18.13sa.9
25.51
29.80
49.1
57.20
62.70
SoftMatch
40.52
29.49
SoftMatch
22.93 16.55
27.40:am
56.24:0.51 48.09
61.08:am
Table 5: Topl error rate (%) on text datasets of 3 different random seeds. Best numbers are in bold.
Datascts
AG News
DBpedia
IMDb
Amazon-5
Yclp-5
Labels
40
200
70
280
100
1000
1000
UDA
16.8321.8
14.34
4.11
6,932385
18.33=0.1
50.29
47.49
FixMatch
17.10
11.242114
2.1820.9
1.42
7.59
42.70
39.56207
FlexMatch
15.49
10,95
2.69
1.691002
2.80
42.341062
39.01
SoftMatch
12.68
10.41:013
1.68:0.4
1.27:101
7.48:
42.14:092
39.31
4.2 LONG-TAILED IMAGE CLASSIFICATION
Setup. We evaluate Match on a more realistic and challenging setting of imbalanced SSL (Kim
jet al., 2020 Wei et al. 2021 Lee et al. 2021 Fan al. 2022), where both the labeled and the unlabeled data exhibit long-tailed distributions. Following (Fan et al. 2022), the imbalance ratio y ranges from 50 to 150 and 20 100 for CIFAR-IO-LT and CIFAR- 100-LT, respectively. Here, ? is used to exponentially decrease the number of samples from class 0 to class C (Fan et al. 2022). We compare SoftMatch with two strong baselines: FixMatch (Sohn et al. 2020) and Facxh (Zhang eet al., 2021]. All experiments use the same WRN-28-2 (Zagoruyko & Komodakis 2016) as the backbone and the same set of common hyper-parameters. Each experiment repeated five times with different data splits, and we report the average test accuracy and the standard deviation. More details are Appendix A3..2
Results. As is shown in Table SoftMatch achieves the best test error rate across all long-tailed settings. The performance improvement over the previous state-of-the-an large imbalance ratios. For example, SoftMatch outperforms the second-best by 2.4% at 7 = 150 on CIFAR-IO-LT, which suggests the superior robustness of our method against data imbalance.
still significant even at
Discussion. Here study the design choice of uniform alignment it plays key role in Soft- Match's performance on imbalanced SSL. We conduct experiments with different target distribu- tions for aligmment. Specifically, the default uniform target distribution u(C) can be replaced by ground-truth class distribution or the empirical class distribution estimated by seen labeled data dur- ing training. The results in Fig. 3(a) Bha) clear advantage of using uniform distribution. Uniform target distribution enforces the class marginal to become uniform, which has a strong regularization effect of balancing the head and tail classes in imbalanced classification settings.
4.3 TEXT CLASSIFICATION
Setup. In addition to image classification tasks, we further evaluate SoftMatch on text topic clas- sification tasks of AG News and DBpedia, and sentiment tasks of IMDb, Amazon-5, and Yelp-5 (Maas al. 201T Zhang et al. 2015). We split a validation the algorithms. For 5mazon-5 and Yelp-5, we randomly sample 50,000 samples per class from the training data to reduce the training time. We fine-tune the pre- trained BERT-Bass (Devlin et al., 2018) model for all datasets using UDA (Xie et al., 2020), FixMatch (Sohn et al. 2020). FlexMatch (Zhang et al., (02T). J. and SoftMatch. We use Adam W (Kingma & Ba 2014 Loshchilov & Hutter) 2017) optimizer with initial learning rate of 1e - and the same cosine scheduler image classi- fication tasks. All algorithms are trained for a total iteration of 218 The fine-tuned model is directly used for evaluation rather than the EMA version. To reduce the GPU memory usage, we set both BL and By to 16. Other algorithmic hyper-parameterr parameters stay the same as image classification tasks. Details of the data splitting and the paper-parameter used are in Appendix AA..3
from the training data evaluate
Results. The results on text datasets are shown in Table SoftMatch consistently outperforms other methods, especially on the topic classifications tasks. For instance, Mothh achieves an error rate
7

---

Published as a conference paper at ICLR 2023
FidMash Sabttaak
Qualty of Phondolaaad Labols
Quantiy of Peeal Labols
Quantry Poeedo
0.5
Elner Bote
60-
0.0
Wh The Ck
0.4
Solt
PieMarc
IhMashh
Wive
Manch
Selt. (ildd Cle
10k
Ree.
Iter
(d) Cls. Quality Figure 2: Qualitative analysis of FixMatch, FlexMatch, and SoftMatch CIFAR-10 with 250 la- bels. (a) Evaluation error; (b) Quantity of Pseudo-Labed (c) Quality of Pseudo-Labets (d) Quality of Pseudo-Labels from the best and worst learned class. Quality computed according to the un- derlying ground truth labels. SoftMatch achieves significantly better performance.
(a) Eval. Error
(b) Quantity
(c) Quality
of 12.68% on AG news with only 40 labels and 1.68% on DBpedia with 70 labels, surpassing the second best by a margin of 2.81% and 0.5% respectively. On sentiment tasks, Soft.atch also shows the best results on Amazon-5 and IMDb, and comparable results to its counterpart Yelp-5.
4.4 QUALITATIVE ANALYSIS
In this section, we provide a qualitative comparison on CIFAR-10 with 250 labels of FixMatch (Sohn et al. 2020), FlexMatch (Zhang al., [202], and SoftMatch from different aspects, as shown in Fig. [2 We compute the error rate, the quantity, and the quality of pseudo-labels to analyze the proposed method, using the ground truth of unlabeled data that is unseen during training.
SoftMatch utilizes the unlabeled data better. From Fig. 2(b) and Fig. 2(c) one can observe that SoftMatch obtains highest quantity and quality of pseudo-labels across the training. Larger error with more fluctuation is present in quality of FixMatch and FlexMatch due to the nature of confidence thresholding, where significantly more wrong pseudo-labels are enrolled into training, leading to larger variance in quality and thus unstable training. While attaining a high quality, SoftMatch also substantially improves the unlabeled data utilization ratio, i.e., the quantity, as shown in Fig. 2 demonstrating the design of truncated Gaussian function could address the quantity- quality trade-off of the pseudo-labels. We also present the quality of the best and worst learned classes, shown in Fig. [ where both retain the highest along training in SoftMatch. The well- solved quantity-quality trade-off allows Match achieves better performance on convergence and error rate, especially for the first 50k iterations, as in Fig. 2(a)
4.5 ABLATION STUDY
Sample Weighting Functions. We validate different instantiations of A(p) to verify the effective- ness of the truncated Gaussian assumption on PMF X(p), as shown in Fig. ((b) Both linear function and Quadratic function fail generalize and present large performance gap between Gaussian due to the naive assumption on PMF as discussed before. Truncated Laplacian assumption also works well on different settings, but truncated Gaussian demonstrates the most robust performance.
Gaussian Parameter Estimation. SoftMatch estimates the Gaussian parameters μ and o² directly from the confidence generated from all unlabeled data along the training. Here we compare it (All- Class) with two alternatives: (1) Fixed: which uses pre-defined and x² of 0.95 and 0.01. (2) Per-Class: where a Gaussian for each class instead of a global Gaussian weighting function. As shown in Fig. 3(c) the inferior performance of Fixed justifies the importance of adaptive weight adjustment in SoftMatch. Moreover, Per-Class achieves comparable performance with SoftMatch at 250 labels, but significantly higher error rate 40 labels. This is because accurate parameter estimation requires many predictions for each class, which is not available for Per-Class.
Uniform Alignment on Gaussian. To verify the impact of UA we compare the performance of SoftMatch with and without UA, denoted as all-class with UA and all-class without UA in Fig. 3( Since the per-class estimation standalone can also be viewed way to achieve fair class utilization (Zhang et al., 202T). we also include it in comparison. Removing UA from SoftMatch has a slight performance drop. Besides, per-class estimation produces significantly inferior results S SVHN.
8

---

Published as a conference paper at ICLR 2023
Fand
Prn-Ches
Quadiskk Tran. Lepkokis
Paa-Clee oy UA
Ernor Bate (55
Al-Clew
Eirur to (N)
AllChaw syjn UA
Erno Bale
Tira. Grasdar
AllChaa my
Rede
0.4
LBA0011 02
(a) L.T. UA
(b) Weight, Func.
(c) Gau. Param.
(d) UA
Figure 3: Ablation study of SoftMatch. (a) Target distributions for Uniform Alignment (UA) on long-tailed setting: (b) Error of different sample functions; (c) Error rate of different Gaussian parameter estimation, with UA enabled; (d) Ablation on UA with Gaussian parameter estimation;
We further include the detailed ablation of sample functions and several additional ablation study in Appendix A.5 due to space limit. These studies demonstrate that SoftMatch stays robust to different EMA momentum, variance range, and UA target distributions on balanced distribution settings.
RELATED WORK
Pseudo-labeling (Lee et al. 2013) generates artificial labels for unlabeled data and trains the model
in self self-training manner. Consistency regularization (Samuli & Timo 2017) is proposed to achieve the goal of producing consistent predictions for similar data points. A variety of works focus
improving the pseudo-labeling and consistency regularization from different aspects, such as loss
weighting (Samuli & Timo 2017: Tarvainen & Valpola 2017: [scen et al.) 2019 Ren al.. 2020)
data augmentation (Grandvalet et al. |005] Sajjadi et al. 2016 Miyato et al. 2018:) 88; Berthelot et al.
2019b a Xie et al.
2020 Cubuk et al. 22200) [Sajaaii et al.] (2166). label allocation (Tai al. [021]]
Teature consistency
Li et al. 2021 Zheng et al.
2022 Fan et al. 2021) and confidence thresholding
(Sohn et
2020 hang et al. 2021 Xu et al.
2021b).
Loss weight ramp-up strategy is proposed to balance the learning on labeled and unlabeled data. (Samuli & Timo 2017 Tarvainen & Valpola 2017) Berthelot et al., [019b]a) a). By progressively
increasing the loss weight for the unlabeled data, which prevents the model involving too much ambiguous unlabeled data at the early stage of training, the model therefore learns in a curriculum fashion. Per-sample loss weight is utilized to better exploit the unlabeled data (Iscen et al. 2201 Ren (ct al.) 2020). The previous work "Influence" shares a similar goal with us, which aims to calculate the loss weight for each sample but for the motivation that not all unlabeled data are equal (Ren ctt al. 2020). SAW (Lai et al. 2022) utilizes effective weights (Cui et al. 2019) to overcome the ibblaaeed issues in SSL. Modeling of loss weight has also been exp in semi-supervised segmentation (Hu et al.) 2021). De-biased self-training (Chen et al. 2022 Wang et al. (002aa) study the data bias and training bias brought by involving pseudo-labels into training, which is similar exploration of quantity and quality in SoftMatch. Kim et al. (2022) proposed to use small network to predict the loss weight, which is orthogonal to our work.
Confidence thresholding methods (Sohn et al. 2020 Xie et al. 2020 Zhang et al. 2021 Xu et al., 2021b) adopt threshold enroll the unlabeled samples with high confidence into training. Fix- Match (Sohn et al., (020) uses a fixed threshold to select pseudo-labels with high quality, which limits the data utilization ratio and leads to imbalanced pseudo-label distribution. Dash (Xu et al.) 2021b) gradually increases the threshold during training to improve the utilization of unlabeled data. Flex (Zhang et al., (02T) designs class-wise thresholds and lowers the thresholds for classes that are more difficult to learn, which alleviates class imbalance.
CONCLUSION
In this paper, we revisit the quantity-quality trade-off of pscudo-labeling and identify the core rea- son behind this trade-off from unified sample weighting. We propose SoftMaich with truncated Gaussian weighting function and Uniform Aligmment that overcomes the trade-off, yielding both high quantity and quality of pseudo-labels during training. Extensive experiments demonstrate the effectiveness of our method on various tasks. We hope more works can be inspired in this direction, such designing better weighting functions that can discriminate correct pseudo-labels better.

---

Published as a conference paper at ICLR 2023
REFERENCES
Eric Arazo, Diego Ortego. Paul Albert, Noel E O' O'Connor, and Kevin McGuinness, Pscudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2020.
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmenta- tion anchoring. In International Conference on Learning Representations, 2019a.
David Berthelot, Nicholas Carlini, lan Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in Neural Infor- mation Processing Systems, 32, 2019b.
David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alex Kurakin. Adamatch:
A unified approach to semi-supervised learning and domain adaptation. ICLR, 2021.
Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien (eds.). Semi-Supervised Learning. The MIT Press, 2006.
Baixu Chen, Junguang Jiang, Ximei Wang, Jianmin Wang, and Mingsheng Long. Debiased pseudo labeling in self-training. arXiv preprint arXlv:2202.07155 2022.
Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmen- tation with cross pseudo supervision. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2613-2622, 2021.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli gence and statistics, pp. 215 123. JMLR Workshop and Conference Proceedings, 2011.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. on effective number samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019.
loss based
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- crarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-248222 255. leee, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1rXiv:18100004805, 2018.
Yue Fan, Anna Kukleva, and Bernt Schicle. Revisiting consistency regularization for semi- supervised learning. In DAGM German Conference Pattern Recognition, pp. 63-78. Springer, 2021.
Yue Fan, Dengxin Dai, and Bernt Schiele. Cossl:
Co-learning of representation and classifier for imbalanced semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Yves Grandvalet. Yoshua Bengio, et al. Semi-supervised learning by entropy minimization. volume 367, pp. 281-296, 2005.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330. PMLR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
10

---

Published as a conference paper at ICLR 2023
Hanzhe Hu, Fangyun Wei, Han Hu, Qiwei Ye, Jinshi Cui, and Liwei Wang. Semi- sumi-vupe seman- tic segmentation via adaptive equalization learning. Advances in Neural Information Processing Systems, 34:22106-22 22- 18, 2021.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi- supervised learning. In CVPR, 2019.
Jachyung Kim, Younghum Hur, Sejun Park, Funho Yang, Sung Ju Hwang, and Jinwoo Shin. Dis- tribution aligning refinery of pseudo-label for imbalanced semi-supervised learning. Advances in Neural Information Processing Systems, 33: 13:14497-14579 2020.
Jiwon Kim, Youngjo Min, Dachwan Kim, Gyuseong Lee, Junyoung Seo, Kwangrok Ryoo, and Seungryong Kim. Conmatch: Semi-supervised learning with confidence-guidde consistency reg- ularization. In European Conference on Computer Vision 2022.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint
arXiv:1412.6980, 2014.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Zhengfeng Lai, Chao Wang. Henrry Gunawan, Sen-Ching S Cheung, and Chen-Nee Chuah. Smoothed adaptive weighting for imbalanced semi-supervised learning: Improve reliability against unknown distribution data. In International Conference on Machine Learning, pp. 11828- 11843, PMLR, 2022.
Dong-Hyun Lee al. Pseudo-label: The simple and efficient semi-super learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, pp. 896, 2013.
Hyuck Lee, Seungjae Shin, and Heeyoung Kim. Abe: Auxiliary balanced classifier for class- imbalanced cemi-superiised learning. Advances in Neural Information Processing Systems, 34, 2021.
Junnan Li, Caiming Xiong, and Steven CH Hoi. Comatch: Semi-supervised learning with con- trastive graph regularization. In Proceedings the 2021.
the IEEE/CVF International Conference on Com-
puter Vision, pp.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05000 2017.
Andrew Maas, Raymond Daly, Peter Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 201
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 1979-19979 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Real- istic evaluation of deep semi-supervised learning algorithms. Advances in neural information processing systems, 31, 2018.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-IILT 2019: Demonstrations, 2019.
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. Proceedings of the IEEE/CV Conference Computer Vision and Pattern Recognition, pp- 11557-11568, 2021.
Zhongzheng Ren, Raymond A. Yeh, and Alexander G. Schwing. Not all unlabeled data equal: Learning to weight data in semi-supervised learning. In Neural Information Processing Systems (NeurIPS). 2020.
11

---

Published as a conference paper at ICLR 2023
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-superrised learning. Advances in neural information processing systems, 29:1 19:1163-1171, 2016.
Laine Samuli and Aila Timo. Temporal ensembling for semi-supervised learning. In International Conference on Learning Representations (ICLR), volume 4, pp. 6. 2017.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying superviee learning with consistency and confidence. Advances in Neural Information Processing Systems, 33, 2020.
Kai Sheng Tai, Peter Bailis, and Gregory Valiant. Sinkhorn label allocation: Semi-supervised clas- sification via annealed self elffttrainnn, 2021.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results. In Proceedings of the 31st Interna- tional Conference on Neural Information Processing Systems, pp. 1195-1204, 2017.
Xudong Wang, Zhirong Wu, Long Lian, and Stella Yu. Debiased learning from naturally im- balanced pscudo-labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14647-14657, 2022a.
Yidong Wang, Hao Chen, Yue Fan, Wang Sun, Ran Tao, Wenxin Hou, Renjie Wang, Linyi Yang. Zhi Zhou, Lan-Zhe Guo, Heli Qi, Zhen Wu, Yu-Feng Li, Satoshi Nakamura, Wei Ye, Marios Savvides, Bhiksha Raj, Takahiro Shinozaki, Bernt Schiele, Jindong Wang, Xing Xie, and Yue Zhang. Usb: A unified semi-supervised learning benchmark. In Neural Information Processing Systems (NeurIPS), 2022b.
Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang. Crest: A class- rebalancing self-training framework for imbalanced semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10857-10866, 2021.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems, 33, 2020.
Mengde Xu, Zheng Zhang, Han Hu, Jianfeng Wang, Lijuan Wang. Fangyun Wei, Xiang Bai, and Zicheng Liu. End-to-end semi-supervised object detection with soft teacher. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 3060-3069, 2021a.
Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-supervised learning with dynamic thresholding. In International Conference Machine Learning, pp. 1 11525-11533 11536. PMLR, 2021b.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference 2016. British Machine Vision Association, 2016.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo la- beling. Advances in Neural Information Processing Systems, 34, 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 1700007700 2017.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. Advances in neural information processing systems, 28:649-657, 2015.
Mingkai Zheng, Shan You, Lang Huang, Fei Wang, Chen Qian, and Chang Xu. Simmatch: Semi- supervised learning with similarity matching. arXiv preprint arXiv:2203.06915, 2022.
Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Time-consistent self-supervision for cemi-supervised learning. In International Conference Machine Learning, pp. 11523 11533. PMLR, 2020.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 QUANTITY-QUALITY TRADE-OFF
In this section, we present the detailed definition and derivation of the quantity and quality for- mulation. Importantly, we identify that the sampling weighting function A(p) [0, Amax di- rectly related to the (implicit) assumption of probability mass function (PMF) over p for E (p(y/x";;*" x² }. i.e., the distribution p. From the unified sample weighting function per- spective, we show the analysis of quantity and quality of the related methods and SoftMatch.
A.1.1 QUANTITY AND QUALITY
Derivation Definition 2.1
The definition and derivation of quantity (p) of pseudo-labels rather straightforward. We define the quantity as the percentage/ratio of unlabeled data enrolled in the weighted unsupervised loss. In other words, the quantity the average sample weights on unlabeled data:
Ny A(p:) Σ
f(p)
No ED (A(p:)),
(10)
where each unlabeled data is uniformly sampled from Dy and f(p) E [0, Amax].
Derivation Definition 2.2
We define the quality g(p) of pseudo-labels as the percentage/ratio of correct pseudo-labels enrolled in the weighted unsupervised loss, assuming the ground truth label y" of unlabeled data is known. With the 0/1 correct indicator function (()) being defined as:
(p) = 1(p y") e {0, 1}.
(II)
where P is the one-hot vector pseudo-label argmax(p). We can formulate quality as:
g(p) = - (p.) ((p:)
No
Σ, A(p,)
Σ (P::)((P:)
(12)
= Exmh(()) =
y"]] E [0.1].
We denote x(p) as the probability mass function (PMF) of p. with A(p) 2 and Σ Exp) = 1.0.
This indicates that, once X(p) is set to function, the assumption the PMF of p is made. In most of the previous methods (Tarvainen & Valpola 2017: Berthelot et al. 2019b|a Sohn et al. 2020 Zhang et al., 2021 Xu et al. 2022b), although they do not explicitly set A(p). the introduction of loss weight schemes implicitly relates to the PMF of p. While the ground truth label p is actually unknown in practice,
still use it for theoretical analysis.
In the following sections, we explicitly derive the sampling weighting function A(p), probability mass function X(p). quantity (p), and quality g(p) for each relevant method.
13

---

Published as a conference paper at ICLR 2023
A.1.2 NAIVE PSEUDO-LABELING
In naive pseudo-labeling LLe et al. 2013), the pseudo-labels are directly used to the model itself.
This is equivalent to set (p) to fixed value Amax, which is hyper-parameter, We can write:
X(p) Amax:
(13)
Amax
Xp) = NuAmax
(14)
No
No Amax
f(p) Σ
Amax:
(15)
NU
Nu
( y.
g(p)
(16)
NU
We can observe that the naive self-training maximizes the quantity of the pseudo-labels by fully enrolling them into training. However, full enrollment results in pseudo-labels of low quality. At beginning of training, large portion of the pseudo-labels would be wrong, i.e., 7(p) 0. since the model is not well-learned. The wrong pseudo-labels usually leads to confirmation bias (Guo et al.)
2017 Arazo et al. 2020) training progresses, where the model memorizes the wrong pseudo- labels and becomes very confident on them. We can also notice that, by setting ((p) to a fixed value Amax. we implicitly assume the PMF of the model's prediction p is uniform, which far away from the realistic distribution.
A.1.3 Loss WEIGHT RAMP UP
In the earlier attempts of semi-supervised learning, a bunch of work (Tarvainen & Valpola 2017 Berthelot et al., 2019b a) exploit the loss weight ramp up technique avoid involving too much erroneous p labels in the early training and let the model focus on learning from labeled data first. In this case, the sample weighting function is formulated as function of training iteration t, which is linearly increased during training and reaches its maximum Ama after T warm-up itera- tions. Thus have:
t A(p) Amax min( T
1),
(17)
Amax min( 4, 1)
A(p) No Amax min 4,1)
(18)
Nu
Amax min( t T 1),
f(p)
(19)
Ny 1(p; = =y")
g(p) =
(20)
Nu
which demonstrates the same uniform assumption of PMF and same quality function as naive self- training. It also indicates that, long as same sample weight used for all unlabeled data, uniform assumption of PDF over is made.
A.1.4 FIXED CONFIDENCE THRESHOLDING
Confidence thresholding introduces a filtering mechanism, where the unlabeled data whose predic- tion confidence max(p) is above the pre-defined threshold is fully enrolled during training, and others being ignored (Xie et al., 22000; /Sohn et al. 2020). The confidence thresholding mechanism can be formulated by setting A()) as step function when the confidence is above threshold, the
14

---

Published as a conference paper at ICLR 2023
sample weight is set Amax, and otherwise O. We can derive:
Mmaxx 0.0
if max(p) 2 T, otherwise.
X(p)
(21)
1 (m) 2T
if max(p) 2 T,
X(p)
No
(22)
Σ
(max(())) 2 T)
0.0,
otherwise.
NE
(max(p:)) 2 A)
Ne
f(p)
Amax
(23)
Nv
Nu
NC
( y
((P)
(24)
NU
8
(25)
where we set Nú: = Σ (p)a((p:) 2 T), i.e., number of unlabeled samples whose prediction confidence max(p) above threshold 7.
Interestingly, one can find that confidence thresholding directly modeling the PMF over the predic- tion confidence max(p). Although it still makes the uniform assumption, as shown in Eq. (22), constrains the probability mass to concentrate in the range of [T, 1]. As the model is more confident about the pseudo-labels, and the unconfident ones are excluded from training, it is more likely that would be close to y", thus ensuring the quality of the pseudo-labels to a high value if a high threshold is exploited. However, a higher threshold corresponds to smaller No, directly reducing the quantity of pseudo-labels. We can clearly observe a trade-off between quantity and quality of using fixed confidence thresholding. In addition, assuming the PMF of max(p) as uniform within a range [T, 1] still does not reflect the actually distribution over confidence during training.
A.1.5 SOFTMATCH
In this paper, we propose SoftMatch to overcome the trade-off between quantity and quality of pseudo-labels. Different from previous methods, which implicitly make over-simplified assumptions on the distribution of p. we directly modelling the PMF of max(p), from which we derive the sample weighting function X(p) used in SoftMatch.
We assume the confidence of model predictions max(p) generally follows the Gaussian distribution N (max(p); Pc, ōz) when max(p) < H4 and the uniform distribution when max(p) Z t4. Note that 14 and de changing along training as the model learns better. One the PMF is similar to that of confidence thresholding, and is the Gaussian part makes SoftMatch distinct from previous methods. In SoftMatch, we directly estimate the Gaussian parameters max(p) using Maximum Likelihood Estimation (MLE), rather than set them to fixed values, which more consistent to the actual distribution of prediction confidence. Using the definition of PMF A(p), we can directly write the sampling weighting function X(p) of SoftMatch as:
that the uniform part of
Amax v (ma Amax:
max(p) < bt
A(p)
(26)
max(p) 2 ft:
where ((1;;1,0) = "2aaa exp( (x-m)² Without loss of generality, we can assume max(p:) <
uffor E [0, No fe No Nou max(p:) (shown in Eq. (6) and thus P(max(p) < (x) 0.5.
15

---

Published as a conference paper at ICLR 2023
Therefore, Σ X(p) is computed as follows:
Nu Σ (pi))
Ny
NO
((p.)
X(p;)
==1
Nic 2
+1
Ny
(27)
Nu
max v o (e) +
Amax
+
Nu
(max(p:) - -11)
Amax
exp(
2
20²
Further,
No
((p) Nu
E A(P))
Ny
No
X(p:)
+
Σ
A(PI)
Nu
(28)
Ny
Amax
Nu
(max(p;)
-1e)2
exp(
Nu
2
207
Nu
Amxx
Amax
exp(-
Nu
207
Since max(p:) < uh for E [0, No
2
( 12)
(max(p:) -102 1
exp(
<= exp(
20
20²
Nu
(2 - -113
(max(p:) - m1)
Nu
exp(
<= exp
2
207
20²
2
Amax
Amax (1 + exp(
( ue)² )) <= f(p) < Amax
2
2
207
Therefore, SoftMatch can guarantee at least half of the possible contribution to the final loss, im- proving the utilization of unlabeled data. Besides, as is also estimated from max(p), the lower bound of f()) would become tighter during training with a better and more confident model.
With the derived Σ X(p), We can write the PDF A(p) in SoftMatch as:
v 2 of unax (max(p))p...o)) Ny.
max (p) < j4
A(p)
+
(29)
max (p) 2 μ4
16

---

Published as a conference paper at ICLR 2023
and further derive the quality of pseudo-labels in SoftMatch as:
Ne.
g(p) Σ L(P: y"))(p)
No
1
Σ (P:))((P))
Eno N(pk)
1
Σ (p:)A(p))) + (P))A(p))
ENo A(pk)
(30)
+1
Nu
Ny
2
Amax 12 oe)
Amax
(p:)
(p,)
Σx Ne X(P*)
Σx X(Ps)
Nu No 1(D; - y() exp( (max((:) )-m1
Nu (p,) =yj)
2(NU (NN
-N)
3 2NU
where No E No (max(p:) 2 Me). From the above equation, we can see that for pseudo-labels whose confidence is above ft,, the quality is as high as in confidence thresholding: for pseudo- labels whose confidence is lower, thus more possible to be erroneous, the quality is weighted by the deviation from f4.
At the beginning of training, where the model is unconfident about most of the pseudo-labels, Soft-
Match guarantees the quantity for at least Amaa and high quality for at least Σn No 1(p== As the model learns better and becomes more confident, i.e,, dr increases and o, decreases, the lower bound of quantity becomes tighter. The increase in Nú leads better quality with pseudo-labels whose confidence below fr are further down-weighted. Therefore, SoftMatch overcomes the quantity- quality trade-off.
2N,
A.2 ALGORITIM
We present the pseudo algorithms of SoftMatch this section. SoftMatch adopts the truncated Gaussian function with parameters estimated from the EMA of the confidence distribution each training step, which introduce trivial computations.
Algorithm SoftMatch algorithm.
1: Input: Number of classes C, labeled batch yx) unlabeled batch {u, {ui)se[Bo) and
EMA momentum m.
2: Define: p; (w:))
3: L. 1 BBz H(y,, pyy (xx)))
Bz
Compute L, on labeled batch b Compute the mean of confidence
Br.
4: A By
Σe4 max(p:)
Bu
5: σ² Bu
LBe (ma fi,))
By
Compute the variance of confidence
6: i = mpp-1
(1
m)) jāb
p Update EMA of mean Update EMA of variance
7: of mo? 2 + (1 m)
Bu -1-1 03
8: for to Bu do
exp (UAx)UU((p)))... R.)² 287
if max(UA((:)) < Pr, Compute loss weight
9:
A(p.) 1.0,
otherwise.
10: end for
C. By ΣBe
> Compute 2, on unlabeled batch
Return: C. L.
17

---

Published as a conference paper at ICLR 2023
A.3 EXPERIMENT DETAILS
A.3.1 CLASSIC IMAGE CLASSIFICATION
We present the detailed hyper- paramperreeess used for the classic image classification setting in Table 6 for reproduction. We use NVIDIA V100 for training of classic image classification. The training time for CIFAR-10 and SVHN on single GPU is around 3 days, whereas the training time for CIFAR- 100 and STL-I0 is around 7 days.
Table 6: Hyper-paramettrs of classic image classification tasks.
Dataset
CIFAR-10
CIFAR-100
STL-10
SVHN
ImageNet
Model
WRN-28-2
WRN-28-8
WRN-37-2
WRN-28-2
ResNet-50
Weight Decay
5e-4
le-3
5c-4
5e-4
3c-4
Labeled Batch size
64
128
Unlabeled Batch size
448
128
Learning Rate
0.03
Scheduler
7) Yo ons( Talk
SGD Momentum
0.9
Model EMA Momentum
0.999
Prediction EMA Momentum
0.999
Weak Augmentation
Random Crop, Random Horizontal Flip
Strong Augmentation
RandAugment (Cubuk al. 2020
A.3.2 LONG-TAILED IMAGE CLASSIFICATION
The hyper-parameter: for long-taiie image classification evaluation is shown in Table We use Adam optimizer instead. For faster training, WRN-28-2 is used for both CIFAR-10 10 and CIFAR-100. NVIDIA V100 used to train long-tailed image classfication, and the training time is around day.
Table 7: Hyper- of long-tailed image classification tasks.
Dataset
CIFAR-10
CIFAR-
Model
WRN-28-2
Weight Decay
4e-5
Labeled Batch size
64
Unlabeled Batch size
128
Learning Rate
0.002
Scheduler
17 1o0 cos( 16.
Optimizer
Adam
Model EMA Momentum
0.999
Prediction EMA Momentum
0,999
Weak Augmentation
Random Crop. Random Horizontal Flip
Strong Augmentation
RandAugment (Cubuk et al. 2020
A.3.3 TEXT CLASSIFICATION
For text classification tasks, we random split validation set from the training set of each dataset used. For IMDb and AG News, we randomly sample 1,000 data and 2,500 data per-class respectively validation set, and other data is used as training set. For Amazon-5 and Yclp-5, we randomly sample 5,000 data and 50,000 data per-class as validation set and training set respectively. For DBpedia, the validation set and training set consist of 1, 1,000 and 10,000 samples per-class.
18

---

Published as a conference paper at ICLR 2023
The training parameters used shown in Table 8 Note that for strong augmentation, back-
translation similar to (Xie et al. 2020). We conduct back-trame tack-translation offline before training, using EN-DE and EN-RU with models provided in fairseq (Ott et al., 2019). We use NVIDIA V100 to train all text classification models, the total training time is around 20 hours.
Table 8: Hyper-parameters of text classification tasks.
Dataset
AG News
DBpedia
IMDb
Amazom-5
Yelp-5
Model
Bert-Base
Weight Decay
le44
Labeled Batch size
16
Unlabeled Batch size
16
Learning Rate
le-5
Scheduler
1) cos(
16K
Tark
Model EMA Momentum
0.0
Prediction EMA Momentum
0..99
Weak Augmentation
None
Strong Augmentation
Trac--rranlation (Xie et al. 2020
A.4 EXTEND EXPERIMENT RESULTS
In this section, we provide detailed experiments on the implementation of the sample weighting function in unlabeled loss, as shown in Table 9 One can observe most fixed functions works sur- prisingly well on CIFAR-IO with 250 labels, yet Gaussian function demonstrate the best results on CIFAR-10 with 40 labels. On the SVHN with 40 labels, Linear and Quadratic function fails to learn while Laplacian and Gaussian function shows better performance. Estimating the function param- eters from the confidence and making the function truncated allow the model learn more flexibly and yields better performance for both Laplacian and Gaussian function. We visualize the functions studied in Fig. 4 where one can observe the truncated Gaussian function most reasonable by assigning diverse weights for samples whose confidence is within its standard deviation.
Table 9: Detailed results of different instantiation of Ap on CIFAR-1O with 40 and 250 labels, and SVHN-10 with 40 labels.
Method
(p)
Learnable
CIFAR-10
CIFAR-IO 250
SVIIN-1040
Lineur
max
11.381392
5.41:019
15.27
Quadratic
(max(p)
1)² +1
12.44
5,94=022
84.1
Laplacian
exp
mund (p)
:4 = 1.0, b= 0.3
13.294333
5,24
12.77
Gaussian
exp
[man] 2 20"
# 1.0, 0.3
7.73
4.98
12.951879
Trun, Laplacian
exp
[maxip)_p
max(p) p. otherwise.
14.b
5.30
5.14
3.12
1.0,
Trun, Gaussian
exp
(msa] p)
if max(p) p. otherwise.
1,.
a.91
4.82+am9
2.33
1.0.
A.5 EXTENDED ABLATION STUDY
We provide the additional ablation study of other components of SoftMatch, including the EMA mo- mentum parameter m, the variance range of truncated Gaussian function, and the target distribution of Uniform Alignment (UA), on CIFAR-10 10 with 250 labels.
EMA momentum. We compare SoftMatch with momentum 0.99, 0.999, and 0.9999 and present the results in Table I0 A momentum of 0.999 shows the best results. While different momentum does not affect the final performance much, they have larger impact on convergence speed, where smaller momentum value results in faster convergence yet lower accuracy and a larger momentum slows down the convergence.
19

---

Published as a conference paper at ICLR 2023
1.0
Quad.
0.8
Lpp
Gan.
Turm. Lags.
0.6
X
Tarm Kamn.
0.4
0.22
0.0
0.0
0.2
0.44
0.6
0.5
max(p:)
Figure 4: Sample weighting function visualization
Table 10: Ablation of EMA momentum CIFAR-10 with 250 labels.
Table 11: Ablation of vari- ance range in Gaussian func- tion on CIFAR-10 with 250 labels.
Table 12: Ablation target distribution of UA CIFAR- 10 with 250 labels.
Momentum
Error Rate
Target Dist.
Error Rate
Variance Range
Error Rate
0.99
1.
(y. P u(C)
4.8310.12
0.999
4.02
0
4.97
4.90.1023
0.9999
4.86=0.12
2n
4.82
4.82+00.9
30
4.84+0.15
Variance range. We study the variance range of Gaussian function. In all experiments of the main paper, we use 2o range, i.e., divide the estimated variance o, by in practice. The variance range directly affects the degree of softness of the truncated Gaussian function. We show in Table 11 that using a directly results in a slight performance drop, while 2o and 3o produces similar results.
UA target distribution. In the main paper, we validate the target distribution of UA on long-tailed setting. We also include the effect of the target distribution of UA on balanced setting. As shown in Table 12 using uniform distribution u(c) or the ground-truth marginal distribution pz (y) produces the same results, whereas using the estimated pz (y) (Berthelot et al. 2021 has a performance drop.
A.6 EXTEND ANALYSIS ON TRUNCATED GAUSSIAN
In this section, provide further visualization about the confidence distribution of pseudo-lahels, and the weighting function, similar to Fig. Taa) but on CIFAR-10. More specifically, we plot the histogram confidence of pseudo-labels and of wrong pseudo-labels, from epoch I 6. We select the first epochs because the difference is more significant. Along with the histogram, we also visualization how the pseudo-labels over
plot the current weighting function over confidence, different confidence interval are used in different methods.
Fig. Fisummarizes the visualization. Interestingly, although FixMatch adopts quite high threshold, the quality of pseudo-labels is very low, i.e., there are more wrong pseudo-labels in each confidence interval. This reflects the important of involving more pseudo-labels into training at the beginning, as in SoftMatch, to let the model learn more balanced on each class to improve quality of pseudo- labels.
A.7 EXTEND ANALYSIS ON UNIFORM ALIGNMENT
In this section, provide explanation regarding the mechanism of Uniform Alignment (UA). UA is proposed to make the model learn more equally on each classes to reduce the pseudo-label imbalance/bias. To do so, align the expected prediction probability to uniform distribution
20

---

Published as a conference paper at ICLR 2023
Spock 13.00
Speck
Spork 22.00
Spock $5.00
Spock
Speck
Wrng kgo
Weee 4go
Weeg
Wree Ago
3go
51
K3 Clebbacc
X1
X7 Cabbbar
(a) FixMatch
Epock
Wros
Wina Xgi
Whon
Wros Ago
Weem 2g1
Ago
A30
Mics
Oedkeer
Oekker
Geekeer
Oebken
(b) FlexMatch
Wrng
Weep
Weer
Ayo
(c) SoftMatch
Figure 5: Histogram of confidence of pseudo-labels, learned by (a) FixMatch; (b) Flexmatch; (c) SoftMatch, for first 6 epochs on CIFAR-10. The weighting function over confidence of each method shown as the blue curve. For FlexMatch, we plot the average threshold. SoftMatch presents better accuracy by utilizing pscudo-labels in a more efficient way.
Before
After
Dafieraaee
1.0
m155-11.56,
m150-11557 std-1150
0.11
0.02
Wingan
0.8
Wongie
0.2
0.27
0.6
0.6
-0.14
Class Index
Claes Index:
Claes Index
Figure 6: Average weight for each class according to pseudo-label, for (a) before UA; and (b) after UA. We also include the difference of them in (c). UA helps to balance the average weight of each class.
when computing the sample weights. A difference of UA and DA is that UA is only used in weight computing, and not used in consistency loss. To visualize this, we plot the average class weight according to pscudo-labels of SoftMatch before UA and after UA at the beginning of training, as shown in Fig. 6 UA facilitates more balanced class-wise sample weight, which would help the model learn more equally on each class.
21


Reviews:
[{"review_id": "ysrtH4Dpse", "reviewer": "Reviewer_ZqdU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes SoftMatch to improve both the quantity and quality of pseudo-labels in semi-supervised learning. Basically, the authors designed a truncated Gaussian function to weigh the unlabeled samples and remedied the imbalanced problem by encouraging a low entropy distribution within a batch.", "summary_of_the_review": "This work needs further improvement from both a thorough investigation of SSL and clearer demonstrations to the main contributions.", "strength_and_weaknesses": "Strength:\n1. It is interesting that the authors understood the limitation of pseudo-labels from the view of quantity-quality trade-off, and some experimental evidence was presented by comparing with FixMatch and FlexMatch.\n2. The weight function for unlabeled samples adjusted the importance of the contributions of different samples during training, which is intuitively better compared with naïve pseudo-labels.\n\nWeaknesses:\n1. The quantity-quality trade-off shown in Fig. 1(a) seems not fair. For Fixmatch, how many unlabeled samples are selected by a hard threshold is dependent on the training process. Experimentally, only a small proportion of unlabeled data will be selected at the beginning of model training, and it should select the most unlabeled data once the model converges. Otherwise, the model performance would not be guaranteed. Similar to Flexmatch. Although the motivation sounds interesting, the experimental observation is not clear or convincing to me.\n2. Following 1, I felt a bit confused about why SoftMatch here is a curve instead of a point.\n3. From Section 2.2, the quantity and quality functions f(p) and g(p) are only used for analysis instead of the SoftMatch method. From Table 1, I can sense the difference between them, but I cannot tell the advantages of SoftMatch directly.\n4. Following 3, the comparison between Pseudo-label and FixMatch in Table 1 is unclear to me. E.g., Pseudo-Label also has the sample selection, but the weight equals a const lambda_max.\n5. Eq. (2) can be viewed as a soft version of sample selection, but in what case it is needed or outperforms other strategies is not clear after I read the paper.\n6. The weight is normalized at a batch level with EMA stabilizing the training, thus it shares the same insight with [1]. However, such a branch of work has not been mentioned.\n7. From the experiments, the improvement of SoftMatch on standard SSL benchmarks is quite limited but it showed some potential in imbalanced class cases, which is however not the key challenge from the introduction of this work.\n\n[1] Sinkhorn label allocation: Semi-supervised classification via annealed self-training, ICML 2021.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "xTL3Ljf7Tp", "reviewer": "Reviewer_QAQp", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of quantity-quality trade-off in pseudo-labeling based semi-supervised learning. Specifically, the authors develop a truncated Gaussian function to weight samples based on their confidence, which can be viewed as a soft version of the confidence threshold. They also enhance the utilization of weakly-learned classes by proposing a uniform alignment approach. The experiments on various types of datasets demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "I do not have major concerns on this paper. I only have several minor concerns as listed above. I hope the authors can provide the reply regarding my previous minor concerns.", "strength_and_weaknesses": "Pros:\n\n1. This paper is written in a clear way.\n\n2. The idea of this work is reasonable, and the details of the proposed algorithm are provided.\n\n3. The experiments are convincing.\n\nI do not have major concerns on this work. I only have several questions:\n1. This work targets the quantity-quality trade-off problem in semi-supervised learning, which is meaningful. I agree that in each iteration, this problem does exist. But for the entire learning process which consists of many iterations, I’m not sure whether this is still an important problem. In traditional pseudo-labeling methods, only the unlabeled examples with high confidence are utilized for each iteration. This will decrease the quantity, but the quality will be high, as most of them are with correct labels. Then, when the iterations proceed, the learning performance will be gradually improved with a large probability. However, if we spend too much efforts on the quantity-quality trade-off in each iteration, I’m afraid that this might hurt the accuracy as well as decrease the efficiency from the global view of learning. Therefore, I hope the authors can give more explanations on this point. \n2. For pseudo-labeling methods, the authors are suggested to mention whether their method is transductive or inductive, or is applicable to both conditions. \n3. Some typos should be fixed. For example, in the caption of Fig.1, “(b) Quality of pseudo-labels” should be “(c) Quality of pseudo-labels”. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "rIOz1J5CEW", "reviewer": "Reviewer_ZV9Z", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper points out the inherent quantity-quality trade-off problem of pseudo-labeling with confidence thresholding exists in recent semi-supervised learning (SSL) methods. It then proposes a soft adaptive sample weighting scheme using a truncated Gaussian function to utilize both high quantity and quality of pseudo-labels. Meanwhile, a uniform alignment technique is designed for pseudo-labeling, which exploits original predictions to compute pseudo-labels and normalized predictions to compute sample weights.", "summary_of_the_review": "The main idea is very interesting to me, and the performance is good. However, the clarity and experiments should be improved.", "strength_and_weaknesses": "Pros:\n* The consideration of the quantity-quality trade-off is very interesting and novel.\n* The proposed solution is simple but effective. Imbalanced setting is also taken into consideration in this paper.\n* The evaluation is conducted on image and text datasets, together with imbalanced settings. The performance is also promising.\n\nCons:\n* Even with the summary of formulations in Table 1, why the proposed SoftMatch could achieve both high quality and quantity is not well-explained.\n* More recent studies towards adaptive thresholding and imbalanced setting in SSL should be discussed and compared, e.g., [1-2].\n* Why the truncated version of Gaussian function is used should be explained (although the ablation results show it’s the best choice).\n* It would be better if the learned truncated Gaussian function and the real confidence distribution could be illustrated on real-world datasets (for several epochs). \n* It is claimed that the proposed uniform alignment could alleviate the distribution of generated pseudo-labels to be imbalanced. Then [3] should be included for performance comparison. Besides, the visualization of the pseudo-label distribution (w.r.t. classes) is helpful to understand the effect of uniform alignment, especially on imbalanced datasets.\n* Why only 3 random seeds are used in Table 2? In the original FixMatch paper, the number of random seeds is 5.\n* It is not clear on which dataset and with which number of labeled data the ablation studies are conducted.\n\nRef:\n\n[1] Class-Imbalanced Semi-Supervised Learning with Adaptive Thresholding, ICML22.\n\n[2] Smoothed Adaptive Weighting for Imbalanced Semi-Supervised Learning: Improve Reliability Against Unknown Distribution Data, ICML22.\n\n[3] Debiased Learning from Naturally Imbalanced Pseudo-Labels, CVPR22.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "C3QvsvrpQ0", "reviewer": "Reviewer_Kfxk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the traditional Semi-Supervised Learning (SSL) problem. The authors first revisit the popular pseudo-labeling methods via a unified sample weighting formulation and demonstrate the inherent quantity-quality trade-off problem of pseudo-labeling with thresholding, which may prohibit learning. To this end, The authors propose SoftMatch to overcome the trade-off by maintaining both high quantity and high quality of pseudo-labels during training, effectively exploiting the unlabeled data. The authors derive a truncated Gaussian function to weight samples based on their confidence, which can be viewed as a soft version of the confidence threshold. The authors further enhance the utilization of weakly-learned classes by proposing a uniform alignment approach.\n\n", "summary_of_the_review": "See *Strength And Weaknesses*\n", "strength_and_weaknesses": "Strengths:\n\n1. This paper demonstrates the importance of the unified weighting function by formally defining the quantity and quality of pseudo-labels, and the trade-off between them.\n2. This paper proposes SoftMatch to effectively leverage the unconfident yet correct pseudo-labels, fitting a truncated Gaussian function in the distribution of confidence, which overcomes the trade-off. \n3. This paper demonstrates that SoftMatch outperforms previous methods on various image and text evaluation settings.\n\n\nWeaknesses:\n\n1. One issue of the existing SOTA SSL methods is the issue of efficiency (e.g., Fixmatch is time-consuming for training). Based on figure 2, the proposed SoftMatch has a similar issue.\n2. What is the motivation for Uniform Alignment (UA)? Why the model prediction is aligned to a uniform distribution?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster