FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning
Authors: Yidong Wang
Hao Chen
Qiang Heng
Wenxin Hou
Yue Fan
Zhen Wu
Jindong Wang
Marios Savvides
Takahiro Shinozaki
Bhiksha Raj
Bernt Schiele
Xing Xie
Venue: ICLR 2023 poster (2023)
Keywords: Semi-Supervised Learning
Semi-Supervised Classification

Abstract: Semi-supervised Learning (SSL) has witnessed great success owing to the impressive performances brought by various methods based on pseudo labeling and consistency regularization. However, we argue that existing methods might fail to utilize the unlabeled data more effectively since they either use a pre-defined / fixed threshold or an ad-hoc threshold adjusting scheme, resulting in inferior performance and slow convergence. We first analyze a motivating example to obtain intuitions on the relationship between the desirable threshold and model's learning status. Based on the analysis, we hence propose FreeMatch to adjust the confidence threshold in a self-adaptive manner according to the model's learning status. We further introduce a self-adaptive class fairness regularization penalty to encourage the model for diverse predictions during the early training stage. Extensive experiments indicate the superiority of FreeMatch especially when the labeled data are extremely rare. FreeMatch achieves 5.78%, 13.59%, and 1.28% error rate reduction over the latest state-of-the-art method FlexMatch on CIFAR-10 with 1 label per class, STL-10 with 4 labels per class, and ImageNet with 100 labels per class, respectively. Moreover, FreeMatch can also boost the performance of imbalanced SSL. The codes can be found at https://github.com/microsoft/Semi-supervised-learning.


Published as a conference paper at ICLR 2023
FREEMATCH: SELF-ADAPTIVE THRESHOLDING FOR SEMI-SUPERVISED LEARNING
Yidong Wang 1.2g Hao Chen Qiang Heng". Wenxin Hou Yue Fan Zhen Wu", Jindong Wang Marios Savvides Takahiro Shinozaki?, Bhiksha Raj Bernt Schiele", Xing Xie
Microsoft Research Asia, Tokyo Institute of Technology. Carnegie Mellon University, North Carolina State University, 9 Microsooft STCA, 6 Max Planck Institute for Informatics, Saarland Informatics Campus, Nanjing University, "Mohamed bin Zayed University of Al
ABSTRACT
Semi- semi-ssuprrised Learning (SSL) has witnessed great success owing to the im- pressive performances brought by various methods based on pseudo labeling and consistency regularization. However, we argue that existing methods might fail to utilize the unlabeled data more effectively since they either use a pre-defined fixed threshold or an ad-hoc threshold adjusting scheme, resulting in inferior performance and slow convergence. We first analyze motivating example to ob- tain intuitions on the relationship between the desirable threshold and model's 's learning status. Based on the analysis, we hence propose FreeMatch to ad- just the confidence threshold in a self-adaptive manner according to the model's learning status. We further introduce a self-adaptive class fairness regularization penalty to encourage the model for diverse predictions during the early training stage. Extensive experiments indicate the superiority of FreeMatch especially when the labeled data are extremely rare. FreeMatch achieves 5.78%, 13.59% and 1.28% error rate reduction over the latest state-of of-the-art art method FlexMatch on CIFAR-10 10 with 1 label per class, STL-10 with labels per class, and Im- ageNet with 100 labels per class, respectively. Moreover, FreeMatch can also boost the performance of imbalanced SSL. The codes can be found at whttts:: htt.://ww...oooiiiooonaaaeae
Semi supervised weweeedddeeeainnngg
INTRODUCTION
The superior performance of deep learning heavily relies on supervised training with sufficient la- beled data (He al. 2016) Vaswani al., 2017 Dong al. 2018). However, it remains laborious and expensive to obtain massive labeled data. To alleviate such reliance, semi-supervised learning (SSL) (Zhu 2005| Zhu & Goldberg, 2009; Sohn et al. 2020, Rosenberg al., 2005 Gong et al., Kervadec et al.) 2019: Dai et al.) 2017) is developed to improve the model's l's generalization performance by exploiting a large volume of unlabeled data. Pseudo labeling (Lee et al. 2013] Xie
2016
et al.
2020b McLachlan 1975 Rizve et al. 2020) and consistency regularization (Bachman et al.)
2014
Samuli & Timo 2017: Sajjadi et al. (016) are two popular paradigms designed for modern
SSL
Recently, their combinations have shown promising results (Xie et al.) 2020aa [Sohn et al.)
Pham et al., 2021 Xu et al. 202T Zhang et al. 2021). The key idea is that the model should produce similar predictions the same pscudo labels for the same unlabeled data under different perturbations following the smoothness and low-density assumptions in SSL (Chapelle et al. 2006).
2020
A potential limitation of these threshold-basee methods is that they either need a fixed threshold (Xie) et al. 2020a| Sohn et al. 2020 Zhang et al. 2021 Guo & Li 2022) or an ad-hoc threshold adjusting
Equual Contribution: yidongwang:7 @ com, haoc3 woddew.cmu.edu work done when Yidong was a research intern at MSRA.
"Correspandence to: jindong. wang miiemsofi.cm
"Note the results of this paper are obtained using TorchSSI. 7hang et al. 2021 We also provide codes and
logs in USB (Wang al. 2022)

---

Published as a conference paper at ICLR 2023
3.0
Sunplin Raks
RaMatch
Deaffatch
Betrarch
Ratlarr
Dash
VTL
XTL
Fleellatch
ProPatth
Feaioochh
PresMaathh
0.-
Dash
2.0
Dan
400
003
1200
1997
2002
420
2200 1860 2020
(a) Decision boundary
(b) Self-adaptive fairness
(c) Confi. threshold
(d) Sampling rate
Figure 1: Demonstration of how Free.aath works on the "two-moon" dataset. (a) Decision bound- ary of FreeMatch and other SSL methods. (b) Decision boundary improvement of self-adaptive fairness (SAF) on two labeled samples per class. (c) Class-average confidence threshold. (d) Class- average sampling rate of FreeMatch during training. The experimental details are in Appendix [A] scheme (Xu et al. 2021) to compute the loss with only confident unlabeled samples. Specifically, UDA (Xie et al. 2020a) and FixMatch (Sohn al. 2220)) retain a fixed high threshold to ensure the quality of pseudo labels. However, a fixed high threshold (0.95) could lead to low data utilization in the early training stages and ignore the different learning difficulties of different classes. Dash Xu et al. 202T) and AdaMatch (Berthelot et al. 2022) propose to gradually grow the fixed global (dataset-specific) threshold as the training progresses. Although the utilization of unlabeled data is improved, their ad-hoc threshold adjusting scheme is arbitrarily controlled by hyper-parameters and thus disconnected from model's learning process. FlexMatch (Zhang et al. (222I) demonstrates that different classes should have different local (class-specific) thresholds. While the local thresholds take into account the learning difficulties of different classes, they are still mapped from a pre- defined fixed global threshold. Adsh (Guo & Li| (022) obtains adaptive thresholds from pre-defined threshold for imbalanced Semi-supervised Learning by optimizing the the number of pseudo labels for each class. In nutshell, these methods might be incapable or insufficient in terms of adjusting thresholds according to model's 's learning progress, thus impeding the training process especially when labeled data too scarce to provide adequate supervision.
For example, as shown in Figure I(a] on the "two-moon" dataset with only labeled sample for each class, the decision boundaries obtained by previous methods fail in the low-density assumption. Then, two questions naturally arise: 1) Is it necessary to determine the threshold based on the model learning status? and 2) How to adaptively adjust the threshold for best training efficiency?
In this paper, we first leverage motivating example to demonstrate that different datasets and classes should determine their global (dataset-specific) and local (class-specific) thresholds based on the model's learning status. Intuitively, we need a low global threshold to utilize more unlabeled data and speed up convergence at early training stages. As the prediction confidence increases, a higher global threshold is necessary to filter out wrong pseudo labels to alleviate the confirmation bias (Arazo et al. 2020). Besides, local threshold should be defined on each class based on the model's confidence about its predictions. The "two-moon" example in Figure (aa) shows that the decision boundary is more reasonable when adjusting the thresholds based on the model's ' learning status.
We then propose FreeMatch to adjust the thresholds in a self- addapivdee manner according to learning status of each class (Guo et al.) 2017). Specifically, FreeMatch uses the self-adaptive thresholding (SAT) technique to estimate both the global (dataset-specific) and local thresholds (class-specifor via the exponential moving average (EMA) of the unlabeled data confidence. To handle barely su- pervised settings (Sohn et al., 2020) more effectively, we further propose a class faireess objective to encourage the model to produce fair (i.e., diverse) predictions among all classes (as shown in Figure [(b)p. The overall training objective of FreeMatch maximizes the mutual information be- tween model's input and output (John Bridle) 1991), producing confident and diverse predictions on unlabeled data. Benchmark results validate its effectiveness. To conclude, our contributions are:
Using motivating example, we discuss why thresholds should reflect the model's learning status and provide some intuitions for designing a threshold threshold-adjusting scheme.
We propose a novel approach, FreeMatch, which consists of Self-Adaptive Thresholding (SAT) and Self-Adaptive class Fairness regularization (SAF). SAT is a threshold-acjusting scheme that is free of setting thresholds manually and SAF encourages diverse predictions.
Extensive results demonstrate the superior performance of FreeMatch various SSL bench- marks, especially when the number of labels is very limited (c.g, an error reduction of 5.78% on CIFAR-10 with 1 labeled sample per class).
2

---

Published as a conference paper at ICLR 2023
2 A MOTIVATING EXAMPLE
In this section, we introduce a binary classification example to motivate our threshold-adjusting scheme. Despite the simplification of the actual model and training process, the analysis leads to some interesting implications and provides insight into how the thresholds should be set.
We aim demonstrate the necessity of the self- aof-aadataabliity and increased granularity in confidence thresholding for SSL. Inspired by (Yang& & Xu (2020), we consider a binary classification problem where the true distribution is an even mixture of two Gaussians (i.e., the label Y equally likely to be positive (+1) or negative (-1)) 1)). The input X has the following conditional distribution:
X X = -1 ~ -
X = +1 NN1~N(1p)) (u2,, o3).
We assume 222 > Ma without loss of generality. Suppose that classifier outputs confidence score s(x) 1/ 1x111=xp(=3(x==== == ))]. where /8 is positive parameter that reflects the model learning status and expected to gradually grow during training as the model becomes more confident. Note that d2 is in fact the Bayes' optimal linear decision boundary. We consider the scenario where fixed threshold T E ( 1) is used to generate pseudo labels. A sample x: is assigned pseudo label +1 (()) > т and
sxx) < - T. The pseudo label is 0 (masked) if - s(x) S т.
We then derive the following theorem to show the necessity of self-adaptive threshold:
Theorem 2.1. For a binary classification problem as mentioned above, the pseudo label Yp has the following probability distribution:
P(Y, = 1)
-
1
B
(og(
log(
P1-42
2
log( I-
2
02
2
01
122111
1
loog( 1-7
)
p1
log( 1-7
(2)
P(Y, =
-1) =-
01
2
P(Y, 0) = P(Y, =) p( -11,
where + is the cumulative distribution function of a standard normal distribution. P(Y, = 0) increases as μ2 Pi gets smaller.
Moreover,
The proof is offered in Appendix B Theorem 2.1 has the following implications or interpretations:
(i) Trivially, unlabeled data utilization (sampling rate) 1 - P(Y, = 0) is directly controlled by thresh- old r. As the confidence threshold gets larger, the unlabeled data utilization gets lower. At early training stages, adopting high threshold may lead to low sampling rate and slow convergence since still small.
(ii) More interestingly. P(Y, = 1) + P(Y, = 1) if o1 + 02. In fact, the larger T is, the more imbalanced the pseudo labels are. This is potentially undesirable in the sense that aim to tackle a balanced classification problem. Imbalanced pseudo labels may distort the decision boundary and lead to the so-called pseudo label bias. An easy remedy for this is to use class-specific thresholds and 1 - Ti to assign pseudo labels.
(ii)
The sampling rate 1 P(Y, 0) decreases μ1 μn gets smaller. In other words, the more similar the two classes are, the more likely an unlabeled sample will be masked. As the two classes get more similar, there would be more samples mixed in feature space where the model is less confident about its predictions, thus moderate threshold is needed to balance the sampling rate. Otherwise may not have enough samples train the model to classify the already difficult-to-classifyy classes.
The intuitions provided by Theorem 2.1 is that at the early training stages, should be low to en- courage diverse pseudo labels, improve unlabeled data utilization and fasten convergence. However, as training continues and |3 grows larger, a consistently low threshold will lead to unacceptable con- firmation bias. Ideally, the threshold should increase along with 3 to maintain stable sampling rate throughout. Since different classes have different levels of intra-class diversity (different σ) and some classes are harder classify than others (#2 - μu being small), fine-grat grained class-specific threshold desirable to encourage fair assignment of pseudo labels to different classes. The chal- lenge is how design threshold adjusting scheme that takes all implications into account, which is
3

---

Published as a conference paper at ICLR 2023
the main contribution of this paper. We demonstrate our algorithm by plotting the average threshold trend and marginal pseudo label probability (i.e. sampling rate) during training in Figure [[c] and I(d) To sum up, we should determine global (dataset-specific) and local (class-specifie) thresholds by estimating the learning status via predictions from the model. Then, we detail Free
3 PRELIMINARIES
In SSL, the training data consists of labeled and unlabeled data. Let Dz {((,, Ub) : b E [Nt]] and Du = (up [Nu]]] be the labeled and unlabeled data, where Nr and No is their number of samples, respectively. The supervised loss for labeled data is:
1 B
L.
H(yb, Pm (ylw(())))
where is the batch size, H(;;)) refers to cross-entropy loss, w(-) means the stochastic data aug- mentation function, and Pm is the output probability from the model.
For unlabeled data, we focus on pseudo labeling using cross-entropy loss with confidence threshold for entropy minimization. We also adopt the "Weak and Strong Augmentation" strategy introduced by UDA (Xie et al. 2020a). Formally, the unsupervised training objective for unlabeled data is:
BB
1
Cu
P. Σ (max((e) > r) H(@g. Q6)-
(4)
We use @ and Qs to denote abbreviation of Pm)(y(w(s)) and pm (y/ ((())), respectively. qu the hard "one-hot" label converted from qu, is the ratio of unlabeled data batch size to labeled data batch size, and 1( > T) is the indicator function for confidence-based thresholding with r being the threshold. The weak augmentation (i.e., random crop and flip) and strong augmentation (i.e., RandAugment
al. (2020) is represented by w()) and 12(-) respectively.
Besides, a faimess objective Ly is usually introduced to encourage the model to predict each class at the same frequency, which usually has the form of L, U log En [4o] (Andreas Krause 2010). where U is a uniform prior distribution. One may notice that using a uniform prior not only pre- vents the generalization to non-ume -nniuoffomm data distribution but also ignores the fact that the underlying pseudo label distribution for mini- batch may be imbalanced due the sampling mechanism. The uniformity across a batch is essential for fair utilization of samples with per-class threshold, espe- cially for early-training stages.
FREEMATCH
4.1 SELF-ADAPTIVE THRESHOLDING
We advocate that the key to determining thresholds for SSL is that thresholds should reflect the learn- ing status. The learning effect can be estimated by the prediction confidence of a well-calibrated model (Guo al. 2017p. Hence, defines and adaptively adjusts the confidence threshold for each class by leveraging the model pre- dictions during training. SAT first estimates a global threshold as the EMA of the confidence from the model. Then, SAT modulates the global threshold via the local class-specific thresholds es- timated the EMA of the probability for each class from the model. When training starts, the threshold is low to accept more possibly correct samples into training. As the model becomes more confident, the threshold adaptively increases to filter out possibly incorrect samples reduce the confirmation bias. Thus, as shown in Figure we define SAT as 7, Tc(c) indicating the threshold for class e at the 1-th iteration.
propose self-adaptive thresholding (SAT) that automatically
Self-adaptive Global Threshold We design the global threshold based on the following two prin- ciples. First, the global threshold in SAT should be related to the model's confidence on unlabeled
[N] (1,,...., N).
4

---

Published as a conference paper at ICLR 2023
Selfadapive Threhholdigg
Pscudo Label
Unisbeled Samples
(r(3)
F2(())
Global Threshodd
Local Threshold (t)
Local Threshold (+()
0.50
0.25
0.50
0:25
Training Sennion
0.00 N, (90
0.00
Figure 2: Illustration of Self-Adaptive Thresholding (SAT). FreeMatch adopts both global and local self-adaptive thresholds computed from the EMA of prediction statistics from unlabeled samples. Filtered (masked) samples are marked with red X.
data, reflecting the overall learning status. Moreover, the global threshold should stably increase during training to ensure incorrect pscudo labels are discarded. We set the global threshold ti as av- erage confidence from the model on unlabeled data, where represents the t-th time step (iteration). However, it would be time-consuming to compute the confidence for all unlabeled data at every time step or even every training epoch due to large volume. Instead, we estimate the global confidence as the exponential moving average (EMA) of the confidence each training time step. We initialize Tr as & where indicates the number classes. The global threshold T₃ is defined and adjusted as:
t A + (1- -)) >) nB Eta max(g)),
if I = 0, otherwise,
(5)
where À E (0, 1) is the momentum decay of EMA.
Self-adaptive Local Threshold The local threshold aims modulate the global threshold in class-specific fashion to account for the intra-class diversity and the possible class adjacency. We compute the expectation of the model's predictions on each class e to estimate the class-specific learning status:
( t. Pc(c) (c) +1- (1- Eua (c)),
if = 0,
(6)
otherwise,
where 5; = (c), I is the list containing all (c()). Integrating the global and local thresholds, we obtain the final self-adaptive threshold (c) as:
(c)
(c) MaxNorm(i. (c)) 7 max (c) c [C]}
Th
where MaxNorm is the Maximum Normalization (i.e., x' m training objective fu the 1-th iteration is:
). Finally, the unsupervised
BB
Lu
E (max((9)) > 7i (arg max (q6)) H(qu. Qb).
(8)
BB
4.2 SELF-ADAPTIVE FAIRNESS
We include the class fairness objective as mentioned Section 3 into FreeMatch to encourage the model to make diverse predictions for each class and thus produce meaningful self-adaptive threshold, especially under the settings where labeled data are rare. Instead of using uniform prior in (Arazo et al. 2020), the expectation of prediction distribution over unlabeled data. We optimize the cross-entropy of p. and p F B Ppm [(w))) over mini-batch estimate of H (TF,, [pm. (ylu))) Considering that
the EMA of model predictions Pt from Eq. 6 as estimate of
5

---

Published as a conference paper at ICLR 2023
the underlying pseudo label distribution may not be uniform, we propose modulate the fairness objective self-adaptive way, i.c., normalizing the expectation of probability by the histogram distribution of pseudo labels to counter the negative effect of imbalance as:
(max (q:) 2 Tr (arg max (q:)) Qb:
BB
(9)
H 1 (max (ga) 2 (arg max (g:) Q1))
Similar to Pt, we compute h as:
h Ahi-1 (1 X) Hist uB (06)
(10)
The self-adaptive fairness (SAF) Ly at the t-th iteration is formulated as:
C-=-HSSumNomy (SumNorm
Pr
P
Cj = =
SumNorm
(1I)
where SumNorm = ()/ E(). SAF encourages the expectation of the output probability for each mini-batch to be close to marginal class distribution of the model, after normalized by histogram distribution. It helps the model produce diverse predictions especially for barely supervised settings (Sohn et al. 2020), thus converges faster and generalizes better. This is also showed in Figure [(b)
The overall objective for FreeMatch at (-th iteration is:
L L. WiyCe WyLf,
(12)
where wi, and wy represents the loss weight for Lu and Ly respectively. With Cu and C,. FreeMatch maximizes the mutual information between its outputs and inputs. We present the procedure of FreeMatch in Algorithm T of Appendix.
5
EXPERIMENTS
5.1 SETUP
We evaluate FreeMatch on common benchmarks: CIFAR-10/100 (Krizhevsky al.] 2009).
SVHN (Netzer et al. 2011). STL-10 (Coates et al. lowing previous work (Sohn et al., 2020; Xu et al. 2021 Zhang et al.) 2021 Oliver et al.) 2018),
2011 and ImageNet (Deng et al. 22009) Fol-
we conduct experiments with varying amounts of labeled data. In addition to the commonly-chosen
labeled amounts, following (Sohn et al. 2020), we further include the most challenging case of CIFAR-10: each class has only one labeled sample.
For fair comparison, we train and evaluate all methods using the unified codebase TorchSSL (Zhagg et al. 2021) with the same backbones and hyperparameters. Concretely, 28-2 Zagoruyko & Komodakis 2016) for CIFAR-10, Wide ResNet-28-8 for CIFAR-100, 100, Wide ResNet-37-2 (Zhou et al. 20200) for STI-110, and ResNet-50 (He al., 2016) for ImageNet. We use SGD with a momentum 0.9 as optimizer. The initial learning rate is 0.03 with a cosine learning rate decay schedule as 7 170 cos( Tak ), where 170 is the initial learning rate, k(K) is the current (total) training step and we K 220 for all datasets. At the testing phase, use an exponential moving average with the momentum of 0.999 of the training model to conduct inference for all algorithms. The batch size of labeled data is 64 except for ImageNet where we set 128. We use the same weight decay value, pre-defined threshold т, unlabeled batch ratio / and loss weights introduced for Pseudo-Laaee (Lee et al. (013). model (Rasmus et al., 2015), Mean Teacher (Tarvainen & Valpola 2017), VAT MMyyato al. 2018 MixMatch (Berthelot et al. (019b), ReMixMatch [Berthelot et al. 2019a), UDA (Xie et al. 2020a), FixMatch (Sohn et al. 2020). and Flex (Zhang al. 2021
Wide ResNet-
We implement MPL based on UDA as in (Pham et al. 2021 where we set temperature as 0.8 and wa 10. We do not fine-tune MPL labeled data in (Pham et al.] 2021) since find fine-tuning will make the model overfit the labeled data especially with very few of them. For Dash, we the same parameters as in (Xu et al. 2021) except warm-up labeled data for

---

Published as a conference paper at ICLR 2023
Table 1: Error rates on CIFAR-IO/100 SVHN, and STL-10 datasets. The fully-supervised results of STL-10 10 are unavailable since we do not have label information for its unlabeled data. Bold indicates the best result and underline indicates the second- besttd-bett result. The significant tests and average error rates for each dataset can be found in Appendix B.
10
40
250
1000
400
2500
10000
40
250
1000
-00
1000
101 mm
46.30 79
15185
87.48amm
57.35an1
78.59an3
15.50e11
0.40
76.38amm
12.54
VAT Pooai 1PR
71511.41
1400.11
10.31.01
55.20
45344009
31:14001
34.511.
4.5....1
4.1
14.3420u
31:35
S.10ua2
81:11.10
45.17106
31:75000
360911
345:46
71:21116
36.19.cm
1343
3,69.000
31.50use
RcNG 1osmiig
20.77
6.30
4.540001
42.75.05
3602102
2484199
0.6as
5166000
12.12
6.7
1014
10822311
6:3pmm
32.R4am
512eam
132
1
37/43xa4
Dut 2001
5.
23358633
31.55pm
137
6.3kam
2355-ar1
21:4aam
933eem
2.0
3:28
15.3aa0
0.50.29
6.73
5007
3758
16:2
21,61000
15.5
Full-Ssperried
4.2-44
19:30
2.13
2 epochs since too much warm-up will lead to the overfitting (i.e. 2.048 training iterations). For FreeMatch, we set We 1 for all experiments. Besides, we set wy 0.01 for CIFAR-10 with 10 labels, CIFAR-100 with 400 labels, STL-10 with 40 labels, ImageNet with 100k labels, and all experiments for SVHN. For other settings, we use w = 0.05. For SVHN we find that using a low threshold at early training stage impedes the model to cluster the unlabeled data, thus we adopt two training techniques for SVHN: (1) warm-up the model on only labeled data for epochs as Dash; and (2) restrict the SAT within the range [0.9, 0.95]. The detailed hyperparameters are introduced in Appendix D We train each algorithm times using different random seeds and report the best error rates of all checkpoints (Zhang et al. 2021
5.2 QUANTITATIVE RESULTS
The Top-1 classification error rates of CIFAR-10/100, SVHN and STL-10 are reported in Table I The results on ImageNet with 100 labels per class are in Table 2 precision, recall, F1 score, and confusion matrix in Appendix E.3
We also provide detailed results on These quantitative results demon- strate that FreeMatch achieves the best performance on CIFAR- I0, STL-10, and ImageNet datasets, and it produces very close results on SVHN the best competitor. On CIFAR- FreeMatch is better than ReMixMatch when there are 400 labels. The good performances of ReMixMatch on CIFAR- 100 (2500) and CIFAR- (10000) are probably brought by the mix up (Zhang et al. 2017) technique and the self-supervised learning part. On ImageNet with 100k labels, Free.Match signif- icantly outperforms the latest counterpart Flex by 1.28%F) We also notice that FreeMatch exhibits fast computation in ImageNet from Table 2 Note that FlexMatch is much slower than Fix- Match and FreeMatch because it needs to maintain a list that records whether each sample is clean, which needs heavy indexing computation budget on large datasets.
Noteworthy is that, FreeMatch consistently outperforms other methods by large margin settings with extremely limited labeled data: 5.78% on CIFAR-10 10 with 10 labels, 1.96% on CIFAR-100 with 400 labels, and surprisingly 13.59% on 100 with 40 labels. STL-10 is a more realistic and challenging dataset compared to others, which consists of a large unlabeled set of 100k images. The signif- icant improvements demonstrate the capability and poten- tial of FreeMatch to be deployed in real-world applications.
Table 2: Error rates and runtime on ImageNet with 100 labels per class.
Runtime (scc./iter.)
Top-1
Top-5
FixMatch FlexMatch
43.66
21.80
0.4
41.85
19.48
0.6
FreeMatch
40.57
18.77
0.4
5.3 QUALITATIVE ANALYSIS
We present some qualitative analysis: Why and how does FreeMatch work? What other benefits does it bring? We evaluate the class average threshold and average sampling rate on STL-10 (40) (i.e., 40 labeled samples on STL-10) of FreeMatch to demonstrate how it works aligning with our theoretical analysis. We record the threshold and compute the sampling rate for each batch during
Σx" 1 (max(qu (2-r (arg max(gs)) We
training. The sampling rate is calculated on unlabeled data as
RB
Following (Zhang et al. 2021), J. we train ImageNet for 220 iterations like other datasets for fair compari- son. We use 4 Tesla V100 GPUs ImageNet.
7

---

Published as a conference paper at ICLR 2023
Thowstold
6.66
0.8
Samete Pare
a.n-
0.7
0.5
Farkatch
8.4
Tuhanth
235
3.5
RleMatch
lochatcc
FresHarch
Danh
3.3
Dash
0.2
Flexiaatch
Fiextlatch
FiexMathh
3.0
0.0
5034
100t 5004
Iter.
(a) Confidence threshold
(h) Sampling rate
(c) Accuracy
(d) Confusion matrix
Figure 3: How FreeMatch works in STL-I0 with 40 labels, compared to others. (a) Class-average
confidence threshold; (b) class-average sampling rate; (c) convergence speed in terms of accuracy; (d) confusion matrix, where fading colors of diagonal elements refer to the disparity of accuracy.
also plot the convergence speed in terms of accuracy and the confusion matrix show the proposed
component in FreeMatch helps improve performance. From Figure B(a) and Figure 3(b) one can observe that the threshold and sampling rate change of FreeMatch is mostly consistent with our
theoretical analysis. That is, at the early stage of training, the threshold of FreeMatch is relatively lower, compared to FlexMatch and FixMatch, resulting in higher unlabeled data utilization (sam- pling rate), which fastens the convergence. As the model learns better and becomes more confident, the threshold of FreeMatch increases to a high value to alleviate the confirmation bias, leading to stably high sampling rate. Correspondingly, the accuracy of FreeMatch increases vastly (as shown in Figure 3(c) and resulting better class-wise accuracy (as shown in Figure 3(d) Note that Dash fails to leam properly due to the employment of the high sampling rate until TOOK iterations.
To further demonstrate the effectiveness of the class-specific threshold in FreeMatch, we present the 1-SNE (Van der Maaten & Hinton| 2008) visualization of features of FlexMatch and FreeMatch on STL-10 (40) in Figure 5 of Appendix E.8 We exhibit the corresponding local threshold for each class. Interestingly. FlexMatch has high threshold, i.c.. pre-defined 0.95, for class and class 6, yet their feature variances are very large and are confused with other classes. This means the class- wise thresholds in FlexMatch cannot accurately reflect the learning status. In contrast, FreeMatch clusters most classes better. Besides, for the similar classes 1, 3, 5, 7 that are confused with each other, FreeMatch retains higher average threshold 0.87 than 0.84 of FlexMatch, enabling to mask more wrong pseudo labels. We also study the pseudo label accuracy in Appendix E.9 and shows FreeMatch can reduce noise during training.
5.4 ABLATION STUDY
Self-adaptive Threshold We conduct experiments on the components of SAT in FreeMatch and compare to the components in FlexMatch (Zhang et al., 2021). FixMatch (Sohn et al. 2020). Class-Balanced Self-Training (CBST) (Zou al. 2018), and Relative Threshold (RT) in AdaMatch (Berthelot et al. 2022). The ablation is conducted on CIFAR-10 (40 labels).
As shown in Table 3 SAT achieves the best performance among all the threshold schemes. Self-adaptive global threshold and local threshold MaxNoorm(Pe(e) themselves also achieve comparable results, compared to the fixed threshold r, demonstrating both local and global threshold proposed are good learning effect estimators. When using CPL MM3(c)) to adjust 7,, the result is worse than the fixed threshold and exhibits larger variance, indicating potential instability of CPL. AdaMatch (Berthelot RT, which can be viewed as a global threshold at 1-th iteration computed on the predictions of labeled data without EMA, whereas FreeMatch conducts computation of T. with EMA on unlabeled data that can better reflect the overall data distribution. For class-wise threshold, CBST (Zou et al., 2018) maintains a dre-ne sampling rate, which could be the reason for its bad performance since the sampling rate should be changed during training as we analyzed in Sec. [ Note that we did not include L5 in this ablation for a fair comparison. Ablation study in Appendix E.4 and E.5 on FixMatch and FlexMatch with different thresholds shows SAT serves to reduce syperparameter-tuniig
Table 3: Comparison of dif- ferent thresholding schemes.
Threshokl
CIFAR-10 (40)
(FisMatch))
7 7.47
MM.(e)) (FlexMatch)
4.97asus
MMaNNotttfee(e)) n (Glebal)
5.13
6.062aa5
(M(((())
8. 8.10s224
al.) (022) uses the
CBST
16.65 16.65:229
(AdaMaich)
6.09
SAT (Glebal and Local)
4.92
computation or overall training time in the event of similar performance for an optimally selected threshold.
8

---

Published as a conference paper at ICLR 2023
Self-adaptive Fairness As illustrated in Table 4 also empir- ically study the effect of SAF on CIFAR-10 (10 labels). We study the original version of faimess objective as in (Arazo al. 2020). Based on that, we study the operation of normalization probability by histograms and show that countering the effect of imbalanced underlying distribution indeed helps the model to learn and diverse better. One may notice that adding original fairness regularization alone already helps improve the performance. Whereas adding normalization operation in the log operation hurts the performance, suggesting the underlying batch data are indeed not uniformly distributed. We also evaluate Distribution Alignment (DA) for class fairness in ReMixMatch (Berthelot et al.) 2019a) and AdaMatch (Berthelot et al., 2022), showing inferior results than SAF. A possible reason for the worse performance of DA (AdaMatch) is that only uses labeled batch prediction as the target distribution which cannot reflect the true data distribution especially when labeled data is scarce and changing the target distribution to the ground truth uniform, i.e., DA (Rthhx is better for the case with extremely limited labels. We also proved SAF can be easily plugged into FlexMatch and bring improvements in Appendix [E.6] The EMA decay ablation and performances of imbalanced settings are in Appendix B.5 and Appendix E.7
Table Comparison of dif- ferent class fairness items.
Faimess
CIFAR-10 (10)
w/o fairness
10.37=2.70
U Fog
9.57:s677
U log (
12,07=52
DA (AdaMatch)
12.94
DA (ReMixMatch) SAF
11.06=821
8,07#424
RELATED WORK
To reduce confirmation bias (Arazo al., 2020 in pseudo labeling, confidence- thresholding techniques are proposed to ensure the quality of pseudo labels (Xie et al.) 2020a [Shnn et al.) 2020 Zhang et al.) 2021 Xu et al.] 202T). where only the unlabeled data whose confidences are higher than the threshold are retained. UDA (Xie et al., 2020a) and FixMatch (Sohn et al. (0200) keep the fixed pre-defined threshold during training. FlexMatch (Zhang et al. 2021) adjusts the pre-defined threshold in sass--pecificc fashion according to the per-class learning status estimated by the
number of confident unlabeled data. A co-current work Adsh (Guo & Li 2022) explicitly optimizes the number of pseudo labels for each class in the SSL objective to obtain adaptive thresholds for imbalanced Semi-supervised Learning. However, it still needs a user-predefined threshold. Dash (Xu al. 202T) defines threshold according to the loss on labeled data and adjusts the threshold according to a fixed mechanism. A more recent work, AdaMatch (Berthelot et al. 2022). aims to unify SSL and domain adaptation using pre-defined threshold multiplying the average confidence of the labeled data batch to mask noisy pseudo labels. It needs pre-defined threshold and ignores the unlabeled data distribution especially when labeled data is too rare to reflect the unlabeled data distribution. Besides, distribution alignment (Berthelot et al. 2219a] 2022) is also utilized in Adamatch to encourage fair predictions on unlabeled data. Previous methods might fail to choose meaningful thresholds due to ignorance of the relationship between the model learning status and thresholds. Chen et al. (2020); Kumar et al. (2020) try to understand self-training/ / thresholding from the theoretical perspective. We use motivating example to derive some implications and further adjust meaningful thresholds according to the learning status satisfying the derived implications.
Except consistency regularization, entropy-basse regularization is also used in SSL. Entropy minimization (Grandvalet et al., (005) encourages the model to make confident predictions for all samples disregarding the actual class predicted. Maximization of expectation of entropy (Andreas Krause 2010; Arazo et al. 2020) over all samples is also proposed to induce fairness to the model. enforcing the model to predict each class at the same frequency. But previous ones assume uniform prior for underlying data distribution and also ignore the batch data distribution. Distribution alignment (Berthelot et al. (2019)) adjusts the pseudo labels according to labeled data distribution and the EMA of model predictions.
7 CONCLUSION
We proposed FreeMatch that utilizes self-adaptive thresholding and class-fairness regularization for SSL. FreeMatch outperforms strong competitors across a variety of SSL benchmarks, especially setting. We believe that confidence thresholding has more potential in
in the SSL. A potential limitation is that the adaptiveness still originates from the heuristics of the model prediction, and hope the efficacy of FreeMatch inspires more research for optimal thresholding.

---

Published as a conference paper at ICLR 2023
REFERENCES
Ryan Gomes Andreas Krause, Pietro Perona. Discriminative clustering by regularized information maximization. Advances neural information processing systems, 2010.
Eric Arazo, Diego Ortego, Paul Albert, Noel O' O'Connor, and Kevin McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2020.
Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-cnsembles. Advances in neural information processing systems, 27:3365-3373, 2014.
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmenta- tion anchoring. In International Conference Learning Representations, 2019a.
David Berthelot, Nicholas Carlini, lan Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in Neural Infor- mation Processing Systems, 32, 2019b.
David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alex Kurakin. Adamatch: A unified approach to semi-supervised learning and domain adaptation. In International Conference on Learning Representations (ICLR), 2022.
Nicholas Carlini, Ulfar Erlingsson, and Nicolas Papernot. Distribution density, tails, and outliers in machine learning: Metrics and applications. arXiv preprint arXiv: 13427, 2019.
Olivier Chapelle, Berhhard Schölkopf, and Alexander Zien (eds.). Semi-Supervised Learning. The MIT Press, 2006.
Yining Chen, Colin Wei, Ananya Kumar, and Tengyu Ma. Self-training avoids using spurious features under domain shift. Advances in Neural Information Processing Systems, 33:21061- 21071, 2020.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215-223. JMLR Workshop and Conference Proceedings, 2011.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Zihang Dai, Zhilin Yang, Fan Yang, William W Cohen, and Russ Salakhutdinov. Good semi- supervised learning that requires a bad gan. Advances in neural information processing systems, 30, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255 leee, 2009.
Linhao Dong, Shuang Xu, and Bo Xu. trasshetransformmer:: a no-recurrence sequence-to-sequence model for speech recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5884-5888. IEEE, 2018.
Yue Fan, Dengxin Dai, and Bernt Schiele. Cossl:
Co-leaming of representation and classifier for 2021.
imbalanced semi-supervised learning. arXiv preprint
Chen Gong, Dacheng Tao, Stephen J Maybank, Wei Liu, Guoliang Kang, and Jie Yang. Multi- modal curriculum learning for semi-supervised image classification. IEEE Transactions on Image Processing, 25(7):3249-3260 2016.
Yves Grandvalet. Yoshua Bengio, et al. Semi-supervised learning by entropy minimization. volume 367, pp. 281-296, 2005.
10

---

Published as a conference paper at ICLR 2023
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, PP. 1321-1330. PMLR, 2017.
Lan-Zhe Guo and Yu-Feng Li. Class-imbalanced semi-supervised learning with adaptive threshold- ing. In International Conference on Machine Learning, pp. 8082-8094, PMLR, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
David MacKay John Bridle, Anthony Heading. Unsupervised classifiers, mutual information and "phantom targets. 1991.
Hoel Kervadec, Jose Dolz, Éric Granger, and Ismail Ben Ayed. Curriculum semi-supervised seg- mentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 568-576 Springer, 2019.
Jachyung Kim, Youngbum Hur, Sejun Park, Eunho Yang, Sung Ju Hwang, and Jinwoo Shin. Dis- tribution aligning refinery of pseudo-label for imbalanced semi-supervised learning. Advances in Neural Information Processing Systems, 33:14567-14579, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980) 2014.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Ananya Kumar, Tengyu Ma, and Percy Liang. Understanding self-training for gradual domain adaptation. In International Conference Machine Learning, pp. 5468-448556 5479. PMLR, 2020.
Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges representation learning, ICML, volume 3, pp. 896, 2013.
Hyuck Lee, Seungjae Shin, and Hecyoung Kim. Abe: Auxiliary balanced classifier for class- imbalanced semi-supervised learning. Advances in Neural Information Processing Systems, 34, 2021.
Geoffrey McLachlan. Iterative reclassification procedure for constructing an asymptotically opti- mal rule of allocation in discriminant analysis. Journal of the American Statistical Association, 70(350):365 369, 1975.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi- cupi-super leaming. IEEE transactions on pattern analysis and machine intelligence, 1189:1197,, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Real- istic evaluation of deep semi-supervised learning algorithms. Advances in neural information processing systems, 31, 2018.
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoe V Le. Meta pseudo labels. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11557-11568, 2021.
Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi- supervised learning with ladder networks. Advances in Neural Information Processing Systems, 08:3546-35545 2015.
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo- labeling: An uncertainty-awart aware pseudo-label selection framework for semi-supervised learning. In International Conference Learning Representations, 2020.
11

---

Published as a conference paper at ICLR 2023
Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. Semi-supervised self- self-training of object detection models. 2005.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. Advances in neural information processing systems, 29: 169116611,,, 2016.
Laine Samuli and Aila Timo. Temporal ensembling for semi-supervised learning. In International Conference on Learning Representations (ICLR), volume pp. 6, 2017.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in Neural Information Processing Systems, 33, 2020.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results. In Proceedings of the 3/st Interna- tional Conference on Neural Information Processing Systems, pp. 1195-1204, 2017.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Yidong Wang. Hao Chen, Yue Fan, SUN Wang. Ran Tao, Wenxin Hou, Renjie Wang, Linyi Yang, Zhi Zhou, Lan-Zhe Guo, et al. Usb: A unified semi-supervised learning benchmark for clas- sification. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang. Crest: A class- rebalancing self-training framework for imbalanced supervupe learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10857-10866, 2021.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems, 33, 2020a.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10687-10698, 2020b.
Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-supervised learning with dynamic thresholding. In International Conference Machine Learning. pp. 11525-11536 PMLR, 2021.
Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-ibbalanced learning. In NeurlPS, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference 2016. British Machine Vision Association, 2016.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang. Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting superviie learning with curriculum pseudo la- beling. Advances in Neural Information Processing Systems, 34, 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez- Paz. mixup: Beyond empirical risk minimization. arXiv preprint 17 2017.
Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Time-consistent self-supervision for cemi-supervised learning. In International Conference Machine Learning, pp. 11523-11533. PMLR, 2020.
12

---

Published as a conference paper at ICLR 2023
Xiaojin Zhu and Andrew B Goldberg. Introduction to semi-supervised learning. Synthesis lectures on artificial intelligence and machine learning, 3(1):1-130, 2009.
Xiaojin Jerry Zhu. Semi-supervised learning literature survey. 2005.
Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for se- mantic segmentation via class-balanced self-training. In Proceedings of the European conference on computer vision (ECCV), pp. 289-305, 2018.
13

---

Published as a conference paper at ICLR 2023
A EXPERIMENTAL DETAILS OF THE "TWO-MOON" DATASET.
We generate only two labeled data points (one label per each class, denoted by black dot and round circle) and 1,000 unlabeled data points (in gray) in 2-D space. We train 3-layer MLP with 64 neu- rons in each layer and ReLU activation for 2,000 iterations. The red samples indicate the different samples whose confidence values are above the threshold of FreeMatch but below that of FitMa The sampling rate is computed on unlabeled data as E (max((:) > r)/Nu. Results are aver- aged 5 times.
B PROOF OF THEOREM 2.1
Theorem 2.1 For a binary classification problem as mentioned above, the pseudo label Yp has the following probability distribution:
11
log(
-
log( I-7
P(Y, = 1) = ④
I-T
2
2
02
2
log T-7
P1
log(
(13)
P(Y, -1) = -1
2
$
2
T-
2
01
02
P(Y, = 0) - P(Y, = where 4 the cumulative distribution function of a standard normal distribution. Moreover, P(Y, = 0) = 0 increases as μ4l gets smaller.
P(Y,
--1,
Proof. A sample a will be assigned pseudo label 1 if
1
+ exp (-3(x H2))
> T,
which is equivalent
M1 + H2
1
x
+
log
2
B
Likewise, will be assigned pseudo label -1 if
<1
+ exp (-3(x
11
which is equivalent to we integrate over x, we arrive at the following conditional probabilities:
11 + 112
x
log
2
P2-#1
log(
P(Y, 1|Y =1) $
2
1-7
02
12
log
P(Y, p 1/Y = =-1 = $
7 log
P(Y, 1|Y 11) $
2
1-7
9
log(
P(Y, - 1) +
02
Recall that P(Y 1) P(Y =-1) 0.5, therefore
log
log
P(Y, = 1) =
2 T
$
2
02
01
1/
log
4-
log
T
P(Y; - =
B
1-7
2
1-7
91
14

---

Published as a conference paper at ICLR 2023
Now, let's to denote #2 141 to show that P(Y, 0) increases μ1 14 gets smaller,
only need to show P(X, = -1) + P(Yp = 1) gets bigger. We write P(Yp = =- + P(Y, + = 1) as
-))1)+++
P(Y, = 1)
P(Y, 1)
= 1) =
1 _ - -b1) + pp b1) + ( b2) 2 f - b2)
2
Ing( )-) b2 = B0z
log( (=x) are positive constants. We futher only
where = 2. 02 20g
L bj = Ben
need show that f(z) = } (a, b1) + f b1) is monotone increasing on (0, (0,00), Take
the derivative of 2, we have
( bn) c(-ai b1)), ('(:) 2
where é is the probability density function of a standard normal distribution. Since la12 b1 Q1 - b1). we have (z) > 0, and the proof is complete.
<
ALGORITHM ALGORITHM
C
We present the pseudo algorithm of FreeMatch. Compared to FixMatch, each training step involves updating the global threshold and local threshold from the unlabeled data batch, and computing corresponding histograms. FreeMatchs introduce very trivial computation budget compared to FixMatch, demonstrated also in our main paper.
Algorithm FreeMatch algorithm at f-th iteration.
1: Input: Number of classes C, labeled batch X {(T,, vo) b E (1,2,.... B)}. unlabeled batch U {u, b E (1,2,...,,BB). unsupervised loss weight was fairness loss weight wy, and EMA decay A.
2: Compute L, for labeled data
L. 1/ b= Hyy,, Pm (y]w(x))))
3: Update the global threshold
Tt = A + (1 W E max(g)) {96 is an abbreviation of pm (yll((us)), shape of TỤ:
[1]
4: Update the local threshold
Pr = Apr-1 + (1 A) pz MuB qb (Shape of pe: [C]}
5: Update histogram for ps
he = Ah, + (1 - A) Hist (46) (Shape of he: [C]}
6: for 1 to C do
7: Te(c)= = (ax) n (Calculate SAT)
8: end for
9: Compute L. on unlabeled data
f. = nB Σu (max (2x) 2 rarg max (q.)) H((s. Qb)
10: Compute expectation of probability on unlabeled data
p 1 Σn 1 (max (96) > (arg max (96)) Qu (Qu is abbr. of p. (w))))))) shape of p:
nB
[C]}
11: Compute histogram for p
h= = (I (max (qs) 2 Tararg max (gh)) @.) {Shape of h: [C]
Compute C, on unlabeled data
C = - SSmnNomm{ (t)) SumNNrm( (
13: Return: L. + Wn Lu wy L.j
D HYPERPARAMEN HYPERPARAMETER CAREPARAMEEEE SETTING
For reproduction. we show the detailed hyperparameter setting for FreeMatch in Table 5 and 6 for c
algorithm-dependent and a seorittmm--i independent eeeeparameeees respectively.
15

---

Published as a conference paper at ICLR 2023
Table Algorithm dependent hyperparameters.
Algorithm
FreeMatch
Unlabeled Data Labeled Data Ratio (CIFAR 10/100 STL 10, SVHN
Unlabeled Data to Labeled Data Ratio (ImageNet)
weight wa experiments
I Loss weight wy for CIFAR-I0 (10), CIFAR 10D (400), STI. (40), ImageNet (100k), SVHN
0.01
Loss weight wy others
0.05
Thresholding EMA decay for all experiments
0.999
Table 6: Algorithm independent hyperparameters.
Dataset
CIFAR-10
CIFAR-100
STL-10
SVHN
ImageNet
Model
WRN 28--28-2
WRN 28
WRN-37-2
WRN 28
ResNet 50
Weight decay
le-3
5e-4
5e-4
3e-4
Batch size
64
128
Learning rite
0.03
EMA decay
0,999
Note that for ImageNet experiments, we used the same learning rate, optimizer scheme, and training iterations as other experiments, and a batch size of 128 is adopted, whereas, in FixMatch, a large batch size of 1024 and different optimizer is used. From our experiments, ImageNet with only 2 is not enough, and the model starts converging at the end of training. Longer training iterations on ImageNet will be explored the future. Single NVIDIA V100 used for training on CIFAR-10 10, CIFAR-100 100. SVHN and STL-10. It costs about 2 days to train on CIFAR-10 and SVHN. 10 days are needed for the training on CIFAR- and STL-10.
found that training
E EXTENSIVE EXPERIMENT DETAILS AND RESULTS
We present extensive experiment details and results complementary the experiments in the main paper.
E. SIGNIFICANT TESTS
We did significance test using the Friedman test. We choose the top 7 algorithms on 4 datasets (i.e.. N = 4, = 7). Then, we compute the F value = 3.56, which is clearly larger than the thresholds 2. 2. 0.05) and 2.130 = 0.1). This test indicates that there are significant differences between all algorithms.
To further show our significance, we report the average error rates for each dataset in Table 7 We can see FreeMatch outperforms most SSL algorithms significantly.
E.2 CIFAR-10 10 (10) LABELED DATA
Following (Sohn et al. 2020), we investigate the limitations of SSL. algorithms by giving only one labeled training sample per class. The selected 3 labeled training sets are visualized in Figure which are obtained by (Sohn et al. 2020) using ordering mechanism (Carlini et al. 2019).
E.3 DETAILED RESULTS
To comprehensively evaluate the performance of all methods in classification setting, we further report the precision, recall, fl score, and AUC (area under curve) results CIFAR-10 with the same 10 labels, LIFAR-100 with 400 labels, SVHN with 40 labels, and STL-10 with 40 labels. As shown Table and | FreeMatch also has the best performance on precision, recall, Fl score, and AUC in addition the topl error rates reported in the main paper.
16

---

Published as a conference paper at ICLR 2023
Table 7: The average enror rates for each dataset.
CIFAR-10
SVHN CIFAR-100
STI.-10
Total Average
Model
53.22
60.80
29.31
53.55
49.19
Pseudo Label
54.10
60.58
29.87
53.66
49.59
VAT
51.50
54.73
27.73
56.35
47.17
MeanTeacher
48.01
52.68
14.27
52.81
41.54
MixMatch
30.56
45.04
12.95
38.32
31.07
ReMihMatch
10.45
29.60
11.85
19.43
17.08
UDA
13.65
32.20
2.98
22.03
17.02
FixMatch
10.33
32.22
2.60
21.11
15.67
Dash
11.43
31,28
2.07
20.46
15.56
MPL
10.12
31.90
4.63
21.21
16.04
FlexMatch
7.00
29,44
7.17
17,46
14.40
FrecMatch
5.49
28.71
1.97
10.60
11.26
Figure 4: CIFAR-10 (10) labeled samples visualization, sorted from the most prototypical dataset (first row) to least prototypical dataset (last row).
E.4 ABLATION OF PRE-DEFINED THRESHOLDS FIXMATCH AND FLEXMATCH
As shown in Table 12 the performance of FixMatch and FlexMatch is quite sensitive the changes of the pre-defined threshold r.
E.5 ABLATION ON EMA DECAY ON CIFAR-10 (40)
We provide the ablation study on EMA decay parameter À in Equation (5) and Equation 6 One can observe that different decay À produces the close results on CIFAR-IÓ with 40 labels, indicating that FreeMatch is not sensitive this hyper-parameter. large À is not encouraged since could impede the update of global local thresholds.
E.6 ABLATION OF SAF ON FLEXMATCH AND FREEMATCH
In Table we present the comparison of different class fairness objectives on CIFAR-10 with 10 labels. FreeMatch is better than Match in both settings. In addition, SAF is also proved effective when combined with FlcxMatch.
E.7 ABLATION OF IMBALANCED SSL
To further prove the effectiveness of FreeMatch, We evaluate FreeMatch on the imbalanced SSL setting Kim et al. (2020); Wei et al. 2021]: Lee et al. (202T); Fan et al. (2021), where the labeled and the unlabeled data both imbalanced. We conduct experiments
LIFAR-10-L1 and CIFAR- 100-LT with different imbalance ratios. The imbalance ratio used on CIFAR datasets is defined as
7 Nuax Nmmin where N the number of samples the head (frequent) class and Nmin the
17

---

Published as a conference paper at ICLR 2023
Table 8: Precision, recall, fl score and AUC results on CIFAR-10/100.
Datascts
CIFAR-I0 (10)
CIFAR-100
Criteria
Procision
Recall
FI Score
AUC
Procision
Recall
FI Score
AUC
UDA
0.5304
0.5121
0.4754
0.8258
0.5813
0.5484
0.5087
0.9475
FixMatch
0.6436
0.6622
0.6110
0,8934
0.5574
0.5430
0.4946
0.9363
Dash
0.6409
0,5410
0.4955
0,8458
0.5835
0.5649
0.5215
0.9456
MPL.
0.6286
0.6857
0.6178
0.7993
0.5799
0.5606
0.5193
0.9316
FlexMatch
0.6769
0.6861
0.6780
0,9126
0.6135
0.6193
0.6107
0.9675
FrecMatch
0.8619
0.8593
0.8523
0.9843
0.6243
0.6261
0.6137
0.9692
Table 9: Precision, recall, fl score and AUC results on SVHN and STL-
Datnsets
SVHN (40)
STL- (40)
Criteria
Precision
Recall
FI Score
AUC
Precision
Recall
Fl Score
AUC
UDA
0.9777
0.9977
0.6385
0.5319
0.4765
FixMatch
0.9731
0.9706
0.9716
0.9962
0.6590
0.5830
0.5405
0.8362
Dash
0.8117
0.8827
MPI.
0.9564
0.9513
0.9512
0.9844
0.6191
0.5740
0.4999
0.8529
FlexMarch
0.9566
0.9691
0.9625
0.9975
0.6403
0.6755
0.6518
0.9249
FreeMatch
0.9783
0.9791
0.5489
0.8439
0.9792
tail (rare). Note that the number of samples for class ie computed as N₂- Nmaxz" _ where i' the number of classes. Following (Lee et al. 2021 Fan et al.) 2021). set Nmor 1500
for CIFAR-10 and Nmaz = 150 for CIFAR- 100, and the number of unlabeled data is twice as many for each class. We use WRN-28-2 (Zagoruyko & Komodakis 2016) the backbone. We use Adam (Kingma & Ba 2014) as the optimizer. The initial learning rate is 0.002 with cosine learning rate decay schedule 77 To cos( 7ak ), where To is the initial learning rate, k(K) is the ToX
current (total) training step and we set K 2.5 x 10 for all datasets. The batch size of labeled and unlabeled data is 64 and 128, respectively. Weight decay is set as 4e-5, Each experiment is run on three different data splits, and report the average of the best error rates.
Compared with other standard SSL methods, FreeMach achieves the best performance across all settings. Especially on CIFAR-IO at imbalance ratio 150, FreeMatch outperforms the second best by 2.4%. Moreover, when plugged in the other imbalanced SSL method (Lee et al. 202T), FreeMatch still attains the best performance most of the settings.
The results are summarized in Table 14
E.8 T-SNE VISUALIZATION ON S0L- (40)
We plot the T-SNE visualization of the features on STL-10 with 40 labels from FlexMatch (Zhang ct al. 2021) and FreeMatch. FreeMatch shows better feature space than FlexMatch with less con- fusing clusters.
E.9 PSEUDO LABEL ACCURACY ON CIFAR- 10 (10)
We average the pseudo label accuracy with three random seeds and report them in Figure 6 This indicates that mapping thresholds from high fixed threshold like FlexMatch did can prevent un- labeled samples from being involved in training. In this case, the model can overfit on labeled data and a small amount of unlabeled data. Thus the predictions on unlabeled data will incorporate
Table 10: FixMatch and FlexMatch with different thresholds on CIFAR-10 (40).
FixMatch
FlexMatch
0.25
11.7620.60
18.84+0.36
0.5
16.29#0.31
14.16+0.21
0.75
+0.63
6.08+0.17
0.95
7.47+0.28
4.97+0.06
0.98
8.01+0.91
1.40+0.11
18

---

Published as a conference paper at ICLR 2023
Table 11: Error rates of different thresholding EMA decay.
Thresholding EMA decay
CIFAR-10 (40)
0.9
4.94:005
0.99
4.92=0..5
4.90
0.9999
5.03=0007
Table 12: FixMatch and FlexMatch with different thresholds on CIFAR-10 (40).
T
FixMatch
FlexMatch
0.25
11.76++0.60
18.84+0.36
0.5
16.29+0.31
14.16#0.21
0.75
15.61#0.23
6.08+0.1 17
0.95
7.47=0.28
4.97=0.06
0.98
8.01
5,40=0.11
N00000:00
5001-154
100--044
361-314
NS5-0004
5551-3.)
1000-1:00
5181-111
NO1-0.00 0.00
-01-233
(a) FlexMatch (train, test)
(h) FreeMatch (train, test)
Figure 5: T-SNE visualization of FlexMatch and FreeMatch features on STL-10 (40). Unlabeled data is indicated by gray color. Local threshold (c(c) for each class shown on the legend.
more noise. Introducing appropriate unlabeled data training time can avoid overfitting labeled datasets and small amount of unlabeled data and bring more accurate pseudo labels.
E. CIFAR-10 (10) CONFUSION MATRIX
We plot the confusion matrix of FreeMatch and other SSL methods on CIFAR- (10) Figure It is worth noting that even with the least prototypical labeled data in our setting, FreeMatch still gets good results while other SSL methods fail separate the unlabeled data into different clusters, showing inconsistency with the low-density assumption in SSL.
Table 13: Ablation of SAF on FlexMatch and FreeMatch on CIFAR-10 (10)
Fairness Objective
FlexMatch
FreeMatch
w/o SAF w/ SAF
13.85+12.04 12.60+8.16
10.37+7.70 8.07+4.24
19

---

Published as a conference paper at ICLR 2023
Table 14: Error rates (%) of imbalanced SSL using 3 different random seeds.
Dataset
CIFAR-IO.A 10-IT
CIFAR-I00-IT
100-I.T
Imbalance 7
50
150
20
100
FixMarch
18.5
31.2.108
FlexMarch
17,8:014
29.5:m4:
48.9au71
62.7=um
FrocMatch
17.7=033
28.8=064
62.5-023
FixMatch w/ ABC
14.0_uu2
22.
46.6_009
58.3_n0
FlesMatch
14.2=0034
58.9-051
FrocMatch w/ ABC
13.9
22.3-026
45.6
58.9
0.7
0.6
FreeMatch
0.4
0.3
250k
500k
750k
1000k
Iter. CIFAR-10
Figure 6: CIFAR- 10 (10) Pseudo Label accuracy visualization.
0.8
0.5
> 5/9 0.1 9.90
S/.5 Mibsee 0
V/ 6. 61 69.0 S13b 9.00
545 b9 1199999999
19999 5/0
1 9.90
Predicted labol
Predicted labei
Predicted labol
Predicted label
Predicted labol
Predicted label
(a) The most prototypical labeled samples
0.5
0.4
J/4 61999999
Vi. Mibsee 69,0
1/.5 5933661 69.0 S19use1 0 14.5 696661999 40364
9
Predicted label
Predicted label
Predicted label
Predicted label
Predicted labal
Predicted label
(b) The second-most prototypical labeled samples
0.8
0.6
Trus
5/4 A- 01 9.0
Vibbee1 69 593560 69,0
5135511 99,
14569610
522 243646 61 9
Pradicted label
Predicted labei
Predictad label
Prodicted label
Predicted labal
Prodicted labol
(c) The least prototypical labeled samples
Figure 7: Confusion matrix on the test set of CIFAR-IO (10). Rows correspond to the rows in
Figure Columns correspond to different SSL methods.
20


Reviews:
[{"review_id": "RRvVN_fzMH", "reviewer": "Reviewer_Ebth", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "For the semi-supervised learning task, this paper argues that existing methods might fail to utilize the unlabeled data more effectively since they either use a fixed threshold or an ad-hoc threshold adjusting scheme, resulting in inferior performance and slow convergence. Then they propose FreeMatch to adaptively adjust the confidence threshold. Experiments on several related datasets demonstrate the superiority.", "summary_of_the_review": "The overall novelty of self-adaptive threshold learning strategy is unsatisfying. Improvement over FlexMatch on CIFAR is marginal. More experiments are suggested to support some statements.", "strength_and_weaknesses": "Strengths:\n\n(1) The motivation is clear.\n\n(2) The presentation is good, and the paper is easy to follow.\n\n(3) The proposed method is simple and effective.\n\nWeaknesses:\n\n(1) The main contribution lies in learning the threshold in an adaptive way. They assign various thresholds for different categories and iterations. Given the fact FlexMatch already set different weights for different classes, I think the overall novelty is unsatisfying. The contribution of self-adaptive fairness over existing strategy is also marginal.\n\n(2) The improvements over FlexMatch on CIFAR10 and CIFAR100 are marginal.\n\n(3) Please compare with FlexMatch under the same fairness regularization. For example, if you replace the SAF with that in FlexMatch, will the results are still better than FlexMatch?\n\n(4) The authors claim that the proposed method can speed up the convergence. However, according to Figure 3(c), in 500k iteration, the proposed method is still not converged. Please give more explanation.\n\n(5) For different datasets, different training strategy is adopted to improve the performance. For example, the authors define specific training requirement on STL-10, which is not consistent to the so-called self-adaptive property throughout the paper.\n\n(6) In FixMatch and FlexMatch, the predefined threshold is very high to promise that highly-confident samples are selected for training. However, the proposed FreeMatch assigns low threshold at the beginning, which might incorporate some incorrect samples for training. Please explain this.\n\n(7) There are several typos, such as \"Eq equation 6\" in page 5.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2XiaingJtXW", "reviewer": "Reviewer_zMTV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper the authors have proposed FreeMatch, a novel Semi Supervised Learning (SSL) technique that samples unlabeled data (the goal is to add pseudo labels and use them as supplementary data for model learning) through a Self Adapting (confidence) Thresholding (SAT) mechanism. The hypothesis is that an adaptive threshold (depending on the status of the model) is likely to be more effective than a predefined or ad-hoc threshold,  usually used by existing techniques.  \n\nAt each training iteration, FreeMatch adjusts confidence thresholds (for each class in the context of a classification problem) by leveraging model prediction information from the corresponding step. SAT first estimates a global threshold, then modulates via the local class-specific thresholds, estimated as the Exponential Moving Average (EMA) of the probability for each class. The threshold is kept low at the beginning of the training process to accept as many weak(possibly correct) samples as possible. As the model becomes more and more confident in subsequent iterations, the threshold is gradually increased to identify and reject incorrect samples (which reduces confirmation bias). \n\nThe proposed model has been tested on CIFAR-10/100, SVHN, STL-10, and ImageNet benchmarks and the results look to be encouraging. \n", "summary_of_the_review": "I have gone through the paper more than once including the appendices. Overall, the idea is quite sound, well articulated through the document, which is found easy to follow. The experiment is thorough and the reported results look encouraging. Self Adaptive Threshold (SAT), Self Adaptive Fairness, and the FreeMatch approach as a whole may benefit the SSL research. \n", "strength_and_weaknesses": "Strength: Usually, semi-supervised learning (SSL) methods use either some predefined or ad-hoc threshold adjustment mechanisms  which are challenged  by FreeMatch through the idea of Self Adaptive Threshold (SAT). FreeMatch provides an automated iterative unlabelled data sampling methodology using  threshold adjustments based on the learning status of the model and using  prediction confidence scores at the latest. The methodology also ensures class  fairness and learns a generalized solution (in the context of classification).\n \nFreeMatch has been tested on CIFAR-10/100, SVHN, STL-10, and ImageNet benchmarks and compared against existing techniques such as FixMatch, ReMixMatch, and FlexMatch. Reported results are found to be encouraging. \n \nWeakness: Although the reported results in section 5 look promising, they lack some statistical tests. It is suggested that authors perform proper statistical tests to justify whether the achieved gains are (statistically) significant or not.  If we zoom in to the results (Table 1 and 2) we find that gains are marginal for larger (such as CIFAR-100, ImageNet) and more complex (ImageNet) datasets when compared to smaller and simpler datasets such as CIFAR-10 (or data with less number of classes). This may have exhibited scalability limitations of the proposed approach. \n\nAlso, it is surprising that for smaller datasets (such as CIFAR-10) the reported results are sometimes even better than models trained in a fully supervised fashion.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0p5LScpToF", "reviewer": "Reviewer_axb6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper present an adaptive approach to threshold selection for labeling of unannotated data during semi-supervised model training. The proposed approach combining adaptive threshold selection and a combination of losses leads to state of the art performance for a variety of SSL classification tasks.", "summary_of_the_review": "Overall I'm in favor of accepting this paper as it combines novelty related to both the thresholding of unlabeled data and formulation of the loss function with empirically strong results.", "strength_and_weaknesses": "The authors to a good job of laying out th problem and presenting their proposed approach. The approach is clean and well motivated and the empirical results are strong.\n\nOne experimental result that would have been helpful to demonstrate the importance/impact of the components of the proposed approach would have been to show performance of the loss function (12) for a fixed threshold selected by optimizing validation performance. This would help show whether the adaptive thresholding scheme is serves to reduce hyperparameter-tuning computation or overall training time (in the event of similar performance for an optimally selected threshold) or whether the adaptive threshold outperforms any fixed threshold. Additionally, this would help demonstrate the improvement from the proposed self-adaptive fairness loss term independent of the adaptive threshold.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster