In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning
Authors: Mamshad Nayeem Rizve
Kevin Duarte
Yogesh S Rawat
Mubarak Shah
Venue: ICLR 2021 Poster (2021)
Keywords: Semi-Supervised Learning
Pseudo-Labeling
Uncertainty
Calibration
Deep Learning

Abstract: The recent research in semi-supervised learning (SSL) is mostly dominated by consistency regularization based methods which achieve strong performance. However, they heavily rely on domain-specific data augmentations, which are not easy to generate for all data modalities. Pseudo-labeling (PL) is a general SSL approach that does not have this constraint but performs relatively poorly in its original formulation. We argue that PL underperforms due to the erroneous high confidence predictions from poorly calibrated models; these predictions generate many incorrect pseudo-labels, leading to noisy training. We propose an uncertainty-aware pseudo-label selection (UPS) framework which improves pseudo labeling accuracy by drastically reducing the amount of noise encountered in the training process. Furthermore, UPS generalizes the pseudo-labeling process, allowing for the creation of negative pseudo-labels; these negative pseudo-labels can be used for multi-label classification as well as negative learning to improve the single-label classification. We achieve strong performance when compared to recent SSL methods on the CIFAR-10 and CIFAR-100 datasets. Also, we demonstrate the versatility of our method on the video dataset UCF-101 and the multi-label dataset Pascal VOC.

Published as a conference paper at ICLR 2021
IN DEFENSE OF PSEUDO-LABELING: AN UNCERTAINTY- SWAAR AWARE PSEUDO-LABEL SELEC- TION FRAMEWORK FOR SEMI-SUPERVISED LEARNING
Mamshad Nayeem Rizvel, Kevin Duarte Yogesh Rawat & Mubarak Shah
Center for Research in Computer Vision
University of Central Florida, Orlando, Florida, USA [Iaayemmrizve kevin.duarte (awrrtteeeenggttten (ennights edu $ (yogesh, chah)ecrev.uee
ABSTRACT
The recent research in semi--sperrvsed learning (SSL) is mostly dominated by consistency regularization based methods which achieve strong performance. How- ever, they heavily rely on domain-specific data augmentations, which are not easy generate for all data modalities. Pseudo-labeling (PL.) is general SSL approach that does not have this constraint but performs relatively poorly in its original formulation. We argue that PL underperforms due to the erroneous high confidence predictions from poorly calibrated models; these predictions generate many incor- rect pseudo-labels, leading to noisy training. We propose an uncertainty-awar pseudo-label selection (UPS) framework which improves pseudo labeling accuracy by drastically reducing the amount of noise encountered in the training process. Fur- thermore, UPS generalizes the pseudo-labeling process, allowing for the creation of negative gpeudo-labels, these negative pseudo-labels can used for multi-label classification well negative learning improve the single-label classification. We achieve strong performance when compared to recent SSL methods on the CIFAR-10 and CIFAR- 100 datasets. Also, we demonstrate the versatility of our method on the video dataset UCF-101 and the multi-label dataset Pascal VOC.
INTRODUCTION
The recent extraordinary success of deep learning methods can be mostly attributed to advancements in learning algorithms and the availability of large-scale labeled datasets. However, constructing large labeled datasets for supervised learning tends to be costly and is often infeasible. Several approaches have been proposed to overcome this dependency on huge labeled datasets; these include
semi-supervised learning (Berthelot al. 2019 Tarvainen & Valpola, 2017; Miyato et al. 2018 Lec
2013), self-supervised learning (Doersch et al. 2015 Noroozi & Favaro| 2016 Chen et al. 2020a), and few-shot learning (Finn et al.) 2017 [Snell et al. 2017| et al. 2016). Semi-supervised learning (SSL) is one of the most dominant approaches for solving this problem, where the goal is to leverage a large unlabeled dataset alongside small labeled dataset.
One common assumption for SSL is that decision boundaries should lie in low density regions (Chapelle& & Zien 2005). Consistency-reggllriiaaooo based methods achieve this by making the network outputs invariant to small input perturbations (Verma el al. 2019). However, one issue with these methods is that they often rely on a rich set of augmentations, like affine transformations, cutout VDees & Taylor 2017). and color jittering in images, which limits their capability for domains where these augmentations are less effective (e.g. videos and medical images). labelig based methods select unlabeled samples with high confidence as training targets (pseudo-labels, this can be viewed as form of entropy minimization, which reduces the density of data points at the decision boundaries (Grandvalet & Bengio 2005 Lee 2013). One advantage of pseudo-labeling over consistency regularization is that it does not inherently require augmentations and can be generally applied to most domains. However, recent consistency regularization approaches tend to outperform -abuldig on SSL benchmarks. This work defense of pseudo- labelinlb we demonstrate that seeuo-labbllng based methods perform on par with consistency regularization methods.

---

Published as a conference paper at ICLR 2021
Although the selection of unlabeled samples with high confidence predictions moves decision bound- aries to low density regions in pseudo-labeling based approaches, many of these selected predictions are incorrect due to the poor calibration of neural networks (Guo et al. 2017). Since, calibration measures the discrepancy between the confidence level of a network's 's individual predictions and its overall accuracy (Dawid 1982 Degroot & Fienberg] 1983); for poorly calibrated networks, an incorrect prediction might have high confidence. We argue that conventional pseudo-labeling based methods achieve poor results because poor network calibration produces incorrectly led samples, leading to noisy training and poor generalization. To remedy this, we empirically study the relationship between output prediction uncertainty and calibration. We find that selecting predictions with low uncertainty greatly reduces the effect of poor calibration, improving generalization.
pseudo-label selection (UPS) framework that leverages the prediction uncertainty to guide the pseudo-label selection procedure. We believe pseudo- labeling has been impactful due to its simplicity, generality, and ease of implementation; to this end, our proposed framework attempts to maintain these benefits, while addressing the issue of calibration to drastically improve PL performance. UPS does not require modality-speciffe augmentations and can leverage most uncertainty estimation methods in its selection process. Furthermore, the proposed framework allows for the creation of negative pseudo-labels (i.e. labels which specify the absence of specific classes). If network predicts the absence of class with high confidence and high certainty, then a negative label can be assigned to that sample. This generalization is beneficial for both single-label and multi-label learning. In the single-label case, networks can use these labels for negative learning (Kim et al. 2019 in the multi-label case, class presence is independent so both positive and negative labels are necessary for training.
Motivated by this, we propose an uncertainty-awant
Our key contributions include the following: (1) We introduce UPS, a novel uncertainty-aware pseudo-label selection framework which greatly reduces the effect of poor network calibration on the pseudo-labeling process, (2) While prior SSL methods focus on single-label classification, we generalize pseudo-labeling to create negative labels, allowing for negative learning and multi-label classification, and (3) Our comprehensive experimentation shows that the proposed method achieves strong performance on commonly used benchmark datasets CIFAR- 10 and CIFAR-100. In addition, we highlight our method's flexibility by outperforming previous -tate-of-the-art approaches on the video dataset, UCF-101, and the multi-label Pascal VOC dataset.
RELATED WORKS
Semi-supervised learning is a heavily studied problem. this work, we mostly focus on pseudo- labeling and consistency regularization based approaches currently, these are the dominant ap- proaches for SSL. Following (Berthelot et al. 2019), we refer to the other SSL approaches for interested readers which includes: "transductive" models (Gammerman et al. 1998 Joachims 1999 2003), graph- ing (Belkin & Niyogi 2002 Lasserre et 2006 Kingma al. [014]] Pu al. 2016). Furthermore, several recent self self-supervised approaches (Grill et al. 2020 Chen et al. 2020b Caron et al. 2020),
graph-based based methods (Zhu et al.] [0003 [Bengio et al. |0066 Liu et al. (2019), generative model-
have shown strong performance when applied the SSL task. For a general overview of SSL, we point to (Chapelle et al. 2010: Zhu 2005).
Pseudo-labeling The goal of pseudo-labeling Lee 2013; Shi et al., 2018) and self-training
(Yarowsky) | McClosky et al.| 2006) is to generate pscudo-labels for unlabeled samples with model trained on labeled data. In (Lee 2013), pseudo-labels are created from the predictions of a trained neural network. Pseudo- labels can also be assigned to unlabeled samples based on neighborhood graphs (Iscen al. 2019). Shi et al. (2018) extend the idea of pseudo-labeling by incorporating confidence scores for unlabeled samples based on the density of a local neighborhood. Inspired by noise correction work (Yi & Wu 2019), Wang & Wu (2020) attempt to update the pseudo-labels labels through an optimization framework. Recently, (Xie et al. 2019)) show self-training can be used improve the performance of benchmark supervised classification tasks. A concurrent
'The motivations for using negative learning (NL) in this work differs greatly from Kim et al. 2019). In this work, NL is used to incorporate more unlabeled samples into training and to generalize secudo-labeling to the multi-label classification setting, whereas Kim et al. (2019) use negative learning primarily to obtain good network initializations to lear with noisy labels. Further discussion about NI. can be found in Appendix K
2

---

Published as a conference paper at ICLR 2021
work (Haase- Schutz et al. 2020) partitions an unlabeled dataset and trains re-initialized networks on each partition.. They use previously trained networks to filter the labels used for training newer networks. However, most of their experiments involve learning from noisy data. Although previous pseudo-labeling based SSI. approaches are general and domain-agnostic, they tend to under-perform due to the generation of noisy pseudo-lahels; our approach greatly reduces noise by minimizing the effect of poor network calibration, allowing for competitive state-of-the-art results.
Consistency Regularization The main objective of consistency regularization methods obtain perturbatio//augmentttioo invariant output distribution. In (Sajjadi et al. 2016) random max max- moo-looiing. dropout, and random data augmentation are used as input perturbations. In (Miyato et al. 2018 perturbations are applied to the input that changes the output predictions maximally. Temppral ensembling (Laine & Aila 2017) forces the output class distribution for a sample to be consistent over multiple epochs. Tarvainen & Valpola (2017) reformulate temporal ensembling
teacher-student problem. Recently, the Mixup (Zhang et al. 2018) augmentation, has been used for consistency regularization in (Verma et al. 2019). Several SSL works combine ideas from both consistency regularization and labudl-labeling (Berthelot et al. 2019: 2020 Zhou et al., 2020). In (Berthelot al. 2019), pseudo-labels are generated by averaging different predictions of augmented versions of the same sample and the Mixup augmentation is used to train with these pseudo-labels. The authors in (Berthelot al.) 2020) extend this idea by dividing the set of augmentations into strong and weak augmentations. Also, (Zhou et al. 2020) incorporate a tmme-cossistnncy metric to effectively select consissent samples for consistency regularization. The success of recent consistency regularization methods can be attributed to domain-specite specific augmentations; our approach does not inherently rely on these augmentations, which allows for application to various modalities. Also, our pseudo-labeling method is orthogonal to consistency regularization techniques; therefore, these existing techniques can be applied alongside UPS further improve network performance.
Uncertainty and Calibration
Estimating network prediction uncertainty has been deeply studied topic (Graves 2011 Blundell et al. 2015 Louizos & Welling, 2016 Lakshminarayanaan et al.
2017 Malinin & Gales 2018| Maddox et al. 2019: Welling & Teh 2011). In the SSL domain, 2019 Xia et al. 2020 use uncertainty to improve consistency regularization learning for the segmentation of medical images. A concurrent work (Mukherjee & Awadallah 2020), selects pscudo-labels predicted by pretrained language model using uncertainty for downstream SSL task. One difference between our works is the selection of hard samples. Whereas Mukherjee et al. select a certain amount of hard samples (i.e. those which are not confident or certain) and learn from these using positive learning, we decide to use negative learning on these samples which reduces the amount of noise seen by the network. Zheng & Yang (2020) show strong performance on the domain adaptive semantic segmentation task by leveraging uncertainty. However, to the best of our knowledge, uncertainty has not been used to reduce the effect of poor network calibration in the pscudo-labeling process. In this work, instead of improving the calibration of the network (Guo et al., 2017 Xing al. 2020). we present general framework which can leverage most uncertainty estimation methods to select a better calibrated subset of pseudo-labels.
YYu
et et
3 PROPOSED METHOD
3.1 PSEUDO-LABELING FOR SEMI-SUPERVISED LEARNING
i=1 be a labeled dataset with N₂ samples, where a((() is the S 10 is the corresponding label with C' class categories (note that multiple elements in ( can be non-zero in multi-label datasets). For a sample i, yo denotes that class e is present in the corresponding input and و 0 represent the class's 'ss absence. Let Dr {z j be an unlabeled dataset with No samples, which does not contain labels
Notation Let Dr = (x)(), ((0(t)))() () No.
input and (0) = [1 ..... 3o
(1)
corresponding its input samples. For the unlabeled samples, pscudo-labels f(() are generated. Pseudo-labeling based SSL approaches involve learning a parameterized model fe on the dataset
ò (x(0), (())) ) No.
with ())) = ( for the Nr. labeled samples.
3

---

Published as a conference paper at ICLR 2021
Generalizing Pseudo-label Generation There several approaches to create the pseudo-labels g(), which have been described in Section We adopt the approach where hard pseudo-labels are obtained directly from network predictions. Let p()) be the probability outputs of trained network on the sample _0(), such that ped represents the probability of class being present in the sample. Using these output probabilities, the pseudo-label can be generated for 20)) as:
=
1
PC
(1)
where 7 E (0,1) is a threshold used to produce hard labels. Note that conventional single-label pseudo-labeling can be derived from equation when 7y max. Pe y 0.5 would lead to binary pseudo-labels, in which multiple classes can be present one sample.
(6) For the multi-label case,
3.2 PSEUDO-LABEL. SELECTION
Although pscudo-labelingg is versatile and modality-agnoosee it achieves relatively poor performance when compared to recent SSL methods. This is due to the large number of incorrectly pseudo-labeles samples used during training. Therefore, we aim reducing the noise present in training to improve the overall performance. This can be accomplished by intelligently selecting a subset of pseudo- labels which are less noisy; since networks output confidence probabilities for class presence (or class absence), we select those pseudo-labels corresponding with the high-confidence predictions.
Let g = 9 Ic
S 10 be binary vector representing the selected pseudo-labels in (+) is selected and ge (x)
sample i, where ge when ye obtained as follows:
(i) 0 when ye is not selected. This vector is
= 1 p Tp +1 <
(2)
where Tp and Ta are the confidence thresholds for positive and negative labels (here, Tp Tn)) If the probability score is sufficiently high (p.") Tp) then the positive label is selected; conversely,
network is sufficiently confident of a class's s absence (po (5) S Tn)), in which case the negative label is selected.
The parameterized model fo trained on the selected subset of pseudo-labels For single-label classification, cross-entropy loss is calculated on samples with selected positive pseudo-labels. If no positive label is selected, then negative learning is performed, using negative cross-entropy loss:
LSCE
Σ
9e ( (1 (1-1)))
log - o..t)
(3)
where (1)) = Ec Jc is the number of selected pseudo-labels for sample i. Here, ((()) = so (z)(1) is the probability output for the model fo- For multi-label classification, a modified binary eross-entropy loss is utilized:
(6)
LBCE 5( = 5(8)
UC
log
((1) De
(1 5 log
(4)
In both cases. the selection of high confidence pseudo-labels removes noise during training, allowing for improved performance when compared to traditional pseudo-labeling.
3.3 UNCERTAINTY-AWARY PSEUDO-LABEI SELECTION
Although confidence-bret basee selection reduces pseudo-labed error rates, the poor calibration of neural networks renders this solution insufficient in poorly calibrated networks, incorrect predictions can have high confidence scores. Since calibration can be interpreted as a notion of a network's s overall prediction uncertainty (Lakshminarayanan et al. 2017), the question then arises: Is there relationship between calibration and individual prediction uncertainties? To answer this question. we empirically analyze the relationship between the Expected Calibration Error (ECE) score")
? Guo
dn-th description of the ECE score included in section G of the Appendix.
4

---

Published as a conference paper at ICLR 2021
Calbraton Brie 1822 Stone)
0.20
Number of Seleetad
10
0.10
Pseusdo
0.
Opnvertional PL ONe sddctiont
10000
Larworsonal One octien
0.00
62
Uncortainty (STD)
Use
0.5
Pooudo Laboling iterat
Labeling Earotion
(a)
(b)
Figure 1: (a) The relationship between prediction uncertainty and expected calibration error (ECE). In all datasets, as the uncertainty of the selected pscudo-labels decreases, the ECE of that selected subset decreases. (b) Comparison of pscudo-label selection accuracy between conventional labudo-aabelig (PL), confidence- based selection (Confidence PL.), and UPS. Comparison of the number of selected pseudo-labels between conventional pseudo-labeling (PI.), confidence-bart initially selects a smaller set of pscudo-lahels, by the final pseudo-laheling iterations it incorporates the majority of pseudo-label labels in training. while maintaining higher label-la accuracy (as seen in (b)). Figures (b) and (c) are generated from the CIFAR- 10 dataset with 1000 labels.
selection (Confidence PL.) and UPS. Although UPS
al. 2017) and output prediction uncertainties. Figure la illustrates the direct relationship between the ECE score and output prediction uncertainty; when pseudo-labels with more certain predictions selected, the calibration error is greatly reduced for that subset of pseudo-labels. Therefore, for that subset of labels, a high-confidence prediction is more likely to lead to a correct pseudo-label.
From this observation, effects of poor calibration. Thus, we propose an uncertainty-awan pseudo-label selection process: by utilizing both the confidence and uncertainty of network prediction, a more accurate subset of pseudo-labels are used in training. Equation 2
conclude that prediction uncertainties can be leveraged negate the
now becomes,
gedh
PE
S Ki
Pe
Tp
+
PC
S
Kin
Pe
S Tp
(5)
where up)) is the uncertainty of a prediction p. and Kip and An are the uncertainty thresholds. This additional term, involving u()). ensures the network prediction is sufficiently certain to be selected. Figure Tb shows that this uncertainty-awe aware selection process greatly increases pseudo-label accuracy when compared to both traditional pseudo-labeling and confidence-basse selection.
3.4 LEARNING WITH UPS
First, neural network Se.o is trained on the small labeled set, Dr.- Once trained, the network generates predictions for all unlabeled data in Dv: pscudo-labels are created from these predictions following [quation[] A subset of the pscudo-labels is selected using UPS (equation 5). Next, another network fo.1, is trained using the selected pseudo-labelh as well as the labeled set. This process is continued iteratively until convergence is observed in terms of the number of selected pseudo-labels. Figure Ic illustratee that most pseudo-labels are selected by the end of the training. To limit error propagation in our iterative training and pseudo-labeling process, unlabeled samples and reinitialize the neural network after each pseudo-labeling step. The complete training procedure is described in Algorithm in Section of the Appendix.
generate new labels for all
EXPERIMENTAL EVALUATION
Datasets To show the versatility UPS we conduct experiments on four diverse datasets: CIFAR- 10. 100 (Krizhevsky et al.) 2009). Pascal VOC2007 (Everingham et al.), and UCF-101 (Soomro et al. 2012). CIFAR-IO and CIFAR- 100 are standard benchmark datasets, with 10 and 100 class categories respectively; both contain 60, 000, 32 x 32 images, split into 50, 000 training images and 10, 000 test images. Pascal VOC2007 a multi-label dataset with 5, 011 training images and 4, 952 test images. It consists of 20 classes and each sample contains between and 6 class categories. We also evaluate our method on the video dataset UCF-101, which contains 101 action classes. We the standard train/test split with 9, 537 videos for training and 3, 783 videos for testing.
5

---

Published as a conference paper at ICLR 2021
Table I: Error rate (%) on the CIFAR- 10 and CIFAR- 100 test set. Methods with f are pseudo-labeling based, whereas others are consistency regularization methods.
CIFAR-II
CIFAR-10
Method
1000 labels
4000 labels
4000 labels
10000 labels
Deepl.Pi
22.02 + 0.88
12.69 + 0.29
16.20 +: 0.76
38.43 1.88
TSSDL
21.13 + 1.17
10.90 + 0.23
MT
19.01 + 0.51
11.41 + 0.25
15.36 + 0.49
36.08 + 0.51
MT + DeepLP
16.93 + 0.70
10.61 + 0.28
43.73 + 0.20
35.92 0.47
ICT
15.48 + 0.78
7.29 + 0.02
DualStudent
14.17 + 0.38
8.89 + 0.09
32.77 + 0.24
R2-D2
32.87 = 0.51
MixMatch
6.84
UPS
8.18 + 0.15
6.39 + 0.02
40.77 + 0.10
32.00 + 0.49
4.1 IMPLEMENTATION DETAILS
For CIFAR-10 and CIFAR- 100 experiments, we use the CNN-13 architecture that is commonly used benchmark SSL methods (Oliver et al. 2018 Sa Luo al. 2018). For the Pascal VOC2007 experiments, we use ResNet-50 (He et al. 201 6) with an input resolution of 224 x 224. Finally, for UCF-101, we follow the experimental setup of Jing al. 2020) by using 3D ResNet-18 (Hara 2018) with a resolution of 112 x 112 112 and 16 frames. For all experiments, we set dropout et al.
rate of 0.3. We use SGD optimizer with an initial learning rate of 0.03 and cosine annealing (Loshchilov & Hutter (017) for learning rate decay. We set the confidence thresholds Tp 0.7 and Ty 0.05 for all experiments, except on the Pascal VOC dataset, where Tp = 0.5 as it a multi-label dataset and strict confidence threshold significantly reduces the number of positive pseudo-labels for difficult classes. Furthermore, for the uncertainty thresholds we use Kp = 0.05 and K₂ = 0.005
for all experiments. The threshold, 7 used to generate the pseudo-labels is set to 7 max. pe in single-label experiments and 7 0.5 in multi-label experiments.
Our framework utilize most uncertainty estimation method for selecting pseudo-labele (see section for experiments with different uncertainty estimation methods). Unless otherwise stated, we use MC-Dropout (Gal & Ghahramani (0166) obtain uncertainty measure by calculating the standard deviation of 10 stochastic forward passes. To further reduce the calibration error and to make the negative pseudo-label selection more robust, we perform temperature scaling to soften the output predictions we set T = (Guo et al. (017)). Moreover, networks trained on datasets with few classes tend to be biased towards easy classes in the initial pseudo-labeling iterations. This leads to the selection of more pseudo-labels for these classes, causing a large class imbalance. To address this issue, we balance the number of selected pseudo-labels for all classes; this constraint removed after 10 pseudo-labeling iterations in CIFAR-I0 and after I pseudo-labeling iteration in Pascal VOC. The effect of this balancing presented in section F of the Appendix.
4.2 RESULTS
CIFAR-10 and CIFAR-100 We conduct exper- iments CIFAR-10 for two different labeled set sizes (1000 and 4000 labels), as well as on CIFAR- 100 with labeled set sizes of 4000 and 10000 la- bels. For a fair comparison, we compare against methods which report results using the CNN-13
Table 2: Error rate (%) CIFAR-10 with differ- ent backbones Wide ResNet-28-2 (WRN) and Shake- Shake (S-S).
Labells
Method
Backbone
1000
4000
architecture: DeepL. (Iscen et al.] 2019), TSSDL (Shi et 2018 MT (Tarvainen & Valpola 2017).
MixMatch
WRN
7.75
6.24
Mix
S-S
4.95
ReMixMatch
WRN
5.73
5.14
MT + De ICT (Verma et 2019). DualStu-
TC-SSL
WRN
6.15
5.07
dent Ke et al., 2019), and MixMatch (Berthelot et al. 2019). The results reported in Table 1 are the mean and standard deviation from experiments across three different random splits. We achieve comparable results to the state-of of- art holistic
R2-D2
S-S
5.72
UPS
WRN
7.95
6.42
UPS
S-S
4.86
method MixMatch (Berthelot et al.) 2019) for CIFAR-10 10 with 4000 labels, with 0.45% improve- ment. Our CIFAR- 10 experiment with 1000 labeled samples outperforms previous methods which use the C3N-13 architecture. Also, we outperform previous methods in the CIFAR-10 100 experiments.

---

Published as a conference paper at ICLR 2021
Table 3: Accuracy (%) on the UCF-101 101 test set. Meth- ods with a use scores reported in (Jing et al. 2020).
Table 4: mAP scores on the Pascal VOC2007 test set.
Method
10% labeled
20% labeled
Method
20% labeled
50% laheled
Supervised
18.36 1 0.65
28.84 1.68
Supervised
33.5
45.6
PL
27.44 1 0.55
34.84 + 1.88
MT
36.3
45.8
MixMatch
29.57 0.78
37.02 + 0.97
PL*
37.0
47.5
MT
32.55 + 1.48
39.62 1.66
S4L*
37.7
47.9
UPS
34.22 1 0.79
40.34 0.08
UPS
39.4
50.2
We present additional results on CIFAR-IO with better backbone networks (Wide ResNet 28-2 Zagoruyko & Komodakis 2016) and Shake-Shake (Gastaldi 2017)) in Table and compare with methods: MixMatch, ReMix.atth (Berthelot et al., 2020), TC-SSL (Zhou et al., 2020), R2-D2 (Wang & Wu 2020). We find that UPS is not backbone dependent, and achieves further performance improvements when a stronger backbone is used.
UCF-101 For our UCF-101 experiments, we evaluate our method on using 20% and 50% of the training data as the labeled set. A comparison of our method and several SSL baselines is reported in [3] The results reported for the baselines PL (Lee) 2013), MT (Tarvainen & Valpola 2017). and S4I. (Zhai et al. 2 are obtained from [Ding et al. 2020). as it uses the same network architecture and similar training strategy. We do not compare directly with (Jing et al. 2020), since they utilize a pretrained 2D appearance classifier which makes it an unfair comparison. Even though none of the reported methods, including UPS, are developed specifically for the video domain, UPS outperforms all SSL methods. Interestingly, both aseudo-labeling (PL) and UPS achieve strong results, when compared to the massistency-reuularizzen
based method, MT (Tarvainen & Valpola 2017).
Pascal VOC2007 We conduct two experiments with 10% (500 samples) and 20% (1000 samples) of the train-val split as the labeled set. Since there is no prior work on multi-label semi-supervised classification, we re-implement three methods: srcudo-labeling (PL) (Lee 2013). Mean Teacher (MT) (Tarvainen & Valpola 2017). and MixMatch (Berthelot et al. 2019). For fair comparison, we use the same network architecture and similar training strategy for all baselines. Table shows UPS outperforming all methods with 1.67% and 0. 0.72% improvements when using 10% and 20% of the labeled data, respectively. One reason why UPS and MT performs strongly multi-lahel classification that neither approach has a single-label assumption; meanwhile, recent SSL methods like MixMatch and MeMixMatch are designed for single-label classification (e.g. temperature sharpening assumes a softmax probability distribution), which make them difficult to apply to multi- datasets 3
4.3 ABLATIONS
We present an ablation study to measure the con- tribution of the method's different components. We run experiments on CIFAR- 10 with 1000 and 4000 labeled samples. Table dissplays the results from our ablation study. Standard pseudo-labeling (UPS, no selection) slightly improves upon the supervised baseline; this is caused by the large number of incorrect pseudo-labels present train- ing. Through confidence-basee selection (UPS, no uncertainty-awan (UA) selection), many in- correct lseudo-label are ignored, leading to 6.1% and 3.71% error rate reductions for 1000 and 4000
Table 5: Ablation Study on CIFAR-10 dataset (Er- Rate (%)). UPS with uncertainty-aware (UA) selection, selects using only confidence- based criteria.
Method
1000 labels
4000 labels
Supervised
27.66
16.65
UPS, no selection
22.60
12.94
UPS, no UA
16.50
10.02
UPS, no UA (Cal.)
13.68
8.09
UPS, no NL UPS, full method
9.46
6.64
8.14
6.36
labels respectively. When MC-Dropout is used to improve network calibration (UPS, no UA (Cal.)), there is a slight improvement; however, this calibration is not adequate to produce sufficiently accurate pseudo-labels. Hence, by including uncertainty into the selection process, further improvement of 5.54% is observed for 1000 samples and 1. 1.73% for 4000 samples. We also find that negative learning (NL) is beneficial in both experimental setups. By incorporating more unlabeled samples in training (i.e. samples which do not have a confident and certain positive label), NL leads to 1.32% and 0.32% error rate reductions for 1000 and 4000 samples respectively.
Details on how MixMatch was adapted for this experiment be found in section H of the Appendix.
7

---

Published as a conference paper at ICLR 2021
4.4 ANALYSIS
Robustness to Hyperparameters Our frame- work introduces new threshold hyperparameters and K. Following (Oliver al., 2018b) we do not "over-tweak"" the hyperparameters we select thresholds based on a CIFAR-I0 10 validation set of 1000 sample:"] Although our experiments set Ri = 0.05, we find that UPS relatively robust to this hyperparameter. Figure 2 shows the test error produced when using various uncertainty thresh- olds. We find that using Kp < 0.1 leads to com- parable performance, and further increases of the threshold lead to predictable performance drops (as the threshold increases, more noisy pseudo- labels are selected leading to higher test error). Once the uncertainty threshold is selected, confidence thresholds Tp > 0.5 also lead to similar performance. UPS requires little hyperparameter tuning: although our thresholds were selected using the CIFAR-10 10 validation set, these same thresholds were used successfully on the other datasets (CIFAR-106) 100, UC1- and Pascal VOC) and splits.
201
% 50001
C.10 C.10
5-0.03
Error
20
00.05
15-
Exeudo-labeling Iteration
Figure 2: Robustness to uncertainty threshold. Thresh-
olds helow 0.1 lead to similar test error on CIFAR- (1000 labels), showing that UPS is not reliant single threshold.
UPS vs Confidence-based PL To investigate the performance difference between our approach and conventional pseudo-labeling. we analyse the subset of labels which UPS selects. Our analysis performed on CIFAR-10 with 1000 labeled samples. Both confidence based selection and UPS improve pseudo-labeling accuracy, while still utilizing the majority of samples by the end of the training, as seen Figures and Ic. Specifically, after 10 pseudo-labeling iterations, UPS selects about 13% of the positive pseudo-labels with an accuracy of 90.29%. Of the remaining 3371 samples which do not have a selected positive pseudo-label, over 88% of them are still used in training through negative learning: negative pscudo-labels are selected for 2988 samples, with an average of 6.57 negative pseudo-labels per sample. Even though confidence-basee selection improves upon conventional PL in terms of label accuracy (77.44% vs. 72.03% in the initial pseudo-labeling step), it is insufficient to achieve strong overall performance. UPS overcomes this problem by initially selecting smaller, more accurate subset (20217 positive labels with accuracy of 94.87%), and gradually increasing the number of selected pseudo-labels while maintaining high accuracy.
DISCUSSION
Uncertainty Estimation UPS is a general framework, it does not depend on a particular un- certainty measure. In our experiments, we use
Table 6: Comparison methods uncertainty esti- mation on CIFAR-IO (1000 labels) (Error Rate (%)
MC-Dropout (Gal & Ghahramani] (016)) ob-
Method
1000 labels
4000 labels
tain the uncertainty measure. Ideally, approximate Bayesian inference methods (Graves) 201T Blun-
MC-Dropout
8.14
6.36
MC- SC-SSpaialDroppuut
8.28
6.60
dell et al. 2015 Louizos & Welling (0166) can
MC- DroppBoook
9.76
7.50
DataAug
8.28
6.72
be used to obtain prediction uncertainties; how-
ever, Bayesian NNs computationally costly
and more difficult to implement than non-Bayesian NNs. Instead, methods like (Wan et af. 2013
al. 2017 Tompson et al. 2015 Ghiasi et al., (0188) can be used without extensive network modification to obtain an uncertainty measure directly (or through Monte Carlo sampling) that can easily be incorporated into UPS. To this end, we evaluate UPS using three other uncertainty estimation methods using MC sampling with SpatialDropout (Tompson et al. 2013) and DropBlock (Ghiasi et al., 2018), as well random data augmentation (DataAug). The experimental settings are described in Section of of the Appendix. Without using uncertainty estimation method specific hyperparameters, we find that UPS achieves comparable results when using any of these methods, as seen in T
Lakshminarayanan
Addiiiional information hyperparameter selection can he found in section of the Appendix.
8

---

Published as a conference paper at ICLR 2021
Data Augmentation in SSL One major advantage of UPS recent state-of-ther of-thee-rt consisteney regularization based SSL methods is that does not inherently rely on Nomain-specific data aug- mentations. For different data modalities like video. text, and speech it is not always possible to obtain a rich set of augmentations. This is evident in Table 3 where both standard pseudo-labeling approach (Leel 2013) and UPS outperform MT (Tarvainen & Valpola (017) the video dataset, UCF-101. Recent state-of-the-ar SSL methods, like (Berthelot et al. 2020 [Sohn et al., 2020). divide the augmentation space into the sets of strong and weak augmentations, which possible for the image domain as it has many diverse augmentations. However, it is not straightforward to extend these methods other data modalities; for instance, in the video domain, the two dominant augmentations are spatial crop and temporal jittering, which are difficult to divide into strong and weak subcategories. Moreover, the Mixup (Zhang et al. 2018) data augmentation used several recent SSL. methods (Verma et al. 2019 Berthelot et al. 2019 2020); ; it achieves strong results on single-label data, but extensions to multi- label-labell classification have been ineffective (Wang et al. 2019).
Although using augmentations during training improves network performance (see section B in the Appendix), the existence of domain specific augmentations is not a prerequisite for UPS.
CONCLUSION
In this work, we propose UPS, an uncertainty-awan pseudo-labe selection framework that maintains the simplicity, generality. and ease of implementation of pseudo-labeling, while performing on par with consistency regularization based SSL. methods. Due to poor neural network calibration, conventional pseudo-labeling methods trained on large number of incorrect pseudo-labels result in noisy training; pscudo-label selection process utilizes prediction uncertainty to reduce this noise. This results in strong performance on multiple benchmark datasets. The unique properties of UPS that it can be applied to multiple data modalities and to both single-label and multi-label classification. We hope that in the future, the machine learning community will focus on developing general SSL algorithms which do not have inherent limitations like domain-specifie augmentations and single-label assumptions.
This research is based upon work supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA RD Contract No. D17PC00345. The views and conclusions contained herein those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Govermental purposes notwithstanding any copyright annotation thereon.
ACKNOWLEDGEMENT
REFERENCES
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps and spectral techniques for embedding and clustering. In Advances in neural information processing systems, pp. 585-591, 2002.
Yoshua Bengio, Olivier Delalleau, and Nicolas Le Roux. 11 label propagation and quadratic criterion. 2006.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-But Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp.
Curran Associates, Inc., 2019.
David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Supi-supe learning with distribution matching and augmentation anchoring. In International Conference on Learning Representations, 2020.
Charles Blundell, Julien Comebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In Proceedings of the 32nd International Conference on International Conference Machine Learning Volume 37, ICML. pp. 1613-1622. JMLR.org, 2015.

---

Published as a conference paper at ICLR 2021
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33, 2020.
Olivier Chapelle and Alexander Zien. Semi-supervised classification by low density separation. In AISTATS, volume 2005, pp. 57-64. Citeseer, 2005.
Olivier Chapelle, Bernhard Schlkopf, and Alexander Zien. Semi-Supervised Learning. The MIT Press, 1st edition, 2010. ISBN 0262514125.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in Neural Information Processing Systems, 33, 2020b.
Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V. Le. Randaugment: Practical data augmentation with no separate search. CoRR, htt/1199.137199 2019.
A. Dawid, The well-calibrated bayesian. Journal of the American Statistical Association, 77(379): 605-610, 1982. doi: 10. 10..0/016621499.111 1982.10477856
Morris Degroot and Stephen E. Fienberg. The comparison and evaluation of forecasters. 1983.
Terrance Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: /7ckiv:/700.000 /w/.//00000800 04552, 2017.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1422-1430, 2015.
M. Everingham, L. Van Gool, K. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pascal
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, pp. 1126-1135. JMLR. org, 2017.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1059, New York, New York, USA, 20-22 Jun 2016. PMLR.
A. Gammerman, V. Vovk, and V. Vapnik. Learning by transduction. In Proceedings of the Fourteenth Conference on Uncertainty Artificial Intelligence, UAI'98, pp. 148-155, San Francisco, CA. USA, 1998. Morgan Kaufmann Publishers Inc. ISBN 155860555X.
Xavier Gastaldi. Shake-shake regularization. arXiv preprint arXiv: 17050007750078555 2017.
Golnaz Ghiasi, Tsung- Yi Lin, and Quoc V Le. Dropblock: A regularization method for convolutional networks. In Bengio, H. Wallach, H. Larochelle, K. Grauman, Cesa-Biann Maa--Bianchi, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems 31, pp. 10727-10737. Curran Associates, Inc., 2018
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Advances in neural information processing systems, pp. 329-536, 2005.
Alex Graves. Practical variational inference for neural networks. In J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, Pereira, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 24, pp. 2358-2356. Curran Associates, Inc., 2011.
10

---

Published as a conference paper at ICLR 2021
-ean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a approach to self- learning. Advances in Neural Information Processing Systems, 33, 2020.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1321-1330, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR.
Christian Haase-Schutz, Rainer Stal, H. Hertlein, and B. Sick. Iterative label improvement: Robust training by confidence based filtering and dataset partitioning. arXiv: Learning, 2020.
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d cmns and imagenet? In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 6546-6555, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondfej Chum. Label propagation for deep semi- supervised learning. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5065-5074, 2019.
Longlong Jing, Toufiq Parag, Zhe Wu, Yingli Tian, and Hongcheng Wang. Videossl: Semi-supervised learning for video classification. arXiv preprint arXiv:2003.00197 2020.
Thorsten Joachims. Transductive inference for text classification using support vector machines. In Ieml, volume 99, pp. 200-2099.10 1999.
Thorsten Joachims. Transductive learning via spectral graph partitioning. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pp. 290-297, 2003.
Zhanghan Ke, Daoye Wang, Qiong Yan, Jimmy Ren, and Rynson W.H. Lau. Dual student: Breaking the limits of the teacher in semi-supervised learning. In The IEEE International Conference Computer Vision (ICCV), October 2019.
Youngdong Kim, Junho Yim, Juseung Yun, and Junmo Kim. Nlul: Negative learning for noisy labels. In Proceedings of the IEEE International Conference Computer Vision, pp. 101-110, 2019.
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in neurul information processing systems, pp. 3581-3589, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In ICLR (Poster). OpenReview.net, 2017.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances Neural Information Processing Curran Associates, Inc., 2017.
Systems 30, pp.
Julia A Lasserre, Christopher M Bishop, and Thomas P Minka. Principled hybrids of generative and discriminative models. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 06), volume 1, pp. 87-94. IEEE, 2006.
Dong-Hyun Lee. Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. 2013.
Bin Liu, Zhirong Wu, Han Hu, and Stephen Lin. Deep metric transfer for label propagation with limited annotated data. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 0-0, 2019.
11

---

Published as a conference paper at ICLR 2021
1. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations (ICLR) 2017 Conference Track, April 2017.
Christos Louizos and Max Welling. Structured and efficient variational deep learning with matrix gaussian posteriors. Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Rexearch, pp. 1708-1716, New York, New York, USA, 20-22 Jun 2016. PMLR.
Yucen Luo, Jun Zhu, Mengxi Li, Yong Ren, and Bo Zhang. Smooth neighbors on teacher graphs for semi-supervised learning. CVPR, 2018.
Wesley Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems. pp. 13164, 2019.
Andrey Malinin and Mark Gales, Predictive uncertainty estimation via prior networks. In Advances in Neural Information Processing Systems, pp. 7047-7058, 2018.
David McClosky, Eugene Charniak, and Mark Johnson. Effective self-training for parsing. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, New York City, USA, June 2006. Association for Computational Linguistics.
Takeru Miyato, Shin ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41:1979-1993, 2018.
Subhabrata Mukherjee and Ahmed Awadallah. Uncertainty-aware self-training for few-shot text classification. Advances in Neural Information Processing Systems, 33, 2020.
Mchdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pp. 69-84. Springer, 2016.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Besa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 3/, pp. 0235-3246
Curran Associates, Inc., 2018a.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In Advances in Neural Information Processing Systems, pp. 3235-3246, 2018b.
Yunchen Pu, Zhe Gan, Ricardo Henao, Xin Yuan, Chunyuan Li, Andrew Stevens, and Lawrence Carin. Variational autoencoder for deep learning of images, labels and captions. In Advances in neural information processing systems, pp. 2352-2360, 2016.
Mchdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances Neural Information Processing Systems 29, pp. 1163-1171 Curran Associates, Inc., 2016.
Weiwei Shi, Yihong Gong, Chris Ding, Zhiheng MaXiaoyu Tao, and Nanning Zheng. Transductive semi-supervised deep learning using min-max features. In The European Conference on Computer Vision (ECCV), September 2018.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pp. 4077-4087, 2017.
Kihyuk Sohn, David Berthelot. Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying suiisuuer learning with consistency and confidence. arXiv preprint arXiv:2001.07685. 2020.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset 101 human actions classes from videos in the wild. arXiv preprint arXiv: /2/..0////0000 2012.
12

---

Published as a conference paper at ICLR 2021
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- Bright-averaged consisteney targets improve semi-supervised deep learning results. In Guyon, U. V. Luxburg. S. Bengio. H. Wallach, R. Fergus. S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp- 1 1194-1204. Curran Associates, Inc., 2017.
Sunil Thulasidasan, Gopinath Chennupati, Jeff Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. In Advances in Neural Information Processing Systems, pp. 13888-13899, 2019.
Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann LeCun, and Christoph Bregler. Efficient object localization using convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 648 656, 2015.
Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for semi-supervised learning. In IJCAI, 2019.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference on machine learning, pp. 1058-1066, 2013.
Guo-Hua Wang and Jianxin Wu. Repetitive reprediction deep decipher for semi-supervised learning. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
Qian Wang. Ning Jia, and Toby P Breckon. A baseline for multi-label image classification using an ensemble of deep convolutional neural networks. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 644-648. IEEE, 2019.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-II), pp. 681-688, 2011.
Yingda Xia, Fengze Liu, Dong Yang, Jinzheng Cai, Lequan Yu, Zhuotun Zhu, Daguang Xu, Alan Yuille, and Holger Roth. 3d semi-supervised learning with uncertainty-aware multi-view co- training. In The IEEE Winter Conference on Applications of Computer Vision, pp. 3646-3655, 2020.
Qizhe Xie, Eduard Hovy, Minh-Thang Luong, and Quoe V Le. Self-training with noisy student improves imagenet classification. arXiv preprint urXiv:19/11 19t11.421..22 2019.
Chen Xing. Sercan Arik, Zizhao Zhang, and Tomas Pfister. Distance-basse learning from errors for confidence calibration. In International Conference on Learning Representations, 2020.
David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd Annual Meeting of the Association for Computational Linguistics, Cambridge, Massachusetts, USA, June 1995. Association for Computational Linguistics.
Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7017-7025, 2019.
Lequan Yu, Shujun Wang, Xiaomeng Li, Chi-Wing Fu, and Pheng-Ann Heng. Uncertainty-aware self- ensembling model for semi- sumeriupere 3d left atrium segmentation. International Conference on Medical Image Computing and Computer-Assister Intervention, pp. 605-613 613. Springer, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith (eds.), Proceedings the British Machine Vision Conference (BMVC), pp. 87.1-87.12 12. BMVA Press, September 2016. ISBN 1-901725-59-6 doi: 10.5244/C. 30.87.
13

---

Published as a conference paper at ICLR 2021
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S41:
Self superviper semi- supervised learning. In Proceedings of the IEEE international conference on computer vision, pp. 1476-1485, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Zhedong Zheng and Yi Yang. Rectifying pseudo label learning via uncertainty estimation for domain adaptive semantic segmentation. arXiv preprint arXiv:2003.03777 2020.
Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Time-consistent self-supervision for semi-supervised learning. 2020.
Xiaojin Zhu, Zoubin Ghahramani, and John D Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In Proceedings of the 20th International conference on Machine learning (ICML-03), pp. 912-919, 2003.
Xiaojin Jenry Zhu. Semi-supervised learning literature survey. Technical report, University of Wisconsin-Madisee Department Computer Sciences, 2005.
A APPENDIX
In the appendix include the following: training procedure of UPS (section B). additional experi- ments on Pascal VOC, UCF-101, and CIFAR- (section( implementation details of the uncertainty estimation (section D) experiments with additional data augmentation (section) H analysis of the
effects of class balancing in pseudo-label selection (section F). additional details on ECE (section G
details pertaining the of MixMatch in multi-label classification (section H). additional details
about hyperparameter selction (section [ and some qualitative results (section
UPS TRAINING PROCEDURE B
The training procedure for our proposed UPS framework described in Algorithm
Algorithm 1 The proposed method takes set of labeled data, Dr. and set of unlabeled data, Du and returns trained model, fe. using samples from both D2 and Du
1: Train a network, fe.o. using the samples from Dr.
2: for == I.Maxxteessson do
b Repeats until convergence b Equation Eqqutiin
3: Pseudo-label Du using fa.i-
4:
Dselected + Select pseudo-labels using UPS
5:
Đ De Dr. U Drelected
6:
Initialize new network fo.:
7:
Train fa.; using the samples from D. Using cross-entropy loss losses in Equations 4-5
8: So + fo.i
9: return fo
C ADDITIONAL RESULTS
C. FULLY SUPERVISED CLASSIFICATION SCORES
For Pascal VOC2007 and UCF-101 datasets the fully supervised classification scores are presented T these scores can be viewed as an upper-bound for our method, as our method trains in supervised manner the pseudo-labeled samples and percentage of the labeled data. These results should not be confused with the "Supervised" baseline in Tables 4 [nddd of our paper, which involve training with only the listed percentage of data. With UPS obtain mAP of 40.34 and 34.22 when 20% and 10% the Pascal VOC2007 training-valicn validation data available respectively. Using all labeled data, a fully supervised network achieves 52.62% mAP with the ResNet-50 network.
14

---

Published as a conference paper at ICLR 2021
For UCF-101, fully supervised 3D ResNet- 18 achieves 50.4% accuracy. Even with only 50% of the labeled data, UPS is able to achieve a similar accuracy (50.2%).
Table 7: Fully supervised classification score Pascal VOC2007 and UCF-101 dataset. The metries used for each dataset are mAP and accuracy, respectively.
Dataset
UPS, 10% data
UPS, 20% data 40.34 0.08
UPS, 50% data
Supervised, all data
Pascal VOC2007 UCF-101
34.22 + 0.79
52.62
39.4
50.2
50.4
C.2 CIFAR-10 RESULTS
We conduct experiments the CIFAR-10 dataset with 250 and 500 labeled examples. The results are presented in table We achieve similar results (within 1.5%) to MixMatch (Berthelot et al. 2019) when using CNN-13 on 250 labels. Most other pseudo-labeling based remi-supervised learning methods do not present results using 250 and 500 labeled samples.
Table 8: Error rates CIFAR-IO dataset with very few training samples.
Method
250 labels
500 labels
UPS
15.90 + 0.61
10.64 + 0.18
D UNCERTAINTY ESTIMATION: IMPLEMENTATION DETAILS
The proposed UPS framework can leverage most uncertainty estimation methods select better calibrated subset of pseudo-labels. In 6f of the main text we show that UPS performs comparably when uncertainty is estimated using (Tompson al. 2013 Ghiasi al. 2018) through Monte Carlo sampling during inference time. For MC-SpatialDropour we set the dropout rate to 0.3 and performed 10 stochastic forward passes to obtain an uncertainty measure from the standard deviation of the output probabilities. We follow the same MC sampling strategy with MC-DropBlock, Following (Ghiasi et al., 2018)) we set the keep probability to 0.9 for the experiment with MC-DropBlock. For estimating uncertainty with random data augmentation perform 10 forward passes while performing random crop and random horizontal flip.
E DATA AUGMENTATION
Since is common practice to use data augmentation on image datasets, we use RandAugment (Cubuk et al. 2019) for experiments CIFAR-10, CIFAR-100, and Pascal VOC2007 datasets. For UCF-101 dataset experiments, we use random crop and temporal jittering following (Jing et al.) 2020). The Mixup augmentation (Zhang et al.) 2018) has become widely used both supervised and semi- supervised classification. We test how the addition of this powerful augmentation technique could improve UPS single-label classification. As the extension of Mixup negative learning is non- trivial, we do not include negative learning in this experiment. For Mixup, we set the hyper-pane parameter 0.50. Since the output prediction with mixup augmentation is better calibrated (Thulasidasan et al. 2019) we use relaxed thresholds for Tp (0.50) and Kp (0.10). The results are presented in table expected the improved augmentation leads an improvement: achieves 2.16% reduction error when compared to UPS without negative learning on the 1000 label experiment. Notably, UPS+Mixup even outperforms UPS with negative learning.
Since our method is not inherently reliant on specific data augmentations. we additional ex- periments on CIFAR-10 10, with no input augmentations. The results are shown in the table I0 Our method achieves error rate of 28. 14% and 14.98% for 1000 and 4000 labels respectively. This is respectable score that improves upon other SSL methods II model (Laine & Aila 2017) and Mean Teacher (Tarvainen & Valpola 2017) in the same experimental setting (i.e. no data augmentations).
15

---

Published as a conference paper at ICLR 2021
Table 9: The effect of using the Mixup augmentation with UPS CIFAR-10 dataset with 1000 labels.
Method
Error Rate (%)
UPS, without negative learning
9.46
UPS, with negative learning
8.14
MPS+ without negative learning
7.30
Table 10: Error rates the CIFAR- 10 test with no input augmentations used during training.
Method
1000 labels
4000 labels
IT model
32.18
17.08
MT
17.74 30.62
UPS
28.14
14.98
F EFFECT OF CLASS BALANCING
In our experiments, we observe that generating pseudo-labels with only limited number of labeled samples difficult especially for the datasets with a small number of classes CIFAR-10 10 and Pascal VOC2007. For these datasets, when the number of training samples is limited, the network tends to be biased towards easy classes which leads class imbalance during the pseudo-label selection; this is specifically true for the initial pseudo-labelie selected pseudo-labels for each class of CIFAR- 10 during the first eseudo-labelligg iteration. For 1000 labeled samples, there is an imbalance between the number of selected pseudo-labels for each class; the cat class has only 1065 pseudo-labels selected whereas the automobile class has 3734 selected pscudo-labels, leading to an imbalance ratio of 3.5. For the CIFAR- 10 experiment with 4000 labeled samples the imbalance ratio is not that severe but it's still 2.35. To address this issue we make the pseudo-label selection class balanced for CIFAR-10 10 for the first 10 heeudolabbling iterations. Even though Pascal VOC2007 itself training with limited data results in trained classifier biased towards easy classes. Therefore, for the first iteration of pseudo-label selection, we enforce class balancing for Pascal VOC2007 dataset, which results in improvement. The results with and without class balancing are presented in table 12 is evident that class balancing is more impactful when fewer labeled samples are used training.
iterations. Table 11 presents the number of
a class balanced dataset, it has only 20 object classes and
Table 11: Number of selected pseudo-label -lbells from each class of CIFAR- 10.
Class ID and Name
1000 labels
4000 labels
0 (airplane)
2707
3274
(automobile)
3734
3900
(bird)
1929
2377
3 (cat)
1065
1658
4 (deer)
2145
2898
(dog)
1924
2273
(frog)
3224
3468
(horse)
3266
3403
8 (ship)
3083
3538
(truck)
3214
3920
G EXPECTED CALIBRATION ERROR (ECE) COMPUTATION
In our work, we analyse the effect of prediction uncertainty and network calibration. A standard metric for measuring network calibration is Expected Calibration Error (ECE) (Guo et al. 201 Xing
et al. 2020 score,
ECF
max
Σ 1
arg max (f arg max T(()
(6)
D
1=1
16

---

Published as a conference paper at ICLR 2021
Table 12: Performance the CIFAR-10 10 and Pascal VOC2007 test sets.
CIFAR- 10 (aceuracy)
Pascal VOC2007 (mAP)
Method
1000 labels
4000 labels
10% labeled 20% labeled
UPS, with class halance
91.86
93.64 93.14
34.72
40.33
UPS, without class balance
88.77
31.88
40.06
where the confidence predictions on dataset D are partitioned into equally-spaced bins. are the samples present in a particular bin I, The discrepancy between the average confidence and average accuracy gives the calibration gap of each bin. The average over the calibration gap of all the bins results in ECE score. In our ECE score calculation we have set L = 15. Conventionally, ECE is calculated over an entire test set. our case, we select subset of unlabeled samples based on their prediction uncertainty and calculate the ECE on this subset. As shown in the main text, we find that as the prediction uncertainty decreases, the ECE score tends to decrease. This implies that neural networks tend to be more calibrated on samples for which it has lower uncertainty.
H MIXMATCH FOR MULTI-LABEL CLASSIFICATION
MixMatch (Berthelot et al. 2019) recent popular SSL method that performs well in the single- label case. We find that this performance does not transfer to the multi-label case. For implementing MixMatch for multi-label classification in the Pascal VOC2007 dataset mentioned in their paper. We set the value of a to be 0.75 and we use K = 2. Label sharpening for single-label predictions defined below, cannot be applied multi-label predictions, which assume class independence.
the default parameters
1/T P3
Sharpen(p, T);
L 1/T Σラ= Pp
Sharpening can be performed independently on each output by dividing the logits by temperature T. and applying sigmoid operation to obtain the probabilities p. In our experiments, we found that this did not lead significant change in results, so our main paper the MixMatch experiments on Pascal VOC2007, we report results without label sharpening.
I HYPERPARAMETER SELECTION
In this section, present how hyperparameters are selected in our experiments. As stated in the main text, our syper-parameter are selected based on a 1000 sample CIFAR-10 validation set. The distribution of pseudo-labels for this validation set can be found in Figures Baj and [3b] The distributions with respect to confidence and uncertainty are skewed toward and ( respectively.
Therefore, selecting confidence and uncertainty thresholds which encompass the majority of pseudo- labels, while maintaining relatively high accuracy would be sufficient. We find that similar pattern emerges on the full unlabeled set (Figures 3el and 3d).
On the CIFAR-10 validation set, we select the confidence threshold of 0.7 and uncertainty threshold of Kp 0.05 leading the selection of 531 labels with 92.28% accuracy. We find that once we select this uncertainty threshold, Kp, changes in the confidence threshold yields similar numbers of selected pseudo-labels with similar accuracy. Although, if we select a less strict uncertainty threshold, then changes in the confidence threshold have larger impacts. Since we did not want to over-tweak the confidence threshold from dataset to dataset, we maintained this stricter uncertainty threshold of 0.05 throughout our experiments. However, for any drastically different dataset, we can perform this analysis to obtain a new set thresholds.
Using fixed set of hyper-paramedt performance without dataset specific pprammee tuning: we achieve strong performance on CIFAR-100 and UCF-101 with thresholds obtained from the CIFAR-10 validation set. It is true that better set of hyper- parameters for particular dataset can always be found, which is also the case for existing SSL methods (e.g. loss weighting for unlabeled samples and a in the MixUp augmentation
in experiments demonstrates that UPS can give reasonable
ayper-parameters
17

---

Published as a conference paper at ICLR 2021
1002
# Selected Prcudo Labels
Correct Pseudo
Correct Fsoudb-lbells
Incorrect "seudo-lebe's Accuracy
9.
302
Incorrect Pse beedoolebells Accuracy
Poeudo
b021
Accurecy
nccurats
233
02
600
610
E70
6.0
000
600
600
100
END
6,00
EUS
630
6.00
0.00
Confidence
Uncertainty
(b)
19030
19333
Selected Pocudo Labels
Connect Psaudo-labels
Correct Pseudo-iebels
Pscudo lebels Accuracy
53330
Pseudd-labels Accuracy
23020
33020
23033
Accurolty
13333
13339
922
$033
0.5
0.2
6.1
022
0.5
0.35
675
0.7
0,30
0.20
0.325
0.75
0,355
0.65
0.00
Confidence
uncertainty IS1D
(c)
(d)
Figure 3: (a) The distrihution of correct and incorrect pseudo-labels with respect the confidence 1000 sample CIFAR-10 validation set. (h) The distribution of correct and incorrect pscudo-lahel, with respect to uncertainty on a 1000 sample CIFAR-10 validation set. (c) The distribution of correct and incorrect pscudo-lahels with respect to the confidence the set of unlabeled samples. (d) The distribution of correct and incorrect pseudo-labels with respect the uncertainty the of unlabeled samples. Both the full unlabeled set and the small validation have similar skewed confidence and uncertainty distributions.
are both saper-parameters which can be tuned for improved performance), but the robustness of our method (shown multiple datasets) and reasonable hyper-parameter selection strategy enables our method to be applicable to many different datasets.
J CALIBRATION AND THRESHOLD SELECTION
We show in our ablations (Table 5) that having a calibrated network with confidence based thresh- olding achieves better performance than without calibration, when the confidence threshold the same (r=0.7). However, we find that even with adjusted thresholds the calibrated network is unable achieve sufficient pseudo-labeling accuracy outperform the uncertainty- present the accuracy for the first set of selected pseudo-labels (CIFAR-100, 4000 labels) in Table
selection. We
13 Increasing the threshold for the calibrated network leads to increased accuracy (which to be expected), but even with this high threshold of 0.9, it is unable to achieve the 83% selected pseudo-label accuracy of UPS.
Table 13: Pscudo-Labeling accuracy on the CIFAR-100 (4000 labels)
Method
Tp 0.7
Tp 0.8
Tp
Conf--Based Selection
54.92
58.75
64.18
Conf.-Based Selection (Cal)
64.75
70.48
77.18
UPS
83.16
83.37
83.09
Based on the trend present this table, is feasible that there exists some confidence-based threshold for network (uncalibrated or calibrated) that could achieve strong pseudo-labeling performance. However, finding such a threshold would be difficult and would not reasonably transfer across datasets. moves distribution of confidences, but the
Calibrating the network may make the task simpler, use of UPS allows us to find robust set thresholds (both confidence and uncertainty) which can be applied across different label splits and datasets.
18

---

Published as a conference paper at ICLR 2021
K NEGATIVE LEARNING
There are several distinctions between the use of NI. in previous works (Kim et al. 2019) and its use in this work. First, the motivation of using negative labels in this work incorporate more unlabeled samples into the training procedure, and 2) to generalize pseudo-labeling for the multi-label classification setting. the other hand, Kim et (2019) use negative learning primarily obtain good network initializations to learn with noisy labels.
Furthermore, our negative labels are selected in an uncertainty-awart process (equation 5), whereas Kim et al. (2019) initially generates negative labels randomly (NL. step) to train network and then use that network to selectively generate negative labels using confidence scores (SeINL.). Their use of selective positive learning (SeIPL) also relies on manfidnnce-based positive pseudo-labe creation. In our work, we show that relying on confidence-based selection is insufficient, and our proposed uncertainty-ane selection is beneficial for the pseudo-labeling task. In general, our method is not built upon negative learning we achieve strong performance without NL (see Table 5). but best performance is achieved when the additional negatively pseudo-labeled samples used during training.
L QUALITATIVE RESULTS
In Figure we show some incorrect pseudo-labels obtained from the network trained with 1000 labeled samples from the CIFAR-10 dataset. As the confidence scores for all these images is greater than 0.9, no reasonable confidence based selection criteria would be able to filter out these incorrect predictions. However, UPS easily filters these incorrect pseudo-labels by leveraging the prediction uncertainties. This demonstrates the adverse effect of poor network calibration in pseudo-labeling, and the benefit of using uncertainty in the selection process.
19

---

Published as a conference paper at ICLR 2021
Airpinne
Automobile
DVF
Bird
8122
DV EXST
DEI CIR
MO 6U00
DVG DUNN
LINO
CIK CHH
Cot
CV CaM4
DIL CATE
Dog
0122
Frog
ON
DO CAM
Ship
Truck
Figure 4: CIFAR-10 samples with incorrect high confidence predictions, that filtered out by UPS when prediction uncertainty leveraged.
20


Reviews:
[{"review_id": "vh8hpbu5l-r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strength:\n\nThe paper notices the problem in pseudo-labeling methods of semi-supervised learning, which is the erroneous prediction of pseudo-labeling. Pseudo-labeling is an easy-to-implement method for semi-supervised learning does not require constraints needed by consistency regularization methods, so improving pseudo-labeling methods can promote the practical use of semi-supervised learning.\n\nThe paper is well-written and easy to follow. \n\nThe experimental results on datasets of different domains such as image, video show that the proposed method outperforms previous semi-supervised learning methods.\n\n\n\nConstraint:\n\nThe paper proposes a new method to predict pseudo-labels of unlabeled data by confident threshold. However, confidence threshold is a widely-used method to decide pseudo-labels. And the threshold of confidence is hard to decide since different backbone network and different datasets have different confidence levels.\n\nFurthermore, using both positive and negative labels for classification is also used in (Kim et al., 2019). The authors fails to discuss the difference of the usage of positive and negative labels between their method and (Kim et al., 2019). The authors also need to discuss what is the special challenge of using positive and negative labels in pseudo-labeling for semi-supervised learning.\n\nThe paper argues that the calibration error is greatly reduced with more certain predictions. However, the paper only empirically shows the relation but fails to demonstrate the claims or give intuition on the relation. Also, confidence itself is also an uncertainty measure, but the authors do not use the confidence for uncertainty but use a new uncertainty measure. Could the authors explain what uncertainty measurement do they use and why using the new measurement instead of the confidence?\n\nThe paper uses a fixed set of threshold hyper-parameters for all the experiments. However, the confidence-level for different datasets should be different. For example, the confidence for a real difficult dataset should be much lower than an easy dataset. The authors need to show that why using a set of hyper-parameters is enough and how to select the hyper-parameters. For example, showing the relation between the accuracy of pseudo-labels and the confidence.", "extra_scores": {}, "extra_text": {}}, {"review_id": "sDwl8uA4iJp", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is in defense of simple semi-supervised learning (SSL) with pseudo-labeling (PL): authors demonstrate with experiments on 4 vision datasets (CIFAR-10, CIFAR-100, Pascal VOC and UCF-101) that pseudo-labeling can perform on par with consistency regularization methods. Authors argue that PL doesn't work well because of poor network calibration: because of that the high confident predictions are wrong leading to noisy training and poor generalization. The main contribution of the paper is the usage of prediction uncertainty selection in addition to the confidence-based selection which provides high accuracy of PL used in the further training. Besides this PL is generalized to create negative labels: with this authors perform and show effectiveness of negative learning and multi-label classification. Proposed approach performs in the same ballpark as state-of-the-art methods on CIFAR-10 and CIFAR-100, while it achieves the new state-of-the-art results on video dataset and multi-label task. It is worth to notice that proposed approach is independent from the domain while consistency regularization methods extensively are based on the specific augmentation techniques for the vision/datasets.\n\nPros:\n- Cool idea on predictions uncertainty based selection for pseudo-labeling, no any dependence on domain-specific augmentations for the method\n- Analysis of correlation between model calibration and prediction uncertainty\n- Analysis of UPS compared to the conventional PL and confidence-based PL\n- Ablation study on each component of the method, and dependence on the network architecture\n- Well-designed (fair comparison) extensive experiments and comparisons on 4 dataset for multiclass and multi-label classification with better performance than other methods\n\nCons:\n- Absent of large-scale experiments with ImageNet\n\nComments:\n- typo page 3 \"would lead to binary psuedo-labels\" -> \"would lead to binary pseudo-labels\"\n- \"For the multi-label case, $\\gamma = 0.5$ would lead to binary pseudo-labels, in which multiple classes can be present in one sample.\" - this sentence is not clear. If $\\gamma = 0.5$ it could be only one or two classes presented in the pseudo-label vector.\n- Eq. (2), it is obvious but still please specify that $\\tau_p >= \\tau_n$\n- Figure 1 (b) and (c) - on which data is this analysis done?\n- Do authors use the same network $f_{\\theta, k}$ on each PL iteration $k$ and just randomly reinitialize it?\n- What is $\\gamma$ value in experiments?\n- For Table 1 would be good to have a clarification on baselines: which are consistency regularization based, which are PL. \n- typo in footnote 2 on page 7: add dot at the end of sentence.\n- typo page 7 \"experimental set-ups.\" -> \"experimental setups.\", \"labeled samples Both\" -> \"labeled samples. Both\"\n- Did authors try experiments on ImageNet too?\n- On which data is study in Fig.2 performed?\n- typo page 8 \"unique in that it can be easily\" -> \"unique in that: it can be easily\" (or any punctuation here)\n- For ECE computation is percentile binning used?\n- some possible relevant works: https://arxiv.org/pdf/2003.03773.pdf, https://arxiv.org/abs/2006.07733, simCLR v2, https://arxiv.org/abs/2006.09882\n\nIt is very well written paper with extensive experiments and ablations (except large-scale experiment), which prove the method efficiency and generalization. Hope, this will push the study of simple SSL approach, pseudo-labeling, with the new competitive results not only in vision but in other domains too.", "extra_scores": {}, "extra_text": {}}, {"review_id": "pQwfLiBsf-U", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\nThis paper proposes uncertainty aware pseudo-labelling for semi-supervised learning, extending previously known methods by negative labels.\n\n# Score justification\nWell written paper and with extensive experiments with some points of improvement. The method should be positioned against others using confidence filtering and the role of calibration needs to be studied further.\n\n\n# Strong and weak points\n## Pros\nExtensive experiments on different datasets and domains.\nBroadly applicable method, that aims to improve conventional pseudo-labelling. Method is independent of uncertainty estimate and data augmentation.\nCombining uncertainty estimates and confidence estimates, as well as adding negative learning are interesting approaches.\n\n## Cons\nConfidence filtering for pseudo-labels has already been suggested, see e.g. suggestions given in detailed comments.\n\nI am unfortunately not convinced by the role of calibration, details given below. The authors could enhance the ablations studies with and without calibration and -importantly- adjusted thresholds, to clarify that point.\n\n\n\n# Questions to the authors\nThe authors should consider e.g. https://arxiv.org/pdf/2002.02705 and https://www.microsoft.com/en-us/research/uploads/prod/2020/06/uncertainty_self_training_neurips_2020.pdf and position their work against those.\n\n\"Learning with UPS\" have you treated pseudo-labels and original labels equally when training $f_{\\theta,1}$, if so, how about fine-tuning on the original labels?\n\nPlease elaborate on the effect of different thresholds and how they were chosen. Especially with regards to calibration. \n\n# Detailed comments\nThe authors should consider e.g. https://arxiv.org/pdf/2002.02705 and https://www.microsoft.com/en-us/research/uploads/prod/2020/06/uncertainty_self_training_neurips_2020.pdf\n\nAugmentations for other domains are only not effective, if they are not suitable for that domain. Augmentations should use domain specific invariances. From the paper it becomes clear, you did use data augmentation, not sure why you argue so strongly against it.\n\nI am not sure if calibration plays a role here. Calibration only moves the distribution in shape, by setting a confidence threshold $\\tau$ the threshold would be changed, but a suitable threshold could be found before and after calibration.\n\nSection 3.1 small type: \"psuedo\" --> \"pseudo\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "4ySL35T4XT", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "As noted in (Guo et al., 2017), modern neural networks are often miscalibrated. Pseudo-labeling based Semi-Supervised learning schemes are predicated on high confidence predictions from these neural networks. This paper posits that this miscalibration may lead to inferior results in confidence-based pseudo-labeling approaches. By taking into account the uncertainty of models and only using pseudo labels from high-confidence instances with low uncertainty this work presents a model that significant improves on other PL strategies and is competitive with consistency-based regularization strategies that comprise the current state-of-the-art.\n\nNot only does this approach generate competitive results from with pseudo labeling strategy (which is compelling due to the history of PL approaches), but it is also less reliant on domain-specific augmentations that current consistency-based regularization approaches rely on. This is important for extending approach beyond domains where specific types of augmentation have been extensively studied. The results on the video domain offer some evidence of the usefulness in under-explored domains.\n\n\nFor the CIFAR (10/100) results presented it is unclear what data augmentations where used to produce the UPS results and how important these augmentations are to presented performance. As a key claim in this paper is de-emphasis of domain-specific augmentations would be beneficial to see a highlighted result strengthening this claim. \n\nAlso, it seems the model is re-initialized and trained to convergence after each round of pseudo labeling. This would extremely compute intensive for large-scale semi-supervised learning tasks, which is a motivating use-case for semi-supervised learning. Providing results and comparisons on the compute requirements for this approach and its comparisons with competitors would be beneficial. As the re-initialization, is likely a major component in compute resources would also be beneficial for community to understand how essential this is to the improved performance numbers presented in the paper.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is written in defense of pseudo-labeling. The authors `aim at demonstrating that pseudo-labeling based methods can perform on par with consistency regularization methods which have been show to achieve strong performance.

The paper is well-written and easy-to-follow. The reviewers are generally positive about the contribution. It has to be underline, however, that pseudo-labeling is still a controversial approach with a very limited theoretical understanding. This paper does not provide any further understanding of it, but proposes several "heuristics", intuitively well-motivated, but justified only experimentally. Nevertheless, the results are promising and the paper is an important voice in the general discussion around learning with weak labels and semi-supervised learning, two crucial problems in many practical applications. Taking this into account I recommend to accept the paper as a poster. 

The reviewers have raised several problems that the authors have been exhaustively discussing in their rebuttals. The one remaining issue is the interaction between calibration and the threshold. This problem has to be clarified in the final version of the paper, as indeed calibration usually does not change the order.
Decision: Accept (Poster)