Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning
Authors: Eric Arazo
Diego Ortego
Paul Albert
Noel E. O'Connor
Kevin McGuinness
Venue:  (2020)
Keywords: Semi-supervised learning
pseudo-labeling
deep semi-supervised learning
confirmation bias
image classification

Abstract: Semi-supervised learning, i.e. jointly learning from labeled an unlabeled samples, is an active research topic due to its key role on relaxing human annotation constraints. In the context of image classification, recent advances to learn from unlabeled samples are mainly focused on consistency regularization methods that encourage invariant predictions for different  perturbations of unlabeled samples. We, conversely, propose to learn from unlabeled data by generating soft pseudo-labels using the network predictions. We show that a naive pseudo-labeling overfits to incorrect pseudo-labels due to the so-called confirmation bias and demonstrate that mixup augmentation and setting a minimum number of labeled samples per mini-batch are effective regularization techniques for reducing it. The proposed approach achieves state-of-the-art results in CIFAR-10/100 and Mini-ImageNet despite being much simpler than other state-of-the-art. These results demonstrate that pseudo-labeling can outperform consistency regularization methods, while the opposite was supposed in previous work. Code will be made available.

Under review as a conference paper at ICLR 2020
PSEUDO-LABELING AND CONFIRMATION BIAS IN DEEP SEMI-SUPERVISED LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Semi-supervised learning, i.e. jointly learning from labeled an unlabeled samples, is an active research topic due to its key role on relaxing human supervision. In the context of image classification, recent advances to learn from unlabeled samples are mainly focused on consistency regularization methods that encourage invariant predictions for different perturbations of unlabeled samples. We, conversely, propose to learn from unlabeled data by generating soft pseudo-labels using the network predictions. We show that naive pseudo-labeling overfits to incorrect pseudo-labels due to the so-called confirmation bias and demonstrate that mixup augmentation and setting minimum number of labeled samples per mini-bate matih are effective regularization techniques for reducing it. The proposed approach achieves state-of-the-art results in CIFAR-10/100, SVHN, and Mini-ImageNee despite being much simpler than other methods. These results demonstrate that pseudo-labeling alone can outperform consistency regularization methods, while the opposite was supposed in previous work. Code will be made available.
INTRODUCTION
Convolutional neural networks (CNNs) have become the dominant approach in computer vision Lin
al. 2017 Liu al. 2018 Kim al. 2018 Xie al. 2018). To best exploit them, vast amounts of labeled data are required. Obtaining such labels, however, is not trivial, and the research community
exploring alternatives alleviate this (Li et al. 2017 Oliver et al. 2018 Liu al., 2019).
Knowledge transfer via deep domain adaptation (Wang & Deng 2018) is a popular alternative that seeks to learn transferable representations from source to target domains by embedding domain adaptation in the learning pipeline. Other approaches focus exclusively on learning useful repre- sentations from scratch
target domain when annotation constraints are relaxed (Oliver al.. 2018 Arazo al.) 2019 Gidaris et al. 2018). Semi-supervised learning (SSL) (Oliver et al. 2018) focuses on scenarios with sparsely labeled data and extensive amounts of unlabeled data; learning with label noise (Arazo et al., 2019) seeks robust learning when labels obtained automatically and may not represent the image content; and self-supervised learning (Gidaris et al. 2018) uses data supervision to learn from unlabeled data in supervised manner. This paper focuses on SSL for image classification, a recently very active research area (Li et al. 2019).
SSL is transversal task for different domains including images Oliver et al.) 2018). audio (Zhang (et al. 2016), time series (González et al.) 2018), and text (Miyato et al.) 201 6). Recent approaches image classification primarily focus on exploiting the consistency in the predictions for the same sample under different perturbations (consistency regularization) (Sajiadi et al. 2016 Li et al., 2019). while other approaches directly generate labels for the unlabeled data to guide the learning process (pseudo-labeling (Lee [013] Iscen et al.) 2019). These two alternatives differ importantly in the mechanism they use to exploit unlabeled samples. Consistency regularization and pseudo-labeling approaches apply different strategies such as warm-up phase using labeled data (Tarvainen & Valpolal 2017; Iscen et al. 2019), uncertainty weighting (Shi al. 2018 Li et al. 2019). adversarial attacks Miyatto et al. 2018 Qiao et al. 2018), or graph-consistency (Luo et al.
2018 Iscen et al.)
2019). These strategies deal with confirmation bias (Tarvainen & Valpola| 2017 Li et al. 2019), ) also
known noise accumulation (Zhang al. 2016). This bias stems from using incorrect predictions on unlabeled data for training in subsequent epochs and, thereby increasing confidence in incorrect predictions and producing model that will tend to resist new changes.

---

Under review as a conference paper at ICLR 2020
This paper explores pseudo-labeling for semi-superrvsed deep learning from the network predictions and shows that, contrary to previous attempts on pseudo-labeling (Iscen et al. 2019 Oliver et al. 20188)
Shi et al.] 2018), simple modifications to prevent confirmation bias lead to state-of-the the-art art performance
without adding consistency regularization strategies. We adapt the approach proposed by [Tanaka et al. (2018) in the context of lahel noise and apply it exclusively on unlabeled samples. Experiments show that this naive pseudo-labeling is limited by confirmation bias as prediction errors are fit by the network. To deal with this issue, we propose to use mixup augmentation (Zhang et al. 2018 as an effective regularization that helps calibrate deep neural networks (Thulasidasan et al. 2019) and, therefore, alleviates confirmation bias. We find that mixup alone does not guarantee robustness against confirmation bias when reducing the amount of labeled samples or using certain network architectures (see Subsection 4.3), and show that, when properly introduced, dropout regularization (Srivastava et al.] (014) and data augmentation mitigates this issue. Our purely pseudo-labeling approach achieves oftth-of-the-rt results (see Subsection (.4) without requiring multiple networks (Tarvainen & Valpola 2017: Qiao et al. 2018 Li et al. 2019] Vermaa et all 2019). does it require over a thousand epochs of training to achieve peak performance in every dataset (Athiwaratkun et al. 2019 Berthelot et al.] 2019), nor needs many (ten) forward passes for each sample (Li et al.) (2019). Compared to other pseudo-labeling approaches, the proposed approach is simpler in that it does not require graph construction and diffusion (Iscen et al., 2019) or combination with consistency regularization methods (Shi et al. 2018), but still achieves state-of-thee of-thheart results.
2
RELATED WORK
This section reviews closely related SSL methods, i.c. those using deep learning with mini-bateh optimization over large image collections. Previous work on deep SSL differ in whether they use consistency regularization or pseudo-labeling to learn from the unlabeled set (Iscen et al., 2019), while they all share the use of a cross-entropy loss (or similar) on labeled data.
Consistency regularization Imposes that the same sample under different perturbations must produce the same output. This idea used in (Sajjadi et al. (2016) where they apply randomized data augmentation, dropout, and random max- max-pooling while forcing softmax predictions to be similar. A similar idea is applied in (Laine& & Aila 2017), which also extends the perturbation to different epochs, i.e. the current prediction for a sample has to be similar to an ensemble of predictions of the same sample in the past. Here the different perturbations come from networks at different states, dropout, and data augmentation. In (Tarvainen & Valpola| (0177). the temporal ensembling method is interpreted as a teacher-studee for the unlabeled data as a temporal ensemble, and a student that learns the generated targets by imposing the consistency regularization, Tarvainen & Valpola (2017) naturally re-define the problem iatrraaatoonnTTTaiieennnnn to deal with confirmation bias by separating the teacher and the student. The teacher is defined as a different network with similar architecture whose parameters are updated as an exponential moving average of the student network weights. This method is extended in (Li et al. 2019), where they apply an uncertainty weight over the unlabeled samples to learn from the unlabeled samples with low uncertainty (i.e. entropy of the predictions for each sample under random perturbations). Additionally, lly, MMyato et al. (2018) use virtual adversarial training to carefully introduce perturbations to data samples as adversarial noise and later impose consistency regularization on the predictions. More recently. Luo et al. (2018) propose to use a contrastive loss on the predictions as regularization that forces predictions be similar (different) when they are from the same (different) class. This method extends the consistency regularization previously considered only in ne-between the same data samples to in-between different samples. Their method can naturally be combined with (Tarvainen & Valpola 2017) or (Miyato et al. 2018) to boost their performance. Similarly, Verma et al. (2019) propose interpolation consistency training, method inspired by (Zhang et al. (2018) that encourage predictions at interpolated unlabeled samples to be consistent with the interpolated predictions of individual samples. Also, authors in (Berthelot et al.) 2019) apply consistency regularization by guessing low-entropy labels, generating lata-auummnted un unlabeled examples and mixing labeled and unlabeled examples using mixup (Zhang et al. (2018). Both (Verma et al. 2019) and (Berthelot et al. 2019) adopt (Tarvainen & Valpola 2017) to estimate the targets used in the consistency regularization.
problem where the network is both teacher that produces targets
Co-training QQiao et al., 2018) uses two (or more) networks trained simultaneously to agree on their predictions (consisteney regularization) and disagree their errors. Errors are defined as
2

---

Under review as a conference paper at ICLR 2020
different predictions when exposed to adversarial attacks, thus forcing different networks to learn complementary representations for the same samples. Recently, [Chen et al. (2018) measure the consistency between the current prediction and an additional prediction for the same sample given by an external memory module that keeps track of previous representations. They additionally introduce an uncertainty weighting of the consistency term to reduce the contribution of uncertain predictions. Consistency regularization methods such as (Laine& & Aila, [017 Tarvainen & Valpola [2017; Miyato al. 2018 have all been shown benefit from stochastic weight averaging method (Athiwaratkun et al. 2019), that averages network parameters at different training epochs to move the SGD solution on borders of flat loss regions to their center and improve generalization.
Pseudo-labeling Seeks the generation of labels or pseudo-labels for unlabeled samples to guide the learning process. An early attempt at lseudo-labeling proposed in (Lee) 2013) uses the network predictions as labels. However, they constrain the pseudo-labeling to fine-tuning stage, i.e. there is a pre-training or warm-up to initialize the network. A recent pseudo-labeling approach proposed in (Shi et al. 2018) uses the network class prediction as hard labels for the unlabeled samples. They also introduce an uncertainty weight for each sample loss, it being higher for samples that have distant k-nearest neighbors in the feature space. They further include a loss term to encourage intra-class compactness and inter-class separation, and a consistency term between samples with different perturbations. Improved results are reported in combination with (Tarvainen & Valpola 2017). Finally. a recently published work (Iscen et al., 2019) implements pseudo-labeling through graph-based label propagation. The method alternates between two steps: training from labeled and pseudo-labeled data and using the representations of the network to build a nearest neighbor graph where label propagation applied to refine hard pseudo-labels. They further add an uncertainty score for every sample (softmax prediction entropy based) and class (class population based) to deal, respectively, with the unequal confidence network predictions and class-imbalance.
PSEUDO-LABELING
We formulate SSL as learning a model (ar) from set of N training samples D. These samples are split into the unlabeled set D.,- = { N.N., and the labeled set Di = {(x,, y:) ) being yi E {0, 1 the one-hot encoding label for C' classes corresponding to I; and N = N; + Nu. In our case, he is a CNN and 9 represents the model parameters (weights and biases). As we seek to perform pseudo-labeling, we assume that pseudo-label ÿ is available for the N. unlabeled samples. We can then reformulate SSL as training using D { ( being ğ = y for the N; labeled samples.
The CNN parameters can be optimized using categorical cross-entropy:
(0) = Σ oo log (ho(z;)),
where ha(r)) are the softmax probabilities produced by the model and log(-) is applied element-wise. A key decision is how to generate the pseudo-labels ÿ for the N,, unlabeled samples. Previous approaches have used hard pseudo-labels (i.e. one-hot vectors) directly using the network output class (Lee] 2013) Shi et al. 2018) or the class estimated using label propagation on a nearest neighbor graph (Iscen et al. 2019). We adopt the former approach, but use soft pseudo-labed this outperforms hard labels, confirming the observations noted (Tanaka al. 2018) in the context of relabeling when learning with label noise. In particular, we store the softmax predictions hg (2;) of the network in every mini-batch of an epoch and use them to modify the soft pseudo-label ý for the Nr, unlabeled samples at the end of every epoch. We proceed as described from the second to the last training epoch, while in the first epoch we use the softmax predictions for the unlabeled samples from model trained in 10 epochs warm-up phase using the labeled data subset D...
as we have seen
We use the two regularizations applied in (Tanaka et al., 2018) to improve convergence. The first regularization deals with the difficulty of converging early training stages when the network's 's predictions are mostly incorrect and the CNN tends to predict the same class to minimize the loss. Assignment of all samples a single class is discouraged by adding:
RA =
Σ
Po log
Pe hee
(2)
3

---

Under review as a conference paper at ICLR 2020
where Pe is the prior probability distribution for class and hee denotes the mean softmax probability of the model for class с across all samples in the dataset. As in (Tanaka et al. 2018). we assume a uniform distribution P. 1/C for the prior probabilities (RA stands for all classes regularization) and approximate hhe using mini-batches. The second regularization is needed to concentrate the probability distribution of each soft pseudo-label on a single class, thus avoiding the local optima in which the network might get stuck due a weak guidance:
N ΣΣ c=1
RH = N
(z,) log (hG(2;)),
(3)
where (1xz))) denotes the e class value of the softmax output (z4) and again using
(i.e. N is replaced by the mini-batch size) to approximate this term. This second regularization is the average per-sample entropy (R stands for entropy regularization), a well-known regularization in SSL (Grandvalet & Bengio 2004). Finally, the total semi-supervised loss is:
(= AARA + Ag RH,
where AA and λи control the contribution of each regularization term. We stress that this pseudo- labeling approach adapted from Tanaka et al. (2018) is far from the state-of-the-art for SSL (see Subsection 4.2). 1.2, and are the mechanisms proposed in Subsection 3.1 which make pseudo-labeling a suitable alternative.
3.1 CONFIRMATION BIAS
Network predictions are, of course, sometimes incorrect. This situation is reinforced when incorrect predictions used labels for unlabeled samples, as it is the case in pseudo-labeling. Overfitting to incorrect pseudo-labels predicted by the network is known as confirmation bias. It is natural to think that reducing the confidence of the network on its predictions might alleviate this problem and improve generalization. Recently, mixup data augmentation (Zhang et al.) (2018) introduced a strong regularization technique that combines data augmentation with label smoothing, which makes it potentially useful to deal with this bias. Mixup trains on convex combinations of sample pairs (xp and xx)) and corresponding labels (уp and yd):
x 5x, + (1 (1- (1-d)ag
(5)
y 8yp + (1 8)Hn-
(6)
where E E (0, 1} is randomly sampled from a beta distribution Be (a, with a = 3 (e.g. a =1 uniformly selects d). This combination regularizes the network to favor linear behavior in-between training samples, reducing oscillations in regions far from them. Additionally, Eq. can be re- interpreted in the loss as l = 8lp
o) thus re-defining the loss l used in Eq. 4 as:
8 TT log (ho (x:))] (1 - 6) [V, log (ho(s:))]
As shown in (Thulasidasan et al. 2019), overconfidence in deep neural networks is a consequence of training hard labels and it the label smoothing effect from randomly combining yμ and yy during mixup training that reduces prediction confidence and improves model calibration. In the supervipe context with pseudo-labeling, using soft-labels and mixup reduces overfitting to model predictions, which is especially important for unlabeled samples whose predictions are used as soft-labels. Note that training with mixup generates softmax outputs hap ha(a) for mixed inputs x, thus requiring a second forward pass with the original images to compute unmixed predictions. Mixup data augmentation alone might not effectively deal with confirmation bias when few labeled examples are provided. For example, when training with 500 labeled samples in CIFAR- 10 and mini-batch size of 100, clean sample per batch seen, which especially problematic at early stages of training where little correct guidance is provided. We find that setting minimum number of labeled samples per mini-batch (as done in other works (Tarvainen & Valpola 2017, Chen et al. 2018 Berthelot et al. 2019 Iscen et al. 2019)) provides constant reinforcement with correct labels during training. reducing confirmation bias and helping to produce better pseudo-labes experimentally show that mixup. minimum number of samples per mini batch, and other techniques (dropout and data augmentation) reduce confirmation bias and make aeeudo-labeling an effective alternative to consistency regularization.
Subsections 4.2 and 4.3
4

---

Under review as a conference paper at ICLR 2020
4 EXPERIMENTAL WORK
4.1 DATASETS AND TRAINING
We use four image classification datasets, CIFAR-10/100 (Krizhvssky Hinton 2009). SVHN (Ntt zer et al., 2 and Mini-ImageNet (Vinyals et al. 2016). to validate our approach. Part of the training images are labeled and the remaining are unlabeled. Following (Oliver et al. 2018), we use a validation set of 5K samples for CIFAR-10/ 100 for studying hyperparameters in Subsections 4.2 and 4.3 However, as done in (Athiwaratkum et al. 2019), we add the 5K samples back to the training set for comparisons in Subsection 4.4 where report test results (model from the best epoch).
CIFAR-10, CIFAR-100, and SVHN These datasets contain 10, 100, and 10 classes respectivelly, with 50K color images for training and 10K for testing in CIFAR-10/100 and 73257 images for training and 26032 for testing in SVHN. The three datasets have resolution 32x32. We perform experiments with a number of labeled images N₁ = 0.25K, 0.5K, and 1K for SVHN and Ni = 0.25K, 0.5K, IK, and 4K (4K and 10K) for CIFAR- 10 (CIFAR- 100). We use the well-known "13-CNN" architecture (Athiwaratkun et al. 2019) for CIFAR- 10/100 and SVHN. We also experiment with Wide ResNet-28-2 (WR-28) (Oliver et al.) 2018) and PreAct ResNet-18 (PR-18) (Zhang et al. 2018) in Subsection 4.3 to study the generalization to different architectures.
Mini-ImageNet We emulate the semi-supervised learning setup Mini-Imagete (Vinyals et al. 2016) (a subset of the well-known ImageNet (Deng al. (009) dataset) used in (Iscen et 1019). Train and test sets of 100 classes and 600 color images per class with resolution 84 x 84 are selected from ImageNet, as in (Ravi & Larochelle| 2017). 500 (100) images per-class are kept for train (test) splits. The train and test sets therefore contain 50k and 10k images. As with CIFAR-100. we experiment with a number of labeled images N; 4K and 10K. Following (Iscen al. 2019), we use a ResNet-18 (RN-18) architecture (He et al. 2016a).
Hyperparameters We use the typical configuration for CIFAR 10/100 and SVHN (Laine & Aila 2017), and the same for Mini-ImageNet Image normalization using dataset mean and standard deviation and subsequent data augmentation (Laine & Aila 2017) by random horizontal flips and Additionally, color jitter is applied as in (Asano et al. 2019) in Subsections 4.3 and 4.444.4t for higher robustness against confirmation bias. We train using SGD with momentum of 0.9, weight decay of 10-4, and batch size of 100. Training always starts with a high learning rate (0.1 in CIFAR and SVHN, and 0.2 in Mini- Mimi-emmaeNNe), dividing it by ten twice during training. We train for CIFAR and Mini- Imaieeeteeeet 400 epochs (reducing learning rate in epochs 250 and 350) and use 10 epoch warm-up with labeled data, while for SVHN we train 150 epochs (reducing learning rate in epochs 50 and 100) and use a longer warm-up of 150 epochs to start the pscudo-labeling with good predictions and leading to reliable convergence. We do not attempt careful tuning of the regularization weights XA and λo and just set them to 0.8 and 0.4 as done in (Tanaka et al., 2018)) (see Appendix A.2 for an ablation study of these parameters). When using dropout, is introduced between consecutive convolutional layers of ResNet blocks in WR-28, PR-18, and RN-18 18, while for 13-CNN we introduce it as in Laine& & Aila (2017). Following (Athiwaratkun et al. 2019 we use weight normalization (Salimans & Kingma 2016) in all networks.
(6) pixel translations for CIFAR and SVHN (Mini-ImageNNee
4.2
EFFECT OF MIXUP ON CONFIRMATION BIAS
This section demonstrates that carefully regularized pscudo-labeling is a suitable alternative for SSL. Figure[illustren Tillustrates our approach on the "two moons" toy data. Figure [ (left) shows the limitations of a naive pseudo-labeling adapted from (Tanaka et al. 2018), which fails to adapt to the structure in the unlabelled examples and results in linear decision boundary. Figure effect of mixup, which alleviates confirmation bias to better model the structure and gives a smoother boundary. Figure (right) shows that combining mixup with minimum number of labeled samples k per mini- batch improves the semi-supervised decision boundary.
(middle) (middle) shows the
Naive pseudo-labeling leads to overfitting the network predictions and high training accuracy in CIFAR-10/100. Table [ (left) reports mixup effect in terms of validation error. Naive pseudo-labeling
http::://gitubb.ccmm
semi ssup
5

---

Under review as a conference paper at ICLR 2020
U
Figure I:
Pseudo-labeling in the "two moons" data (4 labels/class) for 1000 samples. From left to right: no mixup, mixup, and mixup with minimum number of labeled samples per use an NN classifier with one hidden layer with 50 hidden units as in (Miyato et al.) 2018).
We
Table 1: Confirmation bias alleviation using mixup and minimum number of k labeled samples per mini-batch. Left: Validation error for naive pseudo-labeling without mixup (C), mixup (M), and alternatives with minimum k. Right: Study of the effect of on the validation error
CIFAR-10
CIFAR-100 100RR-100
CIFAR-10
CIFAR-100
Labeled images
500
4000
4000
Labeled images
500
4000
4000
C
52.44
11.40
48.54
k = 8
13.14
7.18
42.32
C* (k: = 16)
35.08
10.90
46.60
k = 16
13.68
6.90
38.78
M
32.10
7.16
41.80
k 32
14.58
7.06
39.62
M
*(k- = 16)
13.68
6.90
38.78
k =
19.40
8.20
46.28
leads an error of 11.40/48.54 for CIFAR-10/100 when training with cross-entropy (C) loss for 4000 labels. This error can be greatly reduced when using mixup (M) to 7.16/41.80 1.00. However, when further reducing the number of labels 500 in CIFAR-10. (32.10). We propose to set minimum number of samples A per mini-batch to tackle the problem. Table (right) studies this parameter when combined with mixup, showing that samples per mini-batch works well for both CIFAR-10 and CIFAR-100, dramatically reducing error in all cases (e.g. in CIFAR- for 500 labels error is reduced from 32.10 to 13.68).
is insufficient to ensure low-error
Confirmation bias causes a dramatic increase in the certainty of incorrect predictions during training. To demonstrate this behavior we compute the average cross-entropy of the softmax output with a uniform U across the classes in every epoch & for all incorrectly predicted samples {zm, Joh as: Fr = 1 ΣM: 2 log (he (Im,)). Figure shows that mixup and minimum are effective regularizers for reducing Tr, i.e. confirmation bias reduced. also experimented with using label noise regularizations (Xie et al. 2016), but setting minimum & proved more effective.
Mr
4.3 GENERALIZATION TO DIFFERENT ARCHITECTURES
There are examples in the recent literature (Koeesnkovv et al. 2019) where moving from one ar- chitecture to another modifies the belief of which methods have higher potential. Kolesnikov et al. (2099) show that skip- scip-connections ResNet architectures play a key role on the quality of learned representations, while most approaches in previous literature were systematically evaluated using AlexNet (A. Krizhevsky 2012). Ulyanov et al. (2018) showed that different architectures lead different and useful image priors, highlighting the importance of exploring different networks. We, therefore, test our method with two more architectures: a Wide ResNet-288 (WR-28) (S. 'S.Zagoruyko 2016) typically used in SSL (Olver al. 20188) (1.5M parameters) and a PreAct ResNet-18 (PR-18) (He et al.) 2016b) used in the context of label noise (Zhang et al.) 2018) (11M parameters). Table 2 presents the results for the 13-CNN (AlexNet-type) and these network architectures (ResNet-type). Our pseudo-labeling with mixup and & = 16 (M*) works well for 4000 and 500 labels across architectures, except for 500 labels for WR-28 where there large error increase (29.50). This is due to a stronger confirmation bias in which labeled samples are not properly learned, while incorrect lseudo-labels are fit. Interestingly, PR-18 (11M) is more robust to confirmation bias than WR-28 (1.5M), while the 13-layer network (3M) has fewer parameters than PR- 18 and achieves better performance. This suggests that the network architecture plays an important role, being a relevant prior for SSL with few labels.

---

Under review as a conference paper at ICLR 2020
10
C
C*
8
M
M
8
6
T. 6
4-
4
100
200
300
400
100
200
300
400
Epoch
Epoch
Figure 2: Example of certainty of incorrect predictions r during training when using 500 (left) and 4000 (right) labeled images in CIFAR-10. Moving from cross-entropy (C) to mixup (M) reduces rzs whereas adding minimum number of samples per mini-batch (*) also helps in 500 labels, where M* (with slightly lower rs than M) is the only configuration that converges, as shown in Table (left).
Table 2: Validation error across architectures is stabilized using dropout and data augmentation (A).
M*
M (p 0.1)
M (p = 0.3)
M (p = 0.1, A)
Labeled images
500
4000
500
4000
500
4000
500
4000
13-layer
13.68
6.90
12.62
6.58
11.94
6.66
9.16
6.22
WR-28
29.50
6.40
14.14
7.06
30.56
11.44
10.94
6.74
PR-18
13.90
5.94
14.78
5.90
14.78
6.62
14.96
6.32
We found that dropout (Srivastava et al. 2014) and data augmentation needed for good performance across all architectures. Tsoos that dropout p = 0.1, 0.3 helps in achieving better convergence in CIFAR- 10, whereas adding color jitter as additional data augmentation (details in Subsection 4.1) further contributes to error reduction. Note that the quality of pseudo-labels is key, so it is essential to disable dropout to prevent corruption when computing these the second forward pass. We similarly disable data augmentation in the second forward pass, which consistently improves performance. This configuration is used for comparison with the tate-of-the-aae in Subsection 4.4
4.4 COMPARISON WITH THE STATE-OF-THE-AN
We compare our lseudo-labeling approach against related work that makes use of the 13-CNN TTrvaiinen & Valpola, 2017) in 1IFAR-10/100 II model (Laine & Aila, (017)) TE (Laine & Aila
2017). MT (Tavvainen & Valpola 2017). II model-SN (Luo et al., 2018), MA-DNN (Chen et al.
2018), Deep-Co (Qiao et al. TSSDL Shi et al. 2018), LP (Iscen et al. 2019, CCL et al.
2019), fast-SWA (Athiwaratkun al. 2019) and ICT (Vema
2019). B methods into those based on consistency regularization and pseudo-labeling. Note that we include pseudo- labeling approaches combined with consistency regularization ones (e.g. MT) in the consistency regularization set. The proposed approach clearly outperforms consistency regularization methods, as well as other purely aseudo-labeling approaches and their combination with consistency regularization methods in CIFAR-10/100 and SVHN (250 labels reported in Table and extended results in the Appendix A.3). These results demonstrate the generalization of the proposed approach compared to other methods that fail when decreasing the number of labels. Furthermore, Table (left) demonstrates that the proposed approach successfully scales to higher resolution images, obtaining an over 10 point margin on the best related work in Mini-ImageNet. Note that all supervised baselines are reported using the same data augmentation and dropout as in the proposed pseudo-
Table 4 (right) compares against recent consistency regularization approaches that use mixup. We achieve better performance than ICT (Verma et al.] 2019), while being competitive with MM (Berthelot et al.] (209)) for 500 and 4000 labels using WR-28. Regarding PR-18, we converge to reasonable performance for 4000 and 500 labels, whereas for 250 we do not. Finally, the 13- CNN robustly converges even for 250 labels where we obtain 9.37 test error (see Appendix A. for some details on different architectures convergence). Therefore, these results suggest that is worth exploring the relationship between number labels, dataset complexity and architecture type. As shown in Subsection 4.3 dropout and additional data augmentation help with 500 labels/class across architectures, but are insufficient for 250 labels. Better data augmentation (Ho et al., 2019) self-suue pre-training (Rebuffi et al. 2019) might overcome this challenge. Furthermore,
7

---

Under review as a conference paper at ICLR 2020
Table 3: Test enror in 1IFAR-10/100 and SVHN for the proposed approach using the 13-CNN network. (*) denotes that we have run the algorithm. Bold indicates lowest error. We report average and standard deviation of runs with different labeled/unlabeled splits.
CIFAR-10
CIFAR-10 100
SVHN
Labeled images
500
1000
4000
4000
10000
250
Supervised (C)*
43.64 = 1.21
34.83 + 1.15
19.26 + 0.26
54.49 + 0.53
41.14 0.26
Supervised (M)*
37.60 = 0.65
28.59 + 1.21
15.94 + 0.26
52.70 + 0.28
39.42 0.37
Consistency regularization methods
model
12.36 + 0.31
39.19 + 0.36
9.69 + 0.92
TE
12.16 + 0.24
38.65 0.51
MT
27.45 + 2.64
19.04 + 0.51
11.41 + 0.25
45.36 + 0.49
36.08 + 0.51
4.35 + 0.50
T! model-SN
21.23 + 1.27
11,00 + 0.13
37.97 0.29
5.07 + 0.25
MA-DNN
11.91 + 0.22
34.51 + 0.61
Deep-Co
9.03 + 0.18
38.77 + 0.28
MT-TSSDL
18.41 +: 0.92
9.30 0.55
4.09 1: 0.42
MT-LP
24.02
16.93 + 0.70
10.61 + 0.28
43.73 + 0.20
35.92 1 0.47
MT-CCL
16.99 + 0.71
10.63 1 0.22
34.81 + 0.52
MT-fast-SWA
15.58 + 0.12
9.05 + 0.21
34.10 0.31
ICT
15.48 + 0.78
7.29 t 0.02
4.78 + 0.68
Pscudo-labeling methods
TSSDL
21.13 + 1.17
10.90 + 0.23
5.02 +: 0.26
LP
32.40 = 1.80
22.02 + 0.88
12.69 + 0.29
46.20 + 0.76
38.43 +: 1.88
Ours*
8.80 = 0.45
6.85 0.15
5.97 + 0.15
37.55 = 1.09
32.15 0.50
3.66 + 0.12
Table 4: Test error in Mini- Imagelt (left) and CIFAR-10 with few labeled samples (right). (*) denotes that we have run the algorithm. Bold indicates lowest error. We report average and standard deviation of 3 runs with different labeled/unlabeled splits.
Labeled images
4000
10000
Supervised (C)*
75.69 + 0.24
63.24 0.33
Supervised (M)*
72.03 + 0.21
59.96 0.40
Labeled images
250
4000
Consistency regularization methods
MM (WR-28)
11.08
0.87
9.65 0.94
6.24 = 0.06
ICT (WR-28)
52.19
42.33 0.08
7.26 0.04
72.51 +: 0.22 $7.55
Ours (WR-28)
24.81
5.35
14.25 0.86
6.28 + 0.3
MT-LP
72.78 1 0.15 57.35 1.66
Ours (13-CNN)
9.37 1: 0.12
8.80 0.45
5.97 0.15
Pscudo-labeling methods
Ours (PR-18)
23.86
4.82
12.16 = 1.06
5.86 0.17
LP
70.29 + 0.81
57.58 1.47
Ours
56.49 + 0.51
46.08 0.11
hyperparameters such the regularization weights Aa 0.8 and A 0.4 from Eq, 4 and the mixup a require further study. However, it is already interesting that straightforward modification of lebelo, designed to tackle confirmation bias, gives competitive supirvspee learning approach, without any consistency regularization, and future work should take this into account.
CONCLUSIONS
This paper presented semi-supervised learning approach for image classification based on pseudo- labeling. We proposed directly use the network predictions as soft pseudo-labels for unlabeled data together with mixup augmentation, minimum number of labeled samples per mini- batch, dropout and data augmentation to alleviate confirmation bias. This conceptually simple approach outperforms related work in four datasets, demonstrating that pseudo-labeling eesiistenc--reuulriiiiinn
suitable alternative The proposed approach is,
to the dominant approach in recent literature: to the best of our knowledge, both simpler and more accurate than most recent approaches. Future work should explore SSL in class-unbale unbalanced and large-scale datasets, synergies of pseudo-labelling and consistency regularization, and careful hyperparameter tuning.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
I. Sutskever G. Hinton A. Krizhevsky. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems (NeurIPS). 2012. 4.3
E. Arazo, D. Ortego, P. Albert, N. O'Connor, and K. McGuinness. Unsupervised Label Noise Modeling and Loss Correction. In International Conference on Machine Learning (ICML), 2019.
Y.M. Asano, C. Rupprecht, and A. Vedaldi. Surprising Effectiveness of Few-Image Unsupervised Feature Learning. In IEEE International Conference on Computer Vision (ICCV), 2019. 4.1
B. Athiwaratkun, M. Finzi, P. Izmailov, and A.G. Wilson. There Are Many Consistent Explana- tions Unlabeled Data: Why You Should Average. International Conference on Learning Representations (ICLR), 2019.
D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. Raffel. MixMatch: A Holistic Approach to Supe-Sup Learning. In Advances in Neural Information Processing Systems (NeurlPS), 2019.
4.4
Y. Chen, X. Zhu, and S. Gong. Semi-Supervised Deep Learning with Memory. In European Conference on Computer Vision (ECCV), 2018.
2 3.1 4.4
J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference Computer Vision and Patlern Recognition (CVPR), 2009.
S. Gidaris, P. Singh, and N. Komodakis. Unsupervised Representation Learning by Predicting Image Rotations. In International Conference on Learning Representations (ICLR), 2018. !
M. González, Bergmeir, I. Triguero, Y. Rodríguez, and J.M. Benítez. Self-labeling techniques for semi-super time series classification: an empirical study. Knowledge and Information Systems, 55(2):493-528, 2018. [
Y. Grandvalet and Y. Bengio. Semi-supervised Learning by Entropy Minimization. In International Conference Neural Information Processing Systems (NIPS), 2004.
K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016a. 4.1
K. He, X. Zhang, S. Ren, and Sun. Identity Mappings in Deep Residual Networks. European Conference Computer Vision (ECCV), 2016b. 4.3
D. Ho, E. Liang, X. Chen, I. Stoica, and P. Abbeel. Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules. In International Conference on Machine Learning (ICML), 2019. 4.4
A. Iscen, G. Tolias, Y. Avrithis, and O. Chum. Label Propagation for Deep Semi-supervised Learning.
In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, I
4.4
C. Kim, F. Li, and J.M. Rehg. Multi-object Tracking with Neural Gating Using Bilinear L.STM. In European Conference on Computer Vision (ECCV), 2018. 8.
A. Kolesnikov, X. Zhai, and L. Beyer. Revisiting Self-Supervised Visual Representation Learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
S. Laine and T. Aila. Temporal Ensembling for Semi-Supervised Learning. In International Conference on Learning Representations (ICLR), 2017.
D.H. Lee. Pseudo-label: The simple and efficient sumi-supervised learning method for deep neural networks. In International Conference Machine Learning Workshops (ICMLW), 2013. 2

---

Under review as a conference paper at ICLR 2020
W. L. Wang. W. Li, E. Agustsson, and L. Van Gool. WebVision Database: Visual Learning and Understanding from Web Data. arXiv: 1708.02862, 2017.
Y. Li, L. Liu, and R.T Tan. Decoupled Certainty- Driven Consistency Loss for Semi-supervised Learning. arXiv: 1901.05657, 2019. 12 4.4
T.Y Lin, P. Goyal, R. Girshick, K. He, and P. Dollar. Focal Loss for Dense Object Detection. In IEEE International Conference on Computer Vision (ICCV), 2017.
W. Liu, R. Lin, Z. Liu, L. Liu, Z. Yu, B. Dai, and L. Song. Learning towards Minimum Hyperspherical Energy. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
X. Liu, Van De Weijer, and A. D. Bagdanov. Exploiting Unlabeled Data in CNNs by Self- supervised Learning to Rank. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019. 0
Y. Luo, J. Zhu, M. Li, Y. Ren, and B. Zhang. Smooth Neighbors on Teacher Graphs for Semi- Supervised Learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2018.
2
Miyato, A.M. Dai, and Goodfellow. Adversarial Training Methods for Seni-Supervised Text Classification. arXiv: 1605.07725, 2016.
Miyato, S. Maeda, S. Ishii, and M. Koyama. Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning. IEEE Transactions Pattern Analysis and Machine Intellignce, 2018. 2
Y. Netzer, T. Wang, A. Coates, A. Bissacco, Wu, B, and A.Y. Ng. Reading digits in natural images with unsupervised feature learning. In Advances in Neural Information Processing Systems (NeurIPSS, 2011. 4.1
A. Oliver, A. Odena, C.A. Raffel, E.D. Cubuk, and I. Goodfellow. Realistic Evaluation of Deep Sepi-Supervised Learning Algorithms. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
S. Qiao, W. Shen, Z. Zhang, B. Wang, and A. Yuille. Deep Co-Training for Semi-Supervised Image Recognition. In European Conference on Computer Vision (ECCV), 2018.
2 4.4
Ravi and H. Larochelle. Optimization as model for few-shot learning. In International Conference on Learning Representations (ICLR), 2017. 4.1
S.-A. Rebuffi, Ehrhardt, K. Han, A. Vedaldi, and A. Zisserman. Semi-Supervised Learning with Scarce Annotations. In IEEE International Conference on Computer Vision (ICCV), 2019. 4.4
N. Komodakis S. Zagoruyko. Wide Residual Networks. In British Machine Vision Conference
(BMVC), 2016. 4.3
M. Sajjadi, M. Javanmardi, and T. Tasdizen. Regularization With Stochastic Transformations and Perturbations for Deep Semi- cemi-Supervised Learning. In Advances in Neural Information Processing Systems (NeurIPS), 2016. 2
T. Salimans and D.P. Kingma. Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks. In Advances in Neural Information Processing Systems (NeurIPS), 2016. 4.1
W. Shi, Y. Gong, C. Ding, Z. Ma. T. Xiaoyu, and N. Zheng. Transductive Semi-Supervised Deep Learning using Min-Max Features. In European Conference Computer Vision (ECCV), 2018. I
4.4
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15: 1929-1958, 2014.
4
10

---

Under review as a conference paper at ICLR 2020
D. Tanaka, D. Ikami, T. Yamasaki, and K. Aizawa. Joint Optimization Framework for Learning with Noisy Labels. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
4.2 A.2
A. Tarvainen and Valpola. Mean teachers better role models: Weight-aweraged consistency gets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
2 3.1 4.4
Thulasidasan, G. Chennupati, Bilmes, T. Bhattacharya, and S. Michalak. On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks. arXiv: 1905. 11001,
2019. 3.1
D. Ulyanov, A. Vedaldi, and V. Lempitsky. Deep Image Prior. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 4.3
V. Verma, A. Lamb, Kannala, Y. Bengio, and D. Lopez-Paz. Interpolation Consistency Training for Semi-Superviise Learning. In International Joint Conference on Artificial Intelligence (IJCAI), 2019.
2 4.4
O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, and Wierstra. Matching Networks for One Shot Learning. In Advances in Neural Information Processing Systems (NeurIPS), 2016. 4.1
M. Wang and W. Deng. Deep visual domain adaptation: A survey. Neurocomputting, 312: 135 153, 2018. 1
L. Xie, J. Wang, Z. Wei, M. Wang, and Q. Tian. DisturbLabel: Regularizing CNN on the Loss Layer In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 4.2
S. Xie, C. Sun, J. Huang, Z. Tu, and K. Murphy. Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-off's in Video Classification. In European Conference Computer Vision (ECCV), September 2018. T
H. Zhang, M. Cisse, Y.N. Dauphin, and D. Lopez-Paz. mixup: Beyond Empirical Risk Minimization.
In International Conference on Learning Representations (ICLR), 2018. 3.1
4.3
Z. Zhang, Ringeval, B. Dong. E. Coutinho, E. Marchi, and B. Schüller. Enhanced semi-supervised learning for multimodal emotion recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2016.

---

Under review as a conference paper at ICLR 2020
2,5
2.5
2.0
2.0
2.0
1.5
1.5
1.0
1.0
1.0
0.5
0.5
0.5
0.0
0.0
0.0
100
200 Epoch
300
400
0
100
200 Epoch
300
400
100
200 Epoch
300
400
3.0
2.5
2.5
2.5
2.0
2.0
2.0
1.5
1.5
1.5
1.0-
1.0
1.0
0.5
0.5
0.5
0.0
0.0
0.0
100
200
300
400
0
100
200
300
400
100
200
300
400
Epoch
Epoch
Epoch
Figure 3: Cross-entropy loss for labeled samples. First (second) row show 500 (250) labels in CIFAR-10. From left right: 13-CNN, WR-28 and PR-18. The heavy lines represent the median losses and the shaded areas are the interquartile ranges.
A APPENDIX
A.1 CONVERGENCE FOR FEW LABELS
Figure 3 presents the cross-entropy loss for labeled samples when training with 13-CNN, WR-28 and PR-18 18 and using 500 and 250 labels in CIFAR-10. This loss is good indicator of robust convergence to reasonable performance as the interquartile range for cases failing (250 labels for WR-28 and PR-18) is much higher.
A.2 EXTENDED HYPERPARAMETERS STUDY
This subsection studies the effect of XA. and An hyperparameters of our pseudo-labeling approach. Tables [ and 6 report the validation error in CIFAR-10 10 using 500 and 4000 labels for, respectively, a and AA and λи Note that keep the same configuration used in Subsection 4.2 with A 16, i.c. no dropout or additional data augmentation is used. Table 5 results suggest that a = 4 and a = 8 values experimented on CIFAR-10 with 500 labels using the final configuration (adding dropout and additional data augmentation) and observed marginal differences (8.54 with 4, which is within the error range of the 8.80 + 0.45 obtained with a 1), thus suggesting that stronger mixup regularization might not be additive to dropout and extra data augmentation our case. Table d shows that our configuration (AΛ 0.8 and λи = 0.4) adopted from (Tanaka et al. 2018) 8) is very close to the best performance in this experiment where marginal improvements are achieved. In conclusion, more careful hyperparameter tuning might slighitly improve reported results in the paper, while the configuration selected is already good and generalizes across datasets.
might further improve the reported results using a I. However,
A.3 SVHN EXPPERIMENTS
Table 7 reports comparison of different state-of-the-art algorithms in SVHN using the 13-CNN network. We train 150 epochs using labeled samples (i.e. warm-up) to compute initial pseudo-labels. Then, we train 150 epochs reducing the learning rate in epochs 50 and 100. The long warm-up makes the method robust to lower levels of labeled samples in SVHN, We also experimented in CIFAR- 10 with longer warm-up (our results are reported using 10 epochs) and found that results are in the same error range already reported.
12

---

Under review as a conference paper at ICLR 2020
Table 5: Validation error for different values of the parameter from Mixup. Bold indicates lowest error. Underlined values indicate the results of the configuration used.
Labeled images:
500
4000
a
0.1
0.1
8
23.18
13.68
10.60
11.04
8.58
6.90
6.56
6.68
Table 6: Validation error for different values of Åa and λиr Bold indicates lowest error. Underlined values indicate the results of the configuration used.
Labeled images:
500
4000
Aa/AH
0.1
0.4
0.8
0.1
0.4
0.8
2
0.1
22.94
29.64
60.76
83.96
7.22
6.88
7.74
33.98
0.4
20.92
12.88
17.62
38.40
7.18
6.96
7.18
8.82
0.8
23.50
13.68
14.72
25.92
7.24
6.90
7.18
8.78
31.30
14.80
14.62
23.40
8.16
7.28
7.40
8.64
Table 7: Test error in SVHN for the proposed approach using the 13-CNN network. (*) denotes that we have run the algorithm. Bold indicates lowest error. We report average and standard deviation of 3 runs with different labeled/unlabeled splits.
Labeled images
250
500
1000
Supervised (C)*
22.67.2.80 53.15116.54
43.601...35
13.32.10.89 20.74.10.80
Supervised (My*
11.66.10.17
Consistency regularization methods
II model
9.69 0.92
6.83 0.66
4.95 + 0.26
TE MT IT model-SN
5.12 + 0.13
4.42 + 0.16
4.35 + 0.50
4.18 + 0.27
3.95 0.19
5.07 0.25
4.52 + 0.30
4.21 + 0.25
3.82 + 0.25
MA-DNN
Deep-Co
3.61 + 0.15
MT-TSSDL.
4.09 + 0.42
3.90 + 0.27
3.35 + 0.27
ICT
4.78 0.68
4.23 + 0.
3.89 + 0.04
Pseudo-labeling
methods
TSSDL
5.02 0.26
4.32 + 0.30
3.80 + 0.27
Ours*
3.66 0.12
3.64 + 0.04
3.55 + 0.08
13


Reviews:
[{"review_id": "Syla9DLY_r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "OVERALL:\nI think this paper is worth accepting.\nAll modern semi-supervised learning techniques use consistency regularization somehow,\nand this paper shows that you can get away with just using pseudo-labeling combined with some\nengineering to route around the main issue with pseudo labeling (which is apparently called confirmation bias,\nthough I hadn't heard that, and I don't like it as a name because it's confusing).\n\nNeither MixUp nor the idea of fixing some number of labeled elements in a minibatch is new,\nbut that's not the point - we thought one thing, and this paper suggests that\nwe were wrong about that thing - to me this is exactly the sort of paper it's good to have at conferences.\n\nI would change the framing slightly.\nYou're not showing that pseudo-labeling can be useful, because many techniques already incorporate a form of pseudo-labeling.\nInstead, you're showing you can get away without consistency regularization.\n\nA potential improvement:\nIf you add up this techique with some of the most recent SLL techniques based on consistency regularization somehow,\ndoes it do better, or are they both acting via the same mechanism?\n\nDETAILED COMMENTS:\n> , contrary to previous evidences on pseudo-labeling capabilities (Oliver et al., 2018),\nIt's not really contrary to the findings of that paper, since you've totally changed the\ntechnique compared to what's evaluated in that paper.\n\n> n (Berthelo et al., 2019) \nIt's Berthelot\n\n> and are the mechanisms proposed in Subsection 3.1\nDoesn't quite parse\n\n> Network predictions are, of course, sometimes incorrect.\nThis is a great line.\n\n> We use three image classification datasets...\nWhy not use SVHN, which is by now super standard for SSL papers?\n\n> , we add the 5K samples back to the training set for comparison\nwith the state-of-the-art in Subsection 4.4,\nThis is *allowed* from the perspective of reporting a valid test accuracy,\nbut if other papers don't do that, it kind of mucks up the comparison, no?\n\nFig 1 is nice, but why does the effect not seem to be symmetric about the\nblue and the red blobs?\n\n> architecture plays and important role\n\n\n> However, it is already interesting that...  and that future work should take this into account.\nThis sentence doesn't quite make sense\n\nRe table 4:\nI'm curious how e.g. MixMatch would fare w/ the 13-CNN network.\nI am surprised that the change from WRN -> 13-CNN matters so much.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SJlg3L55tH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper focuses on the semi-supervised learning problem, and proposes a way to improve previous pseudo-labeling methods.  In pseudo-labeling, there is an issue called confirmation bias, which accumulates the early errors of wrong pseudo labels.  By adding some simple tricks such as adding mixup augmentation and setting a minimum number of labeled samples per mini-batch, the confirmation bias is shown to be reduced, leading to an improvement in accuracy.  Experiments demonstrate that the additional tricks are meaningful and makes pseudo-labeling better than many baseline methods for semi-superivsed learning, including state-of-the-art consistency regularization methods.\n\n\nPros: This is an interesting paper with a clear motivation, which is to fix the so-called confirmation bias that appears in pseudo-labeling methods for semi-supervised learning.  Although the tricks introduced in the paper (mixup and changing the mini-batch selection rules) themselves are not novel, they make the proposed method simple.  It is also shown to be meaningful in reducing the confirmation bias in Table 1 and Figure 2, achieving the original goal of the paper.\n\n\nCons: The weakness of the paper is that the intuition or the motivation behind the design of the proposed method is not so clear.  Using mixup is justified by the reason that mixup gives better confidence calibration.  This is important for pseudo-labeling methods, because soft-label output predictions are used as pseudo labels.  On the other hand, however, it was not so obvious why a minimum number of labeled samples per mini-batch was considered.  Can we consider further extensions such as minimum number of labeled samples per mini-batch & per class?  (Perhaps the discussions about mixup and soft labels in the last paragraph of Section 3 should be more emphasized, for example in the last paragraph of the Introduction section.)\n\nRelated to the weakness above, it is hard to see how far the regularization effects of adding mixup and mini-batch sampling rules are contributing to add synergy to the pseudo-labeling methods.  This is partially answered with Figure 2, but it would make this easier to see if the experiments included stronger baselines, e.g., by adding the same regularization tricks to consistency regularization methods, perhaps in Table 3.\n\nFinally, since future work on pseudo labels will follow this paper’s setup, hyperparameters such as lamba_A, lambda_H, and alpha should be chosen carefully instead of fixing them.\n\n\nOther minor comments (that did not impact the score):\n\n- In reference section, \"Z. MaXiaoyu Tao\" seems to combine two authors.\n\n- Table 3 never appears in the text.  In Section 4.4, \"The table\" in the second sentence can be changed to \"Table 3\".\n\n- \"architecture plays and important role\" --> \"architecture plays an important role\"\n\n- In Table 2, \"+\" signs make it look like an equation.  I suggest using commas instead.\n\n-  \"ResNet arquitectures\" --> \"ResNet architectures\"\n\n~~~~~\nThank you for the response and for the additional discussions that were included in the updated paper.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1g97ORFtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to combine pseudo-labelling with MixUp to tackle the semi-supervised classification problem. My problem is that \"MixMatch: A Holistic Approach to Semi-Supervised Learning\" by Berthelot et al. is very similar with just a few differences on the pseudo-labelling part. Could you stress more the difference between your paper and their paper ? Because I might be wrong about it.\n\nPros:\n* Good results on C10\n* A clear related work section that divides the existing works in pseudo labelling vs consistency\n* Interesting results about the effects of using different architectures. I also like the ablation study.\n\nWeaknesses:\n* Usually, SVHN is also among the tested datasets\n* The pseudo labelling part is a bit unclear.For example, do you just refresh the pseudo-labels at the end of each epoch ?\n* minor: a typo with \"and important role\"\n\nIf there was not an existing paper already using MixUp, I would have leaned towards acceptance. You can still motivate the differences with the MixMatch paper.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper focuses on semi-supervised learning and presents a pseudo labeling-based approach with i) mixup (Zhang et al. 2018); ii) keeping $k$ labelled examples in each minibatch.

The paper is clear and well-written; it presents a simple and empirically effective idea. Reviewers appreciate the nice proof of concept on the two-moons dataset, the fact that the approach is validated with different architectures. Some details would need to be clarified, e.g. about the dropout control.

A main contribution of the paper is to show that pseudo-labelling plus the combination of mixup and certainty (keeping $k$ labelled examples in each minibatch) can outperform the state of the art based on consistency regularization methods, while being simpler and computationally much less demanding. 

While the paper does a good job of showing that "it works", the reader however misses some discussion about "why it works". It is most interesting that the performances are not improving with $k$ (Table 1). An in-depth analysis of the trade-off between the uncertainty (through mix-up and the entropy of the pseudo-labels) and certainty, and how it impacts the performance, would be appreciated. You might consider monitoring how this trade-off evolves along learning; I suspect that evolving $k$ along the epochs might make sense;  the question is to find a simple way to control online this hyper-parameter.  

The area chair encourages the authors to continue this very promising path of research, and dig a little bit deeper, considering the question of optimizing the trade-off between certainty and uncertainty along the training trajectory.
Decision: Reject