Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning

1 INTRODUCTION . Convolutional neural networks ( CNNs ) have become the dominant approach in computer vision ( Lin et al. , 2017 ; Liu et al. , 2018 ; Kim et al. , 2018 ; Xie et al. , 2018 ) . To best exploit them , vast amounts of labeled data are required . Obtaining such labels , however , is not trivial , and the research community is exploring alternatives to alleviate this ( Li et al. , 2017 ; Oliver et al. , 2018 ; Liu et al. , 2019 ) . Knowledge transfer via deep domain adaptation ( Wang & Deng , 2018 ) is a popular alternative that seeks to learn transferable representations from source to target domains by embedding domain adaptation in the learning pipeline . Other approaches focus exclusively on learning useful representations from scratch in a target domain when annotation constraints are relaxed ( Oliver et al. , 2018 ; Arazo et al. , 2019 ; Gidaris et al. , 2018 ) . Semi-supervised learning ( SSL ) ( Oliver et al. , 2018 ) focuses on scenarios with sparsely labeled data and extensive amounts of unlabeled data ; learning with label noise ( Arazo et al. , 2019 ) seeks robust learning when labels are obtained automatically and may not represent the image content ; and self-supervised learning ( Gidaris et al. , 2018 ) uses data supervision to learn from unlabeled data in a supervised manner . This paper focuses on SSL for image classification , a recently very active research area ( Li et al. , 2019 ) . SSL is a transversal task for different domains including images ( Oliver et al. , 2018 ) , audio ( Zhang et al. , 2016 ) , time series ( González et al. , 2018 ) , and text ( Miyato et al. , 2016 ) . Recent approaches in image classification primarily focus on exploiting the consistency in the predictions for the same sample under different perturbations ( consistency regularization ) ( Sajjadi et al. , 2016 ; Li et al. , 2019 ) , while other approaches directly generate labels for the unlabeled data to guide the learning process ( pseudo-labeling ) ( Lee , 2013 ; Iscen et al. , 2019 ) . These two alternatives differ importantly in the mechanism they use to exploit unlabeled samples . Consistency regularization and pseudo-labeling approaches apply different strategies such as a warm-up phase using labeled data ( Tarvainen & Valpola , 2017 ; Iscen et al. , 2019 ) , uncertainty weighting ( Shi et al. , 2018 ; Li et al. , 2019 ) , adversarial attacks ( Miyato et al. , 2018 ; Qiao et al. , 2018 ) , or graph-consistency ( Luo et al. , 2018 ; Iscen et al. , 2019 ) . These strategies deal with confirmation bias ( Tarvainen & Valpola , 2017 ; Li et al. , 2019 ) , also known as noise accumulation ( Zhang et al. , 2016 ) . This bias stems from using incorrect predictions on unlabeled data for training in subsequent epochs and , thereby increasing confidence in incorrect predictions and producing a model that will tend to resist new changes . This paper explores pseudo-labeling for semi-supervised deep learning from the network predictions and shows that , contrary to previous attempts on pseudo-labeling ( Iscen et al. , 2019 ; Oliver et al. , 2018 ; Shi et al. , 2018 ) , simple modifications to prevent confirmation bias lead to state-of-the-art performance without adding consistency regularization strategies . We adapt the approach proposed by Tanaka et al . ( 2018 ) in the context of label noise and apply it exclusively on unlabeled samples . Experiments show that this naive pseudo-labeling is limited by confirmation bias as prediction errors are fit by the network . To deal with this issue , we propose to use mixup augmentation ( Zhang et al. , 2018 ) as an effective regularization that helps calibrate deep neural networks ( Thulasidasan et al. , 2019 ) and , therefore , alleviates confirmation bias . We find that mixup alone does not guarantee robustness against confirmation bias when reducing the amount of labeled samples or using certain network architectures ( see Subsection 4.3 ) , and show that , when properly introduced , dropout regularization ( Srivastava et al. , 2014 ) and data augmentation mitigates this issue . Our purely pseudo-labeling approach achieves state-of-the-art results ( see Subsection 4.4 ) without requiring multiple networks ( Tarvainen & Valpola , 2017 ; Qiao et al. , 2018 ; Li et al. , 2019 ; Verma et al. , 2019 ) , nor does it require over a thousand epochs of training to achieve peak performance in every dataset ( Athiwaratkun et al. , 2019 ; Berthelot et al. , 2019 ) , nor needs many ( ten ) forward passes for each sample ( Li et al. , 2019 ) . Compared to other pseudo-labeling approaches , the proposed approach is simpler in that it does not require graph construction and diffusion ( Iscen et al. , 2019 ) or combination with consistency regularization methods ( Shi et al. , 2018 ) , but still achieves state-of-the-art results . 2 RELATED WORK . This section reviews closely related SSL methods , i.e . those using deep learning with mini-batch optimization over large image collections . Previous work on deep SSL differ in whether they use consistency regularization or pseudo-labeling to learn from the unlabeled set ( Iscen et al. , 2019 ) , while they all share the use of a cross-entropy loss ( or similar ) on labeled data . Consistency regularization Imposes that the same sample under different perturbations must produce the same output . This idea was used in ( Sajjadi et al. , 2016 ) where they apply randomized data augmentation , dropout , and random max-pooling while forcing softmax predictions to be similar . A similar idea is applied in ( Laine & Aila , 2017 ) , which also extends the perturbation to different epochs , i.e . the current prediction for a sample has to be similar to an ensemble of predictions of the same sample in the past . Here the different perturbations come from networks at different states , dropout , and data augmentation . In ( Tarvainen & Valpola , 2017 ) , the temporal ensembling method is interpreted as a teacher-student problem where the network is both a teacher that produces targets for the unlabeled data as a temporal ensemble , and a student that learns the generated targets by imposing the consistency regularization . Tarvainen & Valpola ( 2017 ) naturally re-define the problem to deal with confirmation bias by separating the teacher and the student . The teacher is defined as a different network with similar architecture whose parameters are updated as an exponential moving average of the student network weights . This method is extended in ( Li et al. , 2019 ) , where they apply an uncertainty weight over the unlabeled samples to learn from the unlabeled samples with low uncertainty ( i.e . entropy of the predictions for each sample under random perturbations ) . Additionally , Miyato et al . ( 2018 ) use virtual adversarial training to carefully introduce perturbations to data samples as adversarial noise and later impose consistency regularization on the predictions . More recently , Luo et al . ( 2018 ) propose to use a contrastive loss on the predictions as a regularization that forces predictions to be similar ( different ) when they are from the same ( different ) class . This method extends the consistency regularization previously considered only in-between the same data samples to in-between different samples . Their method can naturally be combined with ( Tarvainen & Valpola , 2017 ) or ( Miyato et al. , 2018 ) to boost their performance . Similarly , Verma et al . ( 2019 ) propose interpolation consistency training , a method inspired by ( Zhang et al. , 2018 ) that encourage predictions at interpolated unlabeled samples to be consistent with the interpolated predictions of individual samples . Also , authors in ( Berthelot et al. , 2019 ) apply consistency regularization by guessing low-entropy labels , generating data-augmented unlabeled examples and mixing labeled and unlabeled examples using mixup ( Zhang et al. , 2018 ) . Both ( Verma et al. , 2019 ) and ( Berthelot et al. , 2019 ) adopt ( Tarvainen & Valpola , 2017 ) to estimate the targets used in the consistency regularization . Co-training ( Qiao et al. , 2018 ) uses two ( or more ) networks trained simultaneously to agree on their predictions ( consistency regularization ) and disagree on their errors . Errors are defined as different predictions when exposed to adversarial attacks , thus forcing different networks to learn complementary representations for the same samples . Recently , Chen et al . ( 2018 ) measure the consistency between the current prediction and an additional prediction for the same sample given by an external memory module that keeps track of previous representations . They additionally introduce an uncertainty weighting of the consistency term to reduce the contribution of uncertain predictions . Consistency regularization methods such as ( Laine & Aila , 2017 ; Tarvainen & Valpola , 2017 ; Miyato et al. , 2018 ) have all been shown to benefit from stochastic weight averaging method ( Athiwaratkun et al. , 2019 ) , that averages network parameters at different training epochs to move the SGD solution on borders of flat loss regions to their center and improve generalization . Pseudo-labeling Seeks the generation of labels or pseudo-labels for unlabeled samples to guide the learning process . An early attempt at pseudo-labeling proposed in ( Lee , 2013 ) uses the network predictions as labels . However , they constrain the pseudo-labeling to a fine-tuning stage , i.e . there is a pre-training or warm-up to initialize the network . A recent pseudo-labeling approach proposed in ( Shi et al. , 2018 ) uses the network class prediction as hard labels for the unlabeled samples . They also introduce an uncertainty weight for each sample loss , it being higher for samples that have distant k-nearest neighbors in the feature space . They further include a loss term to encourage intra-class compactness and inter-class separation , and a consistency term between samples with different perturbations . Improved results are reported in combination with ( Tarvainen & Valpola , 2017 ) . Finally , a recently published work ( Iscen et al. , 2019 ) implements pseudo-labeling through graph-based label propagation . The method alternates between two steps : training from labeled and pseudo-labeled data and using the representations of the network to build a nearest neighbor graph where label propagation is applied to refine hard pseudo-labels . They further add an uncertainty score for every sample ( softmax prediction entropy based ) and class ( class population based ) to deal , respectively , with the unequal confidence in network predictions and class-imbalance . 3 PSEUDO-LABELING . We formulate SSL as learning a model hθ ( x ) from a set of N training samples D. These samples are split into the unlabeled setDu = { xi } Nui=1 and the labeled setDl = { ( xi , yi ) } Nl i=1 , being yi ∈ { 0 , 1 } C the one-hot encoding label for C classes corresponding to xi and N = Nl + Nu . In our case , hθ is a CNN and θ represents the model parameters ( weights and biases ) . As we seek to perform pseudo-labeling , we assume that a pseudo-label ỹ is available for the Nu unlabeled samples . We can then reformulate SSL as training using D̃ = { ( xi , ỹi ) } Ni=1 , being ỹ = y for the Nl labeled samples . The CNN parameters θ can be optimized using categorical cross-entropy : ` ∗ ( θ ) = − N∑ i=1 ỹTi log ( hθ ( xi ) ) , ( 1 ) where hθ ( x ) are the softmax probabilities produced by the model and log ( · ) is applied element-wise . A key decision is how to generate the pseudo-labels ỹ for the Nu unlabeled samples . Previous approaches have used hard pseudo-labels ( i.e . one-hot vectors ) directly using the network output class ( Lee , 2013 ; Shi et al. , 2018 ) or the class estimated using label propagation on a nearest neighbor graph ( Iscen et al. , 2019 ) . We adopt the former approach , but use soft pseudo-labels , as we have seen this outperforms hard labels , confirming the observations noted in ( Tanaka et al. , 2018 ) in the context of relabeling when learning with label noise . In particular , we store the softmax predictions hθ ( xi ) of the network in every mini-batch of an epoch and use them to modify the soft pseudo-label ỹ for the Nu unlabeled samples at the end of every epoch . We proceed as described from the second to the last training epoch , while in the first epoch we use the softmax predictions for the unlabeled samples from a model trained in a 10 epochs warm-up phase using the labeled data subset Du . We use the two regularizations applied in ( Tanaka et al. , 2018 ) to improve convergence . The first regularization deals with the difficulty of converging at early training stages when the network ’ s predictions are mostly incorrect and the CNN tends to predict the same class to minimize the loss . Assignment of all samples to a single class is discouraged by adding : RA = C∑ c=1 pc log ( pc hc ) , ( 2 ) where pc is the prior probability distribution for class c and hc denotes the mean softmax probability of the model for class c across all samples in the dataset . As in ( Tanaka et al. , 2018 ) , we assume a uniform distribution pc = 1/C for the prior probabilities ( RA stands for all classes regularization ) and approximate hc using mini-batches . The second regularization is needed to concentrate the probability distribution of each soft pseudo-label on a single class , thus avoiding the local optima in which the network might get stuck due to a weak guidance : RH = − 1 N N∑ i=1 C∑ c=1 hcθ ( xi ) log ( h c θ ( xi ) ) , ( 3 ) where hcθ ( xi ) denotes the c class value of the softmax output hθ ( xi ) and again using mini-batches ( i.e . N is replaced by the mini-batch size ) to approximate this term . This second regularization is the average per-sample entropy ( RH stands for entropy regularization ) , a well-known regularization in SSL ( Grandvalet & Bengio , 2004 ) . Finally , the total semi-supervised loss is : ` = ` ∗ + λARA + λHRH , ( 4 ) where λA and λH control the contribution of each regularization term . We stress that this pseudolabeling approach adapted from Tanaka et al . ( 2018 ) is far from the state-of-the-art for SSL ( see Subsection 4.2 ) , and are the mechanisms proposed in Subsection 3.1 which make pseudo-labeling a suitable alternative .

Summary: This paper focuses on the semi-supervised learning problem, and proposes a way to improve previous pseudo-labeling methods.  In pseudo-labeling, there is an issue called confirmation bias, which accumulates the early errors of wrong pseudo labels.  By adding some simple tricks such as adding mixup augmentation and setting a minimum number of labeled samples per mini-batch, the confirmation bias is shown to be reduced, leading to an improvement in accuracy.  Experiments demonstrate that the additional tricks are meaningful and makes pseudo-labeling better than many baseline methods for semi-superivsed learning, including state-of-the-art consistency regularization methods.