Robust Temporal Ensembling

1 INTRODUCTION . Deep neural networks have enjoyed considerable success across a variety of domains , and in particular computer vision , where the common theme is that more labeled training data yields improved model performance ( Hestness et al. , 2017 ; Mahajan et al. , 2018 ; Xie et al. , 2019b ; Kolesnikov et al. , 2019 ) . However , performance depends on the quality of the training data , which is expensive to collect and inevitably imperfect . For example , ImageNet ( Deng et al. , 2009 ) is one of the most widely-used datasets in the field of deep learning and despite over 2 years of labor from more than 49,000 human annotators across 167 countries , it still contains erroneous and ambiguous labels ( FeiFei & Deng , 2017 ; Karpathy , 2014 ) . It is therefore essential that learning algorithms in production workflows leverage noise robust methods . Noise robust learning has a long history and takes many forms ( Natarajan et al. , 2013 ; Frenay & Verleysen , 2014 ; Song et al. , 2020 ) . Common strategies include loss correction and reweighting ( Patrini et al. , 2016 ; Zhang & Sabuncu , 2018 ; Menon et al. , 2020 ) , label refurbishment ( Reed et al. , 2014 ; Song et al. , 2019 ) , abstention ( Thulasidasan et al. , 2019 ) , and relying on carefully constructed trusted subsets of human-verified labeled data ( Li et al. , 2017 ; Hendrycks et al. , 2018 ; Zhang et al. , 2020 ) . Additionally , recent methods such as SELF ( Nguyen et al. , 2020 ) and DivideMix ( Li et al. , 2020 ) convert the problem of learning with noise into a semi-supervised learning approach by splitting the corrupted training set into clean labeled data and noisy unlabeled data at which point semisupervised learning methods such as Mean Teacher ( Tarvainen & Valpola , 2017 ) and MixMatch ( Berthelot et al. , 2019 ) can be applied directly . In essence , these methods effectively discard a majority of the label information so as to side-step having to learning with noise at all . The problem here is that noisy label filtering tactics are imperfect resulting in corrupted data in the small labeled partition and valuable clean samples lost to the large pool of unlabeled data . Moreover , caution is needed when applying semi-supervised methods where the labeled data is not sampled i.i.d . from the pool of unlabeled data ( Oliver et al. ) . Indeed , filtering tactics can be biased and irregular , driven by specification error and the underlying noise process of the label corruption . Recognizing the success of semi-supervised approaches , we ask : can we leverage the underlying mechanisms of semi-supervised learning such as entropy regularization for learning with noise without discarding our most valuable asset , the labels ? 2 ROBUST TEMPORAL ENSEMBLING . 2.1 PRELIMINARIES . Adopting the notation of Zhang & Sabuncu ( 2018 ) , we consider the problem of classification where X ⊂ Rd is the feature space and Y = { 1 , . . . , c } is the label space where the classifier function is a deep neural network with a softmax output layer that maps input features to distributions over labels f : X → Rc . The dataset of training examples containing in-sample noise is defined as D = { ( xi , ỹi ) } ni=1 where ( xi , ỹi ) ∈ ( X × Y ) and ỹi is the noisy version of the true label yi such that p ( ỹi = k|yi = j , xi ) ≡ ηijk . We do not consider open-set noise ( Wang et al. , 2018 ) , in which there is a particular type of noise that occurs on inputs , x̃ , rather than labels . Following most prior work , we make the simplifying assumption that the noise is conditionally independent of the input , xi , given the true labels . In this setting , we can write ηijk = p ( ỹi = k|yi = j ) ≡ ηjk which is , in general , considered to be class dependent noise1,2 . To aid in a simple and precise corruption procedure , we now depart from traditional notation and further decompose ηjk as pj · cjk , where pj ∈ [ 0 , 1 ] is the probability of corruption of the j-th class and cjk ∈ [ 0 , 1 ] is the relative probability that corrupted samples of class j are labeled as class k , with ci6=j ≥ 0 , cjj = 0 and ∑ k cjk = 1 . A noisy dataset with m classes can then be described as transition probabilities specified by F = diag ( P ) · C + diag ( 1− P ) · I ( 1 ) where C ∈ Rm×m defines the system confusion or noise structure , P ∈ Rm defines the noise intensity or ratio for each class , and I is the identity matrix . When cjk = ckj the noise is said to be symmetric and is considered asymmetric otherwise . If ratio of noise is the same for all classes then pj = p and the dataset is said to exhibit uniform noise . For the case of uniform noise , equation ( 1 ) interestingly takes the familiar form of the Google matrix equation as Fp = p · C + ( 1− p ) · I ( 2 ) Note that , by this definition , ηjj = p · cjj = 0 which prohibits ỹi = yi . This ensures a true effective noise ratio of p. For example , suppose there are m = 10 classes and we wish to corrupt labels with 80 % probability . Then if corrupted labels are sampled from Y rather than Y \ { y } , 110 · 0.8 = 8 % of the corrupted samples will not actually be corrupted , leading to a true corruption rate of 72 % . Therefore , despite prescribing p = 0.8 , the true effective noise ratio would be 0.72 , which in turn yields a 0.081−0.8 = 40 % increase in clean labels , and this is indeed the case in many studies ( Zhang & Sabuncu , 2018 ; Nguyen et al. , 2020 ; Li et al. , 2020 ; Zhang et al. , 2020 ) . 2.2 METHODS . At a very high level , RTE is the combination of noise-robust task loss , augmentation , and pseudolabeling for consistency regularization . We unify generalized cross entropy ( Zhang & Sabuncu , 2018 ) , AugMix stochastic augmentation strategy ( Hendrycks et al. , 2020 ) , an exponential moving average of model weights for generating pseudo-labels ( Tarvainen & Valpola , 2017 ) , and an augmentation anchoring-like approach ( Berthelot et al. , 2020 ) to form a robust approach for learning with noisy labels . 2.2.1 NOISE-ROBUST TASK LOSS . Generalized cross entropy ( GCE ) ( Zhang & Sabuncu , 2018 ) is a theoretically grounded noise-robust loss function that can be seen as a generalization of mean absolute error ( MAE ) and categorical cross entropy ( CCE ) . The main idea is that CCE learns quickly , but more emphasis is put on difficult samples which is prone to overfit noisy labels , while MAE treats all samples equally , providing noise-robustness but learning slowly . To exploit the benefits of both MAE and CCE , a negative Box-Cox transformation ( Box & Cox , 1964 ) is used as the loss function Lq ( f ( xi ) , yi = j ) = ( 1− fj ( xi ) q ) q ( 3 ) 1See Lee et al . ( 2019 ) for treatment of conditionally dependent semantic noise such that ηijk 6= ηjk . 2Note that Patrini et al . ( 2016 ) define the noise transition matrix T such that Tjk ≡ ηjk . where q ∈ ( 0 , 1 ] , and fj denotes the j-th element of f . Note that GCE becomes CCE for limq→0 Lq and becomes MAE/unhinged loss when q = 1 . 2.2.2 ENSEMBLE CONSISTENCY REGULARIZATION . Consistency regularization works under the assumption that a model should output similar predictions given augmented versions of the same input . This regularization strategy is a common component of semi-supervised learning algorithms with the general form of ‖pθ ( y|xaug1 ) −pθ ( y|xaug2 ) ‖ where pθ ( y|x ) is the predicted class distribution produced by the model having parameters θ for input x ( Zheng et al. , 2016 ; Sajjadi et al. , 2016 ) . We build upon numerous variations from semisupervised learning ( Laine & Alia , 2017 ; Tarvainen & Valpola , 2017 ; Berthelot et al. , 2019 ; 2020 ) and leverage an ensemble consistency regularization ( ECR ) strategy as ECR = 1 |Y|N∗ N∗∑ i=1 ‖pθ′ ( y|x ) − pθ ( y|A ( x ) ) ‖ ( 4 ) where x is the training example , A is stochastic augmentation function reevaluated for each term in the summation , θ′t = αθ ′ t−1 + ( 1 − α ) θt is a temporal moving average of model weights used to generate pseudo-label targets , and inputs are pre-processed with standard random horizontal flip and crop . In practice , this consists of initializing a copy of the initial model and maintaining an exponential moving average as training progresses . Some methods directly average multiple label predictions together at each optimization step to form a single pseudo-label target ( Berthelot et al. , 2019 ; Li et al. , 2020 ) but we find pseudo-label target distributions generated by θ′ to be better suited for the learning with noise problem due to the intrinsic ensemble nature of the weight averaging process over many optimization steps ( Tarvainen & Valpola , 2017 ) . In semi-supervised learning techniques , it is common to leverage a large batch-size of unlabeled data for consistency regularization . However , we found that modulating N∗ , rather than the batch size of the consistency term , yields a monotonic increase in model performance consistent with related works ( Berthelot et al. , 2020 ) . Moreover , in semi-supervised learning , different batches are used for between supervised and unsupervised loss terms but we find ( see section 4.3 ) that for the case of learning with noise , batches synchronized with GCE task loss term yields superior performance . 2.2.3 AUGMENTATION . AugMix ( Hendrycks et al. , 2020 ) is a data augmentation technique which utilizes stochasticity , diverse augmentations , a Jensen-Shannon divergence consistency loss , and a formulation to mix multiple augmented inputs . Other augmentation strategies such as RandAugment ( Cubuk et al. , 2020 ) , augmentations are applied sequentially with fixed intensity which can degrade input quickly . In AugMix , to mitigate input degradation but retain augmentation diversity , several stochastically sampled augmentation chains are layered together in a convex combination to generate highly diverse transformations . These mixing coefficients are randomly sampled from a Dirichlet distribution with shared concentration parameters , and the resulting augmented version of the input is combined with the original input through a second random convex combination sampled from a beta distribution , again with shared parameters . 2.2.4 JENSEN-SHANNON DIVERGENCE . The Jensen-Shannon consistency loss is used to enforce a flat response of the classifier by incentivizing the model to be stable , consistent , and insensitive across a diverse range of inputs ( Zheng et al. , 2016 ) . The Jensen-Shannon divergence ( JSD ) is minimized across distributions porig , paug1 , and paug2 of the original sample xorig and its augmented variants xaug1 and xaug2 which can be understood to measure the average information that the sample reveals about the identity of its originating distribution ( Hendrycks et al. , 2020 ) . This JSD term is computed withM = ( porig +paug1 +paug2 ) /3 and is then JSD = 1 3 ( KL ( porig ‖M ) + KL ( paug1 ‖M ) + KL ( paug2 ‖M ) ) ( 5 ) where KL ( p ‖ q ) is Kullback–Leibler divergence from q to p. In this way , the JSD term improves the stability of training in the presence of noisy labels and heavy data augmentation with a modest contribution to final classifier test accuracy as shown in Table 5 .

Summary: Real-world data contains noise in the annotated labels. To mitigate, the authors propose a supervised learning approach, Robust Temporal Ensembling (RTE). RTE combines 1) task loss correction, which is a generalized cross entropy loss, 2) different augmentations resulting from AugMix technique and the Jensen-Shannon divergence (JSD), 3) the ensemble consistency regularization and pseudo labeling.