On Non-Random Missing Labels in Semi-Supervised Learning

1 INTRODUCTION . Semi-supervised learning ( SSL ) aims to alleviate the strong demand for large-scale labeled data by leveraging unlabeled data ( Zhu , 2008 ; Yang et al. , 2021 ) . Prevailing SSL methods first train a model using the labeled data , then uses the model to impute the missing labels with the predicted pseudolabels for the unlabeled data ( Van Buuren , 2018 ) , and finally combine the true- and pseudo-labels to further improve the model ( Sohn et al. , 2020 ; Berthelot et al. , 2019a ) . Ideally , the “ improve ” is theoretically guaranteed if the missing label imputation is perfect ( Grandvalet & Bengio , 2005 ) ; otherwise , imperfect imputation causes the well-known confirmation bias ( Arazo et al. , 2019 ; Sohn et al. , 2020 ) . In particular , the bias is even more severe in practice because the underlying assumption that the labeled and unlabeled data are drawn from the same distribution does not hold . We term this scenario as label Missing Not At Random ( MNAR ) ( Hernán & Robins , 2020 ) , as compared to the naı̈ve assumption called label Missing Completely At Random ( MCAR ) . MNAR is inevitable in real-world SSL due to the limited label annotation budget ( Rosset et al. , 2005 ) —uniform label annotations that keep MCAR are expensive . For example , we usually deploy low-cost data collecting methods like crawling social media images from the Web ( Mahajan et al. , 2018 ) . However , the high-quality labels appear to be severely imbalanced over classes due to the imbalanced human preferences for the “ class ” ( Misra et al. , 2016 ; Colléony et al. , 2017 ) . For example , people are more willing to tag a “ Chihuahua ” rather than a “ leatherback ” ( a huge black turtle ) since the former is more lovely and easier to recognize . Thus , the imputation model trained by such biased supervision makes the model favoring “ Chihuahua ” rather than “ leatherback ” . The “ Chihuahua ” -favored pseudo-labels , in turn , negatively reinforce the model confidence in the false “ Chihuahua ” belief and thus exacerbate the confirmation bias . We design an experiment on CIFAR-10 to further illustrate how existing state-of-the-art SSL methods , e.g. , FixMatch ( Sohn et al. , 2020 ) , fail in the MNAR scenario . As shown in Figure 1 ( a ) , the overall training data is uniformly distributed over classes , but the labeled data is long-tailed distributed , which simulates the imbalanced class popularity . Trained on such MNAR training data , FixMatch even magnifies the bias towards the popular classes and ignores the rare classes ( Fig- ure 1 ( b ) ) . We point out that the devil is in the imputation . As shown in Figure 1 ( c ) , the confidence scores of FixMatch for popular classes are much higher than those for rare classes ( i.e. , averagely > 0.95 vs. < 0.05 ) . Since FixMatch adopts a fixed threshold for the imputation samples selection , i.e. , only the samples with confidence larger than 0.95 are imputed , FixMatch tends to impute the labels of more samples from the popular classes , which still keeps the updated labels with the newly added pseudo-labels long-tailed . As a result , FixMatch is easily trapped in MNAR . In fact , MNAR has attracted broad attention in statistics and Inverse Propensity Weighting ( IPW ) is one of the commonly used tools to tackle this challenge ( Seaman & White , 2013 ; Jones et al. , 2006 ) . IPW introduces a weight for each training sample via its propensity score , which reflects how likely the label is observed ( e.g. , its popularity ) . In this way , IPW makes up a pseudo-balanced dataset by duplicating each labeled data inversely proportional to its propensity—less popular samples should draw the same attention as the popular ones—a more balanced imputation . To combine the IPW true labels and imputed missing labels , a Doubly Robust ( DR ) estimator is used to guarantee a robust SSL model if either IPW or imputation is unbiased ( Seaman & Vansteelandt , 2018 ; Vansteelandt et al. , 2010 ) . However , the above framework overlooks the role of “ class ” in causing the MNAR problem of SSL ( Section 3 ) , and we therefore propose a unified Class-Aware Doubly Robust ( CADR ) framework to address this challenge ( Section 4 ) . CADR removes the bias from both the supervised model training and the unlabeled data imputation ends . For the former end , we propose Class-Aware Propensity ( CAP ) that exploits the unlabeled data to train an improved classifier using the biased labeled data ( Section 4.1 ) . We use an efficient moving averaging strategy to estimate the propensity . For the latter end , we design Class-Aware Imputation ( CAI ) that dynamically adjusts the pseudolabel assignment threshold with class-specific confidence scores to mitigate the bias in imputation ( Section 4.2 ) . Intuitively , CAI lowers the threshold requirement of rare class samples to balance with frequent class samples . Experiments on several image classification benchmarks demonstrate that our CADR framework achieves a consistent performance boost and can effectively tackle the MNAR problem . Besides , as MCAR is a special case of MNAR , our method maintains the competitive performance in the conventional SSL setting . The key contributions of this work are summarized as follows : • We propose a realistic and challenging label Missing Not At Random ( MNAR ) problem for semisupervised learning , which is not extensively studied in previous work . We systematically analyze the bias caused by non-random missing labels . • We proposes a unified doubly robust framework called Class-Aware Doubly Robust ( CADR ) estimator to remove the bias from both the supervised model training end—by using Class-Aware Prospesity ( CAP ) , and the unlabeled data imputation end—by using Class-Aware Imputation ( CAI ) . • Our proposed CADR achieves competitive performances in both MNAR and conventional label Missing Completely At Random ( MCAR ) . 2 RELATED WORKS . Missing Not At Random . Missing data problems are ubiquitous across the analysis in social , behavioral , and medical sciences ( Enders , 2010 ; Heckman , 1977 ) . When data is not missing at random , estimations based on the observed data only results in bias ( Heckman , 1977 ) . The solutions are three-fold : 1 ) Inverse probability weighting ( IPW ) assigns weights to each observed datum based on the propensity score , i.e. , the probability of being observed . A missing mechanism-based model is used to estimate the sample propensity , such as a logistic regression model ( Rosset et al. , 2005 ; Wang et al. , 1997 ) or a robit regression model ( Kang & Schafer , 2007 ; Liu , 2005 ) . 2 ) Imputation methods aim to fill in the missing values to produce a complete dataset ( Kenward & Carpenter , 2007 ; Little & Rubin , 2014 ) . An imputation model is regressed to predict the incomplete data from the observed data in a deterministic ( Kovar & Whitridge. , 1995 ) or stochastic ( Lo et al. , 2019 ) way . 3 ) As the propensity estimation or the regressed imputation are easy to be biased , doubly robust estimators propose to integrate IPW and imputation with double robustness ( Seaman & Vansteelandt , 2018 ; Vansteelandt et al. , 2010 ; Jonsson Funk et al. , 2011 ) : the capability to maintain unbiased when either the propensity or imputation model is biased . In this work , as the image labels are missing not at random , we design the class-aware propensity and imputation considering the causal role of label classes in missing , avoiding the uncertain model specification and generating unbiased estimation . These two class-aware modules can be naturally integrated into a doubly robust estimator . Semi-Supervised Learning ( SSL ) . It aims to exploit unlabeled data to improve the model learned on labeled data . Prevailing SSL methods ( Sohn et al. , 2020 ; Berthelot et al. , 2019b ; a ) share a similar strategy : training a model with the labeled data and generating pseudo-labels for unlabeled data based on the model predictions . Pseudo-labeling methods ( Lee , 2013 ; Xie et al. , 2020 ; Rizve et al. , 2021 ) predict pseudo-labels for unlabeled data and add them to the training data for re-training . Consistency-regularization methods ( Sajjadi et al. , 2016 ; Laine & Aila , 2016 ; Berthelot et al. , 2019b ) apply a random perturbation to an unlabeled image and then use the prediction as the pseudo-label of the same image under a different augmentation . Recent state-of-the-art SSL methods ( Sohn et al. , 2020 ; Berthelot et al. , 2019a ) combine the two existing techniques and predict improved pseudolabels . However , when labels are missing at random , these methods can be inefficient as the model learned on labeled data is biased and significantly harms the quality of pseudo labels . Though some works also notice the model bias ( Wei et al. , 2021 ; Kim et al. , 2020 ) , they neglect the causal relationship between the bias and the missing label mechanism in SSL , which is systematically discussed in our work . 3 MISSING LABELS IN SEMI-SUPERVISED LEARNING . In semi-supervised learning ( SSL ) , the training dataset D is divided into two disjoint sets : a labeled dataset DL and an unlabeled data DU . We denote DL as { ( x ( i ) , y ( i ) ) } NLi=1 ( usually NL≪N ) where x ( i ) ∈ Rd is a sample feature and y ( i ) ∈ { 0 , 1 } C is its one-hot label over C classes , and DU as { ( x ( i ) ) } Ni=NL+1 where the remaining N−NL labels are missing in DU . We further review SSL as a label missing problem , and define a label missing indicator set M with m ( i ) ∈ { 0 , 1 } , where m ( i ) =1 denotes the label is missing ( i.e. , unlabeled ) and m ( i ) =0 denotes the label is not missing ( i.e. , labeled ) . In this way , we can rewrite the dataset as D= ( X , Y , M ) = { ( x ( i ) , y ( i ) , m ( i ) ) } Ni=1 . Traditional SSL methods assume that M is independent with Y , i.e. , the labels are Missing Completely At Random ( MCAR ) . Under this assumption , prevailing SSL algorithms can be summarized as the following multi-task optimization of the supervised learning and the unlabeled imputation : θ̂ = argmin θ ∑ ( x , y ) ∈DL Ls ( x , y ; θ ) + ∑ x∈DU Lu ( x ; θ ) , ( 1 ) where Ls and Lu are the loss function designed separately for supervised learning on labeled data ( M =0 ) and regression imputation on unlabeled data ( M =1 ) . In general , SSL methods first train a model with parameters θ̂ using Ls on DL , then impute the missing labels ( Van Buuren , 2018 ) by predicting the pseudo-labels ŷ = f ( x ; θ̂ ) , and finally combine the true- and pseudo-labels to further improve the model for another SSL loop , e.g. , another optimization iteration of Eq . ( 1 ) . The implementation of Ls and Lu are open . For example , Ls is normally the standard cross-entropy loss , and Lu can be implemented as squared L2 loss ( Berthelot et al. , 2019b ) or cross-entropy loss ( Berthelot et al. , 2019a ; Sohn et al. , 2020 ) . As you may expect , the key to stable SSL methods is the unbiasedness of the imputation model . When M is independent with Y , we have P ( Y |X=x , M=0 ) = P ( Y |X=x ) , and thus E [ ŷ ] = E [ y|θ̂ ] = ∑ ( x , y ) ∈DL y · P ( y|x ) = ∑ ( x , y ) ∈D y · P ( y|x , M = 0 ) = ∑ ( x , y ) ∈D y · P ( y|x ) = E [ y ] , ( 2 ) which indicates that the model is an ideally unbiased estimator , and the imputed labels are unbiased . However , such MCAR assumption is too strong and impractical . For example , annotators may tend to tag a specific group of samples due to their preferences or experiences . This motivates us to ask the following realistic but overlooked question : what if M is dependent with Y , i.e. , the labels are missing not at random ( MNAR ) ? In this case , we have P ( Y |X = x , M = 0 ) ̸= P ( Y |X = x ) , and thus E [ ŷ ] = ∑ ( x , y ) ∈D y · P ( y|x , M = 0 ) = ∑ ( x , y ) ∈D y · P ( y|x ) · P ( M = 0|x , y ) P ( M = 0|x ) ̸= ∑ ( x , y ) ∈D y · P ( y|x ) = E [ y ] . ( 3 ) We can see it is no longer unbiased , further leading to biased imputed labels . As a result , the bias will be increasingly enhanced with the joint optimization loop in Eq . ( 1 ) moving forward . This analysis indicates that previous SSL methods are not applicable for the MNAR problem as they overlook the role of “ class ” Y in causing the non-randomness of M . In the following , we will introduce our Class-Aware Doubly Robust framework that targets at an unbiased estimator .

Summary: This paper formulates the problem of semi-supervised learning where the likelihood of an example being labeled is dependent on the class of the example. Given this assumption, the authors propose an approach to mitigate the resulting distribution shifts that occur in both the labeled and unlabeled data sets. Finally, experimental results are shown for the class-dependent labeling scenario, with the proposed algorithm outperforming standard SSL approaches that assume random labeling probability.