Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data

1 INTRODUCTION . Though supervised learning with neural networks has become standard and reliable , it still often requires massive labeled datasets . As labels can be expensive or difficult to obtain , leveraging unlabeled data in deep learning has become an active research area . Recent works in semi-supervised learning ( Chapelle et al. , 2010 ; Kingma et al. , 2014 ; Kipf & Welling , 2016 ; Laine & Aila , 2016 ; Sohn et al. , 2020 ; Xie et al. , 2020 ) and unsupervised domain adaptation ( Ben-David et al. , 2010 ; Ganin & Lempitsky , 2015 ; Ganin et al. , 2016 ; Tzeng et al. , 2017 ; Hoffman et al. , 2018 ; Shu et al. , 2018 ; Zhang et al. , 2019 ) leverage lots of unlabeled data as well as labeled data from the same distribution or a related distribution . Recent progress in unsupervised learning or representation learning ( Hinton et al. , 1999 ; Doersch et al. , 2015 ; Gidaris et al. , 2018 ; Misra & Maaten , 2020 ; Chen et al. , 2020a ; b ; Grill et al. , 2020 ) learns high-quality representations without using any labels . Self-training is a common algorithmic paradigm for leveraging unlabeled data with deep networks . Self-training methods train a model to fit pseudolabels , that is , predictions on unlabeled data made by a previously-learned model ( Yarowsky , 1995 ; Grandvalet & Bengio , 2005 ; Lee , 2013 ) . Recent work also extends these methods to enforce stability of predictions under input transformations such as adversarial perturbations ( Miyato et al. , 2018 ) and data augmentation ( Xie et al. , 2019 ) . These approaches , known as input consistency regularization , have been successful in semi-supervised learning ( Sohn et al. , 2020 ; Xie et al. , 2020 ) , unsupervised domain adaptation ( French et al. , 2017 ; Shu et al. , 2018 ) , and unsupervised learning ( Hu et al. , 2017 ; Grill et al. , 2020 ) . Despite the empirical successes , theoretical progress in understanding how to use unlabeled data has lagged . Whereas supervised learning is relatively well-understood , statistical tools for reasoning about unlabeled data are not as readily available . Around 25 years ago , Vapnik ( 1995 ) proposed the transductive SVM for unlabeled data , which can be viewed as an early version of self-training , yet there is little work showing that this method improves sample complexity ( Derbeko et al. , 2004 ) . Working with unlabeled data requires proper assumptions on the input distribution ( Ben-David et al. , 2008 ) . Recent papers ( Carmon et al. , 2019 ; Raghunathan et al. , 2020 ; Chen et al. , 2020c ; Kumar et al. , 2020 ; Oymak & Gulcu , 2020 ) analyze self-training in various settings , but mainly for linear models and often require that the data is Gaussian or near-Gaussian . Kumar et al . ( 2020 ) also analyze self-training in a setting where gradual domain shift occurs over multiple timesteps but assume a small Wasserstein distance bound on the shift between consecutive timesteps . Another line of work leverages unlabeled data using non-parametric methods , requiring unlabeled sample complexity that is exponential in dimension ( Rigollet , 2007 ; Singh et al. , 2009 ; Urner & Ben-David , 2013 ) . This paper provides a unified theoretical analysis of self-training with deep networks for semisupervised learning , unsupervised domain adaptation , and unsupervised learning . Under a simple and realistic expansion assumption on the data distribution , we show that self-training with input consistency regularization using a deep network can achieve high accuracy on true labels , using unlabeled sample size that is polynomial in the margin and Lipschitzness of the model . Our analysis provides theoretical intuition for recent empirically successful self-training algorithms which rely on input consistency regularization ( Berthelot et al. , 2019 ; Sohn et al. , 2020 ; Xie et al. , 2020 ) . Our expansion assumption intuitively states that the data distribution has good continuity within each class . Concretely , letting Pi be the distribution of data conditioned on class i , expansion states that for small subset S of examples with class i , Pi ( neighborhood of S ) ≥ cPi ( S ) ( 1.1 ) where and c > 1 is the expansion factor . The neighborhood will be defined to incorporate data augmentation , but for now can be thought of as a collection of points with a small ` 2 distance to S. This notion is an extension of the Cheeger constant ( or isoperimetric or expansion constant ) ( Cheeger , 1969 ) which has been studied extensively in graph theory ( Chung & Graham , 1997 ) , combinatorial optimization ( Mohar & Poljak , 1993 ; Raghavendra & Steurer , 2010 ) , sampling ( Kannan et al. , 1995 ; Lovász & Vempala , 2007 ; Zhang et al. , 2017 ) , and even in early versions of self-training ( Balcan et al. , 2005 ) for the co-training setting ( Blum & Mitchell , 1998 ) . Expansion says that the manifold of each class has sufficient connectivity , as every subset S has a neighborhood larger than S. We give examples of distributions satisfying expansion in Section 3.1 . We also require a separation condition stating that there are few neighboring pairs from different classes . Our algorithms leverage expansion by using input consistency regularization ( Miyato et al. , 2018 ; Xie et al. , 2019 ) to encourage predictions of a classifierG to be consistent on neighboring examples : R ( G ) = Ex [ max neighbor x′ 1 ( G ( x ) 6= G ( x′ ) ) ] ( 1.2 ) For unsupervised domain adaptation and semi-supervised learning , we analyze an algorithm which fits G to pseudolabels on unlabeled data while regularizing input consistency . Assuming expansion and separation , we prove that the fitted model will denoise the pseudolabels and achieve high accuracy on the true labels ( Theorem 4.3 ) . This explains the empirical phenomenon that self-training on pseudolabels often improves over the pseudolabeler , despite no access to true labels . For unsupervised learning , we consider finding a classifier G that minimizes the input consistency regularizer with the constraint that enough examples are assigned each label . In Theorem 3.6 , we show that assuming expansion and separation , the learned classifier will have high accuracy in predicting true classes , up to a permutation of the labels ( which can ’ t be recovered without true labels ) . The main intuition of the theorems is as follows : input consistency regularization ensures that the model is locally consistent , and the expansion property magnifies the local consistency to global consistency within the same class . In the unsupervised domain adaptation setting , as shown in Figure 1 ( right ) , the incorrectly pseudolabeled examples ( the red area ) are gradually denoised by their correctly pseudolabeled neighbors ( the green area ) , whose probability mass is non-trivial ( at least c−1 times the mass of the mistaken set by expansion ) . We note that expansion is only required on the population distribution , but self-training is performed on the empirical samples . Due to the extrapolation power of parametric methods , the local-to-global consistency effect of expansion occurs implicitly on the population . In contrast , nearest-neighbor methods would require expansion to occur explicitly on empirical samples , suffering the curse of dimensionality as a result . We provide more details below , and visualize this effect in Figure 1 ( left ) . To our best knowledge , this paper gives the first analysis with polynomial sample complexity guarantees for deep neural net models for unsupervised learning , semi-supervised learning , and unsuper- vised domain adaptation . Prior works ( Rigollet , 2007 ; Singh et al. , 2009 ; Urner & Ben-David , 2013 ) analyzed nonparametric methods that essentially recover the data distribution exactly with unlabeled data , but require sample complexity exponential in dimension . Our approach optimizes parametric loss functions and regularizers , so guarantees involving the population loss can be converted to finite sample results using off-the-shelf generalization bounds ( Theorem 3.7 ) . When a neural net can separate ground-truth classes with large margin , the sample complexities from these bounds can be small , that is , polynomial in dimension . Finally , we note that our regularizer R ( · ) corresponds to enforcing consistency w.r.t . adversarial examples , which was shown to be empirically helpful for semi-supervised learning ( Miyato et al. , 2018 ; Qiao et al. , 2018 ) and unsupervised domain adaptation ( Shu et al. , 2018 ) . Moreover , we can extend the notion of neighborhood in ( 1.1 ) to include data augmentations of examples , which will increase the neighborhood size and therefore improve the expansion . Thus , our theory can help explain empirical observations that consistency regularization based on aggressive data augmentation or adversarial training can improve performance with unlabeled data ( Shu et al. , 2018 ; Xie et al. , 2019 ; Berthelot et al. , 2019 ; Sohn et al. , 2020 ; Xie et al. , 2020 ; Chen et al. , 2020a ) . In summary , our contributions include : 1 ) we propose a simple and realistic expansion assumption which states that the data distribution has connectivity within the manifold of a ground-truth class 2 ) using this expansion assumption , we provide ground-truth accuracy guarantees for self-training algorithms which regularize input consistency on unlabeled data , and 3 ) our analysis is easily applicable to deep networks with polynomial unlabeled samples via off-the-shelf generalization bounds . 1.1 ADDITIONAL RELATED WORK . Self-training via pseudolabeling ( Lee , 2013 ) or min-entropy objectives ( Grandvalet & Bengio , 2005 ) has been widely used in both semi-supervised learning ( Laine & Aila , 2016 ; Tarvainen & Valpola , 2017 ; Iscen et al. , 2019 ; Yalniz et al. , 2019 ; Berthelot et al. , 2019 ; Xie et al. , 2020 ; Sohn et al. , 2020 ) and unsupervised domain adaptation ( Long et al. , 2013 ; French et al. , 2017 ; Saito et al. , 2017 ; Shu et al. , 2018 ; Zou et al. , 2019 ) . Our paper studies input consistency regularization , which enforces stability of the prediction w.r.t transformations of the unlabeled data . In practice , these transformations include adversarial perturbations , which was proposed as the VAT objective ( Miyato et al. , 2018 ) , as well as data augmentations ( Xie et al. , 2019 ) . For unsupervised learning , our self-training objective is closely related to BYOL ( Grill et al. , 2020 ) , a recent state-of-the-art method which trains a student model to match the representations predicted by a teacher model on strongly augmented versions of the input . Contrastive learning is another popular method for unsupervised representation learning which encourages representations of “ positive pairs ” , ideally consisting of examples from the same class , to be close , while pushing negative pairs far apart ( Mikolov et al. , 2013 ; Oord et al. , 2018 ; Arora et al. , 2019 ) . Recent works in contrastive learning achieve state-of-the-art representation quality by using strong data augmentation to form positive pairs ( Chen et al. , 2020a ; b ) . The role of data augmentation here is in spirit similar to our use of input consistency regularization . Less related to our setting are algorithms which learn representations by solving self-supervised pretext tasks , such as inpainting and predicting rotations ( Pathak et al. , 2016 ; Noroozi & Favaro , 2016 ; Gidaris et al. , 2018 ) . Lee et al . ( 2020 ) theoretically analyze self-supervised learning , but their analysis applies to a different class of algorithms than ours . Prior theoretical works analyze contrastive learning by assuming access to document data distributed according to a particular topic modeling setup ( Tosh et al. , 2020 ) or pairs of independent samples within the same class ( Arora et al. , 2019 ) . However , the assumptions required for these analyses do not necessarily apply to vision , where positive pairs apply different data augmentations to the same image , and are therefore strongly correlated . Other papers analyze information-theoretic properties of representation learning ( Tian et al. , 2020 ; Tsai et al. , 2020 ) . Prior works analyze continuity or “ cluster ” assumptions for semi-supervised learning which are related to our notion of expansion ( Seeger , 2000 ; Rigollet , 2007 ; Singh et al. , 2009 ; Urner & BenDavid , 2013 ) . However , these papers leverage unlabeled data using non-parametric methods , requiring unlabeled sample complexity that is exponential in the dimension . On the other hand , our analysis is for parametric methods , and therefore the unlabeled sample complexity can be low when a neural net can separate the ground-truth classes with large margin . Co-training is a classical version of self-training which requires two distinct “ views ” ( i.e. , feature subsets ) of the data , each of which can be used to predict the true label on its own ( Blum & Mitchell , 1998 ; Dasgupta et al. , 2002 ; Balcan et al. , 2005 ) . For example , to predict the topic of a webpage , one view could be the incoming links and another view could be the words in the page . The original co-training algorithms ( Blum & Mitchell , 1998 ; Dasgupta et al. , 2002 ) assume that the two views are independent conditioned on the true label and leverage this independence to obtain accurate pseudolabels for the unlabeled data . By contrast , if we cast our setting into the co-training framework by treating an example and a randomly sampled neighbor as the two views of the data , the two views are highly correlated . Balcan et al . ( 2005 ) relax the requirement on independent views of co-training , also by using an “ expansion ” assumption . Our assumption is closely related to theirs and conceptually equivalent if we cast our setting into the co-training framework by treating neighboring examples are two views . However , their analysis requires confident pseudolabels to all be accurate and does not rigorously account for potential propagation of errors from their algorithm . In contrast , our contribution is to propose and analyze an objective function involving input consistency regularization whose minimizer denoises errors from potentially incorrect pseudolabels . We also provide finite sample complexity bounds for the neural network hypothesis class and analyze unsupervised learning algorithms . Alternative theoretical analyses of unsupervised domain adaptation assume bounded measures of discrepancy between source and target domains ( Ben-David et al. , 2010 ; Zhang et al. , 2019 ) . Balcan & Blum ( 2010 ) propose a PAC-style framework for analyzing semi-supervised learning , but their bounds require the user to specify a notion of compatability which incorporates prior knowledge about the data , and do not apply to domain adaptation . Globerson et al . ( 2017 ) demonstrate semisupervised learning can outperform supervised learning in labeled sample complexity but assume full knowledge of the unlabeled distribution . ( Mobahi et al. , 2020 ) show that for kernel methods , self-distillation , a variant of self-training , can effectively amplify regularization . Their analysis is for kernel methods , whereas our analysis applies to deep networks under data assumptions .

Summary: This work provides a unified framework to analyze the self-training, semi-supervised algorithms. The key assumptions are 1) the “expansion” assumption which characterizes the low-probability data subset must expand to a neighborhood with large probability; and 2) the neighborhoods of samples from different classes have small overlap.  Then the authors established the upper bound of the prediction error on the population when minimizing the self-training and input-consistency based loss on the population. They also extend their results to a finite-sample setting and semi-supervised setting as well.