Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data

1 INTRODUCTION . Though supervised learning with neural networks has become standard and reliable , it still often requires massive labeled datasets . As labels can be expensive or difficult to obtain , leveraging unlabeled data in deep learning has become an active research area . Recent works in semi-supervised learning ( Chapelle et al. , 2010 ; Kingma et al. , 2014 ; Kipf & Welling , 2016 ; Laine & Aila , 2016 ; Sohn et al. , 2020 ; Xie et al. , 2020 ) and unsupervised domain adaptation ( Ben-David et al. , 2010 ; Ganin & Lempitsky , 2015 ; Ganin et al. , 2016 ; Tzeng et al. , 2017 ; Hoffman et al. , 2018 ; Shu et al. , 2018 ; Zhang et al. , 2019 ) leverage lots of unlabeled data as well as labeled data from the same distribution or a related distribution . Recent progress in unsupervised learning or representation learning ( Hinton et al. , 1999 ; Doersch et al. , 2015 ; Gidaris et al. , 2018 ; Misra & Maaten , 2020 ; Chen et al. , 2020a ; b ; Grill et al. , 2020 ) learns high-quality representations without using any labels . Self-training is a common algorithmic paradigm for leveraging unlabeled data with deep networks . Self-training methods train a model to fit pseudolabels , that is , predictions on unlabeled data made by a previously-learned model ( Yarowsky , 1995 ; Grandvalet & Bengio , 2005 ; Lee , 2013 ) . Recent work also extends these methods to enforce stability of predictions under input transformations such as adversarial perturbations ( Miyato et al. , 2018 ) and data augmentation ( Xie et al. , 2019 ) . These approaches , known as input consistency regularization , have been successful in semi-supervised learning ( Sohn et al. , 2020 ; Xie et al. , 2020 ) , unsupervised domain adaptation ( French et al. , 2017 ; Shu et al. , 2018 ) , and unsupervised learning ( Hu et al. , 2017 ; Grill et al. , 2020 ) . Despite the empirical successes , theoretical progress in understanding how to use unlabeled data has lagged . Whereas supervised learning is relatively well-understood , statistical tools for reasoning about unlabeled data are not as readily available . Around 25 years ago , Vapnik ( 1995 ) proposed the transductive SVM for unlabeled data , which can be viewed as an early version of self-training , yet there is little work showing that this method improves sample complexity ( Derbeko et al. , 2004 ) . Working with unlabeled data requires proper assumptions on the input distribution ( Ben-David et al. , 2008 ) . Recent papers ( Carmon et al. , 2019 ; Raghunathan et al. , 2020 ; Chen et al. , 2020c ; Kumar et al. , 2020 ; Oymak & Gulcu , 2020 ) analyze self-training in various settings , but mainly for linear models and often require that the data is Gaussian or near-Gaussian . Kumar et al . ( 2020 ) also analyze self-training in a setting where gradual domain shift occurs over multiple timesteps but assume a small Wasserstein distance bound on the shift between consecutive timesteps . Another line of work leverages unlabeled data using non-parametric methods , requiring unlabeled sample complexity that is exponential in dimension ( Rigollet , 2007 ; Singh et al. , 2009 ; Urner & Ben-David , 2013 ) . This paper provides a unified theoretical analysis of self-training with deep networks for semisupervised learning , unsupervised domain adaptation , and unsupervised learning . Under a simple and realistic expansion assumption on the data distribution , we show that self-training with input consistency regularization using a deep network can achieve high accuracy on true labels , using unlabeled sample size that is polynomial in the margin and Lipschitzness of the model . Our analysis provides theoretical intuition for recent empirically successful self-training algorithms which rely on input consistency regularization ( Berthelot et al. , 2019 ; Sohn et al. , 2020 ; Xie et al. , 2020 ) . Our expansion assumption intuitively states that the data distribution has good continuity within each class . Concretely , letting Pi be the distribution of data conditioned on class i , expansion states that for small subset S of examples with class i , Pi ( neighborhood of S ) ≥ cPi ( S ) ( 1.1 ) where and c > 1 is the expansion factor . The neighborhood will be defined to incorporate data augmentation , but for now can be thought of as a collection of points with a small ` 2 distance to S. This notion is an extension of the Cheeger constant ( or isoperimetric or expansion constant ) ( Cheeger , 1969 ) which has been studied extensively in graph theory ( Chung & Graham , 1997 ) , combinatorial optimization ( Mohar & Poljak , 1993 ; Raghavendra & Steurer , 2010 ) , sampling ( Kannan et al. , 1995 ; Lovász & Vempala , 2007 ; Zhang et al. , 2017 ) , and even in early versions of self-training ( Balcan et al. , 2005 ) for the co-training setting ( Blum & Mitchell , 1998 ) . Expansion says that the manifold of each class has sufficient connectivity , as every subset S has a neighborhood larger than S. We give examples of distributions satisfying expansion in Section 3.1 . We also require a separation condition stating that there are few neighboring pairs from different classes . Our algorithms leverage expansion by using input consistency regularization ( Miyato et al. , 2018 ; Xie et al. , 2019 ) to encourage predictions of a classifierG to be consistent on neighboring examples : R ( G ) = Ex [ max neighbor x′ 1 ( G ( x ) 6= G ( x′ ) ) ] ( 1.2 ) For unsupervised domain adaptation and semi-supervised learning , we analyze an algorithm which fits G to pseudolabels on unlabeled data while regularizing input consistency . Assuming expansion and separation , we prove that the fitted model will denoise the pseudolabels and achieve high accuracy on the true labels ( Theorem 4.3 ) . This explains the empirical phenomenon that self-training on pseudolabels often improves over the pseudolabeler , despite no access to true labels . For unsupervised learning , we consider finding a classifier G that minimizes the input consistency regularizer with the constraint that enough examples are assigned each label . In Theorem 3.6 , we show that assuming expansion and separation , the learned classifier will have high accuracy in predicting true classes , up to a permutation of the labels ( which can ’ t be recovered without true labels ) . The main intuition of the theorems is as follows : input consistency regularization ensures that the model is locally consistent , and the expansion property magnifies the local consistency to global consistency within the same class . In the unsupervised domain adaptation setting , as shown in Figure 1 ( right ) , the incorrectly pseudolabeled examples ( the red area ) are gradually denoised by their correctly pseudolabeled neighbors ( the green area ) , whose probability mass is non-trivial ( at least c−1 times the mass of the mistaken set by expansion ) . We note that expansion is only required on the population distribution , but self-training is performed on the empirical samples . Due to the extrapolation power of parametric methods , the local-to-global consistency effect of expansion occurs implicitly on the population . In contrast , nearest-neighbor methods would require expansion to occur explicitly on empirical samples , suffering the curse of dimensionality as a result . We provide more details below , and visualize this effect in Figure 1 ( left ) . To our best knowledge , this paper gives the first analysis with polynomial sample complexity guarantees for deep neural net models for unsupervised learning , semi-supervised learning , and unsuper- vised domain adaptation . Prior works ( Rigollet , 2007 ; Singh et al. , 2009 ; Urner & Ben-David , 2013 ) analyzed nonparametric methods that essentially recover the data distribution exactly with unlabeled data , but require sample complexity exponential in dimension . Our approach optimizes parametric loss functions and regularizers , so guarantees involving the population loss can be converted to finite sample results using off-the-shelf generalization bounds ( Theorem 3.7 ) . When a neural net can separate ground-truth classes with large margin , the sample complexities from these bounds can be small , that is , polynomial in dimension . Finally , we note that our regularizer R ( · ) corresponds to enforcing consistency w.r.t . adversarial examples , which was shown to be empirically helpful for semi-supervised learning ( Miyato et al. , 2018 ; Qiao et al. , 2018 ) and unsupervised domain adaptation ( Shu et al. , 2018 ) . Moreover , we can extend the notion of neighborhood in ( 1.1 ) to include data augmentations of examples , which will increase the neighborhood size and therefore improve the expansion . Thus , our theory can help explain empirical observations that consistency regularization based on aggressive data augmentation or adversarial training can improve performance with unlabeled data ( Shu et al. , 2018 ; Xie et al. , 2019 ; Berthelot et al. , 2019 ; Sohn et al. , 2020 ; Xie et al. , 2020 ; Chen et al. , 2020a ) . In summary , our contributions include : 1 ) we propose a simple and realistic expansion assumption which states that the data distribution has connectivity within the manifold of a ground-truth class 2 ) using this expansion assumption , we provide ground-truth accuracy guarantees for self-training algorithms which regularize input consistency on unlabeled data , and 3 ) our analysis is easily applicable to deep networks with polynomial unlabeled samples via off-the-shelf generalization bounds . 1.1 ADDITIONAL RELATED WORK . Self-training via pseudolabeling ( Lee , 2013 ) or min-entropy objectives ( Grandvalet & Bengio , 2005 ) has been widely used in both semi-supervised learning ( Laine & Aila , 2016 ; Tarvainen & Valpola , 2017 ; Iscen et al. , 2019 ; Yalniz et al. , 2019 ; Berthelot et al. , 2019 ; Xie et al. , 2020 ; Sohn et al. , 2020 ) and unsupervised domain adaptation ( Long et al. , 2013 ; French et al. , 2017 ; Saito et al. , 2017 ; Shu et al. , 2018 ; Zou et al. , 2019 ) . Our paper studies input consistency regularization , which enforces stability of the prediction w.r.t transformations of the unlabeled data . In practice , these transformations include adversarial perturbations , which was proposed as the VAT objective ( Miyato et al. , 2018 ) , as well as data augmentations ( Xie et al. , 2019 ) . For unsupervised learning , our self-training objective is closely related to BYOL ( Grill et al. , 2020 ) , a recent state-of-the-art method which trains a student model to match the representations predicted by a teacher model on strongly augmented versions of the input . Contrastive learning is another popular method for unsupervised representation learning which encourages representations of “ positive pairs ” , ideally consisting of examples from the same class , to be close , while pushing negative pairs far apart ( Mikolov et al. , 2013 ; Oord et al. , 2018 ; Arora et al. , 2019 ) . Recent works in contrastive learning achieve state-of-the-art representation quality by using strong data augmentation to form positive pairs ( Chen et al. , 2020a ; b ) . The role of data augmentation here is in spirit similar to our use of input consistency regularization . Less related to our setting are algorithms which learn representations by solving self-supervised pretext tasks , such as inpainting and predicting rotations ( Pathak et al. , 2016 ; Noroozi & Favaro , 2016 ; Gidaris et al. , 2018 ) . Lee et al . ( 2020 ) theoretically analyze self-supervised learning , but their analysis applies to a different class of algorithms than ours . Prior theoretical works analyze contrastive learning by assuming access to document data distributed according to a particular topic modeling setup ( Tosh et al. , 2020 ) or pairs of independent samples within the same class ( Arora et al. , 2019 ) . However , the assumptions required for these analyses do not necessarily apply to vision , where positive pairs apply different data augmentations to the same image , and are therefore strongly correlated . Other papers analyze information-theoretic properties of representation learning ( Tian et al. , 2020 ; Tsai et al. , 2020 ) . Prior works analyze continuity or “ cluster ” assumptions for semi-supervised learning which are related to our notion of expansion ( Seeger , 2000 ; Rigollet , 2007 ; Singh et al. , 2009 ; Urner & BenDavid , 2013 ) . However , these papers leverage unlabeled data using non-parametric methods , requiring unlabeled sample complexity that is exponential in the dimension . On the other hand , our analysis is for parametric methods , and therefore the unlabeled sample complexity can be low when a neural net can separate the ground-truth classes with large margin . Co-training is a classical version of self-training which requires two distinct “ views ” ( i.e. , feature subsets ) of the data , each of which can be used to predict the true label on its own ( Blum & Mitchell , 1998 ; Dasgupta et al. , 2002 ; Balcan et al. , 2005 ) . For example , to predict the topic of a webpage , one view could be the incoming links and another view could be the words in the page . The original co-training algorithms ( Blum & Mitchell , 1998 ; Dasgupta et al. , 2002 ) assume that the two views are independent conditioned on the true label and leverage this independence to obtain accurate pseudolabels for the unlabeled data . By contrast , if we cast our setting into the co-training framework by treating an example and a randomly sampled neighbor as the two views of the data , the two views are highly correlated . Balcan et al . ( 2005 ) relax the requirement on independent views of co-training , also by using an “ expansion ” assumption . Our assumption is closely related to theirs and conceptually equivalent if we cast our setting into the co-training framework by treating neighboring examples are two views . However , their analysis requires confident pseudolabels to all be accurate and does not rigorously account for potential propagation of errors from their algorithm . In contrast , our contribution is to propose and analyze an objective function involving input consistency regularization whose minimizer denoises errors from potentially incorrect pseudolabels . We also provide finite sample complexity bounds for the neural network hypothesis class and analyze unsupervised learning algorithms . Alternative theoretical analyses of unsupervised domain adaptation assume bounded measures of discrepancy between source and target domains ( Ben-David et al. , 2010 ; Zhang et al. , 2019 ) . Balcan & Blum ( 2010 ) propose a PAC-style framework for analyzing semi-supervised learning , but their bounds require the user to specify a notion of compatability which incorporates prior knowledge about the data , and do not apply to domain adaptation . Globerson et al . ( 2017 ) demonstrate semisupervised learning can outperform supervised learning in labeled sample complexity but assume full knowledge of the unlabeled distribution . ( Mobahi et al. , 2020 ) show that for kernel methods , self-distillation , a variant of self-training , can effectively amplify regularization . Their analysis is for kernel methods , whereas our analysis applies to deep networks under data assumptions .

Summary: of the paper: The paper gives a theoretical justification of self-training. It proposes a new notion of "expansion" - the amount of data distribution in the neighbor of an example. Here the neighbor means adding perturbations to the example, or augmentations of the example. When the label distribution satisfies nice expansion properties and that classes are properly separated according to the neighbors, the paper proves distributional guarantees of self-training. Combining with generalization bounds of DNNs, the paper also derives finite sample bounds for DNNs. The paper also verifies the expansion assumption via experiments using a GAN.