How unlabeled data improve generalization in self-training? A one-hidden-layer theoretical analysis

√ M , where M is the number of unlabeled samples . Experiments from shallow neural networks to deep neural networks are also provided to justify the correctness of our established theoretical insights on self-training . 1 INTRODUCTION . Self-training ( Scudder , 1965 ; Yarowsky , 1995 ; Lee et al. , 2013 ; Han et al. , 2019 ) , one of the most powerful semi-supervised learning ( SemiSL ) algorithms , augments a limited number of labeled data with unlabeled data so as to achieve improved generalization performance on test data , compared with the model trained by supervised learning using the labeled data only . Self-training has shown empirical success in diversified applications such as few-shot image classification ( Su et al. , 2020 ; Xie et al. , 2020 ; Chen et al. , 2020a ; Yalniz et al. , 2019 ; Zoph et al. , 2020 ) , objective detection ( Rosenberg et al. , 2005 ) , robustness-aware model training against adversarial attacks ( Carmon et al. , 2019 ) , continual lifelong learning ( Lee et al. , 2019 ) , and natural language processing ( He et al. , 2019 ; Kahn et al. , 2020 ) . The terminology “ self-training ” has been used to describe various SemiSL algorithms in the literature , while this paper is centered on the commonly-used iterative self-training method in particular . In this setup , an initial teacher model ( learned from the labeled data ) is applied to the unlabeled data to generate pseudo labels . One then trains a student model by minimizing the weighted empirical risk of both the labeled and unlabeled data . The student model is then used as the new teacher to update the pseudo labels of the unlabeled data . This process is repeated multiple times to improve the eventual student model . We refer readers to Section 2 for algorithmic details . Despite the empirical achievement of self-training methods with neural networks , the theoretical justification of such success is very limited , even in the field of SemiSL . The majority of the theoretical results on general SemiSL are limited to linear networks ( Chen et al. , 2020b ; Raghunathan et al. , 2020 ; Oymak & Gulcu , 2020 ; Oneto et al. , 2011 ) . The authors in ( Balcan & Blum , 2010 ) show that unlabeled data can improve the generalization bound if the unlabeled data distribution and target model are compatible . For instance , the unlabeled data need to be well-chosen such that the target function for labeled data can separate the unlabeled data clusters , which , however , may not be able to be verified ahead . Moreover , ( Rigollet , 2007 ; Singh et al. , 2008 ) proves that unlabeled data can improve the convergence rate and generalization error under a similar clustering assumption , where the data contains clusters that have homogeneous labels . A recent work by Wei et al . ( 2020 ) analyzes SemiSL on nonlinear neural networks and proves that an infinite number of unlabeled data can improve the generalization compared with training with labeled data only . However , Wei et al . ( 2020 ) considers single shot rather than iterative SemiSL , and the training problem aims to minimize the consistency regularization rather than the risk function in the conventional self-training method ( Lee et al. , 2013 ) . Moreover , Wei et al . ( 2020 ) directly analyzes the global optimum of the nonconvex training problem without any discussion about how to achieve the global optimum . To the best of our knowledge , there exists no analytical characterization of how the unlabeled data affect the generalization of the learned model by iterative self-training on nonlinear neural networks . Contributions . This paper provides the first theoretical study of iterative self-training on nonlinear neural networks . Focusing on one-hidden-layer neural networks , this paper provides a quantitative analysis of the generalization performance of iterative self-training as a function of the number of labeled and unlabeled samples . Specifically , our contributions include 1 . Quantitative justification of generalization improvement by unlabeled data . Assuming the existence of a ground-truth model with weights W ∗ that maps the features to the corresponding labels , we prove that the learned model via iterative self-training moves closer to W ∗ as the number M of unlabeled data increases , indicating a better testing performance . Specifically , we prove that the Frobenius distance to W ∗ , which is approximately linear in the generalization error , decreases in the order of 1/ √ M . As an example , Figure 1 shows that the proposed theoretical bound matches the empirical self-training performance versus the number of unlabeled data for image classification ; see details in Section 4.2 . 2 . Analytical justification of iterative self-training over single shot alternative . We prove that the student models returned by the iterative self-training method converges linearly to a model close toW ∗ , with the rate improvement in the order of 1/ √ M . 3 . Sample complexity analysis of labeled and unlabeled data for learning a proper model .. We quantify the impact of labeled and unlabeled data on the generalization of the learned model . In particular , we prove that the sample complexity of labeled data can be reduced compared with supervised learning . 1.1 RELATED WORKS . Semi-supervised learning . Besides self-training , many recent SemiSL algorithms exploit either consistency regularization or entropy minimization . Consistency regularization is based on the as- sumption that the learned model will return same or similar output when the input is perturbed ( Laine & Aila , 2016 ; Bachman et al. , 2014 ; Sajjadi et al. , 2016 ; Tarvainen & Valpola , 2017 ; Reed et al. , 2015 ) . ( Grandvalet & Bengio , 2005 ) claims that the unlabeled data are more informative if the pseudo labels of the unlabeled data have lower entropy . Therefore , a line of works ( Grandvalet & Bengio , 2005 ; Miyato et al. , 2018 ) adds a regularization term that minimizes the entropy of the outputs of the unlabeled data . In addition , hybrid algorithms that unify both the above regularizations have been developed like ( Berthelot et al. , 2019a ; b ; Sohn et al. , 2020 ) . Domain adaptation . Domain adaptation exploits abundant data in the source domain to learn a model for the target domain , where only limited training data are available ( Liebelt & Schmid , 2010 ; Vazquez et al. , 2013 ; Zhang et al. , 2013 ; Long et al. , 2015 ; Tzeng et al. , 2014 ) . Source and target domain are related but different . Unsupervised domain adaptation ( Ganin & Lempitsky , 2015 ; Ganin et al. , 2016 ; Gong et al. , 2013 ; Bousmalis et al. , 2016 ) , where training data in target domain are unlabeled , is similar to SemiSL , and self-training methods have been used for analysis ( Zou et al. , 2018 ; Tang et al. , 2012 ; French et al. , 2018 ) . However , self-training and unsupervised domain adaptation are fundamentally different . The former learns a model for the domain where there is limited labeled data , with the help of a large number of unlabeled data from a different domain . The latter learns a model for the domain where the training data are unlabeled , with the help of sufficient labeled data from a different domain . Generalization analysis of supervised learning . In theory , the testing error is upper bounded by the training error plus the generalization gap between training and testing . These two quantities are often analyzed separately and can not be proved to be small simultaneously for deep neural networks . For example , neural tangent kernel ( NTK ) method ( Jacot et al. , 2018 ; Du et al. , 2018 ; Lee et al. , 2018 ) shows the training error can be zero , and the Rademacher complexity in ( Bartlett & Mendelson , 2002 ) bounds the generalization gap ( Arora et al. , 2019a ) . For one-hidden-layer neural networks ( Safran & Shamir , 2018 ) , the testing error can be proved to be zero under mild conditions . One common assumption is that the input data belongs to the Gaussian distribution ( Zhong et al. , 2017 ; Ge et al. , 2018 ; Kalai et al. , 2008 ; Bakshi et al. , 2019 ; Zhang et al. , 2016 ; Brutzkus & Globerson , 2017 ; Li & Yuan , 2017 ; Soltanolkotabi et al. , 2018 ) . Another line of approaches ( Brutzkus et al. , 2018 ; Li & Liang , 2018 ; Wang et al. , 2019 ) consider linearly separable data . The rest of this paper is organized as follows . Section 2 introduces the problem formulation and self-training algorithm . Major results are summarized in Section 3 , and empirical evaluations are presented in Section 4 . Section 5 concludes the whole paper . All the proofs are in the Appendix . 2 FORMALIZING SELF-TRAINING : NOTATION , FORMULATION , AND ALGORITHM . Problem formulation . Given N labeled data sampled from distribution Pl , denoted by D = { xn , yn } Nn=1 , and M unlabeled data drawn from distribution Pu , denoted by D̃ = { x̃m } Mm=1 . The aim is to find a neural network model g ( W ) , whereW denotes the trainable weights , that minimizes the testing error on data sampled from Pl . Iterative self-training . In each iteration , given the current teacher predictor g ( W ( ` ) ) , the pseudolabels for the unlabeled data in D̃ are computed as ỹm = g ( W ( ` ) ; x̃m ) . The method then minimizes the weighted empirical risk f̂D , D̃ ( W ) of both labeled and unlabeled data through stochastic gradient descent , where f̂D , D̃ ( W ) = λ 2N N∑ n=1 ( yn − g ( W ; xn ) ) 2 + λ̃ 2M M∑ m=1 ( ỹm − g ( W ; x̃m ) ) 2 , ( 1 ) and λ+ λ̃ = 1 . The learned student model g ( W ( ` +1 ) ) is used as the teacher model in the next iteration . The initial model g ( W ( 0 ) ) is learned from labeled data . The formal algorithm is summarized as in Table 1 . Model and assumptions . This paper considers regression1 , where g is a one-hidden-layer fully connected neural network equipped with K neurons . Namely , given the input x ∈ Rd and weights W = [ w1 , w2 , · · · , wK ] ∈ Rd×K , we have g ( W ; x ) : = 1 K K∑ j=1 φ ( wTj x ) , ( 2 ) where φ is the ReLU activation function2 , and φ ( z ) = max { z , 0 } for any input z ∈ R. Here , we fix the top layer weights as 1 for simplicity , and the equivalence of such a simplification is discussed in Appendix K. Moreover , we assume an unknown ground-truth model with weightsW ∗ that maps all the features to the corresponding labels drawn from Pl , i.e. , y = g ( W ∗ ; x ) , where ( x , y ) ∼ Pl . The generalization function ( GF ) with respect to g ( W ) is defined as I ( g ( W ) ) = E ( x , y ) ∼Pl ( y − g ( W ; x ) ) 2 = E ( x , y ) ∼Pl ( g ( W ∗ ; x ) − g ( W ; x ) ) 2 . ( 3 ) By definition I ( g ( W ∗ ) ) is zero . Clearly , W ∗ is not unique because any column permutation of W ∗ , which corresponds to permuting neurons , represents the same function as W ∗ and minimizes GF in ( 3 ) too . To simplify the representation , we follow the convention and abuse the notation that the distance from W to W ∗ , denoted by ‖W −W ∗‖F , means the smallest distance from W to any permutation ofW ∗ . Additionally , some important notations are summarized in Table 2 . We assume the inputs of both the labeled and unlabeled data belong to the zero mean Gaussian distribution , i.e. , x ∼ N ( 0 , δ2Id ) , and x̃ ∼ N ( 0 , δ̃2Id ) . The Gaussian assumption is motivated by the data whitening ( LeCun et al. , 2012 ) and batch normalization techniques ( Ioffe & Szegedy , 2015 ) that are commonly used in practice to improve learning performance . Moreover , training onehidden-layer neural network with multiple neurons is NP-Complete ( Blum & Rivest , 1992 ) without any assumption . The focus of this paper . This paper will analyze three aspects about self-training : ( 1 ) the generalization performance of W ( L ) , the returned model by self-training after L iterations , measured by ‖W ( L ) −W ∗‖F 3 ; ( 2 ) the influence of parameter λ in ( 1 ) on the training performance ; and ( 3 ) the impact of unlabeled data on the training and generalization performance .

Summary: In this paper, the authors make first forrays into the study of the theoretical properties of the following iterative self-training algorithm in a semi-supervised regression setting where we have N labeled samples and M unlabeled samples (and the ground truth is realisable):  (1)  train the model on the labeled data, obtain pseudo labels for the unlabeled data by feeding them through the trained model  (2)  train an auxiliary loss (see equation (1)) on the augmented dataset the includes the unlabeled data with the corresponding pseudolabels (the loss is a convex combination of the empirical loss on the labelled and unlabeled datasets), and  (3) iteratively apply step (2) until convergence.   It is assumed that the ground truth is realisable (though without overparametrization assumptions), the labels are observed exactly (without noise), the model is a simple two-layer neural network with the second layer weights all fixed to one, and the labeled and unlabeled datapoints come from two isotropic Gaussian distributions of variances $\delta$ and $\tilde{\delta}$.   In theorem 2, the following surprising result is shown: under some reasonable assumptions on the initial point and the number of unlabeled datapoints, for a suitable choice of the convex combination parameter $\lambda$, the model converges towards the ground truth with an assumption on the number of samples which is weaker (by a constant factor at best) than what is required in the case where no unlabeled data is provided (which is a particular case of the case studied in [1])  In the more general Theorem 1, it is shown that under much weaker condition (without a requirement on the number of labelled samples), the model still converges towards a given convex combination of the ground truth and the initial point, and is guaranteed to outperform the initial model.    The main idea of the proofs is to define an auxiliary loss referred to as the population risk function (which estimates the risk taking as "labels" the images of the points by a convex combination of the ground truth and the initial point, cf. equation (17)) and to show (1) that the optimization lanscape of this functions is mild  (mainly proved in Lemmas 1 and 11), and (2) that this auxiliary function is close to the empirical risk which is minimized by the model (mainly proved in Lemma 2). The proofs of the techincal content of lemmas 1 and 2 rely both on computational geometry arguments and on existing results from [1].    Experiments are provided which show:  (1) an excellent match between the dependence of the bounds and that of the observed generalisation gaps on various parameters and (2) that the unlabeled data indeed improves performance in data-sparse regimes on some real data.     ======================= Reference: =======================   [1] Kai Zhong, Zhao Song, Prateek Jain, Peter L. Bartlett, Inderjit S. Dhillon. "Recovery Guarantees for One-hidden-layer Neural Networks", JMLR 2017