How unlabeled data improve generalization in self-training? A one-hidden-layer theoretical analysis

√ M , where M is the number of unlabeled samples . Experiments from shallow neural networks to deep neural networks are also provided to justify the correctness of our established theoretical insights on self-training . 1 INTRODUCTION . Self-training ( Scudder , 1965 ; Yarowsky , 1995 ; Lee et al. , 2013 ; Han et al. , 2019 ) , one of the most powerful semi-supervised learning ( SemiSL ) algorithms , augments a limited number of labeled data with unlabeled data so as to achieve improved generalization performance on test data , compared with the model trained by supervised learning using the labeled data only . Self-training has shown empirical success in diversified applications such as few-shot image classification ( Su et al. , 2020 ; Xie et al. , 2020 ; Chen et al. , 2020a ; Yalniz et al. , 2019 ; Zoph et al. , 2020 ) , objective detection ( Rosenberg et al. , 2005 ) , robustness-aware model training against adversarial attacks ( Carmon et al. , 2019 ) , continual lifelong learning ( Lee et al. , 2019 ) , and natural language processing ( He et al. , 2019 ; Kahn et al. , 2020 ) . The terminology “ self-training ” has been used to describe various SemiSL algorithms in the literature , while this paper is centered on the commonly-used iterative self-training method in particular . In this setup , an initial teacher model ( learned from the labeled data ) is applied to the unlabeled data to generate pseudo labels . One then trains a student model by minimizing the weighted empirical risk of both the labeled and unlabeled data . The student model is then used as the new teacher to update the pseudo labels of the unlabeled data . This process is repeated multiple times to improve the eventual student model . We refer readers to Section 2 for algorithmic details . Despite the empirical achievement of self-training methods with neural networks , the theoretical justification of such success is very limited , even in the field of SemiSL . The majority of the theoretical results on general SemiSL are limited to linear networks ( Chen et al. , 2020b ; Raghunathan et al. , 2020 ; Oymak & Gulcu , 2020 ; Oneto et al. , 2011 ) . The authors in ( Balcan & Blum , 2010 ) show that unlabeled data can improve the generalization bound if the unlabeled data distribution and target model are compatible . For instance , the unlabeled data need to be well-chosen such that the target function for labeled data can separate the unlabeled data clusters , which , however , may not be able to be verified ahead . Moreover , ( Rigollet , 2007 ; Singh et al. , 2008 ) proves that unlabeled data can improve the convergence rate and generalization error under a similar clustering assumption , where the data contains clusters that have homogeneous labels . A recent work by Wei et al . ( 2020 ) analyzes SemiSL on nonlinear neural networks and proves that an infinite number of unlabeled data can improve the generalization compared with training with labeled data only . However , Wei et al . ( 2020 ) considers single shot rather than iterative SemiSL , and the training problem aims to minimize the consistency regularization rather than the risk function in the conventional self-training method ( Lee et al. , 2013 ) . Moreover , Wei et al . ( 2020 ) directly analyzes the global optimum of the nonconvex training problem without any discussion about how to achieve the global optimum . To the best of our knowledge , there exists no analytical characterization of how the unlabeled data affect the generalization of the learned model by iterative self-training on nonlinear neural networks . Contributions . This paper provides the first theoretical study of iterative self-training on nonlinear neural networks . Focusing on one-hidden-layer neural networks , this paper provides a quantitative analysis of the generalization performance of iterative self-training as a function of the number of labeled and unlabeled samples . Specifically , our contributions include 1 . Quantitative justification of generalization improvement by unlabeled data . Assuming the existence of a ground-truth model with weights W ∗ that maps the features to the corresponding labels , we prove that the learned model via iterative self-training moves closer to W ∗ as the number M of unlabeled data increases , indicating a better testing performance . Specifically , we prove that the Frobenius distance to W ∗ , which is approximately linear in the generalization error , decreases in the order of 1/ √ M . As an example , Figure 1 shows that the proposed theoretical bound matches the empirical self-training performance versus the number of unlabeled data for image classification ; see details in Section 4.2 . 2 . Analytical justification of iterative self-training over single shot alternative . We prove that the student models returned by the iterative self-training method converges linearly to a model close toW ∗ , with the rate improvement in the order of 1/ √ M . 3 . Sample complexity analysis of labeled and unlabeled data for learning a proper model .. We quantify the impact of labeled and unlabeled data on the generalization of the learned model . In particular , we prove that the sample complexity of labeled data can be reduced compared with supervised learning . 1.1 RELATED WORKS . Semi-supervised learning . Besides self-training , many recent SemiSL algorithms exploit either consistency regularization or entropy minimization . Consistency regularization is based on the as- sumption that the learned model will return same or similar output when the input is perturbed ( Laine & Aila , 2016 ; Bachman et al. , 2014 ; Sajjadi et al. , 2016 ; Tarvainen & Valpola , 2017 ; Reed et al. , 2015 ) . ( Grandvalet & Bengio , 2005 ) claims that the unlabeled data are more informative if the pseudo labels of the unlabeled data have lower entropy . Therefore , a line of works ( Grandvalet & Bengio , 2005 ; Miyato et al. , 2018 ) adds a regularization term that minimizes the entropy of the outputs of the unlabeled data . In addition , hybrid algorithms that unify both the above regularizations have been developed like ( Berthelot et al. , 2019a ; b ; Sohn et al. , 2020 ) . Domain adaptation . Domain adaptation exploits abundant data in the source domain to learn a model for the target domain , where only limited training data are available ( Liebelt & Schmid , 2010 ; Vazquez et al. , 2013 ; Zhang et al. , 2013 ; Long et al. , 2015 ; Tzeng et al. , 2014 ) . Source and target domain are related but different . Unsupervised domain adaptation ( Ganin & Lempitsky , 2015 ; Ganin et al. , 2016 ; Gong et al. , 2013 ; Bousmalis et al. , 2016 ) , where training data in target domain are unlabeled , is similar to SemiSL , and self-training methods have been used for analysis ( Zou et al. , 2018 ; Tang et al. , 2012 ; French et al. , 2018 ) . However , self-training and unsupervised domain adaptation are fundamentally different . The former learns a model for the domain where there is limited labeled data , with the help of a large number of unlabeled data from a different domain . The latter learns a model for the domain where the training data are unlabeled , with the help of sufficient labeled data from a different domain . Generalization analysis of supervised learning . In theory , the testing error is upper bounded by the training error plus the generalization gap between training and testing . These two quantities are often analyzed separately and can not be proved to be small simultaneously for deep neural networks . For example , neural tangent kernel ( NTK ) method ( Jacot et al. , 2018 ; Du et al. , 2018 ; Lee et al. , 2018 ) shows the training error can be zero , and the Rademacher complexity in ( Bartlett & Mendelson , 2002 ) bounds the generalization gap ( Arora et al. , 2019a ) . For one-hidden-layer neural networks ( Safran & Shamir , 2018 ) , the testing error can be proved to be zero under mild conditions . One common assumption is that the input data belongs to the Gaussian distribution ( Zhong et al. , 2017 ; Ge et al. , 2018 ; Kalai et al. , 2008 ; Bakshi et al. , 2019 ; Zhang et al. , 2016 ; Brutzkus & Globerson , 2017 ; Li & Yuan , 2017 ; Soltanolkotabi et al. , 2018 ) . Another line of approaches ( Brutzkus et al. , 2018 ; Li & Liang , 2018 ; Wang et al. , 2019 ) consider linearly separable data . The rest of this paper is organized as follows . Section 2 introduces the problem formulation and self-training algorithm . Major results are summarized in Section 3 , and empirical evaluations are presented in Section 4 . Section 5 concludes the whole paper . All the proofs are in the Appendix . 2 FORMALIZING SELF-TRAINING : NOTATION , FORMULATION , AND ALGORITHM . Problem formulation . Given N labeled data sampled from distribution Pl , denoted by D = { xn , yn } Nn=1 , and M unlabeled data drawn from distribution Pu , denoted by D̃ = { x̃m } Mm=1 . The aim is to find a neural network model g ( W ) , whereW denotes the trainable weights , that minimizes the testing error on data sampled from Pl . Iterative self-training . In each iteration , given the current teacher predictor g ( W ( ` ) ) , the pseudolabels for the unlabeled data in D̃ are computed as ỹm = g ( W ( ` ) ; x̃m ) . The method then minimizes the weighted empirical risk f̂D , D̃ ( W ) of both labeled and unlabeled data through stochastic gradient descent , where f̂D , D̃ ( W ) = λ 2N N∑ n=1 ( yn − g ( W ; xn ) ) 2 + λ̃ 2M M∑ m=1 ( ỹm − g ( W ; x̃m ) ) 2 , ( 1 ) and λ+ λ̃ = 1 . The learned student model g ( W ( ` +1 ) ) is used as the teacher model in the next iteration . The initial model g ( W ( 0 ) ) is learned from labeled data . The formal algorithm is summarized as in Table 1 . Model and assumptions . This paper considers regression1 , where g is a one-hidden-layer fully connected neural network equipped with K neurons . Namely , given the input x ∈ Rd and weights W = [ w1 , w2 , · · · , wK ] ∈ Rd×K , we have g ( W ; x ) : = 1 K K∑ j=1 φ ( wTj x ) , ( 2 ) where φ is the ReLU activation function2 , and φ ( z ) = max { z , 0 } for any input z ∈ R. Here , we fix the top layer weights as 1 for simplicity , and the equivalence of such a simplification is discussed in Appendix K. Moreover , we assume an unknown ground-truth model with weightsW ∗ that maps all the features to the corresponding labels drawn from Pl , i.e. , y = g ( W ∗ ; x ) , where ( x , y ) ∼ Pl . The generalization function ( GF ) with respect to g ( W ) is defined as I ( g ( W ) ) = E ( x , y ) ∼Pl ( y − g ( W ; x ) ) 2 = E ( x , y ) ∼Pl ( g ( W ∗ ; x ) − g ( W ; x ) ) 2 . ( 3 ) By definition I ( g ( W ∗ ) ) is zero . Clearly , W ∗ is not unique because any column permutation of W ∗ , which corresponds to permuting neurons , represents the same function as W ∗ and minimizes GF in ( 3 ) too . To simplify the representation , we follow the convention and abuse the notation that the distance from W to W ∗ , denoted by ‖W −W ∗‖F , means the smallest distance from W to any permutation ofW ∗ . Additionally , some important notations are summarized in Table 2 . We assume the inputs of both the labeled and unlabeled data belong to the zero mean Gaussian distribution , i.e. , x ∼ N ( 0 , δ2Id ) , and x̃ ∼ N ( 0 , δ̃2Id ) . The Gaussian assumption is motivated by the data whitening ( LeCun et al. , 2012 ) and batch normalization techniques ( Ioffe & Szegedy , 2015 ) that are commonly used in practice to improve learning performance . Moreover , training onehidden-layer neural network with multiple neurons is NP-Complete ( Blum & Rivest , 1992 ) without any assumption . The focus of this paper . This paper will analyze three aspects about self-training : ( 1 ) the generalization performance of W ( L ) , the returned model by self-training after L iterations , measured by ‖W ( L ) −W ∗‖F 3 ; ( 2 ) the influence of parameter λ in ( 1 ) on the training performance ; and ( 3 ) the impact of unlabeled data on the training and generalization performance .

Summary: The paper theoretically analyzes self-training in neural networks with one hidden layer. Self-training is a popular semisupervised learning algorithm where a model uses large unlabeled data by training on pseudolabels generated by a teacher model trained on some small set of labels. The paper studies the generalization performance of self-training on a two-layer network under a Gaussian assumption on the inputs, and has some synthetic and CIFAR-10 experiments to show that the predicted theoretical rates (1/\sqrt{ size unlabeled data}) roughly match empirical performance.