Semi-Supervised Boosting via Self Labelling

1 INTRODUCTION . The rise of the Internet has made it easy to collect massive amounts of data to perform machine learning tasks . However , providing quality labels to each of the samples collected within these large datasets is a long and expensive process . There is a rich literature aiming to alleviate this issue , including using techniques from unsupervised machine learning and crowdsourcing . In this paper , we approach the problem by combining concepts from crowdsourcing , learning with noisy data and boosting to propose a novel framework : Boosting via Self Labelling ( BSL ) . Our aim is to develop and leverage i ) machine learning and estimation approaches to create selflabels for unlabelled instances , and ii ) a noise-resistant learning procedure to speed up the performance of the seminal AdaBoost algorithm . The framework consists of mainly three steps : 1 . Accurately predict labels for the unlabelled part of the data using a set of supervised classifiers trained upon a small labelled dataset . Each classifier in our set is analogous to an agent in a crowdsourcing setting . As a result , an inference method can be used to aggregate each of the classifiers predictions and output an accurate noisy label for each of the unlabelled data points . 2 . The second step aims to estimate the noise rate of the generated noisy labels by checking how often the noisy labels generated in step 1 would agree with a particular classifier . This second order statistic suffices to return us the error rates . 3 . The third step of our approach looks at producing a robust boosting method that is trained over the generated noisy data . Classically AdaBoost does not perform well under noisy data compounding errors for each point and progressively creating a worse classifier . The third step introduces a noise resistant version of AdaBoost which relies on the noise and error rate estimated in step 2 . This results in a final classifier which can be compared against different semi-supervised algorithms . BSL builds upon mainly two lines of similar works on boosting without cleanly labelled data : 1 . Semi-supervised boosting ( Fujino et al. , 2005 ; Blum & Mitchell , 1998 ; Laine & Aila , 2016 ; Grandvalet & Bengio , 2005 ) uses semi-supervised algorithms ( e.g. , clustering ) to generate ar- tificial or proxy labels and boost accordingly ( which we compare with ) . Existing methods that apply self-generated labels directly to boosting will fail as the noises in the labels will accumulate while boosting , especially when the noise rates are high - this is what we observed in our experiment results . Our idea is a couple of steps further : we introduce a bias correction procedure into boosting , and explicitly estimate the noises in generated labels . We also introduced an inference framework for generating these labels at first place . 2 . Noise resistant boosting ( Bootkrajang & Kabán , 2013 ) addresses boosting algorithms which are susceptible to noisy data and proposing variants which can perform under a certain noisy conditions . A set of noisy labels , as well as the knowledge of the noises , are often assumed to be known . We do not require neither - we will self-generate the labels for unlabelled instances and learn their error rates . Our contributions summarize as follows : 1 . We propose a novel self-labelling boosting algorithm ( BSL ) which is able to outperform present state-of-the-art semi-supervised algorithms . 2 . As two key components of our self-labelling framework , we contribute i ) a new formulation of a noise resistant Adaboost algorithm which corrects the noises in the labels - this is important in a boosting process , because otherwise the label noises will accurate while boosting ; ii ) a label error estimation procedure without accessing the ground truth labels . 3 . We offer both theoretical guarantees , as well as experimental evidence to the performance of our framework . We conducted an extensive set of experiments to verify the effectiveness of BSL . In the different datasets we ran our algorithm on , our method consistently outperforms many other algorithms by more than 20 % . In the cancer dataset when 10 % of the data was labelled , the second best algorithm ( Semiboost ) under performed our algorithm by 66 % ( relative performance ) . Theoretically we are able to show the convergence of our noise-resistant AdaBoost subroutine under symmetric error rate assumption . The rest of the paper organizes as follows . We survey the most relevant work in the rest of the section . Preliminaries are introduced in Section 2 . Section 3 presents a noise-resistant AdaBoost algorithm . Our Boosting via Self Labelling framework is introduced in Section 4 . Section 5 presents our experiment results . Section 6 concludes our paper . All missing details can be found in Appendix . 1.1 RELATED WORKS . Our work has been inspired by three different lines of research : Semi-Supervised Learning : Research in this avenue looked at creating accurate labels in the presence of limited labelled data . Work started with some basic algorithms ( Cortes & Vapnik , 1995 ; Fujino et al. , 2005 ; Demiriz et al. , 1999 ; Blum & Mitchell , 1998 ) , but escalated to complex systems ( Lee ; Miyato et al. , 2018 ; Laine & Aila , 2016 ; Grandvalet & Bengio , 2005 ) . A good survey in this area was done by Zhu ( Zhu , 2005 ) . Crowdsourcing : Inference crowdsourcing methods have played a huge part in uncovering true labels from multiple noisy labels . Work in the area has ranged from EM algorithms ( Dawid & Skene , 1979 ; Raykar et al. , 2010 ; Smyth et al. , 1995 ; Karger et al. , 2011 ) to variational inference methods ( Liu et al. , 2012 ; Karger et al. , 2014 ; Whitehill et al. , 2009 ; Welinder et al. , 2010 ; Raykar & Yu , 2012 ) . ( Chon et al. , 2012 ) wrote a good survey for this topic . In an ensemble setting , crowdsourcing has appeared in numerous works include , most famously , ( Kim & Ghahramani , 2012 ) . However , many of these results do not consider the noise present in the final aggregated value , which can lead to accumulated errors in a boosting setting . Boosting : Starting with the work ( Freund & Schapire , 1999 ) , research in AdaBoost has expanded to all areas of machine learning . Some of the recent work in the area has looked into improving the performance and pitfalls the original algorithm faced ( Rätsch et al. , 2001 ; Hastie et al. , 2009 ; Domingo et al. , 2000 ; Schapire , 2013 ; Bootkrajang & Kabán , 2013 ) . Our work builds upon previous work done in these areas , using similar ideas to construct a unique method . One notable piece of work is Semiboost ( Zhu , 2005 ) . This algorithm takes a similar approach , introducing a boosting framework that improves upon existing classifiers to provide a good classifier in the semi-labelled setting . However , Semiboost uses an unsupervised learning approach using a similarity matrix based on labelled points and unlabelled points . Semiboost requires a similarity equation to run , and this can be hard to achieve in practice . Pseudo-labelling is one of many neural network approaches to the limited labelled dataset problem . Pseudo-labelling assigns labels to the unlabelled data and then trains a neural network on the combination of the clean and noisy labels . However , Pseudo-labelling and algorithms like it , face many different requirements . Algorithms in this class require a sufficient amount of supervised data to work optimally . They also need the classes to be clustered within the data , and the labelled data to not adhere to the same distribution as the unlabelled data ( Oliver et al. , 2018 ) . Learning in noisy data has also been a research focus that runs parallel to our work . In ( Natarajan et al. , 2013 ) , Natarajan et al . propose a noisy learning algorithm which performs significantly better than other noise resistant algorithms . However , the method requires knowledge on the amount of noise inside of the dataset before training and in a limited label dataset , which is hard to get in practice . In contrast to these related works , the algorithm we introduce in this paper does not need prior knowledge about the dataset for it to perform and the number of labelled points does not adversely affect the accuracy of the classifier generated . 2 PRELIMINARY AND PROBLEM FORMULATION . Assume that D is the underlying true distribution generating n iid examples ( xi , yi ) ni=1 , where each example consists of a feature vector xi = [ xi,1 , xi,2 , . . . , xi , d ] ∈ X ⊆ Rd , and a label yi ∈ { −1 , +1 } ( ∼ Y ) . We assume the true labels of τ examples is available where τ < n such that ( xi , yi ) τ i=1 , denoted as { ( x1 , y1 ) , ( x2 , y2 ) , · · · , ( x|NL| , y|NL| ) } : = NL ; while the rest of n − τ samples ( xi ) ni=τ is unlabelled , { x1 , x2 , · · · , x|NU | } : = NU . Let NU and NL be the set of indices that make up NU and NL respectively . Assume that after an initial classifier f : X → { −1 , +1 } is trained on NL and applied on the unlabelled dataset such that f ( xi ) → ỹi , xi ∈ NU . Then NUnoisy denotes the noisy data set produced ( xi , ỹi ) |NU | i=1 . Let Nnoisy be the combination of NL and NUnoisy such that for i = 1 to |Nnoisy| : { ( xi , ỹi ) , if ( xi , ỹi ) ∈ NUnoisy ; ( xi , yi ) , if ( xi , yi ) ∈ NL } . Let Nnoisy be the set of all indices within Nnoisy . Assume that Nnoisy follows as class-conditional random noise model such that : ∀n = 1 , 2 , ... , |Nnoisy| : P ( ỹi = −1|yi = +1 , xn ) = ρ+ , P ( ỹi = +1|yi = −1 , xn ) = ρ− and ρ+ + ρ− < 1 . Our goal is to learn a classifier f : X → { −1 , +1 } trained on Nnoisy that minimizes the risk of f w.r.t to the 0-1 loss function RD ( f ) = E ( xi , yi ) ∼D [ 1 ( f ( xi ) 6= yi ) ] . 2.1 OUR PROBLEM : BOOSTING VIA SELF LABELLING . We introduce the setting for AdaBoost ( Freund & Schapire , 1999 ) . The key idea is , at step t , • Maintain a weight Di ( t ) for each data instance ( xi , yi ) ∈ NL . • Train a weak learner ft according to the weighted data distribution . • The final hypothesis F is a linear combination of each ft trained at every step t. Let Nmiss be the set of all ( xi , yi ) ∈ NL , such that ft ( xi ) 6= yi . The goal is to increase the weight of mis-classified points Di ( t ) , ( xi , yi ) ∈ Nmiss to encourage classifier ft+1 ( · ) to focus on correctly classifying Nmiss : Dt ( i+ 1 ) = Dt ( i ) · exp ( −αt · ft ( xi ) · yi ) Zt , where Zt = ∑ i∈NL Dt ( i ) ·exp ( −αt ·ft ( xi ) ·yi ) is a normalization factor . AdaBoost creates a final hypothesis in the additive form : F ( xi ) = ∑T t=1 αtft ( xi ) , where xi is a test sample . Our goal , and a short coming with AdaBoost , is classifying a dataset where some of the yi in ( xi , yi ) ∈ Nnoisy are noisy . Because AdaBoost uses a exponential loss function it is inherently susceptible to noisy labels . We propose a new loss function that removes bias : Dt+1 ( i ) = Dt ( i ) · exp ( −αt · ˜̀ ( xi , ỹi ) ) Zt , ∀i ∈ Nnoisy Our goal is to define a function ˜̀ ( · ) that can help us evaluate an unlabelled instance . This function ˜̀ ( · ) will allow us to adjust to noisy labels within Nnoisy . Our algorithm runs in two main stages . It first applies noisy labels for unlabelled instances in our dataset , and then creates a final hypothesis F using a noise-resistant variant of AdaBoost where we define ˜̀ ( · ) .

Summary: The authors propose a new semi-supervised boosting approach. The approach takes a set of supervised learning algorithms to simulate "crowd-source" labels of the unlabeled data, which are then used to generate a noisy label per unlabeled instance. The noise level is then estimated with an agreement-based scheme, and fed to a modified AdaBoost algorithm that is more noise-tolerant given the noise level. Some theoretical guarantee of the modified AdaBoost algorithm is derived and promising experiment results are demonstrated.