Boosting with Multiple Sources

1 Motivation . Ensemble methods such as Bagging , AdaBoost , Stacking , error-correction techniques , Bayesian averaging , AdaNet or other adaptive methods for learning neural networks are general machine learning techniques used to combine several predictors to devise a more accurate one ( Breiman , 1996 ; Freund and Schapire , 1997 ; Smyth and Wolpert , 1999 ; MacKay , 1991 ; Freund et al. , 2004 ; Cortes et al. , 2014 ; Kuznetsov et al. , 2014 ; Cortes et al. , 2017 ) . These techniques are often very effective in practice and benefit from favorable margin-based learning guarantees ( Schapire et al. , 1997 ) . These algorithms assume access to a training sample drawn from the target distribution . But , in many applications , the learner receives labeled data from multiple source domains that it seeks to use to find a good predictor for target domains , which are typically assumed to be mixtures of source distributions ( Mansour et al. , 2008 , 2009a ; Hoffman et al. , 2018 , 2021 ; Cortes et al. , 2021 ; Muandet et al. , 2013 ; Xu et al. , 2014 ; Hoffman et al. , 2012 ; Saito et al. , 2019 ; Wang et al. , 2019a ) . How can we generalize ensemble methods such as boosting to this scenario ? Several related problems have been tackled in previous work . In the special case of a single target domain , boosting solutions have been derived ( Dai et al. , 2007 ; Yao and Doretto , 2010 ) . These algorithms have been further improved by boosting base predictors jointly with source features ( or feature views ) that are predictive in the target ( Yuan et al. , 2017 ; Cheng et al. , 2013 ; Zhang et al. , 2014 ; Xu and Sun , 2012 , 2011 ) . Such methods have been widely adopted in various domain adaptation scenarios for different types of data . For example , Huang et al . ( 2010 , 2012 ) showed that by selecting a base learner jointly with a feature that is predictive across multiple domains at every boosting step , one can achieve higher accuracy than standard transfer learning methods . Moreover , the margin 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . provided by boosting-style algorithms can aid in transfer learning where target domain is unlabelled . Habrard et al . ( 2013 ) have developed an algorithm that jointly minimizes the source domain error and margin violation proportion on the target domain . Wang et al . ( 2019a ) have demonstrated that boosting classifiers from different domains can be carried out online . Taherkhani et al . ( 2020 ) and Becker et al . ( 2013 ) have shown that multi-source boosting can be combined with Deep Neural Networks for multi-task learning on large scale datasets . We give a more detailed discussion of this related prior work in Appendix A . However , as we demonstrate in this paper , several critical questions related to the presence of multiple domains have not been fully addressed by existing boosting methods from both the algorithmic and the theoretical point of view . First , what should be the form of the ensemble solutions ? We show that , in general , the standard convex combinations used in much of previous work may not succeed in this problem ( Section 2 ) . Instead , we put forward Q-ensembles , which are convex combinations weighted by a domain classifier Q , that is , Q ( k|x ) is the conditional probability of domain k given input point x . This is inspired by similar Q-ensembles ( Cortes , Mohri , Suresh , and Zhang , 2021 ) or distributionweighted combinations ( Mansour , Mohri , and Rostamizadeh , 2008 , 2009a ; Hoffman , Mohri , and Zhang , 2018 , 2021 ) used in the context of multiple-source adaptation and crucially differentiates our work from that of past studies of ensemble methods . Our learning scenario strictly generalizes previous algorithms . In the special case of a single domain , the form of our ensemble solutions coincides with that of the familiar ensembles used in AdaBoost . In Appendix J , we further compare our results and guarantees with the multiple-source adaptation algorithms of ( Hoffman , Mohri , and Zhang , 2018 , 2021 ; Cortes , Mohri , Suresh , and Zhang , 2021 ) , when using AdaBoost to derive a predictor for each domain . Second , what should be the form of the objective ? Unlike existing boosting methods which optimize for a specific target distribution or domain , we seek a solution that is accurate for any mixture of the source distributions , where the mixture weights may be constrained to be in a subset of the simplex . This conforms with the learning goal in this scenario where the target distribution is typically assumed to be a mixture of the source ones and is further inspired by the formulation of the agnostic loss in the related context of federated learning ( Mohri , Sivek , and Suresh , 2019 ) . Additionally , this formulation can be viewed as more risk-averse and robust , and it also ensures more favorable fairness properties , since it does not favor any distribution possibly biased towards some subset of the source domains . In Section 3 , we introduce and analyze an algorithm , MULTIBOOST , that is based on such an objective . This further distinguishes our boosting algorithm from related prior work . In Appendix L , we further give a detailed description of a multi-class extension of our algorithm , MCMULTIBOOST , including its pseudocode , the discussion of some of its properties , and the results of several experiments . Third , in Section 4 , we present a theoretical analysis of our MULTIBOOST algorithm , including margin-based generalization bounds that hold for any target mixture of the source distributions . In Appendix F , we derive finer and more flexible learning bounds that can provide more favorable guarantees , in particular when the family of target mixtures is more constrained . Fourth , in Section 5 , we describe FEDMULTIBOOST , an extension of our MULTIBOOST algorithm adapted to the distributed learning scenario of federated learning , which is increasingly important in a wide array of applications ( Kairouz et al. , 2021 ) , including healthcare ( Brisimi et al. , 2018 ) . FEDMULTIBOOST allows the server to train ensemble models that benefit from clients ’ data , while the data remains with clients . We provide a detailed description of our algorithm , as well as a brief analysis of the communication costs , which are critical in this scenario . Appendix G contains experimental results comparing FEDMULTIBOOST with several baselines . Finally , in Section 6 , we report the results of extensive experiments with our MULTIBOOST algorithm . We start with a detailed description of the learning scenario we consider . 2 Learning scenario . Let X denote the input space and Y = { −1 , +1 } the output space associated to binary classification . We consider a scenario where the learner receives labeled samples from p source domains , each defined by a distribution Dk over X × Y , k ∈ [ p ] . We denote by Sk = ( ( xk,1 , yk,1 ) , . . . , ( xk , mk , yk , mk ) ) ∈ ( X× Y ) mk the labeled sample of size mk received from source k , which is drawn i.i.d . from Dk . For any function f : X → R and distribution D over X × Y , let L ( D , f ) be the expected loss of f , that is L ( D , f ) = E ( x , y ) ∼D [ ` ( f ( x ) , y ) ] , where ` is the binary loss ` ( f ( x ) , y ) = I ( yf ( x ) ≤ 0 ) . For any k ∈ [ p ] , let Hk denote a hypothesis set of functions mapping from X to [ −1 , +1 ] , |Hk| = Nk . The objective of the learner is to find a predictor f that is accurate for any target distribution Dλ that is a mixture of the source distributions , where λ may be in a subset Λ of the simplex . Thus , Dλ can written as Dλ = ∑p k=1 λkDk , with λ = ( λ1 , . . . , λp ) ≥ 0 and ∑p k=1 λk = 1 . This leads to the following agnostic loss of a predictor f ( Mohri et al. , 2019 ) : L ( DΛ , f ) = max λ∈Λ L ( Dλ , f ) . ( 1 ) To come up with a predictor f , the learner seeks an ensemble of functions from the base classes Hk . What should be the form of such ensembles ? A natural solution is a convex combinations of the form f = p∑ l=1 Nl∑ j=1 αl , jhl , j , ( 2 ) where hl , j ∈ Hl and αl , j ≥ 0 , ∑p l=1 ∑Nl j=1 αl , j = 1 , as in prominent algorithms such as BAGGING ( Breiman , 1999 ) or ADABOOST ( Freund and Schapire , 1997 ) . It is not hard to show , however , that for some distributions , any such convex combination of base predictors would lead to a poor solution . Proposition 1 . There exist distributions D1 and D2 and hypotheses h1 and h2 with L ( D1 , h1 ) = 0 and L ( D2 , h2 ) = 0 such that L ( 1 2 ( D1 + D2 ) , αh1 + ( 1− α ) h2 ) ≥ 12 for any α ∈ [ 0 , 1 ] , This result is similar to ( Mansour et al. , 2008 , Theorem 1 ) or similar statements in ( Mansour et al. , 2008 , 2009a ; Hoffman et al. , 2018 , 2021 ; Cortes et al. , 2021 ) , however , there are some technical differences . In particular , we are considering here a binary classification loss and not the absolute loss . The proof is given in Appendix B . Note that in the example of the proposition , the base predictors h1 , h2 are both perfect for their respective domains . Nevertheless , unlike the common case of a single source domain , standard convex combinations in general may perform poorly . Thus , we will consider instead the following form for the ensembles of base predictors : ∀x ∈ X , f ( x ) = p∑ l=1 Nl∑ j=1 αl , jQ ( l|x ) hl , j ( x ) , ( 3 ) where Q ( l|x ) denotes the conditional probability of domain l given x . This helps us account for the fact that the learner combines base predictors from different domains . For a given point x , experts from the domains where x is more likely to appear are allocated more weight in the voted combination . The following result further substantiates the hope for this method to succeed . Proposition 2 . For the same distributions D1 and D2 and hypotheses h1 and h2 as in Proposition 1 , the equality L ( Dλ , ( αQ ( 1|· ) h1 + ( 1− α ) Q ( 2|· ) h2 ) ) = 0 holds for any λ ∈ ∆ and any α ∈ ( 0 , 1 ) . The proof is also given in Appendix B . Thus , our Q-ensembles can be substantially more effective in the multiple-source adaptation problem considered . Note that , in the special case of a single domain , the Q-combinations coincide with standard convex combinations , since Q ( k|x ) = 1 for all x . As in the standard boosting scenario , we will adopt a weak-learning assumption . However , our assumption here must hold for each source domain : for each domain k ∈ [ p ] and any distribution D over Sk , there exists a base classifier h ∈ Hk such that the weighted loss of h is γ-better than random : 1 2 [ 1− Ei∼D [ yk , ih ( xk , i ) ] ≤ 1 2 − γ , for some edge value γ > 0 . This is equivalent to the existence of a weak-learning algorithm for each domain , which is a mild assumption . As in the standard boosting scenario , this corresponds to the existence of a a good rule of thumb for each domain . Unlike the standard scenario , however , here , we seek a Q-ensemble and further require that ensemble to be accurate for any target mixture Dλ , λ ∈ Λ . Note that the definition of the hypothesis set Hk and the weak-learning assumption for domain k are intimately related : the elements of Hk are typically simple predictors sufficient to guarantee the weak-learning assumption for that domain . In the next section , we present an algorithm , MULTIBOOST , for deriving an accurate Q-ensemble for any target mixture domain that belongs to the convex combination of the source domains . In Appendix L , we further present a multi-class extension of our algorithm , MCMULTIBOOST .

Summary: The paper addresses the question of learning an accurate classifier using ensemble methods and based on multiple sources. The purpose of the classifier is to achieve good prediction on a set of possible future domains, modelled as linear combinations of the source domains. This is done by minimizing an agnostic loss function over the weights of a linear combination of source predictors, such as traditionally done in ensemble methods. The paper proposes a main algorithm solving this problem, as well as a version for the federated setting. A theoretical analysis and experimental validation are also proposed.