Boosting with Multiple Sources

1 Motivation . Ensemble methods such as Bagging , AdaBoost , Stacking , error-correction techniques , Bayesian averaging , AdaNet or other adaptive methods for learning neural networks are general machine learning techniques used to combine several predictors to devise a more accurate one ( Breiman , 1996 ; Freund and Schapire , 1997 ; Smyth and Wolpert , 1999 ; MacKay , 1991 ; Freund et al. , 2004 ; Cortes et al. , 2014 ; Kuznetsov et al. , 2014 ; Cortes et al. , 2017 ) . These techniques are often very effective in practice and benefit from favorable margin-based learning guarantees ( Schapire et al. , 1997 ) . These algorithms assume access to a training sample drawn from the target distribution . But , in many applications , the learner receives labeled data from multiple source domains that it seeks to use to find a good predictor for target domains , which are typically assumed to be mixtures of source distributions ( Mansour et al. , 2008 , 2009a ; Hoffman et al. , 2018 , 2021 ; Cortes et al. , 2021 ; Muandet et al. , 2013 ; Xu et al. , 2014 ; Hoffman et al. , 2012 ; Saito et al. , 2019 ; Wang et al. , 2019a ) . How can we generalize ensemble methods such as boosting to this scenario ? Several related problems have been tackled in previous work . In the special case of a single target domain , boosting solutions have been derived ( Dai et al. , 2007 ; Yao and Doretto , 2010 ) . These algorithms have been further improved by boosting base predictors jointly with source features ( or feature views ) that are predictive in the target ( Yuan et al. , 2017 ; Cheng et al. , 2013 ; Zhang et al. , 2014 ; Xu and Sun , 2012 , 2011 ) . Such methods have been widely adopted in various domain adaptation scenarios for different types of data . For example , Huang et al . ( 2010 , 2012 ) showed that by selecting a base learner jointly with a feature that is predictive across multiple domains at every boosting step , one can achieve higher accuracy than standard transfer learning methods . Moreover , the margin 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . provided by boosting-style algorithms can aid in transfer learning where target domain is unlabelled . Habrard et al . ( 2013 ) have developed an algorithm that jointly minimizes the source domain error and margin violation proportion on the target domain . Wang et al . ( 2019a ) have demonstrated that boosting classifiers from different domains can be carried out online . Taherkhani et al . ( 2020 ) and Becker et al . ( 2013 ) have shown that multi-source boosting can be combined with Deep Neural Networks for multi-task learning on large scale datasets . We give a more detailed discussion of this related prior work in Appendix A . However , as we demonstrate in this paper , several critical questions related to the presence of multiple domains have not been fully addressed by existing boosting methods from both the algorithmic and the theoretical point of view . First , what should be the form of the ensemble solutions ? We show that , in general , the standard convex combinations used in much of previous work may not succeed in this problem ( Section 2 ) . Instead , we put forward Q-ensembles , which are convex combinations weighted by a domain classifier Q , that is , Q ( k|x ) is the conditional probability of domain k given input point x . This is inspired by similar Q-ensembles ( Cortes , Mohri , Suresh , and Zhang , 2021 ) or distributionweighted combinations ( Mansour , Mohri , and Rostamizadeh , 2008 , 2009a ; Hoffman , Mohri , and Zhang , 2018 , 2021 ) used in the context of multiple-source adaptation and crucially differentiates our work from that of past studies of ensemble methods . Our learning scenario strictly generalizes previous algorithms . In the special case of a single domain , the form of our ensemble solutions coincides with that of the familiar ensembles used in AdaBoost . In Appendix J , we further compare our results and guarantees with the multiple-source adaptation algorithms of ( Hoffman , Mohri , and Zhang , 2018 , 2021 ; Cortes , Mohri , Suresh , and Zhang , 2021 ) , when using AdaBoost to derive a predictor for each domain . Second , what should be the form of the objective ? Unlike existing boosting methods which optimize for a specific target distribution or domain , we seek a solution that is accurate for any mixture of the source distributions , where the mixture weights may be constrained to be in a subset of the simplex . This conforms with the learning goal in this scenario where the target distribution is typically assumed to be a mixture of the source ones and is further inspired by the formulation of the agnostic loss in the related context of federated learning ( Mohri , Sivek , and Suresh , 2019 ) . Additionally , this formulation can be viewed as more risk-averse and robust , and it also ensures more favorable fairness properties , since it does not favor any distribution possibly biased towards some subset of the source domains . In Section 3 , we introduce and analyze an algorithm , MULTIBOOST , that is based on such an objective . This further distinguishes our boosting algorithm from related prior work . In Appendix L , we further give a detailed description of a multi-class extension of our algorithm , MCMULTIBOOST , including its pseudocode , the discussion of some of its properties , and the results of several experiments . Third , in Section 4 , we present a theoretical analysis of our MULTIBOOST algorithm , including margin-based generalization bounds that hold for any target mixture of the source distributions . In Appendix F , we derive finer and more flexible learning bounds that can provide more favorable guarantees , in particular when the family of target mixtures is more constrained . Fourth , in Section 5 , we describe FEDMULTIBOOST , an extension of our MULTIBOOST algorithm adapted to the distributed learning scenario of federated learning , which is increasingly important in a wide array of applications ( Kairouz et al. , 2021 ) , including healthcare ( Brisimi et al. , 2018 ) . FEDMULTIBOOST allows the server to train ensemble models that benefit from clients ’ data , while the data remains with clients . We provide a detailed description of our algorithm , as well as a brief analysis of the communication costs , which are critical in this scenario . Appendix G contains experimental results comparing FEDMULTIBOOST with several baselines . Finally , in Section 6 , we report the results of extensive experiments with our MULTIBOOST algorithm . We start with a detailed description of the learning scenario we consider . 2 Learning scenario . Let X denote the input space and Y = { −1 , +1 } the output space associated to binary classification . We consider a scenario where the learner receives labeled samples from p source domains , each defined by a distribution Dk over X × Y , k ∈ [ p ] . We denote by Sk = ( ( xk,1 , yk,1 ) , . . . , ( xk , mk , yk , mk ) ) ∈ ( X× Y ) mk the labeled sample of size mk received from source k , which is drawn i.i.d . from Dk . For any function f : X → R and distribution D over X × Y , let L ( D , f ) be the expected loss of f , that is L ( D , f ) = E ( x , y ) ∼D [ ` ( f ( x ) , y ) ] , where ` is the binary loss ` ( f ( x ) , y ) = I ( yf ( x ) ≤ 0 ) . For any k ∈ [ p ] , let Hk denote a hypothesis set of functions mapping from X to [ −1 , +1 ] , |Hk| = Nk . The objective of the learner is to find a predictor f that is accurate for any target distribution Dλ that is a mixture of the source distributions , where λ may be in a subset Λ of the simplex . Thus , Dλ can written as Dλ = ∑p k=1 λkDk , with λ = ( λ1 , . . . , λp ) ≥ 0 and ∑p k=1 λk = 1 . This leads to the following agnostic loss of a predictor f ( Mohri et al. , 2019 ) : L ( DΛ , f ) = max λ∈Λ L ( Dλ , f ) . ( 1 ) To come up with a predictor f , the learner seeks an ensemble of functions from the base classes Hk . What should be the form of such ensembles ? A natural solution is a convex combinations of the form f = p∑ l=1 Nl∑ j=1 αl , jhl , j , ( 2 ) where hl , j ∈ Hl and αl , j ≥ 0 , ∑p l=1 ∑Nl j=1 αl , j = 1 , as in prominent algorithms such as BAGGING ( Breiman , 1999 ) or ADABOOST ( Freund and Schapire , 1997 ) . It is not hard to show , however , that for some distributions , any such convex combination of base predictors would lead to a poor solution . Proposition 1 . There exist distributions D1 and D2 and hypotheses h1 and h2 with L ( D1 , h1 ) = 0 and L ( D2 , h2 ) = 0 such that L ( 1 2 ( D1 + D2 ) , αh1 + ( 1− α ) h2 ) ≥ 12 for any α ∈ [ 0 , 1 ] , This result is similar to ( Mansour et al. , 2008 , Theorem 1 ) or similar statements in ( Mansour et al. , 2008 , 2009a ; Hoffman et al. , 2018 , 2021 ; Cortes et al. , 2021 ) , however , there are some technical differences . In particular , we are considering here a binary classification loss and not the absolute loss . The proof is given in Appendix B . Note that in the example of the proposition , the base predictors h1 , h2 are both perfect for their respective domains . Nevertheless , unlike the common case of a single source domain , standard convex combinations in general may perform poorly . Thus , we will consider instead the following form for the ensembles of base predictors : ∀x ∈ X , f ( x ) = p∑ l=1 Nl∑ j=1 αl , jQ ( l|x ) hl , j ( x ) , ( 3 ) where Q ( l|x ) denotes the conditional probability of domain l given x . This helps us account for the fact that the learner combines base predictors from different domains . For a given point x , experts from the domains where x is more likely to appear are allocated more weight in the voted combination . The following result further substantiates the hope for this method to succeed . Proposition 2 . For the same distributions D1 and D2 and hypotheses h1 and h2 as in Proposition 1 , the equality L ( Dλ , ( αQ ( 1|· ) h1 + ( 1− α ) Q ( 2|· ) h2 ) ) = 0 holds for any λ ∈ ∆ and any α ∈ ( 0 , 1 ) . The proof is also given in Appendix B . Thus , our Q-ensembles can be substantially more effective in the multiple-source adaptation problem considered . Note that , in the special case of a single domain , the Q-combinations coincide with standard convex combinations , since Q ( k|x ) = 1 for all x . As in the standard boosting scenario , we will adopt a weak-learning assumption . However , our assumption here must hold for each source domain : for each domain k ∈ [ p ] and any distribution D over Sk , there exists a base classifier h ∈ Hk such that the weighted loss of h is γ-better than random : 1 2 [ 1− Ei∼D [ yk , ih ( xk , i ) ] ≤ 1 2 − γ , for some edge value γ > 0 . This is equivalent to the existence of a weak-learning algorithm for each domain , which is a mild assumption . As in the standard boosting scenario , this corresponds to the existence of a a good rule of thumb for each domain . Unlike the standard scenario , however , here , we seek a Q-ensemble and further require that ensemble to be accurate for any target mixture Dλ , λ ∈ Λ . Note that the definition of the hypothesis set Hk and the weak-learning assumption for domain k are intimately related : the elements of Hk are typically simple predictors sufficient to guarantee the weak-learning assumption for that domain . In the next section , we present an algorithm , MULTIBOOST , for deriving an accurate Q-ensemble for any target mixture domain that belongs to the convex combination of the source domains . In Appendix L , we further present a multi-class extension of our algorithm , MCMULTIBOOST .

Summary: Utilizing recent formulations/results, the authors propose MutliBoost, a multi-source ensemble learning algorithm where: (1) it is assumed that any target domain is a mixture of the source domain distributions and (2) the learned function form is a convex combination of base functions weighted by a domain identification classifier, Q(k|x), and referred to as Q-ensembles. The domain identification classifier can be learned from unlabeled data (other than the domain label) and an important innovation as it leads to a more natural multi-source Boosting objective and generalizes many existing (and widely-used) multi-source Boosting works. This formulation aligns with recent results regarding agnostic loss [Mohri, Siven, and Suresh; Agnostic Federated Learning, ICML19] and is able to use related findings to prove favorable theoretical properties wrt previous work and leads to FedMultiBoost, a federated learning variant. Experiments are conducted on five binary datasets converted to multi-domain settings (sentiment analysis, hand-writing recognition, object recognition, and UCI adult tabular data) against natural single domain transfer and combined domain data baselines, showing statistically significant improvements. Several additional experiments are included in the Appendices to also show further results (e.g., domain-identification classifier, FedMultiBoost) that addresses multiple natural questions in the paper.