Boosting with Multiple Sources
Authors: Corinna Cortes
Mehryar Mohri
Dmitry Storcheus
Ananda Theertha Suresh
Venue: NeurIPS 2021 Poster (2021)
Keywords: boosting
multiple sources
generalization bounds
federated learning

Abstract:  We study the problem of learning accurate ensemble predictors, in
  particular boosting, in the presence of multiple source domains. We
  show that the standard convex combination ensembles in general
  cannot succeed in this scenario and adopt instead a domain-weighted
  combination. We introduce and analyze a new boosting algorithm,
  MULTIBOOST, for this scenario and show that it benefits from
  favorable theoretical guarantees. We also report the results of
  several experiments with our algorithm demonstrating that it
  outperforms natural baselines on multi-source text-based,
  image-based and tabular data. We further present an extension of our
  algorithm to the federated learning scenario and report favorable
  experimental results for that setting as well. Additionally, we
  describe in detail an extension of our algorithm to the multi-class
  setting, MCMULTIBOOST, for which we also report
  experimental results.

Boosting with Multiple Sources
Corinna Cortes Google Research New York, NY 10011
Mehryar Mohri Google Courant Institute New York, NY 10012 mohriggoogle. com
corima@google.com
Dmitry Storcheus
Ananda Theertha Suresh Google Research New York, NY 10011 theertha@google.co com
Courant Institute & Google New York, NY 10012
dstorcheus@google.o
Abstract
We study the problem of learning accurate ensemble predictors, in particular boosting, in the presence of multiple source domains. We show that the standard convex combination ensembles in general cannot succeed in this scenario and adopt instead domain- weighted combination. We introduce and analyze new boosting algorithm, MULTIBOOST, for this scenario and show that it benefits from favorable theoretical guarantees. We also report the results of several experiments with our algorithm demonstrating that it outperforms natural baselines on multi-source text-based, image-based and tabular data. We further present an extension our algorithm to the federated learning scenario and report favorable experimental results for that setting as well. Additionally, we describe in detail an extension of our algorithm to the multi-class setting, MCMUI.TBBOOT, for which we also report experimental results.
Motivation
Ensemble methods such as Bagging, AdaBoost, Stacking, error-correction techniques, Bayesian averaging, AdaNet or other adaptive methods for learning neural networks are general machine learning techniques used combine several predictors to devise
accurate (Breiman, 1996; Freund and Schapire, 1997; Smyth and Wolpert, 1999; MacKay, 1991; Freund et al., 2004; Cortes et al., 2014; Kuznetsov et al., 2014; Cortes et al., 2017). These techniques are often very effective in practice and benefit from favorable margin-based learning guarantees (Schapire et al., 1997). These algorithms assume access to a training sample drawn from the target distribution. But, in many applications, the learner receives labeled data from multiple source domains that it seeks to use find a good predictor for target domains, which are typically assumed to mixtures of source distributions (Mansour et al., 2008, 2009a; Hoffman et al., 2018, 2021; Cortes et al., 2021; Muandet al., 2013: Xu al., 2014; Hoffman et al., 2012: Saito et al., 2019; Wang et al., 2019a). How can we generalize ensemble methods such boosting to this scenario?
Several related problems have been tackled in previous work. In the special case of single target domain, boosting solutions have been derived (Dai et al., 2007; Yao and Doretto, 2010). These algorithms have been further improved by boosting base predictors jointly with source features (or feature views) that are predictive in the target ( (Yuan et al., 2017: Cheng et al., 2013; Zhang et al., 2014; Xu and Sun, 2012, 201 Such methods have been widely adopted in various domain adaptation scenarios for different types of data. For example, Huang et al. (2010, 2012) showed that by selecting base learner jointly with feature that is predictive across multiple domains every boosting step, one can achieve higher accuracy than standard transfer learning methods. Moreover, the margin
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

provided by boosting-styl style algorithms can aid in transfer learning where target domain is unlabelled. Habrard et al. (2013) have developed an algorithm that jointly minimizes the source domain error and margin violation proportion on the target domain. Wang et al. (2019a) have demonstrated that boosting classifiers from different domains can be carried out online. Taherkhani et al. (2020) and Becker et al. (2013) have shown that multi-source boosting can be combined with Deep Neural Networks for multi-task learning on large scale datasets.
We give a more detailed discussion of this related prior work in Appendix A. However, as we demonstrate in this paper, several critical questions related to the presence of multiple domains have not been fully addressed by existing boosting methods from both the algorithmic and the theoretical point of view. First, what should be the form of the ensemble solutions? We show that, in general, the standard convex combinations used in much of previous work may not succeed in this problem (Section 2). Instead, we put forward Q-ensembles, which are convex combinations weighted by a domain classifier Q, that is, Qkk[z)) is the conditional probability of domain & given input point This is inspired by similar Q-ensembles (Cortes, Mohri, Suresh, and Zhang, 2021) or distribution- weighted combinations (Mansour, Mohri, and Rostamizadeh, 2008, 2009a; Hoffman, Mohri, and Zhang, 2018, 2021) used in the context of multiple-source adaptation and crucially differentiates our work from that of past studies of ensemble methods. Our learning scenario strictly generalizes previous algorithms. In the special case of a single domain, the form of our ensemble solutions coincides with that of the familiar ensembles used in AdaBoost. In Appendix J, we further compare our results and guarantees with the multiple-soure sourre adaptation algorithms of (Hoffman, Mohri, and Zhang, 2018, 2021; Cortes, Mohri, Suresh, and Zhang, 2021), when using AdaBoost to derive predictor for each domain.
Second, what should be the form of the objective? Unlike existing boosting methods which optimize for a specific target distribution or domain, we seek a solution that is accurate for any mixture of the source distributions, where the mixture weights may be constrained to be in subset of the simplex. This conforms with the learning goal in this scenario where the target distribution is typically assumed to be a mixture of the source ones and is further inspired by the formulation of the agnostic loss in the related context of federated learning (Mohri, Sivek, and Suresh, 2019). Additionally, this formulation can be viewed as more risk-averse and robust, and it also ensures more favorable fairness properties, since it does not favor any distribution possibly biased towards some subset of the source domains. In Section 3, we introduce and analyze an algorithm, MUI.TIB that is based on such an objective. This further distinguishes our boosting algorithm from related prior work. In Appendix L,, we further give detailed description of multi-class extension of our algorithm, MCMULTIBOOST, including its pseudocode, the discussion of some of its properties, and the results of several experiments.
Third, in Section 4, we present theoretical analysis of our MULTIBOOST algorithm, including margin-based generalization bounds that hold for any target mixture of the source distributions. In Appendix F, we derive finer and more flexible learning bounds that can provide more favorable guarantees, in particular when the family of target mixtures is more constrained.
Fourth, in Section 5, we describe FEDMUL.TIBOOST, an extension of our MULTIBOOST algorithm adapted to the distributed learning scenario of federated learning, which is increasingly important in wide array of applications (Kairouz et al., 2021), including healthcare (Brisimi et al., 201 8). FEDMULTIBOOST allows the server to train ensemble models that benefit from clients' data, while the data remains with clients. We provide detailed description of our algorithm, as well brief analysis of the communication costs, which are critical in this scenario. Appendix contains experimental results comparing FEDMULTIBOOSS with several baselines.
Finally, in Section 6, we report the results of extensive experiments with our MULTIBOOST algorithm. We start with a detailed description of the learning scenario we consider.
2 Learning scenario
Let X denote the input space and y = [-1,+1] the output space associated to binary clas- sification. We consider a scenario where the learner receives labeled samples from p source domains, each defined by a distribution 'Dz over X x y, E [p]. We denote by S, = (( 1)..)..... (m (k,m)) )) E (X x y) the labeled sample of size my received from source
k. which is drawn i...d from Dg. For any function f: X R and distribution D over X x y, let
2

---

f) be the expected loss of f, that is f) = B(n (((f(())( v)], where is the binary loss ((f(z), v) = (z) - 0).
For any k E [p]. let H2 denote hypothesis set of functions mapping from X [-1, +1]. |9Cg| = Ni- The objective of the learner is to find a predictor f that is accurate for any target distribution Dx that is mixture of the source distributions, where À may be in subset A of the simplex. Thus, Dx
can written as > = Di= DRD with = (AI...., Ap) 2 0 and Ei=1 de = 1. This leads to the following agnostic loss of predictor (Mohri et al., 2019):
f) = max f). xEA
(1)
To come up with a predictor f. the learner seeks an ensemble of functions from the base classes Hg- What should be the form of such ensembles? A natural solution is convex combinations of the form
1=1
where hi, E H4 and on, 2 0. Ei= Eド= Oi,j = 1, as in prominent algorithms such as BAGGING (Breiman, 1999) or ADABOOST (Freund and Schapire, 1997). It is not hard to show, however, that for some distributions, any such convex combination of base predictors would lead to a poor solution. Proposition 1. There exist distributions Dj and D2 and hypotheses h1 and h2 with L(D), h1) = and C(D2. h2) = 0 such that ( (D1 + D22, ah1 + (1 - aa)h2) 2 1 for any a E [0, 1].
This result is similar to (Mansour et al.. 2008, Theorem 1) or similar statements in (Mansour et al., 2008, 2009a; Hoffman al., 2018, 2021; Cortes et al., 2021), however, there are some technical differences. In particular, we are considering here binary classification loss and not the absolute loss. The proof is given in Appendix B.
Note that in the example of the proposition, the hase predictors h1, h22 are both perfect for their respective domains. Nevertheless, unlike the common case of single source domain, standard convex combinations in general may perform poorly. Thus, we will consider instead the following form for the ensembles of base predictors:
Ve E X, f(x) = ΣE Σ on, Q(1x))()))
where Q(I[x:) denotes the conditional probability of domain given x. This helps us account for the fact that the learner combines base predictors from different domains. For given point x. experts from the domains where is more likely to appear are allocated more weight in the voted combination. The following result further substantiates the hope for this method to succeed.
For the same distributions D2 and DD and hypotheses hı and hy as in Proposition I, the equality L(Dx, (aQ(1)-)/a + (1 (22(()))))) 0 holds for any X E A and any (0,
Proposition 2.
The proof is also given in Appendix B. Thus, our Q-ensembles can be substantially more effective in the multiple-source adaptation problem considered. Note that, in the special case of single domain, the Q-combinations coincide with standard convex combinations, since Q(k)z)) = for all x.
As in the standard boosting scenario, we will adopt weak-learning assumption. However, our assumption here must hold for each source domain: for each domain E [p] and any distribution D over Se, there exists base classifier h E HG such that the weighted loss of h is 7-better than random: [[ E,- [yk, (y,,,/(rk,. ,((rx)))) ~ 7, for some edge value y > 0. This is equivalent to the existence of weak-learning algorithm for each domain, which is mild assumption. As in the standard boosting scenario, this corresponds to the existence of a good rule of thumb for each domain. Unlike the standard scenario. however, here, we seek a Q-ensemble and further require that ensemble to be accurate for any target mixture Dx, À E A. Note that the definition of the hypothesis set Hy and the weak-learning assumption for domain i are intimately related: the elements of Hg typically simple predictors sufficient to guarantee the weak-learning assumption for that domain.
In the next section, we present an algorithm, MULTIBOOST, for deriving accurate Q-ensemble for any target mixture domain that belongs to the convex combination of the source domains. In Appendix L, we further present multi-class extension of our algorithm, MCMULTIBOOST.
3

---

3 Algorithm
Let + be convex, increasing and differentiable function such that - ((-u) upper-bounds the binary loss u + lugo- + could be the exponential function as in ADABOOST or the logistic function, as in logistic regression. Using Φ to upper-bound the agnostic loss leads to the following objective function for an ensemble f defined by (3) for any a (ons) 2 0:
P
N
F(a)=
max ΛEλ
ykes
ΣΣ an, Al
(4)
m
Since a convex function composed with an affine function is convex and a sum of convex functions is convex, F is convex as the maximum of a set of convex functions. In this section, we will consider the case where the set A coincides with the full simplex, that is A = A. F can then be expressed more
simply as F = max FE. with (a( 1
Σ m + (- E وソ= Mi
O1 ((,),)
ma
It known that ADABOOST coincides with coordinate descent applied to an exponential loss objective function (Duffy and Helmbold, 1999; Schapire and Freund, 2012; Mohri al., 2018). Our algorithm similarly applies coordinate descent to the objective just described, as with other boosting-type algorithms (Mason et al., 1999; Cortes et al., 2014; Kuznetsov al., 2014). While convex, F is not differentiable and, in general, coordinate descent may not succeed in such cases (Tseng, 2001; Luo and Tseng, 1992). This because the algorithm may be stuck at point where no progress is possible along any of the axis directions, while there exists a favorable descent along some other direction. However, we will show that, under the weak-learning assumption we adopted, any point or and for each active function Fk. that is F& such that (x) = F(a), there exists a coordinate direction along which descent progress is possible. We will assume that these directions are also descent directions for F. More generally, it suffices in fact that one such direction admits this guarantee. Alternatively, one can replace the maximum with a soft-max, that the (.i....,, .x) + log(E e(a*) ) for
f > 0, which leads to a continuously differentiable function with close approximation for ja sufficiently large.
Description. Let 1 denote the value of the parameter vector (ar.i) the end of the (l- 1)th iteration and ft-1 the corresponding function: ft-1 = Σ Σx (. hr. Coordinate descent at iteration t consists of choosing a direction Cg.r corresponding to base classifier hgr and a step value 77 > 0 to minimize F(a,-1 + n,,rr))
To select a direction, we consider the subdifferential of along any eg. Since functions F& are differentiable, by the subdifferential calculus for the maximum of functions, the subdifferential of F at a:-1 along the direction eyrr is given by:
OF 0q.-) conv (F
E Kr)..
where F (a Cg.r) ) is the directional derivative of F2 a;-1 along the direction eq. and where K, = [k E p]: (a)-)))) = (a.))))) We will therefore consider the direction egr with the largest absolute directional derivative value [a,-1, ey.r),, & K2, but will restrict ourselves to q = since, as we shall see, that will suffice to guarantee a non-zero directional gradient. To do so, will express (k, in terms of the distribution (,(k, :) over Sy defined by for all i E [ma]]:
(k( = v'(-uesh ((Fa,))) with Ze,K Em
Zekk
me
7t, [2e - 1],
F Fg hk.r) =
yk. Uk. QKk] (on, (-
m
where Er,A,r = 1 E,DD lvv. Q(k[ hh,, (Zx,i)] denotes the weighted error of Q(kl)-)/r.r For any E [mx], since IA,, is a sample drawn from Dr, we have Q(k)=k,s) > 0 and therefore we have: Q Emh Dr(k, s) Q ((k)Ik,)) O. Thus, we can write E..D. [y,, ( Q(kl as
Dr(k, mg
(2x..) Q
E inD)(H.))
(u Qe,.
where (k(,),)) i)
By our weak- leaknllernnng assumption, there exists r E N& such that
Q.e
Ei-D;(K,)) (Zx..)) 2 7
0. For that choice of we have kt < - with 7Q,, 0.

---

In view of that, it suffices for us to search along the directions hg.r and we do not need to consider the directional derivative of Fú. along directions hgg with q * k. The direction chosen by our coordinate descent algorithm is thus defined by: R mh [1 2 Given the direction ek, the optimal step value 7 is a F(a-11 + n,,,))) The pseudocode of our algorithm, MULTIBOOST, is provided in Figure I. In the most general case, 1) can be found via a line search or other numerical methods.
21,"
Step size. some special cases, the line search can be executed using simpler expression by using an upper bound on F(at-1 + 1.Ck,r). ). Using the convexity of tb since yt. (e 1+w. ((2..)
(+1) + 1- on .)...)))).)) (-1), the following holds for all 1 E R:
2
2
$( ((-waff-i(m))) n (I(A) & H.,n, (ng Shr.., ( Vt. (311)) - 11)
2
1-m. L.M..Q( (wL.) T nr (...ff-1(211,)) 1))
2
In the case of exponential and logistic functions, the following upper bounds can then be derived.
Lemma 1. For any k [p], the following upper bound holds when & is the exponential the logistic function:
Ze,1 [[1 - (L,,,,)))) +E..,,,) e.I.,re)
F(a--1 + mox,r) S max LE(p1
my
where f.t.krr 3 E..DD(1.)) Q(k))))))) (xt,)]]
For any k, function 1) (1 et rr, " reaches its minimum for v 1 log 1- When the maximum is achieved for = k. the solution coincides with the familiar expression of the step size = 2 log 1-s,,k.r used in ADABOOST Ce.k.r
Q-function. The conditional probability func-
tions Q(kl-)) are crucial the definition of McLTIIBOSTT(S,..... our algorithm. As pointed out earlier, Q- ensembles can help achieve accurate solutions in some cases that cannot be realized using
an +0
2
for t t- to do
3
fi f- Ep 2i= Nr 11-111 Q(1-)) his
convex combinations. Furthermore, for any
l E [p), since (k,,.) -) is distribution over the sample Sg, is natural to assume that for any j * k we have EgD..((k.)) (oo [Q(k) >
D f- 1 -E ( yk. fi (x
me
KG + { k: k argmax*e(r) &
6
for k E Kr do
E,-D, [Q(j|æR..)) (Qae) lowing lower bound: (...).((.)) [Q(k(=r).))) 2 [
This implies the fol-
Ze + Ema D (- (2)6,))
iil
for + to mg do
p> which in turn implies 7 2 3, since for any x E X, Σn= Q(ilz)
(k &- 4'(-ve (--ve.ff- (1)
1. In the special case where all domains coincide, we have Q(k|zk,e) = p for all and this lower bound is reached. At another extreme, when all domains admit distinct supports, we have Qkk[r.ss) 1 for all E [m+] and thus 7=
Z,
10
(k,r) + argmax
[1 2,r Z:
kEN,, mg
11
Te +- argmin F(a--1 + T)e,r)
12
a, + a1-1 + meC
13
f f Ei= Σn= aT, Q(I-))ii.
return f 14
In practice, we do not have access to the true conditional probabilities -kk--). Instead, we
Figure 1: Pseudocode of the MULTIBOOST algorithm. CrA,r 1/2 [1 BE-DD [ Quk de-
can derive accurate estimates @(kl-) using notes the weighted error of Q(kl-))a...
large unlabeled samples from the source do-
mains, the label used for training being simply the domain index. This can be done using algorithms such conditional maximum entropy models (Berger et al., 1996) (or multinomial logistic regres- sion), which benefit from strong theoretical guarantees (Mohri et al., 2018, Chapter 13), or other rich models based neural networks.
Other variants of MULTIBOOST. In Appendix D, we present and discuss other variants of our algorithm, including their convergence guarantees.
5

---

Theoretical analysis
In this section, we present a theoretical analysis of our algorithm, including margin- margin-based learning bounds for multiple-souree Q-ensembles.
For any k [p], define the family 95 follows: 9x {{((k)-) h: h He). Then, the family of ensemble functions F that we consider be defined as F conv(U Sk). For any À E A, let D be the distribution defined by D> = Ek= ArDxx. where Dr is the empirical distribution associated to an i.i.d. sample Sy drawn from DR- Dx is distinct from the distribution associated to 'Dx- We seek to derive margin-based bounds for ensemble functions f E F with respect to target mixture distributions Xx, while the empirical data is from multiple source distributions DR, or from D,,, Thus, these guarantees differ from standard learning bounds where the source and target distribution coincide.
For any distribution D over X x y and p > 0, let (D, f) denote the expected p-margin loss (e (T)(()) S p)]. We denote by Rm (F) the Rademacher complexity of I' for samples S = (I1....,, (mm) of size m, defined by R., (3) Es-p" [sup Ems ozf((=)], with ogs independent random variables taking values in f-1, +1}.
of with respect to D: (D,
The following gives a margin- bound for our multiple-source learning with Q-ensembles.
Theorem 1. Fix p > Then, for any d > O, with probability at least over the draw of a sample S = (S
................................ Sp) ~ DTt @ ... @ p the following inequality holds for all ensemble functions Ei= Q((()))) I and all A:
P
2
log f 2mT
Ab
max Rax (JG) (E(P)
(5)
Note that the complexity term depends only on the Rademacher complexities of the families of based predictors HG and does not involve the domain classification function Q. Theorem improves upon previous known bounds of Mohri et al. (2019). In particular, it provides a dinension-indeeeedden margin guarantee for the zero-one loss, while the bound of Mohri et al. (2019), when applied to the zero-one loss, is dimenniiondeeeendee holds for Q-ensembles. Our margin bound can be straightforwarcly generalized to hold uniformly for all E (0, 1), at the cost of an additional mild term depending log logg (2/p) (see Theorem 4 in Appendix E). For the ensemble functions returned by our algorithm, the bound applies to x ++ ||l|]
(Mohri et al., 2019, Lemma 3). Additionally, our bound
f(x)
These learning guarantees suggest choosing a 2 0 and p to minimize the following:
T
ai Q(kel-)ho(a)) sp )
Tk
max XEA
A
(z.v)moE
E
|a
+
P
where тą max Rmm (HG). Choosing 1 ||a||] and upper bounding the binary loss using convex function 4, the problem can then be equivalently cast as the following unconstrained minimization, for example in the case of the exponential function:
a (((( rh |alla malllla
min max a2 XEA
E
cxp
y
(x.g)~.s
Thus, the learning guarantees justify posteriori an éi-regularized version of our algorithm. It is straightforward to derive that extension of our algorithm and its pseudocode. This is similar to the extension to the AooSS (Rätsch al., 2001) of the original unregularized ADABoOST (Freund and Schapire, 1997). Let us add that finer learning guarantees such as those for DEEP BOOSTING (Cortes al., 2014) ones can be derived similarly here, which would lead a weighted & f--reulaarizatoo, where the weights are the Rademacher complexities of the families Hg. For the sake of simplicity, do not detail that extension. Let us point out, however, that such an extension would also lead to an algorithm generalizing ADANET (Cortes et al., 2017) to multiple sources.
In some applications, prior knowledge be used to constrain A to strict subset of the simplex A. Finer margin- based learning guarantees can be derived to cover that scenario. Let A be the set of

---

vertices of a subsimplicial cover of A, that is a decomposition of a cover of A into subsimplices and let A₂ be the set of Às c-close to in & fi-distance. Then, the following guarantees hold.
Theorem 2. Fix p > 0 and E >> 0. Then, for any 6 > O. with probability at least 1 over the draw of sample S = (S Sp) ~ D @ ... @ Dp each of the following inequalities holds:
r 1. for all ensemble functions f Di=1 ar QKe((ir-) E 3 and all X Ke:
3E
A
h))x. S (D h) max Rm (H,, A) +
log
(6)
rE(m)
2
2m
2. for all ensemble functions f - 1=1
0 Q(Q(k))) E and all X
-E 1rBBx E ccovv(A): (A):
h)Dx,h) S (D(
2
Σ
P
Pa max Rm (Hr, 31) rE(p]
p ph
p
BP 21TER
A log
P 1=1
The proof of the theorem and further discussion are presented in Appendix F. Here, Rm (JC, A) is the X-weighted Rademacher complexity of H, (see Appendix F). For A a strict subset of the simplex, these learning bounds suggest an algorithm with a regularization term of the form Σx= X mm.- Our analysis and learning guarantees can be similarly extended to cover the multi-class setting.
5 Federated MULTIBOOST algorithm
In this section, we present an ex- tension of our algorithm to the federated learning scenario, called TEDMULTIBOOST. The pseudocode given in Figure 2. Federated learning is a distributed learning scenario where a global model is trained at the server level, based on data remaining at clients (Konečný et al., 2016b,a; McMahan al., 2017; Yang et al., 2019). Clients are typically mobile phones, network sensors, other IoT devices. While the data remains at the clients, the trained
FEDMULTIBOOST(SI, S2, S Qg +0
for t t to do
Si-1 t- Ei=r En an (1.1110(11)) his
Cr. + -SELECCCLIENT((K1
for t 1 do
for Ca, do
LK + mima Eis 2 (---
Ze + Eim 2'( -e (Io,i)
KG + k [p]: E argmaxke(p EcEch @
model significantly benefits from user data, as demonstrated, for example, in next word prediction (Hard al., 2018:
10
Ze, Eoec,, Ze,0s Vk E
11
He.k + SELECTBASECLASSITEERE
12
for k E Kr do
Yang et al., 2018) and healthcare appli- cations (Brisimi et al., 2018). We refer
13
for Cr,k do for + 1 to m do
14
the reader to (Kairouz et al., 2021) for
15
De(c, + 4 ( (x
Zenk
more detailed survey of federated learn- ing.
16
Bter + [1 221
17
(k. + argmax
kEK..EEHe.. Exeo, me Ececr Beer
Zen
widely used algorithm in this sce-
nario is federated averaging, where 18
1) x n////
the server trains the global model via 19
04 - 0t + mee
stochastic updates from the clients
20
Ei= Σ OT, Q(1-))u
(McMahan et al., 2017). It has been argued by Mohri et al. (2019), how- ever, that minimizing the expected loss with respect to particular training dis- Figure 2: Pseudocode of the FEDMULTIBOOST algorithm. tribution, as done by standard feder- tte.r denotes the weighted emor of Q(k)-))e,r. ct.e.r ated learning algorithms. may be risk- ([1-E,_D prone and benefit only a specific subset of clients. This compromises fairness, which is one of the critical questions in federated learning, where clients are often heterogeneous (Bickel et al., 1975; Hardt et al., 2016; Abay et al., 2020; Li et al., 2020). This issue can be further
21
return f
Q(k] (ha., where client e be-
longs to domain k. cobalt steps are carried out by the server, red on the clients.
7

---

accentuated when the set of participating clients during inference may significantly differ from those used in training. Such cases may occur, for example, when clients loose access to network connection during training, which results in distinct training and test distributions.
To overcome these problems, we adopt the agnostic federated learning approach suggested by Mohri et al. (2019) and Ro et al. (2021). Following the ctient-partition approach of Ro et al. (2021), let each client belong to one of p domains. The domains can be, for example, based on the type of device or their geographical location. In this setting, the global model is optimized for all convex combinations of domain distributions as in the objective of MULTIBOOST (Equation 4), by taking the maximum over the mixtures weights X. This ensures that the solution obtained is risk-averse and reduces the worst-case mismatch between the inference and training distributions. We propose FEDMULTIB a communication-efin -effiiitiooeefiiinntt modification of MULTIBOOST, that can overcome the communication bottleneck in federated learning (Konečný al., 201
Before presenting our algorithm, we first describe the learning scenario and the notation we adopt. Let be the number of clients, which in the cross-device setting can be in the order of millions. Let me denote the number of samples in client e and (Zo, Vo.s) denote the شD sample of client c. The server has access to set of pase-predictor classes H2 for k E [p] and a domain classifier Q that assigns for each sample x, the likelihood of it belonging to each of the p domains. The base classifiers can be obtained by training on public data or training on client data with federated learning. The domain classifier Q can be trained using unlabeled client data.
Additional discussion of the algorithm well as experimental results be found in Appendix G. At each round, the algorithm randomly selects r clients and base classifiers from each domain. Next, it chooses the best classifier out of the previously selected subset, based on the data from the subsampled clients. Since FEDMULTIBOOL at each round, the algorithm is communication-efficii ft-1, and s base classifiers to the selected clients. Hence, the server-to-client communication cost at round is O(( + s)K + K"), where K is the number of parameters K' is the number of parameters required to specify Q. Furthermore, the clients communicate only the aggregate statistics ... Zr.c: St.,r to the server and hence the client-to-serve communication is O(p s) real numbers, which is independent of the number of parameters of the base classifiers. We note that FEDMULTIBOOST can be extended to the multi-class setting using techniques presented in Appendix L.
operates only on small set of base classifiers At round t, the server needs send Q,
single base classifier and
6 Experiments
In this section, we present experimental results for the MULTIBOOST algorithm on several multiple- source datasets. Our study is restricted to learning an ensemble of decision stumps Hstamps using the exponential surrogate loss (uu) = e " To estimate the probabilities Q(kl-) for k E [p), we assigned the label & to each sample from domain k and used multinomial logistic regression. This step can be facilitated with unlabeled examples, as only their domain membership information is required.
We compare MULTIBOOST with a set of boosting-styy algorithms that operate on the same hypothe- class Hstmmps Those include ADA BOASOOTT-k for k [p) and ADA The former is standard ADABoOST algorithm trained only on a single source S₁ and the latter is ADA BoOST trained on the union of all sources Sk. It is natural to compare MULTIBOOST against ADA BOoST-all, since both of them have access to all sources during training. The difference is that while all treats - S as a single dataset, MULTIBOOST trains base learners separately for each source and weights examples by domain probabilities Qk)--). We used T = 100 boosting steps for all benchmarks. T up to 1,000 were explored, but did not change any results significantly.
We also compare our results with the a multiple-source adaptation algorithm, DMSA A, designed for a scenario where no labeled data is available, and where only an accurate predictor per domain and unlabeled data are supplied. DMSA A was shown to outperform other algorithms designed for that scenario. To apply DMSA,
the domain predictors ADABOOST-K, & [p)-
We report classification errors on the test data for various mixtures of the source distributions A1..... Ap. including: errors for X the simplex A edges (errors each domain separately); emrors on the uniform mixture Vk: A = f; agnostic error, which is the maximum error across all sources. The errors and their standard deviations are reported based on 10-fold cross validation. Each source
8

---

S₁ is independently split into 10 folds s....., s For the i-th cross-validation step, the test set is while the rest is used for training.
Datasets and preprocessing steps used are described below with additional dataset details provided in Appendix H. Note, that all datasets are public and do not contain any personal identifiers or offensive information. The experiments were performed on Linux and Mac workstations with Quad- Core-Core Intel Core i7 2.9 GHz and Intel Xeon 2.20 GHz respectively. The time taken to perform 10-fold Pross-validation varies per dataset: from roughly 1 hour on CPU for all benchmarks on tabular data to 8 hours on CPU for all benchmarks on digits data. The run-times of a single MULTIBOOST iteration is almost identical to that of ADABOOST when closed form solution is used for step size 1/2 (line 12 in Figure 1) for the exponential surrogate loss function. Alternatively, for some experiments we used line search with 100 steps. If training time is critical, one can use 7 = C/ instead, at the cost of slightly worse convergence guarantee. Convergence plots are illustrated in Appendix I.
Sentiment analysis. The Sentiment dataset (Blitzer et al., 2007) consists of text reviews and rating labels for products sold on Amazon in various categories. We selected p 3 sources from this dataset, where = corresponds to books, = means dvd, and A = 3 is electronicc CE. We converted the 5-star rating to binary labels, assigning y +1 for a positive review and y = -1 for a negative one. Neutral reviews with 3-star labels are removed from the dataset. The benchmarks are trained using bigram features generated from the raw text reviews. Since the Sentiment dataset contains only 2,000 instances for each domain, we used random train/test splits with 10 different seeds instead of the 10-fold validatiii,..nn. To illustrate the importance of the conditional domain probabil- ities ((] for each domain, projections of x onto the first principal component of the joint Sentiment dataset in Figure 3. The figure shows that our domain classifiers are able to provide coherent separation between the three domains and narrow the applicability of the base classifiers. In Appendix K. we further discuss the importance of obtaining a high-quality Q function and illustrate the performance degradation resulting from a lower-quality Q function.
Deraties Daag, preected on the ferst grame corgonor.
illustrate its values along the
60
Figure 3: Mean values of Q(k)-u) for the three domains of the Senti- ment data projected on the first prin- cipal component of the joint data.
Digits recognition. For this problem, we aggregated 32x32 pixel handwritten digits images from p 3 sources: MNIST (k= 1), SVIN (k= 2), MNIST-M (k 3). We compared algorithms on two binary classification problems: digits 4 vs. 9 and digits vs. 7.
Object recognition. We divided images of clothes items from Fashion-MNIST (Xiao et al., 2017) dataset into two classes: tops and bottoms from p = 3 sources. = 1 consists of t-shirts, pullovers, trousers; k = 2 consists of dresses, coats, sandals; k = 3 consists of shirts, bags, sneakers, ankle-boots.
Tabular data. In addition to text and images, we tested MULTIBOOST on tabular data. We used the Adult dataset (Kohavi, 1996), which consists of numerical and categorical features describing an indi- vidual's 's socioeconomic characteristics, given the task to predict if an person's income exceeds USD 50, 000. We divided this dataset into p = 3 sources based on individual's is educational background. Source k- = 1 consisted of individuals with university degree (BSc, MS or PhD), source k = those with only a High School diploma, and source l: 2, none of the above.
Discussion As can be seen from Table 4, MULTIBOOST provides agnostic and uniform errors that are significantly better than the baselines on all datasets. While ADABOOST-k predictors on digits recog- nition and object recognition tasks each show the lowest error on their specific domains D1, D2, Dz, they fail to generalize to other target distribution conv (D D,, Dz). Moreover, as suggested by the discussion in Section 2, the standard convex combination of domain- specific predictors (ADABoosT-all) also performs poorly on several target distributions in conv (D, D2, Da). such as the uniform and agnostic targets. As the experimental results confirm, BULTIBOOST. by leveraging domain predictors Q(kl-) and the agnostic loss, is able produce an ensemble of domain-specine predictors that generalizes to different targets. Appendix I provides additional illustrations of Q(k)-) by projecting each domain on the first principal component of the joint data.
Moreover, for tabular data and the digits (4 vs. 9) classification problem. MULTIBoOST benefits from positive knowledge transfer and obtains an error on individual domains that is even smaller than that

---

Table 1: Test for multiple benchmarks.
Algorithm
Error-1
Error-2
Error-3
Error-Uniform
Etror-Agnosio
Sentiment Analysis
ADABOOST-I
0.326
0.019
0.300 + 0.008
0.360 + 0.017
0.329 0.017
0.360
0.019
ADABOOST-2
0.354 0.020
0.266 + 0.009
0.336 + 0.023
0.318
0.357 0.019
ADABOOST-3
0.402 0.015
0.334 0.008
0.258 + 0.015
0.331 0.016
0.402
ADABOOST-all
0.354 +
0.325
0.313 0.022
0.324 0.021
0.016
DMSA
0.332 + 0.021
0.308 0.017
0.314 0.015
0.318 0.332 0.019
MULTIBOOST
0.332 + 0.027
0.288 + 0.018 0.284 + 0.027
0.301 = 0.027
0.332 + 0.024
Digits Recognition (4 vs. 9)
ADABOOST-1
0.044 0.007
0.615 0.012
0.476 0.022
0.379 0.008
0.015
ADABOOST-2
0,455 0.014
0.299
0.504 + 0.015
0,420
0.00
ADABOOST-3
0.549 0.034
0.300 + 0.013
0.446
0.04
ADABOOST-all
0.060 0.009
0.353 0.374
0.012
0.262
0.014
DMSA
0.069
0.005
0.351
0.310
0.243 0.35 0.009
MULTIBOOST
0.096 + 0.008
0.283 + 0.028 0.246 + 0.014
0.209 = 0.013
0.027
Digits Recognition (1 vs.
ADABOOST-1
0.005 0.002
0.613 + 0.007
0.519 0.012
0.379 0.004
0.00
ADABOOST-2
0.431 0.022
0.252 0.009
0.479 + 0.012
0.387
0.07
ADABOOST-3
0.680 0.031
0.490
0.244 + 0.012
0.474
0.03
ADABOOST-all
0.014 0.003
0.286
+ 0.010
0.306 + 0.012
0.00 0.202
DMSA
0.012 0.003
0.288
0.017
0.286
= 0.015
0.195
= 0.013
MULTIBOOST
0.026 + 0.004
0.261 + 0.013 0.257 + 0.015
0.181
= 0.005
0.261
0.011
Objects Recognition
(Fashion-MNIST)
ADABooST-1
0.015
0.251 + 0.026
0.602 0.028
0.288 0.017
0.602 0.028
ADABOOST:
0.435 4.007
0.015 + 0.002
0.169 + 0.012
0.173
0.43
ADAB
0.311 0.188
0.097 +
0.014 + 0.002
0.140
0.311
ADABOOST-all
0.036 + 0.004
0.020
0.025 + 0.003
0.027
0.002
0.036 + 0.004
DMSA
0.033 0.008
0.015
0.022 0.003
0.023
0.007
0.033 0.009
MULTIBOOST
0.028 + 0.003
0.015 + 0.003
0.022 + 0.002
0.021 = 0.001
0.028 + 0.003
Tabular Data (Adult Data)
ADABOOST-1
0.201
0.249 0.021
0.215
0.014
0.222
0
ADABOOST-2
0.298 0.011
0.131 0.009
0.142 + 0.006
0.190
0.298
ADABOOST-3
0.225
0.138 0.010
0.133 = 0.011
1.155
0.225
ADA BOOTTalll
0.221 + 0.013
0.134 0.007
0.131 + 0.010
0.155 + 0.004
0.221
0.013
DMSA
0.195 + 0.014
0.131
0.008
0.154
0.005
0.195
0.014
MUL.IIBOSST
0.190 + 0.014
0.132 + 0.008
0.130 + 0.009
0.150 + 0.005
0.190 + 0.014
of the domain-specific predictors. From Figure 6 in Appendix it can be seen that the a-weight for the MULTIBOOST classifier in the 4 vs. problem is heavily centered on D2. but contributions from Dj and D3 adds to the stronger performance. The same figure also illustrates the a-mass distribution for the other classification tasks. The standard convex ensemble ADABOOST-all however, does not seem to exhibit the positive transfer property. Similar conclusions carry over for other target distributions in conv Dy, }. for the sake of simplicity we do not present them here.
7 Conclusion
We presented a new boosting algorithm, MULTIBOOST, for multiple-source scenario a common setting in domain adaptation problems- where the target distribution can be any mixture of the source distributions. We showed that our algorithm benefits from strong theoretical guarantees and exhibits favorable empirical performance. We also highlighted the extension of our work to the federated learning scenario, which is critical distributed learning setting in modern applications.
10

---

Acknowledgments
This work was partly supported by NSF CCF-1535987, NSF IIS- 1618661 and Google Research Award.
References
A. Abay, Y. Zhou, N. Baracaldo, S. Rajamoni, E. Chuba, and H. Ludwig. Mitigating bias in federated learning. 2020.
C. J. Becker, C. M. Christoudias, and P. Fua. Non-linear domain adaptation with boosting. In Proceedings of Advances in Neural Information Processing Systems 26: 27th Annual Conference on NIPS, pages 485-493, 2013.
S. Ben-David, J. Blitzer, K. Crammer, and F. Pereira. Analysis of representations for domain adaptation. In Advances in neural information processing systems, pages 137-144, 2007.
S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, Pereira, and J. W. Vaughan. A theory of learning from different domains. Machine learning, 19(1-2):111-175. 2010.
A. L. Berger, S. D. Pietra, and J. Pietra. A maximum entropy approach to natural language processing. Comp. Linguistics, 22(1), 1996.
J. Bickel, E. A. Hammel, and J. W. O'Connell. Sex bias in graduate admissions: Data from Berkeley. Science, 187(4175):398-404 1975. ISSN 0036-8075.
J. Blitzer, M. Dredze, and F. Pereira. Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification. In Proceedings of ACL 2007, Prague, Czech Republic, 2007.
Blitzer, K. Crammer, A. Kulesza, Pereira, and Wortman. Learning bounds for domain adaptation. In Advances in neural information processing systems, pages 129-136, 2008.
K. Bonawitz, H. Eichner, W. Grieskamp. D. Huba, A. Ingerman, V. Ivanov, C. Kiddon, J. Konečný, S. Mazzocchi, H. B. McMahan, et al. Towards federated learning at scale: System design. arXiv preprint 1902v:1902.001 00006 2019.
Breiman. Bagging predictors. Machine Learning. 24(2):123-140, 1996.
L. Breiman. Prediction games and arcing algorithms. Neural Computation, 11:1493-1517, October 1999.
S. Brisimi, Chen, T. Mela, Olshevsky, I. C. Paschalidis, and W. Shi. Federated learning of predictive models from federated electronic health records. International journal of medical informatics, 1 112:59-67, 2018.
Caldas, P. Wu, T. J. Konečny, B. McMahan, V. Smith, and A. Talwalkar, Leaf: benchmark for federated settings. arXiv preprint arXiv: 2018.
Y. Cheng, G. Cao, X. Wang, and Pan. Weighted multi-source TrAdaBoost. Chinese Journal of Electronics, 22(3):505-510 2013.
C. Cortes and M. Mohri. Domain adaptation and sample bias correction theory and algorithm for regression. Theoretical Computer Science, 519:103-126 2014.
C. Cortes, M. Mohri, and U. Syed. Deep boosting. In Proceedings of ICML, pages 1179-1187, 2014.
C. Cortes, M. Mohri, and A. Muñoz Medina. Adaptation algorithm and theory based generalized discrepancy. In Proceedings of the 2 ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 169-178. 2015.
C. Cortes, Gonzalvo, Kuznetsov, M. Mohri, and S. Yang. Adanet: Adaptive structural learning of artificial neural networks. In Proceedings of ICML, pages 874-883, 2017.

---

C. Cortes, M. Mohri, A. T. Suresh, and N. Zhang. A discriminative technique for multiple-soures adaptation. In Proceedings of ICML, volume 139, pages 2132-2143. PMLR, 2021.
W. Dai, Q. Yang. G.-R. Xue, and Y. Yu. Boosting for transfer learning. In Proceedings of ICML, pages 874-883, 2007. 2007.
N. Duffy and D. P. Helmbold. Potential boosters? In Proceedings of NIPS, pages 258-264, 1999.
Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer System Sciences, 55(1):119-15) 1997.
Y. Freund, Y. Mansour, and R. E. Schapire. Generalization bounds for averaged classifiers. The Annals of Statistics, 32: 12:1699-1722, 2004.
Y. Ganin and V. Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pages 180-1188 PMLR, 2015.
R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580-587, 2014.
B. Gong, Y. Shi, F. Sha, and Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE conference on computer vision and pattern recognition, pages 2066-2077 IEEE, 2012.
A. Habrard, J.-P. Peyrache, and M. Sebban. Boosting for unsupervised domain adaptation. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 433 448. Springer, 2013.
Hamer, M. Mohri, and A. T. Suresh. Fedboost: A communication--eficiinnn algorithm for federated learning. In International Conference on Machine Learning, pages 3973-3983, PMLR, 2020.
J. Hampshire and A. Waibel. The meta-pi network: building distributed knowledge representations for robust multisource pattern recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(7):751-769 1992. doi: 1..1099/34.429111.
A. Hard, K. Rao, R. Mathews, F. Beaufays, S. Augenstein, H. Eichner, C. Kiddon, and D. Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.03604 2018.
M. Hardt, E. Price, E. Price, and Srebro. Equality of opportunity supervised learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Proceedings of NIPS, volume 29, 2016.
J. Hoffman, B. Kulis, T. Darrell, and K. Saenko. Discovering latent domains for multisource domain adaptation. In ECCV, volume 7573, pages 702-715, 2012.
J. Hoffman, E. Rodner, J. Donahue, T. Darrell, and K. Saenko. Efficient learning of domain-invariin image representations. arXiv preprint arXiv:1301.3224 2013.
J. Hoffmann, M. Mohri, and N. Zhang. Algorithms and theory for multiple-source adaptation. In Proceedings of NeurlPS, pages 8256-8266, 2018.
J. Hoffman, M. Mohri, and N. Zhang. Multiple-source adaptation theory and algorithms. Annals of Mathematics and Artificial Intelligence, 89(3-4):237-270) 2021.
Huang, G. Wang, and S. Qin. A novel learning approach to multiple tasks based on boosting methodology. Pattern recognition letters, (1000) 2010.
Huang. G. Wang, and S. Qin. Boosting for transfer learning from multiple data sources. Pattern Recognition Letters, 33(5):568-579 2012.
R. A. Jacobs, M. I. Jordan, S. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
12

---

P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji. K. Bonawitz, Z. Charles,
G. Cormode, R. Cummings, R. G. D'Oliveira, H. Eichner, S. E. Rouayheb, D. Evans. J. Gardner, 7. Garrett, A. Gascón, B. Ghazi, P. B. Gibbons, M. Gruteser, 7. Harchaoui, C. He. I. He, Z. Huo, B. Hutchinson, J. Hsu, M. Jaggi, T. Javidi, G. Joshi, M. Khodak, J. Konečný, A. Korolova, F. Koushanfar, S. Koyejo, TT Lepoint, Y. Liu, P. Mittal, M. Mohri, R. Nock, A. Özgür, R. Pagh, M. Raykova, H. Qi. D. Ramage, R. Raskar, D. Song, W. Song, S. U. Stich, Z. Sun, A. T. Suresh, F. Tramèr, P. Vepakomma, J. Wang, L. Xiong, Z. Xu, Q. Yang, F, X. Yu, H. Yu, and S. Zhao. Advances and open problems federated learning, 2021.
R. Kohavi. Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid. In KDD, volume 96, pages 202-207, 1996.
J. Konečný, H. B. McMahan, D. Ramage, and P. Richtárik. Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint arXiv:1610.02527 2016a.
J. Konečný, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:16100005492 2016b.
N. Konstantinov and C. Lampert. Robust learning from untrusted sources. In International Conference Machine Learning, pages 3488-3498, 2019.
V. Kuznetsov, M. Mohri, and U. Syed. Multi-class deep boosting. Proceedings of NIPS, pages 2501-2509, 2014.
Lahoti, A. Beutel, Chen, Lee, Prost, N. Thain, X. Wang, and E. H. Chi. Fairness without demographics through adversarially reweighted learning. arXiv e-prints, pages arXiv-2006, 2020.
M. Ledoux and M. Talagrand. Probability in Banach Spaces: Isoperimetry and Processes. Springer, New York, 1991.
D. Levy, Y. Carmon, J. C. Duchi, and A. Sidford. Large-scale methods for distributionally robust optimization. Advances in Neural Information Processing Systems, 33, 2020.
T. Li, M. Sanjabi, A. Beirami, and V. Smith. Fair resource allocation in federated learning. In Proceedings of ICLR, 2020.
H. Liao. Speaker adaptation of context dependent deep neural networks. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 7947-7951. IEEE, 2013.
M. Long, Y. Cao, J. Wang, and M. Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pages 97-105. PMLR, 2015.
Z.-Q. Luo and P. Tseng. On the convergence of the coordinate descent method for convex differen- tiable minimization. Journal of Optimization Theory and Applications, 72(1):7-35, 1992.
D. J. C. MacKay, Bayesian methods for adaptive models. PhD thesis, California Institute of Technology, 1991.
Y. Mansour, M. Mohri, and A. Rostamizadeh. Domain adaptation with multiple sources. Advances in neural information processing systems, 21:1041-1048, 2008
Y. Mansour, M. Mohri, and A. Rostamizadch. Multiple source adaptation and the Rényi divergence. In UAI, pages 367-374, 2009a.
Y. Mansour, M. Mohri, and A. Rostamizadeh. Domain adaptation: Learning bounds and algorithms. In COLT, 2009b.
L. Mason, J. Baxter, P. L. Bartlett, and M. R. Frean. Boosting algorithms gradient descent. In Proceedings of NIPS, pages 512-518, 1999.
B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. Arcas. Communication-effiiiann learning of deep networks from decentralized data. In Proceedings of AISTATS, pages 1273-1282, 2017.
13

---

M. Mohri, A. Rostamizadch, and A. Talwalkar. Foundations of Machine Learning. Adaptive computation and machine learning. MIT Press, 2018. Second edition.
M. Mohri, G. Sivek, and T. Suresh. Agnostic federated learning. In Chaudhuri and Salakhut- dinov, editors, Proceedings of ICML, volume 97 of Proceedings of Machine Learning Research, pages 4615-4625. PMLR, 2019.
Motiian, Q. Jones, S. Iranmanesh, and G. Doretto. Few-shot adversarial domain adaptation. In Advances in Neural Information Processing Systems, pages 6670-6680, 2017a.
S. Motiian, M. Piccirilli, D. A. Adjeroh, and G. Doretto. Unified deep supervised domain adaptation and generalization. In Proceedings of the IEEE International Conference on Computer Vision, pages 5715-5725,
K. Muandet, D. Balduzzi, and B. Schölkopf. Domain generalization via invariant feature representa- tion. In International Conference on Machine Learning, pages 10-18. PMIR, 2013.
H. Namkoong and J. C. Duchi. Stochastic gradient methods for distributionally robust optimization with f-divergences. In Advances in Neural Information Processing Systems, pages 2208-2216, 2016.
S. J. Pan and Q. Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, (2110)134551133 2009.
Z. Pei, Z. Cao. M. Long. and J. Wang. Multi-adversarial domain adaptation. In AAAI, pages 3934-3941, 2018.
G. Rätsch, S. Mika, and M. K. Warmuth. On the convergence of leveraging. In NIPS, pages 487-494, 2001.
Ro, M. Chen, R. Mathews, M. Mohri, and T. Suresh. Communication-effiiiee agnostic federated averaging. arXiv preprint arXiv:2104.02744 2021.
K. Saito, D. Kim, S. Sclaroff, T. Darrell, and K. Saenko. Semi-supervised domain adaptation via minimax entropy. In Proceedings of the IEEE International Conference on Computer Vision, pages 8050-8058, 2019.
R. E. Schapire and Y. Freund. Boosting: Foundations and Algorithuss. The MIT Press, 2012.
R. E. Schapire, Y. Freund, P. Bartlett, and W. S. Lee. Boosting the margin: A new explanation for the effectiveness of voting methods. In JCML, pages 322-330, 1997.
Z. Shen, H. Hassani, S. Kale, and A. Karbasi. Federated functional gradient boosting. arXiv preprint arXiv:2103.06972, 2021.
Smyth and D. Wolpert. Linearly combining density estimators via stacking. Machine Learning, 36: 59-83, July 1999.
A. Taherkhani, G. Cosma, and T. M. McGinnity. AdaBoost-cnn: An adaptive boosting algorithm for convolutional neural networks elassify multi-class imbalanced datasets using transfer learning. Neurocomputing, 404:351-366, 2020.
A. Torralba and A. A. Efros. Unbiased look at dataset bias. In CVPR 2011, pages 1521-1528. IEEE, 2011.
Tseng. Convergence of block coordinate descent method for nondifferentiable minimization. Journal Optimization Theory and Applications, pages 475-494, 2001.
E. Tzeng, J. Hoffman, T. Darrell, and K. Saenko. Simultancous deep transfer across domains and tasks. In Proceedings of the IEEE International Conference on Computer Vision, pages 4068 4068-4076. 2015.
B. Wang, J. Mendez, M. Cai, and E. Eaton. Transfer learning via minimizing the performance gap between domains. In Advances Neural Information Processing Systems, pages 10645-10655. 2019a.
14

---

T. Wang, X. Zhang, L. Yuan, and J. Feng. Few-shot adaptive faster R-CNN. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7173-7182, 2019b.
H. Xiao, K. Rasul, and R. Vollgraf. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. CoRR, abs/1708.07747, 2017.
R. Xu, Z. Chen, W. Zuo, J. Yan, and L. Lin. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1964-3973, 2018.
Z. Xu and S. Sun. Multi-view transfer learning with adaboost. In 2011 IEEE 23rd International Conference on Tools with Artificial Intelligence, pages 399-402. IEEE, 2011.
Z. Xu and S. Sun. Multi-source transfer learning with multi-view AdaBoost. In International conference on neural information processing, pages 332-339. Springer, 2012.
Z. Xu, W. Li, L. Niu, and D. Xu. Exploiting low-rank structure from latent domains for domain generalization. In European Conference on Computer Vision, pages 628 248-643. Springer, 2014.
J. Yang, R. Yan, and A. G. Hauptmann. Cross-domain video concept detection using adaptive SVMs. In Proceedings of the 15th ACM international conference on Multimedia, pages 188-197, 2007.
Q Yang, Y. Liu, T. Chen, and Y. Tong, Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1-19, 2019.
T. Yang, G. Andrew, H. Eichner, H. Sun, W. Li, N. Kong, D. Ramage, and F. Beaufays. Applied fed- erated learning: Improving google keyboard query suggestions. arXiv preprint arXiv: 18/200933, 2018.
Y. Yao and Doretto. Boosting for transfer learning with multiple sources. 2010 IEEE computer society conference computer vision and pattern recognition, pages 1855-1862. IEEE, 2010.
Z. Yuan, D. Bao, Z. Chen, and M. Liu. Integrated transfer learning algorithm using multi-source tradaboost for unbalanced samples classification. In 2017 international conference on computing intelligence and information system (CIIS), pages 188-195. IEEE, 2017.
Q Zhang, H. Li, Y. Zhang, and M. Li. Instance transfer learning with multisource dynamic tradaboost. The scientific world journal, 2014, 2014.
H. Zhao, S. Zhang, G. Wu, J. M. Moura, J. P. Costeira, and G. J. Gordon. Adversarial multiple source domain adaptation. In Advances neural information processing systems, pages 8559-8570, 2018.
15


Reviews:
[{"review_id": "lSCvrALH2ID", "reviewer": "Reviewer_8DgA", "summary": "Utilizing recent formulations/results, the authors propose MutliBoost, a multi-source ensemble learning algorithm where: (1) it is assumed that any target domain is a mixture of the source domain distributions and (2) the learned function form is a convex combination of base functions weighted by a domain identification classifier, Q(k|x), and referred to as Q-ensembles. The domain identification classifier can be learned from unlabeled data (other than the domain label) and an important innovation as it leads to a more natural multi-source Boosting objective and generalizes many existing (and widely-used) multi-source Boosting works. This formulation aligns with recent results regarding agnostic loss [Mohri, Siven, and Suresh; Agnostic Federated Learning, ICML19] and is able to use related findings to prove favorable theoretical properties wrt previous work and leads to FedMultiBoost, a federated learning variant. Experiments are conducted on five binary datasets converted to multi-domain settings (sentiment analysis, hand-writing recognition, object recognition, and UCI adult tabular data) against natural single domain transfer and combined domain data baselines, showing statistically significant improvements. Several additional experiments are included in the Appendices to also show further results (e.g., domain-identification classifier, FedMultiBoost) that addresses multiple natural questions in the paper.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "As previously stated, this submission primarily builds on recent multi-domain and federated learning finding to revisit and reformulate a multi-domain Boosting algorithm with improved properties relative to existing formulations. Given that this is a well-visited area, the formulation is notable in that it does provide practical advantages (training for all target mixtures) and preferable theoretical properties. Additionally, it is an elegant formulation and has potential to restart work in an area that has become somewhat saturated. The theoretical analysis is sound, concise, and sufficiently thorough. The empirical results are promising, but would benefit from additional comparisons to existing work and, if possible, a ‘real-world’ (non-benchmark) dataset to demonstrate practical strengths of (Fed)MultiBoost. Below, I will evaluate the work along the requested dimensions, but overall think that this is a worthwhile contribution with sufficient originality, quality, clarity, and potential significance (with strengths in theoretical analysis and presentation outweighing weaknesses in empirical evaluations) and lean toward accepting.\n\n== Originality ==\nThe multi-source Boosting problem is not new, but this formulation provides a generalization to existing approaches that is theoretically more broadly applicable (given the weak learning assumption can be satisfied, the domain-identification classifier is sufficiently accurate, and sufficient training data exists). Additionally, the analytical methods are primarily derived from existing work — so this isn’t the strength of originality either. Specifically, core findings are largely derived from earlier margin-based analysis of Boosting or agnostic learning results in federated learning scenarios (i.e., this is not an analytical breakthrough). However, while I think many people could have put these ideas together if I were to say “adapt [Mohri, Siven, and Suresh; Agnostic Federated Learning, ICML19] to create a multi-source Boosting algorithm”, I don’t think making this connection was obvious nor was the analysis trivial.\n\n== Quality ==\nThe scholarship and theoretical analysis is solid. The paper is well-contextualized wrt the first generation of Boosting work, application of this work to domain adaptation/multi-source learning, and application of recent work in multi-source/federated learning. The development of the theory is self-contained and answers most of the relevant questions. From a practical perspective, the underlying assumptions in [lines 101-108] seem potentially overly strong as a weak learner must exist for any target-mixture — which may limit it’s practical application. While the empirical results are good in terms of what is presented, they also don’t establish that this is a clearly practical algorithm since: (1) it is not clear why there aren’t results comparing to [Yao & Doretto, 2010] for the single-target case as it would demonstrate the sensitivity of accurate domain identification to some degree and (2) there is no (non-benchmark) ‘real-world’ dataset to prove it is clearly practical. AdaBoost-all is a natural baseline, but it would be surprising if MutliBoost didn’t perform better. Additionally, while space is a problem, but additional discussion of the experimental results would be useful as it is presently a “we outperform fairly weak baselines”. This is somewhat mitigated by the appendices that add to the results and are of quality similar to the body of the main paper. Also, FedMultiBoost is an extension of this connection additional considerations with softmax explored — which may influence other related works.\n\n== Clarity ==\nOverall, the paper is well-developed and clear. While reading the most directly related works does help with understanding, a famliar reader will likely view the paper as self-contained and well-focused on the most important new results. Again, the appendices add to the paper (without being strictly necessary) although I think there are opportunities to better reference them with short summary statements in context. Also, from a scholarship perspective, better references in Sections 3 (to older Boosting work, etc.) would be helpful and maybe a bit more forward/in context in section 4 (although lines 218-223 are sufficient). However, I believe clarity is a strength of this work.\n\n== Significance ==\nMulti-source is popular and the tie to federated learning increases the potential for impact; I would expect this to be the preferred starting point for multi-source Boosting approaches. The primary limitation that potentially impedes significant is that it isn’t clear that it is the SoTA since thorough empirical comparisons to existing multi-source settings or a ‘real-world’ setting. From a theoretical perspective, the analysis draws heavily from existing work, but after (re-)reading those papers, I liked this presentation and I believe will influence other analyses", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "== Technical Limitations ==\nWhile the primary limitations are stated, there isn’t an explicit section and/or discussion of the ramifications of the relatively strong assumptions. Additionally, as stated, the empirical results don’t have a notable discussion section.\n\n== Potentially Negative Societal Impact ==\nNo concerns in this regard — it is an ML algorithms paper with a theoretical focus. Nothing that isn’t beyond standard ML concerns.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5 hours"}, "extra_text": {}}, {"review_id": "KSICFvQ46Q5", "reviewer": "Reviewer_Nwre", "summary": "The paper studies the problem of learning boosted classifiers with data from multiple source domains. The authors propose Q-ensembles to deal with it and seek a solution for any mixture of the source distribution. They propose the algorithm MultiBoost to solve it inspired by coordinate descent. They present theoretical analysis in terms of margin-based learning bounds. Moreover, they present an extension to the federated learning scenario. In experiments, they compare experimental results with AdaBoost. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "-\tOriginality: The method is new since MultiBoost adopts Q-ensemble. \n\n-\tQuality: For binary classification, the authors propose the complete optimization method. However, I doubt if it can generalize to the multi-class case. The algorithm also needs unlabeled data to estimate the conditional probability $Q(k|\\cdot)$. \n\n-\tClarity: The submission is clearly written except for the Section 3 Algorithm. Some errors occur in the description part of Section 3. For example, the definition of $F_k(\\alpha)$ in line 130 should remove $\\sum_{k=1}^p$. The equation below line 138 is a confusion notation for $F’_k(\\alpha_{t-1}, e_{k,r})$. \n\n-\tSignificance: The theoretical result is not novel and the guarantee can be anticipated.\nFor the multiple source dataset that the authors use to conduct experiments, some people propose the neural network-based methods, which can naturally cope with the multi-classification problem and have higher accuracy than boosting. Moreover, the authors should compare with other boosting methods to cope with multiple source problems, not just AdaBoost. \n\nIf the authors can use boosting to cope with multi-class classification and achieve high accuracy, I will consider changing my rate on the paper. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations. The authors consider a more practical scene, federated learning scenario. Their FedMultiBoost algorithm, to some extent, makes the solution fair and risk-averse.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "JQGC5QdALKs", "reviewer": "Reviewer_2m7P", "summary": "This paper considers the domain adaptation setting where we are given training examples from $k$ different distributions $D_1, \\cdots, D_k$. We assume a hypothesis class $H_k$ the at is good for each domain (in the weak learning sense). At test time, the samples are drawn from a convex combination $D = \\sum_i \\lambda_i D_i$. The goal is to find one linear combination of  weak learners which will perform well for any convex combination. It turns out that this is not possible, but that if you allow the weights to depend on the conditional probability of coming from a particular domain $D(j|x)$, then a single learner is possible. This I believe is already proved in the paper of Mansour-Mohri-Rostemizadeh 2008, although the result there relies on a fixed-point theorem and perhaps does not give an efficient algorithm.    \n\nThe main contribution of this work is a boosting style algorithm that achieves this  guarantee. The algorithm does simple co-ordinate wise gradient descent like in Adaboost. They show theoretical margin-based guarantees for their algorithm, and experimental results showing that it outperforms more naive boosting-based approaches. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am a little concerned about the relation between this paper, and the work of Mansour-Mohri-Rostemizadeh 2008 [MMR]. At the very least, I have problems with how this paper credits that paper without adequate citations. It is possible that there is some substantial overlap in terms of ideas. \n\n1. The notion of $Q$-ensembles:\nThis paper says: *Instead, we put forward Q-ensembles, which are convex combinations weighted by a domain classifier Q, that is, Q(k|x) is the conditional probability of domain k given input point x. This crucially differentiates our work from that of past studies*\n\nI am not sure I agree with this statement. It appears that the notion of $Q$-predictors is present (or at the very least implicit) in [MMR]). In Section 5.2, the predictor $h_z^0$ they use is exactly the $Q$-weighted predictor where the conditionals are computed for the convex combination of distributions with coefficient $z$).  \nIn that paper the setting is that there is a single hypothesis $h_l$ for each $l \\in [k]$, and we do not get access to samples. It seems that if one considers $h_l = \\sum_j \\alpha_{l, j}h_{l,j}$ as a single predictor that does well for domain $D_l$, then you get exactly the form of predictors considered in this paper:\n$$ f(x) = \\sum_{l =1}^pQ(l|x)\\sum_{j \\=1}^{N_l}\\alpha_{l, j}h_{l, j}(x) $$\nIn particular, this paper never spells out explicitly whether these conditionals are being computed with regard to the uniform convex combination of some other combination, it would be nice to clarify this.\n\n2. Existence of good $Q$-ensembles:\nBeyond the notion of $Q$- suggests that the existence of the kind of predictor that this paper gives is implied [MMR]. It seems that their linear combinations are not explicit owing to a fixed point argument, whereas this paper gives an explicit algorithm using boosting.  \n\n3. The negative results:\nProposition 1 in this paper seems identical to Theorem 1 in MMR08, similarly I believe Proposition 2 is just Theorem 2. But there is no citation or acknowledgement, and a reader might think this is a new contribution. While this was probably just oversight, it is not good writing. \n\nWhile I like aspects of this paper, I am going with a weak reject for now based on these issues. This is only meant as a placeholder. I would be open to revising my scores if the authors can give a rebuttal that:\n- Accurately acknowledges what is derived (directly or easily) from prior work.\n- Conditioned on this, emphasizes what is truly novel here.\n\n\n----------------------------------------------------------------------------------------------------------\n\nAfter reading the responses, I am convinced that this paper makes a substantial original contribution. I am raising my score to accept. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "FXm2qm-fIKK", "reviewer": "Reviewer_shXz", "summary": "The paper addresses the question of learning an accurate classifier using ensemble methods and based on multiple sources. The purpose of the classifier is to achieve good prediction on a set of possible future domains, modelled as linear combinations of the source domains. This is done by minimizing an agnostic loss function over the weights of a linear combination of source predictors, such as traditionally done in ensemble methods. The paper proposes a main algorithm solving this problem, as well as a version for the federated setting. A theoretical analysis and experimental validation are also proposed. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Although the overall idea takes inspiration from various existing lines of work, the idea of building an ensemble predictor from multiple sources is to me novel. It builds upon two main well-known results: (1) the theory of ensemble learning, which states that a combination of weak classifiers can be a strong classifier; (2) transfer learning and domain adaptation, in which it is classical to weight the decision by the probability of the sources. The state of the art is briefly introduced in the main paper, but a bit more elaborated in the appendix. Maybe I would encourage the authors to insist a bit more on the transfer learning aspect which is only briefly discussed in the main paper. \n\nQuality: The paper is technically correct, even though I have some reservations about the use of the term “boosting”. The main argument is based upon the result that AdaBoost corresponds to a greedy minimization of an exponential loss function. Following this idea, the authors build their own method around the minimization of a loss (which can be exponential, but not only) and minimization of this objective by application of coordinate descent. This is completely correct, but I wonder whether it is possible to really call that boosting. In its definition, boosting is meant to proceed to a sequential update of the predictor by sequential reweighting of the data according to the misclassification of previous models. It also builds upon the idea of applying a learning algorithm producing weak classifiers on several successive datasets (actually a same reweighted dataset). These aspects are of course present in the paper, but completely hidden in a very complex paragraph in Section 3. This comment is not fundamental and does not affect the overall quality of the paper, but I think a stronger discussion is needed here, and maybe I suggest that the authors weaken the importance of the word “boosting” in the paper. Also, I think that a short explanation about the connection between boosting and coordinate descent over exponential loss would be a very good thing to add, to be more accessible to a general ML audience which may ignore this connection. (While re-reading my review, I realize this comment is maybe more to be read from the “clarity” point of view than from the “quality” angle, but I still think that making the effort to explicitly show that the method is actual boosting would improve the quality of the paper). \nThe theoretical results seem correct to me, even though I have not reviewed (yet) the full proof of the two theorems. \nRegarding the experimental validation, it is, in my opinion, sufficient to validate the interest of the method. However, I think that it may have been even more interesting to present results relative to other transfer learning / domain adaptation techniques which do not rely on boosting. In essence, the addressed problem is a transfer learning question and boosting is only a tool to solve this problem. I would find it more relevant to compare the results to TL algorithms which focus on the transfer of knowledge from the sources to the target, rather than to boosting methods which are designed to be single-task. \nA point which would benefit from further discussion is the choice of the distribution Q(l|x). It is shortly mentioned that its choice is not direct, but I would have expected maybe more comments on that (which would be required for instance for a complete reproducibility). In the theoretical results, I am a bit surprised that the impact of this term is not visible in Theorems 1 and 2. After reading propositions 1 and 2, it was clear that this term was important for the quality of the predictions, and yet it seems not to play any role in the margin bounds. Could the authors comment on that point? Finally, if not done in theory (which I understand would be difficult – if even possible), I regret that the influence of this term is not highlighted in the empirical section, pointing out how a wrong choice of Q could affect the overall algorithm. \n\nClarity: The paper is rather well-written, even though the notations may be sometimes difficult to follow. Unfortunately, this is mostly due to the addressed problem itself and I do not have any real correction suggestions for that. Maybe section 2 could be slightly corrected to introduce the notations in a more readable way? This is only a detail, but I think it would significantly help the reader. During my reading, I had to go back to the notations multiple times and had some difficulties retrieving them. \nIn terms of introduction of the ideas, I particularly enjoyed the first two propositions which are a smart way to introduce the Q(l|x) term. Minor comment, though: “probability of a domain given an input point” seems quite ambiguous. I suggest it to be replaced by a clearer description, for instance “probability that an input point belongs to a domain”. \n\nSignificance: The results are somehow significant. The main scenario is probably the weakest in terms of its induced advances in the state of the art, but the federated learning version is extremely important in my opinion and offers a clean and well-justified solution to a very important problem.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The main limitation of the paper is, as said earlier, the strong role played by Q which is not discussed much. \nThe paper being more about solving a general task rather than focusing on specific applied problems, there is no societal impact in my opinion, neither positive nor negative. ", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concern.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3-4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces and analyzes a new boosting algorithm for learning in the presence of multiple source domains. We were persuaded by the author response to the concerns expressed in the reviews, especially about novelty wrt MMR.
Decision: Accept (Poster)