On Learning Domain-Invariant Representations for Transfer Learning with Multiple Sources
Authors: Trung Quoc Phung
Trung Le
Long Tung Vuong
Toan Tran
Anh Tuan Tran
Hung Bui
Dinh Phung
Venue: NeurIPS 2021 Poster (2021)
Keywords: Domain generalization
domain adaptation
transfer learning
multiple-source domain adaptation
machine learning
deep learning

Abstract: Domain adaptation (DA) benefits from the rigorous theoretical works that study its insightful characteristics and various aspects, e.g., learning domain-invariant representations and its trade-off. However, it seems not the case for the multiple source DA and domain generalization (DG) settings which are remarkably more complicated and sophisticated due to the involvement of multiple source domains and potential unavailability of target domain during training. In this paper, we develop novel upper-bounds for the target general loss which appeal us to define two kinds of domain-invariant representations. We further study the pros and cons as well as the trade-offs of enforcing learning each domain-invariant representation. Finally, we conduct experiments to inspect the trade-off of these representations for offering practical hints regarding how to use them in practice and explore other interesting properties of our developed theory.

On Learning Domain-Invariant Representations for Transfer Learning with Multiple Sources
Trung Phung!
Trung Le² Long Vuong' Toan Tran' Anh Tran' Hung Bui Dinh Phung!
VinAI Research, Vietnam Monash University, Australia
v. t.rrungg3évvinai io, trunglm@monash.co [v. 1ongvt8, v. v.toanmm3, a.anttt12,, v.hungbh1, v.dinhpq2)evinai.cm
Abstract
Domain adaptation (DA) benefits from the rigorous theoretical works that study insightful characteristics and various aspects, e.g., learning Nomain-invariinn representations and its trade-off. However, it seems not the case for the multiple source DA and domain generalization (DG) settings which are remarkably more complicated and sophisticated due to the involvement of multiple source domains and potential unavailability of target domain during training. In this paper, we develop novel upper-bounds for the target general loss which appeal to to define two kinds of domain invariant representations. We further study the pros and cons well as the trade-off's of enforcing learning each domain-invarianl representation. Finally, we conduct experiments to inspect the trade-off of these representations for offering practical hints regarding how to use them in practice and explore other interesting properties our developed theory.
domain-invariant
Introduction
Although annotated data has been shown to be really precious and valuable to deep learning (DL), in many real-world applications, annotating sufficient amount of data for training qualified DL models is prohibitively labour-expensive, time-consuming, and error-prone. Transfer learning is a vital solution for the lack of labeled data. Additionally, in many situations, limited number of annotated data from multiple source domains. Therefore, it desirable to train a qualified DL model primarily based on multiple source domains and possibly together with target domain. Depending on the availability of the target domain during training, we encounter either multiple source domain adaptation (MSDA) domain generalization (DG) problem.
only able to collect
Domain adaptation (DA) is a specific case of MSDA when we need to transfer from a single source domain to another target domain available during training. For the DA setting, the pioneering work [3] and other following work (4] 52 53
54 22] are abundant to study its insightful characterizations and aspects, notably what invariiann (DI) representations how to learn this kind of representations, and the trade-off of enforcing learning DI representations. Those
studies lay the foundation for developing impressive practical DA methods 14 47 52 5T 36).
Due to the appearance of multiple source domains and possibly unavailability of target domain, establishing theoretical foundation and characterizing DI representations for the MSDA and especially DG settings significantly more challenging. A number of state-of-the-a art works in MSDA and
49 35 37 have implicitly exploited and used DI representations in some sense to achieve impressive performances, e.g., [28] 288 III minimize representation divergence during training, 41 decomposes representation to obtain invariant feature,
DG [66 I9
[
41
27
28 34
[5
55|
[55] maximizes domain prediction entropy to learn invariance. Despite the successes, the notion DI representations in MSDA and DG is not fully-understood, and rigorous studies to theoretically
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

characterize DI representations for these settings are still very crucial and imminent. In this paper, we provide theoretical answers to the following questions: (1) what are DI representations in MSDA and DG, and (2) what one should expect when learning them. Overall, our contributions in this work can be summarized as:
In Section 2.2 we first develop two upper-bounds for the general target loss in the MSDA and DG settings, whose proofs can be found in Appendix A. We then base on these bounds to characterize and define two kinds of DI representations: general DI representations and compressed DI representations.
We further develop theory to inspect the pros and cons of two kinds of DI representations in Section 2.3 aiming to shed light on how to use those representations in practice. Particularly, two types of DI representations optimize different types of divergence on feature space, hence serving different purposes. Appendix B contains proofs regarding these characteristics.
Finally, we study in Section 2.4 the trade-off of two kinds of DI representations which theoretically answers the question whether and how the target performance is hurt when enforcing learning each kind of representation. We refer reader to Appendix C for its proof.
We conduct experiments to investigate the trade-off of two kinds of representations for giving practical hints regarding how to use those representations in practice as well as exploring other interesting properties of our developed theory.
is worth noting that although MSDA has been investigated in some works 18 54 our proposed method is the first work that rigorously formulates and provides theoretical analysis for representation learning in MSDA and DG. Specifically, our bounds developed in Theorems I and interweaving both input and latent spaces novel and benefit theoretical analyses in deep learnin. Our theory is developed in general setting of multi-class classification with a sufficiently general loss, which
can be viewed which considers binary classification with specific loss functions. Moreover, our theoretical bounds developed in Theorem is is the first theoretical analysis of the trade-off of learning DI representations in MSDA and DG. Particularly, by considering the MSDA setting with single source and target domains, we achieve the same trade-off nature discovered in [53] but again our setting more general than binary classification setting in that work.
non-trivial generalization of existing works in DA 30 3
53
31 54), each of
2 Our Main Theory
2.1
Notations
Let be a data space on which we endow a data distribution P with a corresponding density function p(x)) We consider the multi-class classification problem with the label set y [C]. where C is the number of classes and [C] := f1..... ..... C}. Denote VA- {a RC :|aal]
2 0} as the C-simplex label space, let f : X' + V be a probabilistic labeling function returning a C-tuple f (x) = [f (x, م) whose element ff (z, i) = p (y = i x) is the probability to assign a data sample P to the class (i.c., i [1,,.. CJ). Moreover, a domain is denoted compactly as pair of data distribution and labeling function D := (P, f). We note that given a data sample x ~ P. its categorical label y E y sampled as y Cat (f (z)) which categorical distribution over f (z) E V/s.
(z) v) with f (x) E Va and y E y specifies the loss (e.g., cross-entropy, Hinge, LI, or L2 loss) to assign data sample to the class by the hypothesis f. Moreover, given a prediction probability f ((x) w.r.t. the ground-truth prediction f (x)). we define the loss E (f (x), (x)) Ey-s(z) [ (f (f) - ا (f (z) f (I, v). This means é (., is convex w.r.t. the second argument. We further define the general loss caused by using classifier f X + Va to predict IB fP as
Let VA x y + R be a loss function, where (f(z),
c (f.f.)) = (F.D) := E.-P (e(F())
We inspect the multiple source setting in which we are given multiple source domains (DS..). K over the common data space X, each of which consists of data distribution and its own labeling

---

function D& : (PS.t, fS.t). Based on these source domains, we need to work out a learner or classifier that requires to evaluate on a target domain DD" := (IIT f"). Depending the knownness
or unknownness of a target domain during training, we experience either multiple source DA (MSDA)
31
29
18
domain generalization (DG) 24 28 27 34 setting.
One typical approach in MSDA and DG is to combine the source domains together 14
11 46] to learn DI representations with hope to generalize well to target domain. When combining the source domains, we obtain mixture of multiple source distributions denoted as D
34
Σ TT..=:; where the mixing coefficients x = [ K can be conveniently set to тद = 25-1 N;
N;
with N, being the training size of the i-th source domain.
In term of representation learning. input is mapped into a latent space Z by feature map X' + 2 and then classifier h E r> V is trained based on the representations (X).. Let f: X' + VA be the original labeling function. To facilitate the theory developed for latent space, we introduce representation distribution being the pushed-forward distribution P. := ggIP, and the
labeling function h : Z + Va induced by g as h(2) [a-1(e) (a)p(())
21 Going back to our
- plx)dz
multiple source setting, the source mixture becomes 10 = E, M.DSs, where each source domain is S (IS.. h and the target domain is DY (PT h²))
Finally, in our theory development, we use Hellinger divergence between two distributions defined as D1/2 (P), PP) 2 (Vo*(()
dr, whose squared d1/2 / D/1y2 is a proper metric.
2.2 Two Types of Domain-Invarian Representations
Hinted by the theoretical bounds developed in Bl. DI representations learning, in which feature extractor 9 maps source and target data distributions to a common distribution on the latent space, is well-grounded for the DA setting. However, this task becomes significantly challenging for the MSDA and DG settings due to multiple source domains and potential unknownness of target domain. Similar to the case of DA, it is desirable to develop theoretical bounds that directly motivate definitions of DI representations for the MSDA and DG settings, as being done in the next theorem.
Theorem 1. Consider a mixture of source domains D = EK ny S and the target domain IDF. Let l be any loss function apper-bounded by positive constant L. For any hypothesis f : X + Va where f = h g with g X 2 and h 2 Vs, the target loss on input space is upper bounded
K
c (F.DT) & Σ
T (f, L max Eps. [IIAp' [+[+)]] + LV221/2 (P) 1P) (E[K]
(1)
1=1
5SA - y) y=1 is the absolute of single point label shift on imput space between source domain S the target domain IT,, .nn [K] 21. KY.
where Ap Ap(x)) = (1)
The bound in Equation Timplies that the target loss in the input or latent space depends on three terms: (i) representation discrepancy: d1/2 (177 PT), (ii) the label shift: [ Eps.. Ap' (v]=))la). and (iii) the general source loss: ΣKr T f. DS..). To minimize the target loss the left side, we need to minimize the three aforementioned terms. First, the label shift term is natural characteristics of domains, hence almost impossible to tackle. Secondly, the representation discrepuncy term can be explicitly tackled for the MSDA setting, while almost impossible for the DG setting. Finally, the general source loss term convenient to tackle, where its mininization results in a feature extractor g and a classifier h.
7 that consider both losses and data discrepancy on data space, our bound connects losses on data space discrepancy on representation space. Therefore, our theory provides natural way to analyse representation learning, especially feature alignment deep learning practice. Note that although DANN [1] explains their feature alignment method using theory developed by Ben-david et al. B. it is not rigorous. In particular, while application of the theory to representation space yield a representation discrepancy term, the loss terms are also that feature space, and hence minimizing these losses is not the learning goal.
Contrary to previous works in DA and MSDA 3
31
54
3

---

Finally, our setting is much more general, which extends to multilabel, stochastic labeling setting, and any bounded loss function.
From the upper bound, we turn to first type of DI representations for the MSDA and DG settings. Here, the objective of is map samples onto representation space in way that the common classifier h can effectively and correctly classifies them, regardless of which domain the data comes from.
Definition 2. Consider class 9 of feature maps and a class H of hypotheses. Let {DS be set of source domains.
(DG with unknown target data) feature map g" G is said to be DG general domain- invariint (DI) feature map if is the solution of the optimization problem (OP): minges minjig E 7iL (h, Moreover, the latent representations z 9" (x) induced by 9" is called general DI representations for the DG setting.
ii) (MSDA with known target data) A feature map 9" E G is said to be a MSDA general DI feature map g" is the solution of the optimization problem (OP); mingeg min E T. (,, (iA)))
which satisfies py. = P (i.e., mingeg d1/2 ( Moreover, the latent representations z g" (x) induced by g" is called general DI representations, for the MSDA setting.
The definition of general DI representations for the DG setting in Definition 2iis transparent in light of Theorem wherein we aim to find and h to minimize the general source loss Σ 7. (f D.S.)) due to the unknownness of Ip_ Meanwhile, regarding the general DI representations for the MSDA setting, we aim to find g* satisfying Pr 1 and h to minimize the general source loss DK J.G. (D.)). Practically, find general representations for MSDA, we solve
Σ Tcl (g. + AD D pa
min min gE0 hEH
where À > 0 is trade-off parameter and D can be any divergence (e.g., Jensen Shannon (JS) divergence [66, f d-iivergence 38 MMD distance (27 or WS distance 42
In addition, the general DI representations have been exploited in some works [57 26 from the practical perspective and previously discussed in (2) from the theoretical perspective. Despite the similar definition to our work, general DI representation in is not motivated from minimization of a target loss bound.
As pointed out in the next section, learning general DI representations increases the span of latent representations g (1)) the latent space which might help to reduce d. /2 (PT
g: TPx) for general 11 have also explored the
target domain. On the other hand, many other works 27 28 15
possibility of enhancing generalization by finding common representation among source distributions. The following theorem motivates the latter, which the second kind of DI representations.
Theorem 3. Consider mixture of source domains D =EK DSD and the target domain ID² Let E be any loss function upper-bounded by a positive constant L. For any hypothesis f X + V where f = h 9 with : X + Z and h Z + VA. the target loss on input space is upper bounded
K
C (f.
Σ
TiL (f, (D) L max Ros.: sE(K]
(App(())))l)]
L LV
d1/2
(PT PS
K
K
LV2T K
da/2 (p (ss).
Evidently, Theorem suggests another kind of representation learning, where source representation distributions are aligned order to lower the target loss't 's bound as concretely defined as follows.
Definition& 4. Consider class G feature maps and class H of hypotheses. Let (TDS.1) K set of source domains.
Jiml be

---

i) (DG with unknown target data) A feature map g" E G is a DG compressed DI represen- tations for source domains {DS.)K F==1 if g" is the solution of the optimization problem (OP):
minges min E TiL ( D2 which satisfies pa = P _... ......... p. (i.e., the pushed
forward distributions of all source domains identical). The latent representations =g" (x) is then called compressed DI representations for the DG setting.
ii) (MSDA with known target data) A feature map g² E G is MSDA compressed DI repre- sentations for source domains (TDS..) Ji=1 if g" is the solution of the optimization problem (OP): (
p.S. mingeg min Ex= 7.G (h.Dg. D.g which satisfies Ip P =... Pp" Fig. (i.e., the
pushed forward distributions of all source and target domains are identical). The latent representa- lions: = 9" (x) is then called compressed DI representations for the MSDA setting.
2.3 Characteristics of
Representations
In the previous section, two kinds of DI representations are introduced, where each of them originates from minimization of different terms the upper bound of target loss. In what follows, we examine and discuss their benefits and drawbacks, We start with the novel development of hypothesis-awar divergence for multiple distributions. which necessary for our theory developed later.
2.3.1 Hypothesis-Awan Divergence for Multiple Distributions
Let consider multiple distributions QI,..., Qc on the same space, whose density functions are
I..... QC. We are given mixture of these distributions Qo = Ee a: with mixing coefficient V/a and desire measure a divergence between Q1, .... Qc. One possible solution is employing a hypothesis from an infinite capacity hypothesis class H. to identify which distribution the data
come from. In particular, given that Qi. Note that we use the notation h in this particular subsection denote domain classifier, while everywhere else in the paper denotes label classifier. Let (2), R be
Q" the function h () e [C] outputs the probability
the loss when classifying z using h provided the ground-truth label [C]. Our motivation is that if the distributions Qi.... Qo are distant, it is easier to distinguish samples from them, hence the minimum classification loss minjieH CQ.:
is much lower than in the case of clutching Q....., Qc. Therefore, we develop the following theorem to connect the loss optimization problem with an f-divergence among Q1 ..., Qc. More discussions about hypothesis-awann aware divergence can be found in Appendix of this paper and in [13]
Theorem 5. Assuming the hypothesis class H has infinite capacity, we define the hypothesis divergence for multiple distributions as
D (QQ, Qc) min CR.
(i)) + Ci
where Ci depends only on the form of loss function and value of a. This divergence is a proper f-divergence among Q1,... QC in the sense that D (Q1,... Qc) 2 O, VQ,,,...,, Qo and a YA.
and D (QI,... Qc) = 0 Of
Qc.
2.3.2 General Invariant Representation
As previously define,, the general DI feature map g" (cf. Definition 2 the one that minimizes the total source loss
g" arg min min DEG hER
T (h, S.t (...t.
Pp)
argmin ung min T T.
(4)
we expect that the minimal loss minjeu L (h, 1 1p5.. should be
From the result of Theorem inversely proportional to the divergence between the class-conditionals PS...c. To generalize this result to the multi-source setting, we further define Q- ΣK Ti T..ipp as the mixture of the class conditional distributions of the source domains the latent space, where a. > Ty
5

---

are the mixing coefficients, and Yi = P (y= c) are label marginals. Then, the inner loop of the objective function in Eq. 4 can be viewed as training the optimal hypothesis h E H to distinguish the samples from Q e E [C] for a given feature map Therefore, by linking to the multi-divergence concept developed in Section 2. we achieve the following theorem.
Theorem 6. Assume that H has infinite capacity, we have the following statements.
1. Da (Q.)
Q2()) = --=-i-inn mingen Ei=1 J.f (.hhS.,,gg)) + const, where a [aeles|o] is
defined as above.
2. Finding the general DI feature map g" via the OP in Eq. 4 is equivalent solving
g = argmax D (Qaa) " 27
(5)
scg
Theorem 6 especially its second claim, discloses that learning general DI representations max- imally expands the coverage of latent representations of the source domains by maximizing
D (Q=a. ..... @2C). Hence, the span of source mixture I = Ees1 a. is also increased.
We believe this demonstrates one of the benefits of general DÍ representations because it implicitly enhance the chance to match general unseen larget domuins in the DG setting by possibly reducing
the souree-target representation discrepancy term d1/2 (TP) Pg)) (cf. Theorem [] Additionally, in MSDA where we wish to minimize d1/2 (42, IP.) explicitly, expanding source representation's 's coverage is also useful.
2.3.3 Compressed Domain-Invarian Representations
is well-known that generalization gap between the true loss and its empirical estimation affects generalization capability of model [48 45). One way to close this gap is increasing sampling density. However, as hinted by our analysis, enforcing general DI representation learning tends to maximize the cross-domain class divergence, hence implicitly increasing the diversity of latent representations from different source domains. This renders learning a classifier h on top of those source representations harder due to scattering samples on representation space, leading to higher generalization gap between empirical and general losses. In contrast, compressed DI representations help making the task of learning h easier with lower generalization gap by decreasing the diversity of latent representations from the different source domains, via enforcing S=S.1 g* = SpS.2 =.. pS.K We now develop rigorous theory to examine this observation.
Let the training set be S = {(2,, 3 consisting of independent pairs (2,, y:) drawn from the source mixture, i.e., the sampling process starts with sampling domain index k ~ Cal (T). then sampling z ~ 1g (i.e., x ~ S.te and 2 = 9 (z)), and finally assigning label with y ~ h (z) (i.e., S. (2) is distribution over (C)). The empirical loss is defined as
K
Nu
C (h.
1
(h(z), h 8.k (z)).
(6)
Σ
where Nk, & [C] is the number of samples drawn from the k-th source domain and N Σk
A== No.
Here, L (h. is an unbiased estimation of the general loss L [D. Ek T& (h. D.A.A. To quantify the quality of the estimation, we investigate the upper bound of the generalization gap L (h, Sh L (h. Dy) with the confidence level 1 ō.
(h,
Theorem 7. For any confident level 5 E [0, over the choice of S, the estimation of loss is in the e-range of the true loss
Pr (( L (f. DT) 56) 2-
where E = = = (4) 1/2 is a function of 8 for which A is proportional to
1
K
K
V
c (F.D
+ L Σ
Th max E.S,A kE(K
A (u|x))
+
L
K K
V2T, d1/2 (pSt, :ps)))

---

Theorem 7 reveals one benefit of learning compressed DI representations, that is, when enforcing compressed DI representation learning, minimize & Σ Σه= V d1/2 (1P5. PP))) which tends to reduce the generalization gap e (6) for a given confidence level - 6. Therefore, compressed DI representations allow us to minimize population loss L Dg)) more efficiently via minimizing empirical loss.
2.4 Trade-off of Learning Domain Invariant Representation
Similar to the theoretical finding in Zhao et al. [53] developed for DA, we theoretically find that compression does come with a cost for MSDA and DG. We investigate the representation trade-off, typically how compressed DI representation affects classification loss. Specifically, consider
data processing chain X' م Z - VA, where X' is the common data space, Z the latent space induced by a feature extractor 9. and is a hypothesis on top of the latent space. We define 17 and f as two distribution over y in which to draw ~ Py. we sample k ~ Cat (T), x ~ p.S.k and y ~ fs. (z), while similar to draw P.. Our theoretical bounds developed regarding the trade-off of learning DI representations are relevant to d1/2 (135, IP}))
Theorem 8. Consider a feature extractor g and a hypothesis h, the Hellinger distance between two label marginal distributions P3, and Po can be upper-bounder as:
1. d1/2 133,13) S
E TH (h f..kk ...A))
)
+ d1/2 [ITT 1.) + L aaachhog fo fT) 1/2
d1/2 (P5, PJ) I EEK 7.
)
)1²
1/2
+ =K EK
vh K d1/2 ())))
v K /1/2 (PT, (g + L (hog. 1/2
ΣK Ej=
Here we note that the general loss L is defined based on the Hellinger loss E which is define
e(f(s), -i = DE (V -((e,))-V((,,))) found in Appendix C).
(more discussion can be
Remark. Compared to the trade-off bound in the work of Zhao et al. 53 our context is more general, concerning MSDA and DG problems with multiple source domains and multi-class probabilistic labeling functions, rather than single source DA with binary-class and deterministic setting. Moreover, the Hellinger distance is more universal, in the sense that does not depend on the choice of classifier family H and loss function l as in the case of H-divergence in [53]
We base on the first inequality of Theorem 8 to analyze the trade-off of learning general DI repre- sentations. The first term on the left hand side is the source mixture's 's loss, which is controllable and tends to be small when enforeing learning general DI representations. With that in mind, if two label marginal distributions PT and PI are distant (i.c., d1/2 (P3, FT) is high), the sum 1/2 tends to be high. This leads to 2 possibilities. The first
11/2 (PT.P)) + L fT,
scenario is when the representation discrepancy d1/2 (PT .P.) has small value, e.g., it is minimized in MSDA setting, or it happens to be small by pure chance in DG setting. In this case, the lower bound of target loss L (h g. 9. JT. TTT) is high, possibly hurting model's generalization ability. On
the other hand, the discrepancy d1/2 (IP2 p.) is large for some reasons, the lower bound of target loss will be small, but its upper-bound is higher, as indicated Theorem [
g.
Based on the second inequality of Theorem g we observe that if two label marginal distributions P3 and Py are distant while enforcing learning compressed DI representations (i.e.. both source loss and source-source feature discrepancy f pp..) 1/2
ΣK T,L ( g, 8
E ΣK A d2 (PS. pPS)) are low), the sum K d1/2 (PT .PS.)) Chog, fr. /T
ΣK K
/E,
1/2 is high. For the MSDA setting, the
7

---

Va x21 (Pry (..) trained to get smaller, meaning that the lower bound discrepancy E Ej=1 of target loss L (hog. PT) is high, hurting the target performance. Similarly, for the DG setting, if the trained feature extractor و occasionally reduces Σ دj=1 some unseen target domain, it certainly increases the target loss C (h o g. fr, .P)). In contrast, if for some target domains, the discrepancy Σ 2ز=1 vT d (Pg P..)) is high by some reasons, by linking to the upper bound in Theorem the target general loss has a high upper-bound, hence is possibly high.
K v K d1/2 (P. P for
This trade-off between representation discrepancy and target loss suggests a sweet spot for just-right feature alignment. In that case, the target loss is most likely to be small.
3 Experiment
To investigate the trade-off of learning DI representations. we conduct domain generalization exper- iments on the colored MNIST dataset (CC BY-SA 3.0) 23]. In particular, the task is to predict binary label Y of colored input images X generated from binary digit feature Za and color feature Ze. We refer readers to Appendix D for more information of this dataset.
We conduct 7 source domains by setting color-label correlation P(Z. = 1|Y 1) = P(Z. Y 0) g8.1 where g Uni ([0.6, 1]) for i 1.... 12, while two target domains are created with TT E {0.05,0, 0.7} for i = 1, 2. Here we note that colored images in the target domain with 8T.2 0.7 are more similar to those in the source domains, while colored images the target domain with TT.1 0.05 are less similar.
We wish to study the characteristics and trade-off between two kinds of DI representations when predicting on various target domains. Specifically, we apply adversarial learning [16] similar to 14. in which a min- max game is played between domain discriminator hd trying to distinguish the source domain given representation, while the feature extractor (generator) g tries to fool the domain discriminator. Simultaneously, a classifier is used to classify label based on the representation. Let fyen and Edise be the label classification and domain discrimination losses, the training objective becomes:
min g
min fien + max Ldise h
hd
where the source compression strength À > 0 controls the compression extent of learned represen- tation. More specifically, general DI representation obtained with À O, while larger À leads to more compressed DI representation. Finally, our implementation is based on DomainBed [17] repository, and all training details as well as further MSDA experiment and DG experiment on real datasets are included in Appendix D
3.1 Trade-off of Two Kinds of Domain-Invarian Representations
Figure 1: Phase transition of representations with increased X. (a) General DI representations. (b)
Just-right compressed DI representations. (c) ceeelyccompeessed DI representations.
'Our code could be found http:///thhub.commiinnAAleessss
8

---

We consider target domain which is similar to source domain in this experiment. To govern the trade- off between two kinds of DI representations, we gradually increase the source compression strength À with noting that À 0 means only focusing on learning general DI representations. According to our theory and as shown in Figure [ learning only general DI representations (A 0) maximizes the cross-domain class divergence by encouraging the classes of source domains more separate arbitrarily, while by increasing A, we enforce compressing the latent representations of source domains together. Therefore, for appropriate X, the class separation from general DI representations and the source compression from compressed DI representations (i.e., just-right compressed DI representations), are balanced as in the case (b) of Figure I
while for overly high X, source compression from compressed DI representations dominates and compromises the class separation from general DI representations (i.c., compreeed DI representations).
Figure 2a shows the source validation and target accuracies when increasing À (i.e., encouraging the source compression). It can be observed that both source validation accuracy and target accuracy have the same pattern: increasing when setting appropriate for just-right compressed representations and compromising when setting overly high values À for overly-compressee DI representations. Figure 2b shows in detail the variation
the source validation accuracy for each specific X. In practice, we should encourage learning two kinds of DI representations simultaneously by finding an appropriate trade-off to balance them for working out just-right compressed DI representations.
3.2
Generalization Capacity on Various Target Domains
0.92
6.40
6.75
0.83
6.72
6:00
Olse Sareet OCENEE
102
6.71
5020
25000 30003
Devosa
(a) Accuracy vs
(b) Accuracy vs iteration
Figure 2: 2ab Source validation accuracy and target accuracy for close target domain (see Section
over compression strength. 2h) Validation accuracy over training step for different values of X.
0.72
0.20
2.73
0.30
1020
2.54
0.00
2000
10000
15383
22020
20000
30300
10000
23000 20000 23003
20000
(a) Far target domain with
(b) Close target domain with 0 0.7.
0.05.
Figure 3: Accuracy of models trained with different À on two test domains. For the far domain with 0 = 0.05, the model corresponding with general DI representations performs better than that with more compact DI representations. For close domain with 0 0.7, the models with compression generalize significantly better than those with general DI representations.
In this experiment, we inspect the generalization capacity of the models trained on the source domains with different compression strengths À to different target domains (i.c., close and far ones). As shown in Figure 3 the target accuracy is lower for the far target domain than for the close one regardless of the source compression strengths X. This observation aligns with our upper-bounds developed in Theorems T and 3 in which larger representation discrepancy increases the upper-bounds of the general target loss, hence more likely hurting the target performance.
Another interesting observation is that for the far target domain, the target accuracy for À = 0 peaks. This can be partly explained as the general DI representations help to increase the span source latent representations for more chance to match far target domain (cf. Section 2.3.2h. Specifically, the source-target discrepancy d2/2 (IP" 9: pgg could be small and the upper bound in Theorem i is lower. In contrast, for the close target domain, the compressed DI representations help to really improve

---

the target performance, while wer-coopressionn degrades the target performance, similar to result on source domains. This because the target domain is naturally close the source domains, the source and target latent representations are already mixed up, but by encouraging the compressed DI representations, we aim to learning more elegant representations for improving target performance.
Related Works
Domain adaptation (DA) has been intensively studied from both theoretical 3 9 53
33 43
and
empirical 44
52
4
7
51
perspectives. Notably, the pioneering work [3] and other theoretical
works 4
52
53
21
30 6
have investigated DA in various aspects which lay foundation for the
success practical works 14
47 44
[22 14, [1]... Multiple-soourc domain adaptation (MSDA)
extends DA by gathering training set from multiple source domains [8] 29, 20 42 43] [22 [54 [50 40)
Different from DA with abundant theoretical works, theoretical study in MSDA is significantly
limited. Notably, the works in [3 32) relies on assumptions about the existence of the same labeling function, which can be used for all source domains; and target domain an unknown mixture of the source domains. They then show that there exists distributional weighted combination of these experts performing well on target domain with loss at most 6, given set of source expert hypotheses (with loss is at most on respective domain). Hoffman al. [8] further develop this idea more general case where different labeling functions corresponding to different source domains, and the target domain is arbitrary. Under this setting. there exists a distributional weighted combination of source experts such that the loss on target domain is bounded by source loss term and a discrepancy term between target domain and source mixture. On the other hand, Zhao et [54] directly extend Ben-david et al.'s work with an improvement on the sample complexity term.
is the most challenging setting among the three due to the unavailability of target data. The studies in 4 10) use kernel method to find feature map which minimizes expected target loss. Moreover, those
Domain generalization (DG) 17 25
27 28
34
49 [T] learn compressed DI [5]
representations by minimizing different types of domain dissimilarity. Other notions of invariance are also proposed, for example, [2] learns a latent space on which representation distributions do not need to be aligned, but share a common optimal hypothesis. Another work in [4] uncovers iamain-invariant hypothesis from low-rank decomposition of domain comain-specific hypotheses. Moreover, there are other works which try to strike balance between two types of representation spaces [5].
5 Conclusion
In this paper, we derive theoretical bounds for target loss MSDA and DG settings which characterize two types of representation: general DI representation for learning invariant classifier which works on all source domains and compressed DI representation motivated from reducing inter-domain representation discrepancy. We further characterize the properties of these two representations, and develop a lower bound on the target loss which governs the trade-off hetween learning them. Finally, conduct experiments Colored MNIST dataset and real dataset illustrate theoretical claims.
10

---

References
[1] AL I., MONTEIRO, J., DARVISHI, M., FALK, T. H., AND MITLIAGKAS, I.
Generalizing to unseen domains via distribution matching, 2020. 2.1
[2] ARJOVSKY, M., BOTTOU, L., GULRAIANI, I., AND LOPEZ-PAZ, D. Invariant risk minimiza- tion, 2020.
2.2
[3] BEN-DAVID S., BLITZER, J., CRAMMER, K., KULESZA, A., PEREIRA, F., AND VAUGHAN,
J. W. A theory of learning from different domains, Mach. Learn. 79, 1-2 (May 2010), 151-175.
2.2 2.2
[4] BLANCHARD. G., LEE, G., AND SCOTT, C. Generalizing from several related classification tasks to new unlabeled sample. In Advances in Neural Information Processing Systems (201 1), vol. 24. 4
[5] CHATTOPADHYAY, BALAJI, AND HOFFMAN, J. Learning to balance specificity and invariance for in and out of domain generalization. arXiv preprint arXiv:2008. 200::000811899 (2020).
[6] CORTES, C., AND MOIIRI, M. Domain adaptation in regression. In Algorithmic Learning Theory 22nd International Conference, ALT 2011, Proceedings (2011), Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes Bioinformatics), pp. 903-323. Copyright: Copyright 2011 Elsevier B.V., All rights reserved.; Conference date:
22nd International Conference on Algorithmic Learning Theory, ALT 2011
05- 10-201101 Through 07-10-2011,
[7] CORTES, C., MOHRI, M.. AND MEDINA, A. M. Adaptation based on generalized discrepancy. Journal of Machine Learning Research 20, 1 (2019), 1-30.
2.2 4
[8] CRAMMER, K., KEARNS, M., AND WORTMAN, J. Learning from multiple sources. Journal
of Machine Learning Research 9, 57 (2008), 1757-1774.
[9] DAVID, S. B., LU, T., LUU, T., AND PAI., D. Impossibility theorems for domain adaptation. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (13-15 May 2010), vol. 9 of Proceedings of Machine Learning Research, PMLR, pp. 129-136.
4
[10] DESHMUKH, A. A., LEI, Y., SHARMA, S., DOGAN, U., CUTLER, J. W., AND SCOTT, C. A generalization error bound for multi-class domain generalization, 2019.
[11] DOU, Q., DE CASTRO, D. C., KAMNITSAS, K., AND GLOCKER, B. Domain generalization via model-agnostic learning of semantic features. In Advances in Neural Information Processing Systems (2019), pp. 6450-6461.
2.2
[12] DUAN, L., XU, D., AND CHANG, S.-F. Exploiting web images for event recognition in consumer videos: A multiple source domain adaptation approach. In 20/2 IEEE Conference on Computer Vision and Pattern Recognition (2012), pp. 1338-1344 1345.
[13] DUCHI, J. C., KHOSRAVI, K., AND RUAN, F. Multiclass classification, information, diver- gence, and surrogate risk, 2017. 2.3.1
[14] GANIN, Y., USTINOVA, E., AJAKAN, H., GERMAIN, P., LAROCHELLE, H., LAVIOLETTE, F., MARCH, M., AND LEMPITSKY, V. anvarraadversarial training of neural networks. Journal of Machine Learning Research 17, 59 (2016), 1-35.
2.2
[15] GUIFARY, M., BASTIAAN KLEIN,, W. ZIIANG, M., AND BALDUZZI, D. Domain gener- alization for object recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision (2015), pp. 2551-2559,
2.2
[16] GOODFELLOW, I., POUGET- POUGET-ABADIES J., MIRZA, M., XU, B., MAARDE-FARLESS D., OZAIR, S., COURVILLE, A., AND BENGIO, Y. Generative adversarial nets. In Advances in Neural Information Processing Systems (2014), vol. 27, Curran Associates, Inc. 2.2
[17) GUERAIANI, I., AND LOPRZ-PAZ, D. In search of lost domain generalization. abs/2007.01434 (2020). 3
CoRR

---

[18] HOFFMAN, J., MOHRI, M., AND ZHANG, N. Algorithms and theory for multiple-souree adap- tation. In Advances in Neural Information Processing Systems (2018), S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianche and R. Garnett, Eds., vol. 31, Curran Associates,
Inc.
2.1 4
[19] HUANG, Z., WANG, H., XING, E. P., AND HUANG, D. Self-challenging improves cross- domain generalization. Proceedings of the European Conference on Computer Vision (ECCV) (2020).
[20] IEVGEN REDKO, AMAURY HABRARD, M. S. On the analysis of adaptability in multi-source domain adaptation. Mach. Learn. 108 (June 2019), 1635-1652, 4
[21] JOHANSSON, F. D., SONTAG, D., AND RANGANATH, R. Support and invertibility in domain-
invariant representations. In Proceedings of Machine Learning Research (2019), vol. 89, pp. 527-536. 2.1 4
[22] LE, T., NGUYEN, T., Ho, N., BUI, H., AND PHUNG, Lamda: Label matching deep domain adaptation. In ICML (2021). [
[23] LECUN, Y., CORTES, C., AND BURGES, C. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yanm.lecm wtw.:lccm..cm//uuuinnnisstt 2 (2010). 3
[24] LI, D., YANG, Y., SONG, Y.-Z., AND HOSPEDALES, T. Learning to generalize: Meta-learning for domain generalization.
2.1
[25] LI, D., YANG, Y., SONG, Y.-Z., AND HOSPEDALES, T. M. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference computer vision (2017), pp. 5542-5550.
[26] LI, D... ZHANG, J., YANG, Y., LIU, C., SONG, Y.-Z., AND HOSPEDALES, T. M. Episodic training for domain generalization. In Proceedings of the IEEE/CVF International Conference
on Computer Vision (ICCV) (October 2019). 2.2
[27] LI, H., PAN, J., WANG, S., AND KOT, A. C. Domain generalization with adversarial feature learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2018). 2.1 2.2
[28] LI, Y., TIAN, X., GONG, M., LIU, Y., LIU, T., ZHANG, K., AND TAO, D. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV) (September 2018). 2.1 2.2 4
[29] MANSOUR, Y., MOHRL, M., Ro, J., THEERTHA SURESH, A., AND WU, K. A theory of multiple-source adaptation with limited target labeled data. In Proceedings of The 24th International Conference Artificial Intelligence and Statistics (13-15 Apr 2021), A. Banerjee and K. Fukumizu, Eds., vol. 130 of Proceedings of Machine Learning Research, PMLR,
pp. 2332 2340. 2.1 4
[30] MANSOUR, Y., MOHRI, M., AND ROSTAMIZADEH, A. Domain adaptation: Learning bounds and algorithms. vol. 22.
[31] MANSOUR, Y.. MOHRI, M., AND ROSTAMIZADEH. A. Domain adaptation with multiple sources. In Advances in Neural Information Processing Systems (2009), D. Koller, D. Schuur- mans, Y. Bengio, and L. Bottou, Eds., vol. 21, Curran Associates, Inc
2. 2.2
[32] MANSOUR, Y., MOHRI, M., AND ROSTAMIZADEH, A. Multiple source adaptation and the
renyi divergence. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (Arlington, Virginia, USA, 2009), UAI '09, AUAI Press, p. 367-374.
[33] MANSOUR, Y., AND SCHAIN, M. Robust domain adaptation. Annals of Mathematics and Artificial Intelligence 71 Issue 4 (2014), 365-380,
[34] MUANDET, K., BALDUZZI, D., AND SCHÖLKOPF, B. Domain generalization via invariant feature representation. In Proceedings of the 30th International Conference on Machine Learning (Atlanta, Georgia, USA, 17-19 Jun 2013), S. Dasgupta and D. McAllester, Eds., vol. 28 of Proceedings of Machine Learning Research, PMLR, pp. 10-18. 2.1
12

---

[35] NGUYEN, T., LE. T., ZHAO, H., TRAN, H. Q.. NGUYEN, T., AND PHUNG, D. Most: Multi- source domain adaptation via optimal transport for student-teacher learning. In UAI (2021).
[36] NGUYEN, T., LE, T., ZHAO, H., TRAN, H. Q., NGUYEN, T., AND PHUNG, D. Tidot: A teacher imitation learning approach for domain adaptation with optimal transport. In IJCAI (2021).
[37] NGUYEN, A., NGUYEN, T., LE, T., TRAN, Q. H., AND PHUNG, D. Stem: An approach to multi-source domain adaptation with guarantees. In Proceedings of the IEEE/CVF International Conference on Computer Vision (2021), pp. 9352-9363.
[38] NGUYEN, X., WAINWRIGHTT M. J., AND JORDAN, M. Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization. In Advances Neural Information Processing Systems (2008), J. Platt, D. Koller, Y. Singer, and S. Roweis, Eds., vol. 20, Curran
Associates, Inc. 2.2
[39] PENG, X., BAI, Q., XIA, X., HCANG, Z., SAENKO, K., AND WANG, B. Moment matching for multi-source domain adaptation. In 2019 IEEE/CVF International Conference Computer
Vision (ICCV) (2019), IEEE Computer Society, pp. 1406-1415. T
[40] PENG, X., BAI, Q.. XIA, X., HCANG, Z., SAENKO, K., AND WANG, B. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (October 2019).
[41] PIRATLA, V., NETRAPALLI, P., AND SARAWAGI, S. Efficient domain generalization via common-specific low-rank decomposition. In Proceedings of the 37th International Conference on Machine Learning (13-18 13-18878 Jul 2020), vol. 119 of Proceedings of Machine Learning Research, PMLR, pp. 7728-7738.
[42] REDKO, I., COURTY, N., FLAMARY, R., AND TUIA, D. Optimal transport for multi-soure domain adaptation under target shift. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statisties (16-18 Apr 2019), K. Chaudhuri and M. Sugiyama, Eds., vol. 89 of Proceedings of Machine Learning Research, PMLR. pp. 849-858.
2.1
2.2
[43] REDKO, I., HABRARD, A., AND SEBBAN, M. Theoretical analysis of domain adaptation with optimal transport, 2017.
[44] SAITO, K., KIM, D., SCLAROFF, S., AND SAENKO, K. Universal domain adaptation through self supervision. In Advances in Neural Information Processing Systems (2020), H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds., vol. 33, Curran Associates, Inc., pp. 16282-16292.
[45] SHALEV-SHWARTZ. S., AND BEN-DAVID, S. Understanding machine learning: From theory algorithms. Cambridge university press, 2014. 2.3.3
[46] SICILIA, A., ZHAO, X., AND HWANG, S. Domain adversarial neural networks for domain generalization: When it works and how to improve, 2021. 2.1
[47] TZENG, E., HOFFMAN, J., SAENKO, K., AND DARRELL, T. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (July 2017).
[48] VAPNIK, V., AND VAPNIK, Statistical Learning Theory. Wiley, 1998. 2.3.3
[49] WANG, H., HE, Z., LIPTON, Z. C., AND XING, E. P. Learning robust representations by projecting superficial statistics out. In International Conference Learning Representations (2019). 2
[50] XU, R., CHEN, Z.,, ZUO, W., YAN, J., AND LIN, L. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2018). 4
13

---

[51] YAN, S., SONG, H., LI, N., ZOU, L., AND REN, L. Improve unsupervised domain adaptation with mixup training, 2020.
[52] ZIANGG, Y., LIU, T., LONG, M., AND JORDAN, M. Bridging theory and algorithm for domain adaptation. In Proceedings of the 36th International Conference Machine Learning (09-15 Jun 2019), K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97 of Proceedings of Machine Learning Research, PMLR, pp. 7404-7413, 14
[53] ZHAO, H., COMBES, R. T. D., ZHANG, K., AND GORDON, G. On learning invariant representations for domain adaptation. In Proceedings of the 36th International Conference on Machine Learning (09-15 Jun 2019), K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97 of Proceedings of Machine Learning Research, PMLR, pp. 7523-7532.
2.4 2.4 4
[54] ZHAO, H., ZHANG, S., WU, G., MOURA, J. M. F., COSTEIRA, J. P., AND GORDON, G. J. Adversarial multiple source domain adaptation. In Advances in Neural Information Processing and
Systems (2018), S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N.
R. Garnett, Eds., vol. 31, Curran Associates, Inc.
2.2
[55] ZIAO,, S., GONG, M., LIU, T., FU, H., AND TAO, D. Domain generalization via entropy regularization. In Advances in Neural Information Processing Systems (2020), H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds., vol. 33, Curran Associates, Inc., pp. 16096-16107.1
[56] Zhao, S., WANG, G., ZHANG, S., GU, Y., LI, Y., SoNG, Z., P., HU, R., CHAI, H., AND KEUTZER, K. Multi-source distilling domain adaptation. Proceedings of the AAAI Conference on Artificial Intelligence 34, 07 (Apr. 2020), 12975-12983.
[57] ZHOU, K., YANG, Y., HOSPEDALES, T., AND XIANG, T. Learning to generate novel domains for domain generalization. In European Conference on Computer Vision (2020), Springer, pp. 561-578. 2.2
14


Reviews:
[{"review_id": "xuqCOEAJFD", "reviewer": "Reviewer_XYGp", "summary": "This paper proposes two novel upper bounds for the target loss in the context of multi-source domain adaptation and generalization and defines two types of domain-invariant representations: general domain invariant and compressed domain invariant. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: \n\nDomain adaptation and generalization are very important issues and as a consequence, many theoretical and practical works have been proposed. This submission completes this large avenue of research with a focus on domain invariant representations in the specific setting of multi-source adaptation and generalization.\n\nThe submission can be completed with more comparisons and positioning with other theoretical works on multi-source domain adaptation such as the one in ref [50]. For instance,  how is Theorem 1 compared with Theorem 3 of ref [50]? \n\n**Quality**\n\n- All core assumptions are made explicit, their motivation and their implications are discussed. Some insights should be given on the choice of considering multiple sources as a mixture of source distributions. For instance, some recent works have been proposed on domain aggregation in the context of multi-source domain adaptation such as [Wen, ICML 2020](http://proceedings.mlr.press/v119/wen20b.html) and this point could be more discussed. In particular, what are the limitations of considering mixtures of distributions? How it behaves in the context of source disagreement (see for instance [Liu et al, 2021](https://hal.archives-ouvertes.fr/hal-02944558))\n\n- Experiments are proposed in order to support the claims. The experimental setup is straightforward, all relevant details are given. Experiments on the PACS dataset are also given to show how the approach behaves on real data. \n\n- The body of supplementary material is impressive. The proofs are sufficiently easy to follow and well detailed.\n\n\n**Clarity**: \nThe paper is well-written and structured. It is a very dense paper. Supplementary materials are mandatory in particular to have the details on the experimental settings. \n\n**Significance**: The submission brings new theoretical results", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors not addressed these aspects. ", "review_text": "", "extra_scores": {"ethical_concerns": "no", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "M7cWCD0FH9", "reviewer": "Reviewer_kHSY", "summary": "This paper studies theoritical aspects of two practical domain adaptation settings, i.e. multiple source domain adaptation and domain generalization settings, where there are multiple source domains and the target domain is potential unavailable during training.\nSpecifically, novel upper-bounds for the target general loss are developed and proved. Based on the upper bounds, two kinds of domain-invariant representations are defined. The pros and cons are also studied and the trade-offs of enforcing learning each domain-invariant representation are investigated.\nExperiments are conducted to inspect the trade-off of these representations for offering practical hints regarding how to use them in practice and explore other interesting properties of the developed theory.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: This paper focuses on the theory analysis of multiple source domain adaptation and domain generalization. The developed upper bounds are novel and make sense. The defined two kinds of domain-invariant representations are reasonable. The pros and cons and the trade-offs of enforcing learning each domain-invariant representation are also well explored. The theory part is sufficient.\n\nQuality: The developed upper bounds sound good and the corresponding technical domain-invariant representation is technically sound. The main claims are supported mainly by theoritical results. The authors prove and explain the theoritical strengths well but the discussion on the weaknesses of the propoded method and the experiment are insufficient.\n\nClarity: Generally, this paper is well organized and clearly written. But there are too many mathetical notations and equations, which make it difficult to understand for general domain adaptation researchers.\n\nSignificance: The novel upper bounds for multiple source domain adaptation and domain generalization are important and can motivate other researchers or practitioners likely to use the upper bound to design novel algorithms.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is insufficient discussion on the limitations and potential negative societal impact of the work.\n1. There already exist some theoritical analysis, such as [18] [50] on multi-source domain adaptation. However, there is no clear comparison between the upper bounds of the proposed method and these existing methods. Are the designed upper bounds tighter than exsisting ones?\n2. The experimental evaluation is weak. Generally, there are many datasets to evaluation the performances of multiple source domain adaptation and domain generalization algorithms, such as Digits-five, Office-31, Office-Home, DomainNet. However, there is no experiment on these datasets and no comparison with existing methods, such as [50] [52].\n3. There is no analysis on the conditions on which the proposed upper bounds hold. If the conditions are not satisfied, what will happen?\n4. Comparison with exisiting domain adaptation in related work section is not satisfying. Adding some recent surveys would make more sense, such as \"A review of domain adaptation without target labels\", \"A Review of Single-Source Deep Unsupervised Visual Domain Adaptation\", \"A Survey of Unsupervised Deep Domain Adaptation\",  \"Multi-source Domain Adaptation in the Deep Learning Era: A Systematic Survey\".", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "GyppX7DBErN", "reviewer": "Reviewer_dgwj", "summary": "This work proposes a principled theoretical framework for Multi-Source Domain Adaptation (MSDA) and Domain Generalization (DG), which is lacked in existing works, particularly under a generally setting of multi-class classification. Specifically, this works propose two target domain generalization bounds corresponding to general domain-invariant (DI) representations and compressed DI representations. On such basis, detailed theoretical analysis are proposed to analyze the characteristics of these two bounds in practice and the trade-off between them. By instantiating these two learning objectives under an adversarial domain adaptation framework, authors conduct experiments on colored MNIST and PACS datasets to verify the existence of trade-off and the various generalization capability on different target domains.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1. This work fills the blank of a general theoretical framework for MSDA and DG problems under a multi-class classification setting.\n2. Various previous practical works can be interpreted under this framework as learning general DI representations and compressed DI representations. More future explorations can be inspired by these developed theorems. \n3. Empirical results on two datasets verify the theoretical results under both MSDA and DG settings.\n\n\nWeakness and question:\n\n1. I'm basically convinced by the two proposed target domain generalization bounds. However, in order to better judge the contribution of these bounds, I think it will be good to more completely discuss the connection and difference with the bounds developed in previous theoretical works for MSDA [a,b].\n2. It is good to see that the instantiation under the adversarial domain adaptation framework (equivalent to JS divergence) matches the proposed theorems about trade-off and target domain generalization ability. However, I think it would be better to implement one or two more instantiations, e.g. using MMD or WS distance, which makes the empirical verification more convincing.\n3. Some typos in the current version:\n     - In Line 88 of main paper, [\\pi_i]_{i=1}^{C}  ->  [\\pi_i]_{i=1}^{K}.\n     - In Line 274 of appendix, \\max_{\\hat{h}^{s-s}}  ->  \\max_{\\hat{h}^{s-t}}.\n\n[a] Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. 'Domain adaptation with multiple sources.' NeurIPS, 2009.\n\n[b] Judy Hoffman, Mehryar Mohri, and Ningshan Zhang. 'Algorithms and theory for multiple- source adaptation.' NeurIPS, 2018.\n\n\nJustification of rating:\n\nI am convinced by the proposed theoretical framework and think it could be a solid foundation for the future practical works. I admire the great efforts of authors on boosting the theoretical understanding of MSDA and DG.\n\n\n**Post Rebuttal**\n\nI have read the response from authors and other reviews. In the response, authors thoroughly discuss the connections and differences with two closely related works, which makes me more convinced about the theoretical contributions of this work. I'll keep my positive rating, and I suggest the authors to supplement additional experimental verifications in the revised version to make these theories more grounded. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the proposed method are less discussed in the current version. Also, the potential social impact is lacked in the paper. Though it is a theoretical work, authors are also encouraged to analyze it can potentially benefit and also hamper the related real-world applications when the theory is implemented in a practical system.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "FJAwC5Rmdpp", "reviewer": "Reviewer_C95H", "summary": "The paper derives theory for invariant representations in unsupervised domain adaptation and domain generalization, in the case when there are multiple labeled source domains. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper provides a large volume of theory for the multiple-source domain adaptation and domain generalization setting. However, this paper lacks clarity and sufficient comparison with existing work to meet the requirements for acceptance. \n\nFirst of all, the paper claims to be the first one to theoretically study in detail the invariant representations for domain adaptation in the multiple-source setting. However, the paper by [Zhao et al. 2018] studied exactly this for domain adaptation, and has provided a generalization bound that, from my understanding, is minimized when we acquire a compressed domain-invariant representation (using the authors' terminology). Given the similarity between the proposed study and the one by [Zhao et al. 2018], it is not clear to me what the novelty is in this paper, as the authors have not provided sufficient discussion and comparison. \n\nFurthermore, the authors derive bounds in the case when P(Y) changes across domains. It is obvious that invariant representation learning is not viable for this setting, and it needs to be modified (relaxed) in order to be suitable. This has been studied in [Wu et al. 2019], and it would be great if the authors could discuss the comparison between their reasoning and the one presented in this study. \n\nFinally, the paper can greatly benefit from better clarity and discussion of individual terms introduced in the theoretical results. In particular, Theorem 5 and the paragraph preceding it are confusing since $h$ here represents a domain discriminator, where before and after this results it represents a classifier for the label $y$. \n\nReferences: \nWu, Yifan, et al. \"Domain adaptation with asymmetrically-relaxed distribution alignment.\" International Conference on Machine Learning. PMLR, 2019.\n\nZhao, Han, et al. \"Adversarial multiple source domain adaptation.\" Advances in neural information processing systems 31 (2018): 8559-8570.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There has not been any discussion regarding limitations in this paper, and any brief discussion would be welcome. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper develops novel upper bounds for the target risk in the multiple source domain adaptation/generalization settings. Based on the theoretical results, the authors further study how to learn proper domain invariant representations. The proposed theory provides new insights and offers practical hints on multiple source domain adaptation. I thus recommend acceptance of this paper.

Nevertheless, there are some concerns that need to be addressed in the final version. The main concern is the comparison with existing theoretical results and methods in Zhao et al. 2018 and Hoffman et al. 2018. The authors have well addressed these in the rebuttal, and I hope the authors could carefully revise the paper to incorporate these discussions.

Decision: Accept (Poster)