On Learning Domain-Invariant Representations for Transfer Learning with Multiple Sources
Authors: Trung Quoc Phung
Trung Le
Long Tung Vuong
Toan Tran
Anh Tuan Tran
Hung Bui
Dinh Phung
Venue: NeurIPS 2021 Poster (2021)
Keywords: Domain generalization
domain adaptation
transfer learning
multiple-source domain adaptation
machine learning
deep learning

Abstract: Domain adaptation (DA) benefits from the rigorous theoretical works that study its insightful characteristics and various aspects, e.g., learning domain-invariant representations and its trade-off. However, it seems not the case for the multiple source DA and domain generalization (DG) settings which are remarkably more complicated and sophisticated due to the involvement of multiple source domains and potential unavailability of target domain during training. In this paper, we develop novel upper-bounds for the target general loss which appeal us to define two kinds of domain-invariant representations. We further study the pros and cons as well as the trade-offs of enforcing learning each domain-invariant representation. Finally, we conduct experiments to inspect the trade-off of these representations for offering practical hints regarding how to use them in practice and explore other interesting properties of our developed theory.

On Learning Domain-Invariant Representations for Transfer Learning with Multiple Sources
Trung Phung!
Trung Le² Long Vuong' Toan Tran' Anh Tran' Hung Bui Dinh Phung!
VinAI Research, Vietnam Monash University, Australia
v. t.rrungg3évvinai io, trunglm@monash.co [v. 1ongvt8, v. v.toanmm3, a.anttt12,, v.hungbh1, v.dinhpq2)evinai.cm
Abstract
Domain adaptation (DA) benefits from the rigorous theoretical works that study insightful characteristics and various aspects, e.g., learning Nomain-invariinn representations and its trade-off. However, it seems not the case for the multiple source DA and domain generalization (DG) settings which are remarkably more complicated and sophisticated due to the involvement of multiple source domains and potential unavailability of target domain during training. In this paper, we develop novel upper-bounds for the target general loss which appeal to to define two kinds of domain invariant representations. We further study the pros and cons well as the trade-off's of enforcing learning each domain-invarianl representation. Finally, we conduct experiments to inspect the trade-off of these representations for offering practical hints regarding how to use them in practice and explore other interesting properties our developed theory.
domain-invariant
Introduction
Although annotated data has been shown to be really precious and valuable to deep learning (DL), in many real-world applications, annotating sufficient amount of data for training qualified DL models is prohibitively labour-expensive, time-consuming, and error-prone. Transfer learning is a vital solution for the lack of labeled data. Additionally, in many situations, limited number of annotated data from multiple source domains. Therefore, it desirable to train a qualified DL model primarily based on multiple source domains and possibly together with target domain. Depending on the availability of the target domain during training, we encounter either multiple source domain adaptation (MSDA) domain generalization (DG) problem.
only able to collect
Domain adaptation (DA) is a specific case of MSDA when we need to transfer from a single source domain to another target domain available during training. For the DA setting, the pioneering work [3] and other following work (4] 52 53
54 22] are abundant to study its insightful characterizations and aspects, notably what invariiann (DI) representations are, how to learn this kind of representations, and the trade-off of enforcing learning DI representations. Those
studies lay the foundation for developing impressive practical DA methods 14 47 52 5T 36).
Due to the appearance of multiple source domains and possibly unavailability of target domain, establishing theoretical foundation and characterizing DI representations for the MSDA and especially DG settings significantly more challenging. A number of state-of-the-a art works in MSDA and
49 35 37 have implicitly exploited and used DI representations in some sense to achieve impressive performances, e.g., [28] 288 III minimize representation divergence during training, 41 decomposes representation to obtain invariant feature,
DG [66 I9
[
41
27
28 34
[5
55|
[55] maximizes domain prediction entropy to learn invariance. Despite the successes, the notion DI representations in MSDA and DG is not fully-understood, and rigorous studies to theoretically
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

characterize DI representations for these settings are still very crucial and imminent. In this paper, we provide theoretical answers to the following questions: (1) what are DI representations in MSDA and DG, and (2) what one should expect when learning them. Overall, our contributions in this work can be summarized as:
In Section 2.2 we first develop two upper-bounds for the general target loss in the MSDA and DG settings, whose proofs can be found in Appendix A. We then base on these bounds to characterize and define two kinds of DI representations: general DI representations and compressed DI representations.
We further develop theory to inspect the pros and cons of two kinds of DI representations in Section 2.3 aiming to shed light on how to use those representations in practice. Particularly, two types of DI representations optimize different types of divergence on feature space, hence serving different purposes. Appendix B contains proofs regarding these characteristics.
Finally, we study in Section 2.4 the trade-off of two kinds of DI representations which theoretically answers the question whether and how the target performance is hurt when enforcing learning each kind of representation. We refer reader to Appendix C for its proof.
We conduct experiments to investigate the trade-off of two kinds of representations for giving practical hints regarding how to use those representations in practice as well as exploring other interesting properties of our developed theory.
is worth noting that although MSDA has been investigated in some works 18 54 our proposed method is the first work that rigorously formulates and provides theoretical analysis for representation learning in MSDA and DG. Specifically, our bounds developed in Theorems I and interweaving both input and latent spaces novel and benefit theoretical analyses in deep learnin. Our theory is developed in general setting of multi-class classification with a sufficiently general loss, which
can be viewed which considers binary classification with specific loss functions. Moreover, our theoretical bounds developed in Theorem is is the first theoretical analysis of the trade-off of learning DI representations in MSDA and DG. Particularly, by considering the MSDA setting with single source and target domains, we achieve the same trade-off nature discovered in [53] but again our setting more general than binary classification setting in that work.
non-trivial generalization of existing works in DA 30 3
53
31 54), each of
2 Our Main Theory
2.1
Notations
Let be a data space on which we endow a data distribution P with a corresponding density function p(x)) We consider the multi-class classification problem with the label set y [C]. where C is the number of classes and [C] := f1..... ..... C}. Denote VA- {a RC :|aal]
2 0} as the C-simplex label space, let f : X' + V be a probabilistic labeling function returning a C-tuple f (x) = [f (x, م) whose element ff (z, i) = p (y = i x) is the probability to assign a data sample P to the class (i.c., i [1,,.. CJ). Moreover, a domain is denoted compactly as pair of data distribution and labeling function D := (P, f). We note that given a data sample x ~ P. its categorical label y E y sampled as y Cat (f (z)) which categorical distribution over f (z) E V/s.
(z) v) with f (x) E Va and y E y specifies the loss (e.g., cross-entropy, Hinge, LI, or L2 loss) to assign data sample to the class by the hypothesis f. Moreover, given a prediction probability f ((x) w.r.t. the ground-truth prediction f (x).. we define the loss E (f (x), (x)
Let VA x y + R be a loss function, where (f(z),
Ey-s(z) [ (f (f) - ا (f (z) f (I, v). This
means é (., is convex w.r.t. the second argument. We further define the general loss caused by using classifier f X + Va to predict IB fP as
c (f.f.P) = (F.D) := E.-P (e(F())
We inspect the multiple source setting in which we are given multiple source domains (DS..). K over the common data space X, each of which consists of data distribution and its own labeling

---

function D& = (PS.t, fS.t). Based on these source domains, we need to work out a learner or classifier that requires to evaluate on a target domain DD" := (IIT f"). Depending the knownness
or unknownness of a target domain during training, we experience either multiple source DA (MSDA)
31
29
18
domain generalization (DG) 24 28 27
34 setting.
One typical approach in MSDA and DG is to combine the source domains together 14
11 46] to learn DI representations with hope to generalize well to target domain. When combining the source domains, we obtain mixture of multiple source distributions denoted as D
34
Σ TT..=:; where the mixing coefficients x = [ K can be conveniently set to тद = 25-1 N;
N;
with N, being the training size of the i-th source domain.
In term of representation learning. input is mapped into a latent space Z by feature map X' + 2 and then classifier h E r> V is trained based on the representations (X).. Let f: X' + VA be the original labeling function. To facilitate the theory developed for latent space, we introduce representation distribution being the pushed-forward distribution P. := ggIP, and the
labeling function h : Z + Va induced by g as h(2) [a-1(e) (a)p(())
21 Going back to our
-o-1ce) plx)dz
multiple source setting, the source mixture becomes 10 = E, M.DSs, where each source domain is S (IS.. h and the target domain is DY (PT h²))
Finally, in our theory development, we use Hellinger divergence between two distributions defined as D1/2 (P), PP) 2 (Vo*(()
dr, whose squared d1/2 / D/1y2 is a proper metric.
2.2 Two Types of Domain-Invarian Representations
Hinted by the theoretical bounds developed in Bl. DI representations learning, in which feature extractor 9 maps source and target data distributions to a common distribution on the latent space, is well-grounded for the DA setting. However, this task becomes significantly challenging for the MSDA and DG settings due to multiple source domains and potential unknownness of target domain. Similar to the case of DA, it is desirable to develop theoretical bounds that directly motivate definitions of DI representations for the MSDA and DG settings, as being done in the next theorem.
Theorem 1. Consider a mixture of source domains D = EK ny S and the target domain IDF. Let l be any loss function apper-bounded by positive constant L. For any hypothesis f : X + Va where f = h g with g X 2 and h 2 Vs, the target loss on input space is upper bounded
K
c (F.DT) & Σ
T (f, L max Eps. [IIAp' [+[+)]] + LV221/2 (P) 1P) (E[K]
(1)
1=1
5SA - y) y=1 is the absolute of single point label shift on imput space between source domain S the target domain IT,, .nn [K] 21. K}.
where Ap Ap(x)) = (1)
The bound in Equation Timplies that the target loss in the input or latent space depends on three terms: (i) representation discrepancy: d1/2 (177 PT), (ii) the label shift: [ Eps.. Ap' (v]=))la). and (iii) the general source loss: ΣKr T f. DS..). To minimize the target loss the left side, we need to minimize the three aforementioned terms. First, the label shift term is natural characteristics of domains, hence almost impossible to tackle. Secondly, the representation discrepuncy term can be explicitly tackled for the MSDA setting, while almost impossible for the DG setting. Finally, the general source loss term convenient to tackle, where its mininization results in a feature extractor g and a classifier h.
7 that consider both losses and data discrepancy on data space, our bound connects losses on data space discrepancy on representation space. Therefore, our theory provides natural way to analyse representation learning, especially feature alignment deep learning practice. Note that although DANN [1] explains their feature alignment method using theory developed by Ben-david et al. B. it is not rigorous. In particular, while application of the theory to representation space yield a representation discrepancy term, the loss terms are also that feature space, and hence minimizing these losses is not the learning goal.
Contrary to previous works in DA and MSDA 3
31
54
3

---

Finally, our setting is much more general, which extends to multilabel, stochastic labeling setting, and any bounded loss function.
From the upper bound, we turn to first type of DI representations for the MSDA and DG settings. Here, the objective of is map samples onto representation space in way that the common classifier h can effectively and correctly classifies them, regardless of which domain the data comes from.
Definition 2. Consider class 9 of feature maps and a class H of hypotheses. Let {DS be set of source domains.
(DG with unknown target data) feature map g" G is said to be DG general domain- invariint (DI) feature map if is the solution of the optimization problem (OP): minges minjig E 7iL (h, Moreover, the latent representations z 9" (x) induced by 9" is called general DI representations for the DG setting.
ii) (MSDA with known target data) A feature map 9" E G is said to be a MSDA general DI feature map g" is the solution of the optimization problem (OP); mingeg min E T. (,, (iA)))
which satisfies py. = P (i.e., mingeg d1/2 ( Moreover, the latent representations z g" (x) induced by g" is called general DI representations, for the MSDA setting.
The definition of general DI representations for the DG setting in Definition 2iis transparent in light of Theorem wherein we aim to find and h to minimize the general source loss Σ 7. (f D.S.)) due to the unknownness of Ip_ Meanwhile, regarding the general DI representations for the MSDA setting, we aim to find g* satisfying Pr 1 and h to minimize the general source loss DK J.G. (D.)). Practically, find general representations for MSDA, we solve
Σ Tcl (g. + AD D pat
min min gE0 hEH
where À > 0 is trade-off parameter and D can be any divergence (e.g., Jensen Shannon (JS) divergence [16, f d-iivergence 38 MMD distance (27 or WS distance 42
In addition, the general DI representations have been exploited in some works [57 26 from the practical perspective and previously discussed in (2) from the theoretical perspective. Despite the similar definition to our work, general DI representation in is not motivated from minimization of a target loss bound.
As pointed out in the next section, learning general DI representations increases the span of latent representations g (1)) the latent space which might help to reduce d. /2 (PT
g: TPx) for general 11 have also explored the
target domain. On the other hand, many other works 27 28 15
possibility of enhancing generalization by finding common representation among source distributions. The following theorem motivates the latter, which the second kind of DI representations.
Theorem 3. Consider mixture of source domains D =EK DSD and the target domain ID² Let E be any loss function upper-bounded by a positive constant L. For any hypothesis f X + V where f = h 9 with : X + Z and h Z + VA. the target loss on input space is upper bounded
K
C (f.
Σ
TiL (f, (D) L max Ros.: sE(K]
(App(())))l)]
L LV
d1/2
(PT PS
K
K
LV2T K
da/2 (p (ss).
Evidently, Theorem suggests another kind of representation learning, where source representation distributions are aligned order to lower the target loss't 's bound as concretely defined as follows.
Definition& 4. Consider class G feature maps and class H of hypotheses. Let (TDS.1) K set of source domains.
Jiml be

---

i) (DG with unknown target data) A feature map g" E G is a DG compressed DI represen- tations for source domains {DS.)K F==1 if g" is the solution of the optimization problem (OP):
minges min E TiL ( D2 which satisfies pa = P. _... ......... p. (i.e., the pushed
forward distributions of all source domains identical). The latent representations =g" (x) is then called compressed DI representations for the DG setting.
ii) (MSDA with known target data) A feature map g² E G is MSDA compressed DI repre- sentations for source domains (TDS..) Ji=1 if g" is the solution of the optimization problem (OP): (
p.S. mingeg min Ex= 7.G (h.Dg. D.g which satisfies Ip P =... =p" Fig. (i.e., the
pushed forward distributions of all source and target domains are identical). The latent representa- lions: = 9" (x) is then called compressed DI representations for the MSDA setting.
2.3 Characteristics of
Representations
In the previous section, two kinds of DI representations are introduced, where each of them originates from minimization of different terms the upper bound of target loss. In what follows, we examine and discuss their benefits and drawbacks, We start with the novel development of hypothesis-awar divergence for multiple distributions. which necessary for our theory developed later.
2.3.1 Hypothesis-Awan Divergence for Multiple Distributions
Let consider multiple distributions QI,..., Qc on the same space, whose density functions are
I..... QC. We are given mixture of these distributions Qo = Ee a: with mixing coefficient V/a and desire measure a divergence between Q1, .... Qc. One possible solution is employing a hypothesis from an infinite capacity hypothesis class H. to identify which distribution the data
come from. In particular, given that Qi. Note that we use the notation h in this particular subsection denote domain classifier, while everywhere else in the paper denotes label classifier. Let (2), R be
Q" the function h () e [C] outputs the probability
the loss when classifying z using h provided the ground-truth label [C]. Our motivation is that if the distributions Qi.... Qo are distant, it is easier to distinguish samples from them, hence the minimum classification loss minjieH CQ.:
is much lower than in the case of clutching Q....., Qc. Therefore, we develop the following theorem to connect the loss optimization problem with an f-divergence among Q1 ..., Qc. More discussions about hypothesis-awann aware divergence can be found in Appendix of this paper and in [13]
Theorem 5. Assuming the hypothesis class H has infinite capacity, we define the hypothesis divergence for multiple distributions as
D (QQ, Qc) min CR.
(i)) + Ci
where Ci depends only on the form of loss function and value of a. This divergence is a proper f-divergence among Q1,... QC in the sense that D (Q1,... Qc) 2 O, VQ,,,...,, Qo and a YA.
and D (QI,... Qc) = 0 Of
Qc.
2.3.2 General Invariant Representation
As previously define,, the general DI feature map g" (cf. Definition 2 the one that minimizes the total source loss
g" arg min min DEG hER
T (h, S.t (...t.
Pp) argmin ang min T T.
(4)
we expect that the minimal loss minjeu L (h, 1 1p5.. should be
From the result of Theorem inversely proportional to the divergence between the class-conditionals PS...c. To generalize this result to the multi-source setting, we further define Q- ΣK Ti T..ipp as the mixture of the class conditional distributions of the source domains the latent space, where a. > Ty
5

---

are the mixing coefficients, and Yi = P (y= c) are label marginals. Then, the inner loop of the objective function in Eq. 4 can be viewed as training the optimal hypothesis h E H to distinguish the samples from Q e E [C] for a given feature map Therefore, by linking to the multi-divergence concept developed in Section 2. we achieve the following theorem.
Theorem 6. Assume that H has infinite capacity, we have the following statements.
1. Da (Q.)
Q2()) = --=-i-inn mingen Ei=1 J.f (..hS.,,gg)) + const, where a [aeles|o] is
defined as above.
2. Finding the general DI feature map g" via the OP in Eq. 4 is equivalent solving
g = argmax D (Qaa) " (.
(5)
scg
Theorem 6 especially its second claim, discloses that learning general DI representations max- imally expands the coverage of latent representations of the source domains by maximizing
D (Q=a. ..... @2C). Hence, the span of source mixture I = Ees1 a. is also increased.
We believe this demonstrates one of the benefits of general DÍ representations because it implicitly enhance the chance to match general unseen larget domuins in the DG setting by possibly reducing
the souree-target representation discrepancy term d1/2 (TP) Pg)) (cf. Theorem [] Additionally, in MSDA where we wish to minimize d1/2 (42, IP.) explicitly, expanding source representation's 's coverage is also useful.
2.3.3 Compressed Domain-Invarian Representations
is well-known that generalization gap between the true loss and its empirical estimation affects generalization capability of model [48 45). One way to close this gap is increasing sampling density. However, as hinted by our analysis, enforcing general DI representation learning tends to maximize the cross-domain class divergence, hence implicitly increasing the diversity of latent representations from different source domains. This renders learning a classifier h on top of those source representations harder due to scattering samples on representation space, leading to higher generalization gap between empirical and general losses. In contrast, compressed DI representations help making the task of learning h easier with lower generalization gap by decreasing the diversity of latent representations from the different source domains, via enforcing S=S.1 g* = SpS.2 =.. pS.K We now develop rigorous theory to examine this observation.
Let the training set be S = {(2,, 3) consisting of independent pairs (2,, y:) drawn from the source mixture, i.e., the sampling process starts with sampling domain index k ~ Cal (T). then sampling z ~ 1g (i.e., x ~ S.te and 2 = 9 (z)), and finally assigning label with y ~ h (z) (i.e., S. (2) is distribution over (C)). The empirical loss is defined as
K
Nu
C (h.
1
(h(z), h S.k (z)).
(6)
Σ
where Nk, & [C] is the number of samples drawn from the k-th source domain and N ΣK
A== No.
Here, L (h. is an unbiased estimation of the general loss L [D. Ek T&L (h. D.A.A. To quantify the quality of the estimation, we investigate the upper bound of the generalization gap L (h, Sh L (h. Dy) with the confidence level 1 ō.
(h,
Theorem 7. For any confident level 5 E [0, over the choice of S, the estimation of loss is in the e-range of the true loss
Pr (( L (f. DT) 56) 2-
where E = =6())= = (4) (/) is a function of 8 for which A is proportional to
1
K
K
V
c (F.D
+ L Σ
Th max E.S,A kE(K
A (u|x))
+
L
K K
V2T, d1/2 (pSt, :ps)))

---

Theorem 7 reveals one benefit of learning compressed DI representations, that is, when enforcing compressed DI representation learning, minimize & Σ Σه= V d1/2 (1P5. PP))) which tends to reduce the generalization gap e (6) for a given confidence level - 6. Therefore, compressed DI representations allow us to minimize population loss L Dg)) more efficiently via minimizing empirical loss.
2.4 Trade-off of Learning Domain Invariant Representation
Similar to the theoretical finding in Zhao et al. [53] developed for DA, we theoretically find that compression does come with a cost for MSDA and DG. We investigate the representation trade-off, typically how compressed DI representation affects classification loss. Specifically, consider
data processing chain X' م Z - VA, where X' is the common data space, Z the latent space induced by a feature extractor 9. and is a hypothesis on top of the latent space. We define 17 and f as two distribution over y in which to draw ~ Py. we sample k ~ Cat (T), x ~ p.S.k and y ~ fs. (z), while similar to draw P.. Our theoretical bounds developed regarding the trade-off of learning DI representations are relevant to d1/2 (135, IP}))
Theorem 8. Consider a feature extractor g and a hypothesis h, the Hellinger distance between two label marginal distributions P3, and Po can be upper-bounder as:
1. d1/2 133,13) S
E TH (h f..kk ...A))
)
+ d1/2 [ITT 1.) + L ahachhon g, fT) 1/2
d1/2 (P5, PJ) I EEK 7.
)
)1²
1/2
+ =K EK
vh K d1/2 ())))
v K /1/2 (PT, (g + L (hog. 1/2
ΣK Ej=
Here we note that the general loss L is defined based on the Hellinger loss E which is define
e(f(s), -i = DE (V -((e,))-V((,,))) found in Appendix C).
(more discussion can be
Remark. Compared to the trade-off bound in the work of Zhao et al. 53 our context is more general, concerning MSDA and DG problems with multiple source domains and multi-class probabilistic labeling functions, rather than single source DA with binary-class and deterministic setting. Moreover, the Hellinger distance is more universal, in the sense that does not depend on the choice of classifier family H and loss function l as in the case of H-divergence in [53]
We base on the first inequality of Theorem 8 to analyze the trade-off of learning general DI repre- sentations. The first term on the left hand side is the source mixture's 's loss, which is controllable and tends to be small when enforeing learning general DI representations. With that in mind, if two label marginal distributions PT and PI are distant (i.c., d1/2 (P3, FT) is high), the sum 1/2 tends to be high. This leads to 2 possibilities. The first
11/2 (PT.P)) + L fT,
scenario is when the representation discrepancy d1/2 (PT .P.) has small value, e.g., it is minimized in MSDA setting, or it happens to be small by pure chance in DG setting. In this case, the lower bound of target loss L (h g. 9. JT. TTT) is high, possibly hurting model's generalization ability. On
the other hand, the discrepancy d1/2 (PP2 p.) is large for some reasons, the lower bound of target loss will be small, but its upper-bound is higher, as indicated Theorem [
g.
Based on the second inequality of Theorem g we observe that if two label marginal distributions P3 and Py are distant while enforcing learning compressed DI representations (i.e.. both source loss and source-source feature discrepancy f pp..) 1/2
ΣK T,L ( g, 8
E ΣK A d2 (PS. pPS)) are low), the sum K d1/2 (PT .PS.)) Chog, fr. /T
ΣK K
/E,
1/2 is high. For the MSDA setting, the
7

---

Va x21 (Pry (..) trained to get smaller, meaning that the lower bound discrepancy E Ej=1 of target loss L (hog. PT) is high, hurting the target performance. Similarly, for the DG setting, if the trained feature extractor و occasionally reduces Σ دj=1 K d1/2 (P. P for some unseen target domain, it certainly increases the target loss C (h o g. fr, .P)). In contrast, if for some target domains, the discrepancy Σ 2ز=1 vT d (Pg P..)) is high by some reasons, by linking to the upper bound in Theorem the target general loss has a high upper-bound, hence is possibly high.
K v
This trade-off between representation discrepancy and target loss suggests a sweet spot for just-right feature alignment. In that case, the target loss is most likely to be small.
3 Experiment
To investigate the trade-off of learning DI representations. we conduct domain generalization exper- iments on the colored MNIST dataset (CC BY-SA 3.0) 23]. In particular, the task is to predict binary label Y of colored input images X generated from binary digit feature Za and color feature Ze. We refer readers to Appendix D for more information of this dataset.
We conduct 7 source domains by setting color-label correlation P(Z. = 1|Y 1) = P(Z. Y 0) g8.1 where g Uni ([0.6, 1]) for i 1.... 12, while two target domains are created with TT E {0.05,0, 0.7} for i = 1, 2. Here we note that colored images in the target domain with 8T.2 0.7 are more similar to those in the source domains, while colored images the target domain with TT.1 0.05 are less similar.
We wish to study the characteristics and trade-off between two kinds of DI representations when predicting on various target domains. Specifically, we apply adversarial learning [16] similar to 14. in which a min- max game is played between domain discriminator hd trying to distinguish the source domain given representation, while the feature extractor (generator) g tries to fool the domain discriminator. Simultaneously, a classifier is used to classify label based on the representation. Let fyen and Edise be the label classification and domain discrimination losses, the training objective becomes:
min g
min fien + max Ldise h
hd
where the source compression strength À > 0 controls the compression extent of learned represen- tation. More specifically, general DI representation obtained with À O, while larger À leads to more compressed DI representation. Finally, our implementation is based on DomainBed [17] repository, and all training details as well as further MSDA experiment and DG experiment on real datasets are included in Appendix D
3.1 Trade-off of Two Kinds of Domain-Invarian Representations
Figure 1: Phase transition of representations with increased X. (a) General DI representations. (b)
Just-right compressed DI representations. (c) ceeelyccompeessed DI representations.
'Our code could be found http:///thhub.commiinnAAleessss
8

---

We consider target domain which is similar to source domain in this experiment. To govern the trade- off between two kinds of DI representations, we gradually increase the source compression strength À with noting that À 0 means only focusing on learning general DI representations. According to our theory and as shown in Figure [ learning only general DI representations (A 0) maximizes the cross-domain class divergence by encouraging the classes of source domains more separate arbitrarily, while by increasing A, we enforce compressing the latent representations of source domains together. Therefore, for appropriate X, the class separation from general DI representations and the source compression from compressed DI representations (i.e., just-right compressed DI representations), are balanced as in the case (b) of Figure I
while for overly high X, source compression from compressed DI representations dominates and compromises the class separation from general DI representations (i.c., compreeed DI representations).
Figure 2a shows the source validation and target accuracies when increasing À (i.e., encouraging the source compression). It can be observed that both source validation accuracy and target accuracy have the same pattern: increasing when setting appropriate for just-right compressed representations and compromising when setting overly high values À for overly-compressee DI representations. Figure 2b shows in detail the variation
the source validation accuracy for each specific X. In practice, we should encourage learning two kinds of DI representations simultaneously by finding an appropriate trade-off to balance them for working out just-right compressed DI representations.
3.2
Generalization Capacity on Various Target Domains
0.92
6.40
6.75
0.83
6.72
6:00
Olse Sareet OCENEE
102
6.71
5020
25000 30003
Devosa
(a) Accuracy vs
(b) Accuracy vs iteration
Figure 2: 2ab Source validation accuracy and target accuracy for close target domain (see Section
over compression strength. 2h) Validation accuracy over training step for different values of X.
0.72
0.20
2.73
0.30
1020
2.54
0.00
2000
10000
15383
22020
20000
30300
10000
23000 20000 23003
20000
(a) Far target domain with
(b) Close target domain with 0 0.7.
0.05.
Figure 3: Accuracy of models trained with different À on two test domains. For the far domain with 0 = 0.05, the model corresponding with general DI representations performs better than that with more compact DI representations. For close domain with 0 0.7, the models with compression generalize significantly better than those with general DI representations.
In this experiment, we inspect the generalization capacity of the models trained on the source domains with different compression strengths À to different target domains (i.c., close and far ones). As shown in Figure 3 the target accuracy is lower for the far target domain than for the close one regardless of the source compression strengths X. This observation aligns with our upper-bounds developed in Theorems T and 3 in which larger representation discrepancy increases the upper-bounds of the general target loss, hence more likely hurting the target performance.
Another interesting observation is that for the far target domain, the target accuracy for À = 0 peaks. This can be partly explained as the general DI representations help to increase the span source latent representations for more chance to match far target domain (cf. Section 2.3.2h. Specifically, the source-target discrepancy d2/2 (IP" 9: pgg could be small and the upper bound in Theorem i is lower. In contrast, for the close target domain, the compressed DI representations help to really improve

---

the target performance, while wer-coopressionn degrades the target performance, similar to result on source domains. This because the target domain is naturally close the source domains, the source and target latent representations are already mixed up, but by encouraging the compressed DI representations, we aim to learning more elegant representations for improving target performance.
Related Works
Domain adaptation (DA) has been intensively studied from both theoretical 3 9 53
33 43
and
empirical 44
52
4
7
51
perspectives. Notably, the pioneering work [3] and other theoretical
works 4
52
53
21
30 6
have investigated DA in various aspects which lay foundation for the
success practical works 14
47 44
[22 14, [1]... Multiple-soourc domain adaptation (MSDA)
extends DA by gathering training set from multiple source domains [8] 29, 20 42 43] [22 [54 [50 40)
Different from DA with abundant theoretical works, theoretical study in MSDA is significantly
limited. Notably, the works in [3 32) relies on assumptions about the existence of the same labeling function, which can be used for all source domains; and target domain an unknown mixture of the source domains. They then show that there exists distributional weighted combination of these experts performing well on target domain with loss at most 6, given set of source expert hypotheses (with loss is at most on respective domain). Hoffman al. [8] further develop this idea more general case where different labeling functions corresponding to different source domains, and the target domain is arbitrary. Under this setting. there exists a distributional weighted combination of source experts such that the loss on target domain is bounded by source loss term and a discrepancy term between target domain and source mixture. On the other hand, Zhao et [54] directly extend Ben-david et al.'s work with an improvement on the sample complexity term.
is the most challenging setting among the three due to the unavailability of target data. The studies in 4 10) use kernel method to find feature map which minimizes expected target loss. Moreover, those
Domain generalization (DG) 17 25
27 28
34
49 [T] learn compressed DI [5]
representations by minimizing different types of domain dissimilarity. Other notions of invariance are also proposed, for example, [2] learns a latent space on which representation distributions do not need to be aligned, but share a common optimal hypothesis. Another work in [4] uncovers iamain-invariant hypothesis from low-rank decomposition of domain comain-specific hypotheses. Moreover, there are other works which try to strike balance between two types of representation spaces [5].
5 Conclusion
In this paper, we derive theoretical bounds for target loss MSDA and DG settings which characterize two types of representation: general DI representation for learning invariant classifier which works on all source domains and compressed DI representation motivated from reducing inter-domain representation discrepancy. We further characterize the properties of these two representations, and develop a lower bound on the target loss which governs the trade-off hetween learning them. Finally, conduct experiments Colored MNIST dataset and real dataset illustrate theoretical claims.
10

---

References
[1] AL I., MONTEIRO, J., DARVISHI, M., FALK, T. H., AND MITLIAGKAS, I.
Generalizing to unseen domains via distribution matching, 2020. 2.1
[2] ARJOVSKY, M., BOTTOU, L., GULRAIANI, I., AND LOPEZ-PAZ, D. Invariant risk minimiza- tion, 2020.
2.2
[3] BEN-DAVID S., BLITZER, J., CRAMMER, K., KULESZA, A., PEREIRA, F., AND VAUGHAN,
J. W. A theory of learning from different domains, Mach. Learn. 79, 1-2 (May 2010), 151-175.
2.2 2.2
[4] BLANCHARD. G., LEE, G., AND SCOTT, C. Generalizing from several related classification tasks to new unlabeled sample. In Advances in Neural Information Processing Systems (201 1), vol. 24. 4
[5] CHATTOPADHYAY, BALAJI, AND HOFFMAN, J. Learning to balance specificity and invariance for in and out of domain generalization. arXiv preprint arXiv:2008. 200::0008.18999 (2020).
[6] CORTES, C., AND MOIIRI, M. Domain adaptation in regression. In Algorithmic Learning Theory 22nd International Conference, ALT 2011, Proceedings (2011), Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes Bioinformatics), pp. 903-323. Copyright: Copyright 2011 Elsevier B. B.V., All rights reserved.; Conference date:
22nd International Conference on Algorithmic Learning Theory, ALT 2011
05- 10-201101 Through 07-10-2011,
[7] CORTES, C., MOHRI, M.. AND MEDINA, A. M. Adaptation based on generalized discrepancy. Journal of Machine Learning Research 20, 1 (2019), 1-30.
2.2 4
[8] CRAMMER, K., KEARNS, M., AND WORTMAN, J. Learning from multiple sources. Journal
of Machine Learning Research 9, 57 (2008), 1757-1774.
[9] DAVID, S. B., LU, T., LUU, T., AND PAI., D. Impossibility theorems for domain adaptation. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (13-15 May 2010), vol. 9 of Proceedings of Machine Learning Research, PMLR, pp. 129-136.
4
[10] DESHMUKH, A. A., LEI, Y., SHARMA, S., DOGAN, U., CUTLER, J. W., AND SCOTT, C. A generalization error bound for multi-class domain generalization, 2019.
[11] DOU, Q., DE CASTRO, D. C., KAMNITSAS, K., AND GLOCKER, B. Domain generalization via model-agnostic learning of semantic features. In Advances in Neural Information Processing Systems (2019), pp. 6450-6461.
2.2
[12] DUAN, L., XU, D., AND CHANG, S.-F. Exploiting web images for event recognition in consumer videos: A multiple source domain adaptation approach. In 20/2 IEEE Conference on Computer Vision and Pattern Recognition (2012), pp. 1338-1344 1345.
[13] DUCHI, J. C., KHOSRAVI, K., AND RUAN, F. Multiclass classification, information, diver- gence, and surrogate risk, 2017. 2.3.1
[14] GANIN, Y., USTINOVA, E., AJAKAN, H., GERMAIN, P., LAROCHELLE, H., LAVIOLETTE, F., MARCH, M., AND LEMPITSKY, V. anvarraadversarial training of neural networks. Journal of Machine Learning Research 17, 59 (2016), 1-35.
2.2
[15] GUIFARY, M., BASTIAAN KLEIN,, W. ZIIANG, M., AND BALDUZZI, D. Domain gener- alization for object recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision (2015), pp. 2551-2559,
2.2
[16] GOODFELLOW, I., POUGET- POUGET-ABADIES J., MIRZA, M., XU, B., MAARDE-FARLESS D., OZAIR, S., COURVILLE, A., AND BENGIO, Y. Generative adversarial nets. In Advances in Neural Information Processing Systems (2014), vol. 27, Curran Associates, Inc. 2.2
[17) GUERAIANI, I., AND LOPRZ-PAZ, D. In search of lost domain generalization. abs/2007.01434 (2020). 3
CoRR

---

[18] HOFFMAN, J., MOHRI, M., AND ZHANG, N. Algorithms and theory for multiple-souree adap- tation. In Advances in Neural Information Processing Systems (2018), S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianche and R. Garnett, Eds., vol. 31, Curran Associates,
Inc.
2.1 4
[19] HUANG, Z., WANG, H., XING, E. P., AND HUANG, D. Self-challenging improves cross- domain generalization. Proceedings of the European Conference on Computer Vision (ECCV) (2020).
[20] IEVGEN REDKO, AMAURY HABRARD, M. S. On the analysis of adaptability in multi-source domain adaptation. Mach. Learn. 108 (June 2019), 1635-1652, 4
[21] JOHANSSON, F. D., SONTAG, D., AND RANGANATH, R. Support and invertibility in domain-
invariant representations. In Proceedings of Machine Learning Research (2019), vol. 89, pp. 527-536. 2.1 4
[22] LE, T., NGUYEN, T., Ho, N., BUI, H., AND PHUNG, Lamda: Label matching deep domain adaptation. In ICML (2021). 0
[23] LECUN, Y., CORTES, C., AND BURGES, C. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yanm.lecm wtw.:lccm..cm//uuuinnnisstt 2 (2010). 3
[24] LI, D., YANG, Y., SONG, Y.-Z., AND HOSPEDALES, T. Learning to generalize: Meta-learning for domain generalization.
2.1
[25] LI, D., YANG, Y., SONG, Y.-Z., AND HOSPEDALES, T. M. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference computer vision (2017), pp. 5542-5550.
[26] LI, D... ZHANG, J., YANG, Y., LIU, C., SONG, Y.-Z., AND HOSPEDALES, T. M. Episodic training for domain generalization. In Proceedings of the IEEE/CVF International Conference
on Computer Vision (ICCV) (October 2019). 2.2
[27] LI, H., PAN, J., WANG, S., AND KOT, A. C. Domain generalization with adversarial feature learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2018). 2.1 2.2
[28] LI, Y., TIAN, X., GONG, M., LIU, Y., LIU, T., ZHANG, K., AND TAO, D. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV) (September 2018). 2.1 2.2 4
[29] MANSOUR, Y., MOHRL, M., Ro, J., THEERTHA SURESH, A., AND WU, K. A theory of multiple-source adaptation with limited target labeled data. In Proceedings of The 24th International Conference Artificial Intelligence and Statistics (13-15 Apr 2021), A. Banerjee and K. Fukumizu, Eds., vol. 130 of Proceedings of Machine Learning Research, PMLR,
pp. 2332 2340. 2.1 4
[30] MANSOUR, Y., MOHRI, M., AND ROSTAMIZADEH, A. Domain adaptation: Learning bounds and algorithms. vol. 22.
[31] MANSOUR, Y.. MOHRI, M., AND ROSTAMIZADEH. A. Domain adaptation with multiple sources. In Advances in Neural Information Processing Systems (2009), D. Koller, D. Schuur- mans, Y. Bengio, and L. Bottou, Eds., vol. 21, Curran Associates, Inc
2. 2.2
[32] MANSOUR, Y., MOHRI, M., AND ROSTAMIZADEH, A. Multiple source adaptation and the
renyi divergence. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (Arlington, Virginia, USA, 2009), UAI '09, AUAI Press, p. 367-374.
[33] MANSOUR, Y., AND SCHAIN, M. Robust domain adaptation. Annals of Mathematics and Artificial Intelligence 71 Issue 4 (2014), 365-380,
[34] MUANDET, K., BALDUZZI, D., AND SCHÖLKOPF, B. Domain generalization via invariant feature representation. In Proceedings of the 30th International Conference on Machine Learning (Atlanta, Georgia, USA, 17-19 Jun 2013), S. Dasgupta and D. McAllester, Eds., vol. 28 of Proceedings of Machine Learning Research, PMLR, pp. 10-18. 2.1
12

---

[35] NGUYEN, T., LE. T., ZHAO, H., TRAN, H. Q.. NGUYEN, T., AND PHUNG, D. Most: Multi- source domain adaptation via optimal transport for student-teacher learning. In UAI (2021).
[36] NGUYEN, T., LE, T., ZHAO, H., TRAN, H. Q., NGUYEN, T., AND PHUNG, D. Tidot: A teacher imitation learning approach for domain adaptation with optimal transport. In IJCAI (2021).
[37] NGUYEN, A., NGUYEN, T., LE, T., TRAN, Q. H., AND PHUNG, D. Stem: An approach to multi-source domain adaptation with guarantees. In Proceedings of the IEEE/CVF International Conference on Computer Vision (2021), pp. 9352-9363. [
[38] NGUYEN, X., WAINWRIGHTT M. J., AND JORDAN, M. Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization. In Advances Neural Information Processing Systems (2008), J. Platt, D. Koller, Y. Singer, and S. Roweis, Eds., vol. 20, Curran
Associates, Inc. 2.2
[39] PENG, X., BAI, Q., XIA, X., HCANG, Z., SAENKO, K., AND WANG, B. Moment matching for multi-source domain adaptation. In 2019 IEEE/CVF International Conference Computer
Vision (ICCV) (2019), IEEE Computer Society, pp. 1406-1415. T
[40] PENG, X., BAI, Q.. XIA, X., HCANG, Z., SAENKO, K., AND WANG, B. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (October 2019).
[41] PIRATLA, V., NETRAPALLI, P., AND SARAWAGI, S. Efficient domain generalization via common-specific low-rank decomposition. In Proceedings of the 37th International Conference on Machine Learning (13-18 18-18878 Jul 2020), vol. 119 of Proceedings of Machine Learning Research, PMLR, pp. 7728-7738.
[42] REDKO, I., COURTY, N., FLAMARY, R., AND TUIA, D. Optimal transport for multi-soure domain adaptation under target shift. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statisties (16-18 Apr 2019), K. Chaudhuri and M. Sugiyama, Eds., vol. 89 of Proceedings of Machine Learning Research, PMLR. pp. 849-858.
2.1
2.2
[43] REDKO, I., HABRARD, A., AND SEBBAN, M. Theoretical analysis of domain adaptation with optimal transport, 2017.
[44] SAITO, K., KIM, D., SCLAROFF, S., AND SAENKO, K. Universal domain adaptation through self supervision. In Advances in Neural Information Processing Systems (2020), H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds., vol. 33, Curran Associates, Inc., pp. 16282-16292.
[45] SHALEV-SHWARTZ. S., AND BEN-DAVID, S. Understanding machine learning: From theory algorithms. Cambridge university press, 2014. 2.3.3
[46] SICILIA, A., ZHAO, X., AND HWANG, S. Domain adversarial neural networks for domain generalization: When it works and how to improve, 2021. 2.1
[47] TZENG, E., HOFFMAN, J., SAENKO, K., AND DARRELL, T. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (July 2017).
[48] VAPNIK, V., AND VAPNIK, Statistical Learning Theory. Wiley, 1998. 2.3.3
[49] WANG, H., HE, Z., LIPTON, Z. C., AND XING, E. P. Learning robust representations by projecting superficial statistics out. In International Conference Learning Representations (2019). 2
[50] XU, R., CHEN, Z.,, ZUO, W., YAN, J., AND LIN, L. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2018). 4
13

---

[51] YAN, S., SONG, H., LI, N., ZOU, L., AND REN, L. Improve unsupervised domain adaptation with mixup training, 2020.
[52] ZIANGG, Y., LIU, T., LONG, M., AND JORDAN, M. Bridging theory and algorithm for domain adaptation. In Proceedings of the 36th International Conference Machine Learning (09-15 Jun 2019), K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97 of Proceedings of Machine Learning Research, PMLR, pp. 7404-7413, 14
[53] ZHAO, H., COMBES, R. T. D., ZHANG, K., AND GORDON, G. On learning invariant representations for domain adaptation. In Proceedings of the 36th International Conference on Machine Learning (09-15 Jun 2019), K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97 of Proceedings of Machine Learning Research, PMLR, pp. 7523-7532.
2.4 2.4 4
[54] ZHAO, H., ZHANG, S., WU, G., MOURA, J. M. F., COSTEIRA, J. P., AND GORDON, G. J. Adversarial multiple source domain adaptation. In Advances in Neural Information Processing and
Systems (2018), S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N.
R. Garnett, Eds., vol. 31, Curran Associates, Inc.
2.2
[55] ZIAO,, S., GONG, M., LIU, T., FU, H., AND TAO, D. Domain generalization via entropy regularization. In Advances in Neural Information Processing Systems (2020), H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds., vol. 33, Curran Associates, Inc., pp. 16096-16107.1
[56] Zhao, S., WANG, G., ZHANG, S., GU, Y., LI, Y., SoNG, Z., P., HU, R., CHAI, H., AND KEUTZER, K. Multi-source distilling domain adaptation. Proceedings of the AAAI Conference on Artificial Intelligence 34, 07 (Apr. 2020), 12975-12983.
[57] ZHOU, K., YANG, Y., HOSPEDALES, T., AND XIANG, T. Learning to generate novel domains for domain generalization. In European Conference on Computer Vision (2020), Springer, pp. 561-578. 2.2
14


Reviews:
[{"review_id": "Ejco6farC9", "reviewer": "Reviewer_ssg2", "summary": "The paper provides a rigorous theory for domain generalization (DG) and multi-source domain adaptation (MSDA). It presents two bounds on the target domain risk of some classifier. The first one suggests that obtaining a general domain invariant (DI) representation can be achieved via minimizing a weighted sum of source domains' risks in the DG setting and that the DI representation should also align the target domain's feature marginal distribution with that of the target for MSDA. The second one requires, in addition, an alignment of all the source domains' feature marginal distribution, resulting in compressed DI representations. The characteristics of both types of DI representations are thoroughly theoretically examined with the advantages and drawbacks of each one being discussed: while the first one increases the span of the class-conditionals of the source mixture, the second guarantees a tighter generalization gap for learning on the mixture of sources. The last theoretical aspect addressed is the effect of a class imbalance between the considered mixture of source domains and the target domain, which generalizes previous work on the drawbacks of learning a DI representation.  Finally, empirical evaluation is carried out to support the theoretical findings.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written, easy to follow, and presents important theoretical findings that help to advance the understanding of both domain generalization and multiple source domain adaptation.\n\n## Strengths\n\n* The theoretical results clearly help to understand both the problem of domain generalization and multiple-source domain adaptation.  It rigorously clarifies the properties of each method of obtaining a DI representation in the case of learning with multiple sources. The bounds in terms of the Hellinger distance are novel to the best of my knowledge, and the loss function is not required to verify the triangle inequality, which corresponds better to the classification setting. \n* Theorem 8 is surprising when looking at Theorems 1 and 3: in these latter two theorems, the non-estimable quantity in the MSDA setting is the maximum expected disagreement between a source labeling function and that of the target. It does not depend on the feature map $g$, so one would expect that it is safe to learn a DI representation (indeed, the result analogous negative result from [3] is intuitively motivated by the fact that the ideal joint error term may increase due to mapping the data via $g$). Despite this, Theorem 8 proves the limit of DI representation learning whenever the class marginals change between the sources mixture and the target.\n\n## Weaknesses\n\n* Although [1] figures in the references (reference [50]), it is not cited in the paper, whereas it is a method that should be compared to at least theoretically, as it tackles multiple source domain adaptation after establishing a theoretical bound on the target domain risk. It should also be mentioned in Section 4 \"Related Work\".\n* The bounds suggest an alignment in the sense of the Hellinger distance. However, in practice optimization, the hard constraint $\\mathbb P_g^T = \\mathbb P_g^\\pi$. Using another distance $D$ int the optimization problem after line 136 would be better justifier if a bound $d_{1/2}(\\mathbb P_g^T, \\mathbb P_g^\\pi) \\leq D(\\mathbb P_g^T, \\mathbb P_g^\\pi)$ was given for example.\n\n## Comments\n* The $\\gamma_{i,c}$ term (line 191) was not defined.\n\n\n\n## Suggestions\n* A summary table defining each type of DI representation with its associated constraints and optimization problem would further facilitate understanding the paper.\n*  Indicating that the second term of the RHS of Equation (3) is just a constant to ensure the non-negativity of the divergence would facilitate reading.\n* It would be better to have a number for the optimization problem just after line 137.\n\n\n## Questions\n* Can the infinite capacity assumption of Theorems 5 and 6 be weakened? In particular, can it lead to a weak form of divergence that can be null even with different distributions, but that is useful for classification (as it is the case with the $\\mathcal H \\Delta \\mathcal H-$divergence [2])? \n* Line 189: \"Intuitively, for each domain $\\mathbb D^{S,i}$, the minimal loss $\\min_{\\hat h \\in \\mathcal H}\\mathcal L(\\hat h, h^{S,i}, \\mathbb P_g^i)$ should be inversely proportional to the divergence between the class-conditional $\\mathbb P^{S,i,c}_g$ \". Is it possible to develop more on why this is intuitive (and true)?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations of the work were not addressed. Societal impact also was not as the paper is theoretical.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "C2zjb1U0O46", "reviewer": "Reviewer_EJQZ", "summary": "This paper studies multi-source domain adaptation (MSDA) and domain generalization (DG). The authors first propose two types of invariant representations in MSDA and DG, and then analyze their benefits and drawbacks theoretically. Furthermore, the authors verify the results with experiments.\n\n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "MSDA and DG are both important problems in transfer learning. The paper aims at understanding the role of invariant representations in MSDA and DG theoretically. The authors state their results and implications clearly. \n\nHowever, I have some questions about the proposed theory:\nFirst, although the upper bounds involve multiple source domains and feature learning, it is of the same spirit as [3]: the target accuracy can be bounded with source accuracy, distribution distance and label shift. (e.g. Theorem 1 of [3]). The extension of [3] to multi-source setting has also occurred in previous works. The lower bounds are also simple extensions of [49] (with some triangle inequalities).\n\nSecond, the two kinds of invariant representations seem not complementary. The “compressed” version is another form of the “general” version with stricter constraint. Theorem 3 is a direct corollary of Theorem 1. Therefore, a representation satisfying the conditions of Theorem 3 (the compressed DI) must satisfy those of Theorem 1 (the general DI). \n\nAnother question is the analysis in Section 2.3.3. Since we only have access to finite samples from the source domains, we can only constrain the empirical version of the feature distribution. Thus, Theorem 7 should take into account the gap between the empirical and population versions of distribution distance. Also, if we use dataset S to minimize the loss of \\hat h on S, we should use uniform convergence or similar results to deal with the generalization gap. \n\nPost-rebuttal:\nI appreciate the endeavor that the authors made to address the concerns. The authors further clarified the difference between the proposed upper bound and previous works. Nonetheless, the following concerns are not fully addressed, and I decide to keep my score.\n\n1. I admit that the algorithms of the two kinds of DI representations are different. However, since Theorem 3 is a direct corollary of Theorem 1 by applying Lemma 4 in the appendix, the “compressed” version is another form of the “general” version with stricter constraint, and it is quite well known to tune the constraint carefully in domain adaptation algorithms (such as carefully tune the $\\lambda$ in line 293 of the main paper).\n\n2. As to the choice of distance, I do admit the Hellinger distance can be applied to the case where KL divergence is infinity. However both of them belong to the F-divergence and satisfy the data processing inequality (line 229 of the appendix). It is quite straightforward to extend Zhao et al. 2019 to get the multi-source lower bound, since it's also based on the data processing inequality. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does not describe the limitations and potential social impact. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "4pRNsWRpJSH", "reviewer": "Reviewer_BYyp", "summary": "This paper proposes new theorems for transfer learning, particularly on multi-source domain adaptation and domain generalization. For domain-invariant representations, authors propose two variants: general DIR and compressed DIR. Experiments have shown its effectiveness.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Domain-invariant representation: is domain-invariant representation enough for learning transferrable representations? Reference [49] pointed out that DIR is not enough. Then, can you show the theorems in this paper can generally work in MSDA and DG settings by only considering DIR?\n\nExperiments are only based on MNIST and PACS datasets, which are not enough. Plus, there needs to be comparisons with existing methods, particularly with DANN to show that the theorem actually works.\n\nFinally, the theorems are based on certain assumptions. Can you show the bridge between the assumptions with real datasets? When does these theorem work and when doesn't?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No negative societal impact", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "0u5wZgt5KL", "reviewer": "Reviewer_RQ3Y", "summary": "The paper proposes a theoretical foundation for learning domain invariant representation.\nTheir theory applies to both multi-source domain adaptation (MSDA) (a generalization of the\nclassical domain adaptation) and domain generalization (DG), the setup where the target\ndomain is unknown during training. It proposes two views of the domain invariant representation learning: a *compressed*\nrepresentation where the source domains are regularized to map onto the same embedding, and a *general* where the source\ndomains are not regularized to map onto the same embedding. The paper studies the trade-offs between these two views\nboth theoretically and empirically.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes an interesting perspective to domain invariant representation learning.\nIn general, the contribution of this work is more theoretical in that it bridges methods already\npresent in the literature. This paper also presents the trade-off between the two approaches explored.\nWhile I enjoyed reading the theoretical perspective, the empirical exploration is unconvincing.\n\nBelow, I list some of my concerns for the authors to address.\n\n### Questions/comment\n* You claim that the task of domain invariant representation learning becomes significantly challenging for MSDA. It is not clear to me why that is true.\n* In Figure 2a. The trade-off is not clear. It would be better to have a confidence interval over, e.g. 10 runs to assess that the variation is not due to noise.\n* Similarly for Figure 2b and Figure 3a. there does not seem to be a benefit to using general DI, except in the first 10k iterations.\n* Please specify if you are in the MSDA or DG setting in your experiments.\n\n========\n## Post-rebuttal: \nI thank the authors for taking the time to answer my questions and the other reviewers'. After reading the reviews and the rebuttal, I am confident to raise my score from 5 -> 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not discuss some limitations of their work, nor the societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studied new theories on multi-source domain adaptation (MSDA) and domain generalization (DG), where two types of invariant representations in MSDA and DG were proposed and analyzed. Reviewers believed that the problems are interesting and the current study is worthwhile to the transfer learning community. However, they still hold some important suggestions for improvement: i) The compressed DI is a special case of the general DI, which does not introduce new surprising insight; ii) More importantly, the multi-source lower bound can be directly extended from Zhao et al. 2019, both falling into the F-divergence; iii) The detailed connection between the theories and existing algorithms is only conceptual, but there is no term-to-term correspondence, hence the gap between them may still be large; iv) The empirical evaluation is relatively shallow, and many recent methods such as DRO and CORAL are not compared with.

AC does hold a high preference of further research into the underlying theoretical aspects of domain adaptation and generalization. After reading the paper and comparing it with Zhao et al., I found that this paper is in its current form not essentially different from existing work. Note that I'm very familiar with Zhao et al., and their conclusions are well-known in our literature. In addition, the paper does not introduce novel algorithms but is limited to explain existing algorithms. As a consequence, the gap between theory and algorithms still remains, and our desire for a truly effective DG algorithm is still far away (as pointed out by the DomainBed work) -- even with this paper published. So authors are highly encouraged to improve their work towards this direction.
Decision: Accept (Poster)