Quantifying and Improving Transferability in Domain Generalization
Authors: Guojun Zhang
Han Zhao
Yaoliang Yu
Pascal Poupart
Venue: NeurIPS 2021 Poster (2021)
Keywords: domain generalization
transferability
transfer learning
out-of-distribution generalization

Abstract: Out-of-distribution generalization is one of the key challenges when transferring a model from the lab to the real world.  Existing efforts mostly focus on building invariant features among source and target domains. Based on invariant features, a high-performing classifier on source domains could hopefully behave equally well on a target domain. In other words, we hope the invariant features to be \emph{transferable}. However, in practice, there are no perfectly transferable features, and some algorithms seem to learn ``more transferable'' features than others. How can we understand and quantify such \emph{transferability}? In this paper, we formally define transferability that one can quantify and compute in domain generalization. We point out the difference and connection with common discrepancy measures between domains, such as total variation and Wasserstein distance. We then prove that our transferability can be estimated with enough samples and give a new upper bound for the target error based on our transferability. Empirically, we evaluate the transferability of the feature embeddings learned by existing algorithms for domain generalization. Surprisingly, we find that many algorithms are not quite learning transferable features, although few could still survive. In light of this, we propose a new algorithm for learning transferable features and test it over various benchmark datasets, including RotatedMNIST, PACS, Office-Home and WILDS-FMoW. Experimental results show that the proposed algorithm achieves consistent improvement over many state-of-the-art algorithms, corroborating our theoretical findings.


Quantifying and Improving Transferability in Domain Generalization
Guojun Zhang
Han Zhao
School of Computer Science
Department of Computer Science University of Illinois at Urbana- ArtbanaChhappaiee hanzhao@ill wwlinois.eenn .du edu
University of Waterloo Vector Institute
guojun, www../aaaaaaeerroo..oo
Yaoliang Yu
Pascal Poupart
School of Computer Science University of Waterloo Vector Institute yaoliang, wow.aaggwwyyuuuuaeoooooooo
School of Computer Science University of Waterloo Vector Institute
ppoupart@uwaterloo. ca
Abstract
generalization is one of the key challenges when transferring model from the lab to the real world. Existing efforts mostly focus on building invariant features among source and target domains. Based on invariant features, high-performing classifier on source domains could hopefully behave equally well on target domain. In other words, we hope the invariant features to be transferable. However, in practice, there are no perfectly transferable features, and some algorithms seem to learn "more transferable" features than others. How understand and quantify such transferability? In this paper, we formally define transferability that one can quantify and compute in domain generalization. We point out the difference and connection with common discrepancy measures between domains, such as total variation and Wasserstein distance. We then prove that transferability can be estimated with enough samples and give bound for the target error based on our transferabilltyy Empirically, we evaluate the transferability of the feature embeddings learned by existing algorithms for domain generalization. Surprisingly, we find that many algorithms are not quite learning transferable features, although few could still survive. In light of this, we propose a new algorithm for learning transferable features and test it over various benchmark datasets, including RotatedMNIST, PACS, Office-Home and WILDS-FMoN Experimental results show that the proposed algorithm achieves consistent improvement over many state-of-the-art algorithms, corroborating our theoretical findings
upper
Introduction
One of the cornerstone assumptions underlying the recent success of deep learning models is that the test data should share the same distribution the training data. However, faced with ubiquitous distribution shifts in various real-world applications, such assumption hardly holds in practice. For example, self-driving recognition system trained using data collected in the daytime may continually degrade its performance during nightfall. The system may also encounter weather or traffic conditions new city that never appear in the training In light of these potentially unseen scenarios, is
Code available httpp///s..aaa.e.ooo///////////r
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

of paramount importance that the trained model can generalize Out-Of Distribbbutt (OOD): even if the target domain is not exactly the same as the source domain(s), the learned model should hopefully behave robustly under slight distribution shift.
To this end, one line of works focuses learning the so-called invariant representations 16 57 588. At a colloquial level, the goal here is to learn feature embeddings that lead to indistinguishable feature distributions from different domains. In practice, both the feature embeddings and the domain discriminators are often parametrized by neural networks, leading to an adversarial game between these two. Furthermore, in order to avoid degenerate solutions, the learned features are required to be informative about the output variable as well. This is enforced by placing predictor over the features and minimize the corresponding supervised loss simultaneously [17 32 48 49
Another line of recent works aims to learn features that can induce invariant predictors, first termed as the invariant risk minimization (IRM) 3 39 paradigm. Roughly speaking, the goal of IRM is to discover a feature embedding, upon which the optimal predictors, i.e., the Bayes predictor, are invariant across the training domains. Again, the same time, the features should be informative about the output variable as well. However, the optimization problem of IRM is rather difficult, and several follow-up works have proposed different relaxations the original formulation 26]
Despite being extensively studied, both theoretical [4 59 and empirical 20 23 works have shown the insufficiency of existing algorithms for domain generalization (DG). Methods based on invariant
features ignore the potential shift in the marginal label distributions across domains (59] and the methods based invariant predictors empirical works have shown that with proper data augmentation and careful model tuning, the very basic algorithm of empirical risk minimization (ERM) demonstrates superior performance
not robust to covariate shift (66.. Perhaps surprisingly,
domain
generalization over existing methods on benchmark image datasets (20, (23). This sharp gap between theory and practice calls for a fundamental understanding of the following question:
What kind of invariance should we look in order to ensure that a good model on source domains also achieves decent accuracy on related target domain?
In this work we attempt to answer the above question by proposing a criterion for models to look at, dubbed as transferabillity which asks for an invariance of the excess risks of a predictor across domains. Different from existing proposals of invariant features and invariant predictors, which seek to find feature embeddings that respectively induce invariant marginal and conditional distributions, our notion of transferability depends on the excess risk, hence it directly takes into account the joint distribution over both the features and the labels. We show how it can be used naturally derive a new upper bound for the target error, and then we discuss how to estimate the transfcrability empirically with enough samples. Our definition also inspires a method that aims to find more transferable features via representation learning using adversarial training.
Empirically, we perform experiments to measure the transferability of several existing algorithms, on both small and large scale datasets. We show that many algorithms. including ERM, are not quite transferable under the definition (Fig. D see more details in $5 when we go away from the optimal classifier (with distance in the parameter space), it could happen that the source accuracy remains high but the target ac- curacy drops significantly. This implies that during the training process, an existing algorithm may find good source classifier with low target accuracy, hence violating the requirement for invariance of excess risks. In contrast, our algorithm is more transferable, and achieves consistent improvement over existing state-of-the-art algorithms, corroborating our findings.
1.0
accuracy
0.9
0.8
0.7
-0-
source
0
5
Figure 1: The target and source (test) accuracies of ERM on MNIST
2 What is Transferability?
In this section we present our definition of transferability in the classification setting, The setup of domain generalization is the following:
Settings and Notation Given n labeled source domains Si, ... S.. the problem of domain gen- eralization is to learn model from these source domains, in the hope that it performs well on an

---

unseen target domain T that is "similar" to the source domains. Throughout the paper, we assume that both the source domains and the unseen target domain share the same imput and output spaces.
denoted as X' and V, respectively. For multi-class classification, the output space J [K] is a set of labels for multi-class classification. For binary classification, we consider y = {-1, +1]. Denote H as the hypothesis class. We define the classification error of classifier h E H on domain 'D (or S for source domains, or T for target domains) as F
eD(h) (( [(((())))) y)].
(1)
For ((h(2),, y) - (1(())) A v), where 1()) is the usual indicator function, we use eh- (h) to denote it is the 0-1 loss.
In domain generalization, we often have several source domains. For the ease of presentation, we only consider single source domain in this section, and later extend to the general case in Section 5 Given two domains, the source domain S and the target domain T. the task of domain generalization is transfer a classifier h that performs well on S to T. We ask: how much of the success of h on S can be transferred to T?
Note that in order to evaluate the transferability from S to T, we need information from the target domain, similar to the test phase in traditional supervised learning. We believe a good criterion of transferabilityy should satisfy the following properties:
1. Quantifiable: the notion should be quantifiable and can be computed in practice;
2. Any near-optimal source classifier should be near-optimal on the target domain.
3. If the two domains are similar, as measured by e.g., total variation, then they are transferable to each other, but the converse may not be true.
At first glance the second criterion above might seem too strong and restrictive. However, we argue that in the task of domain generalization, we only have labeled source data and there is no clue to distinguish classifier from another if both of them perform equally well on the source domain. Based on the second property, we first propose the following definition of transferability:
Definition (transferability). S is (ōs, 8T transffffere to TT ffr óg > 0, there exists 8y such that argmin(es, 6s)n C argminn (T, δT)n. where:
argmin(ed, 5p) == {h E H ep(h) S inf eD(h) + +6p.
NEH
In the literature the set argmin(ep. 8p)H is also known as a 8D 5p-minimal set 124 of eD. which represents the near-optimal set of classifiers. Note that the 5-minimal set depends on the hypothesis class H. Throughout the paper, we omit the subscript H in the definition when there is no confusion. Def. [ says that near-optimal source classifiers are also near-optimal target classifiers. Furthermore, it is easy to verify that our transferability is transitive: S is (ōs, 8p) )tanssfsseee to P, and P is (dp, )transsffee to T, then S is (Ōs, 87 ttransfbrree to T.
Next we define transfer measures, which we will show to be equivalent with Def. D in Prop 5
Definition 2 (quantifiable transfer measures). Given some T C H. Cs = inf <s (h) and Gj ker (h) we define the one-sided transfer measure, symmetric transfer measure and the realizable transfer measure respectively as:
Tr(SIIT) == sup sup (h) E - (es(h) - -ES),
(2)
Tr(S,T) :=
TT(T) sup les(h) - - (ET(() - -e)], heT
(3)
(S, := sup sup [es(() -T (h)))
(4)
The distinction between I' and H. will become apparent in Prop. 5 Note that the one-sided transfer measure is not symmetric. If we want the two domains S and T to be mutually transferable to each other, we can use the symmetric transfer measure. We call both quantities as transfer measures. Furthermore, the symmetric transfer measure reduces to (4) in the realizable case when < C 0. In statistical learning theory, (h) cp is often known as an excess risk (24), which is the relative error compared to the optimal classifier. The transfer measures can thus be represented with the difference of excess risks. With Def. 2 we can immediately obtain the following result that upper bounds the target error:
Throughout the paper, we will use the terms domain and distribution interchangeably.
3

---

Proposition 3 (target error bound). Given I' _ for any h E T, the target error is bounded by: ey (h) S (h) + + Es Tr (S S cs(h) + G cs + (r(S,T).
(5)
The first error bound of such type for a target domain uses H-divergence 8 9 for binary classification (or more rigorously, the H/AH-divergence) The main difference between ours and H-diverreence is that H-divergence only concerns about the marginal input distributions, whereas the transfer measures depend on the joint distributions over both the inputs and the labels. We note that Proposition 3 is general and works in the multi-class case as well. Moreover, even in the binary classification case Proposition 27 in the appendix).
prove that Proposition is tighter than H-divergence (see
In practice may not know the optimal errors. In this case, we can use the realizable transfer measure to upper bound the symmetric transfer measure (note that es or ef may not be zero):
Proposition 4. For I H and domains S, T we have; Tr(S, T) (S, T).
Since Def. essentially asks that the excess risks of approximately optimal classifiers on the source domain are comparable between the source and target domains, we can show that Def. and Def. 2 are equivalent if I' is d-minimal set:
Proposition 5 (equivalence between transferability and transfer measures). Let ós > 0 and == argmin(es, ås) and suppose infner ET (h) infnem ET (h). If (fTT(T)T) S ó Tr (S, then S is (ōg, ó + ôs) -ra-tfsasfee to T. Furthermore, if S is (Ós, ) to T, then (S( S 5y and Tr(S, T) S max(ōg,
In Prop. we do not require = H. since it is unnecessary to impose that all classifiers in H have similar excess risks source and target domains. Instead, we only constrain I be 6-minimal set, ie.. I includes approximately optimal classifiers of S. See also Example [ An additional assumption that I also includes the optimal classifier of 'T which can be ensured by controlling 8s.
2.1 Comparison with other discrepancy measures between domains
In this subsection, we compare the realizable transfer measure 44 with other discrepancy measures between domains and focus the 0-1 loss c-1, We first note that (S, T) can be written as an integral probability metric (IPM) 33 46 The l.h.s. of (4) can be written as:
TS T) = d (S, where d= ()) = sup fEF
Σ (Zz,))(s(a.)) v) pT pT v))dx (6)
and Fr: ( y) 1((x)) # ET: Typical IPMs [46] include MMD, Wasserstein distance, Dudley metric and the Kolmogorov-Smirnv distance (see Appendix B.2 for more details). However, Fr is fundamentally different from these IPMs since relies on an underlying function class F. Our realizable transfer measure shares some similarity with Arora et al. (4, where changeable function class is used, but the exact choices of the function class are different.
Even though the transfcrability can be written in terms of IPM, it is in fact pseudo-metric:
Proposition 6 (pseudo-metric), For a general loss ep as in [ IF T) a pseudo-metric i.e. for any distributions S, TT on the same underlying space, we have (T S) = 0, (S, = (T, (symmetry), and TT (S, T) I (S, P) + (P, T) (triangle inequality).
However in general (S T) is not a metric since (T. T) = 0 even if S + T. For instance, taking I {/²} to be the optimal classifier on both S and T. we have (S, T) = 0, but S and T could differ a lot (see Figure 2). In the next result we discuss the connection between realizable transfer measures and total variation (c.f. Appendix (B22).
Proposition 7 (equivalence with total variation). For binary classification with labels {-1, 1}. given the 0-1 loss éD 6D 0-1 we have (F, S drv(S, (SS,T) T) for domains S, T and any I C H. Denote H₂ be the set of all binary classifiers. Then we have dTy(S,T) < (S,
Prop. 7 tells us that transfer measures (see also Prop. 4 are no stronger than total variation, and the
realizable case, 3 is equivalent to the similarity of domains (as measured by total variation) if I is unconstrained. We can moreover show that transfer measures are strictly weaker, if we choose I to be some 5-minimal set:

---

Example 8 (very dissimilar joint distributions but transferable). We study the distributions de-
scribed in Figure? 2 The joint distributions are very dissimilar, i.e., for any X, Y in the domain, Ips (os(XY) PT Y)I 0.8. Define
{-1 1-1
if - ipss<<1
h (x
(7)
We choose the hypothesis class H (hpp [--1,1]} and I (hp: |p| 6/0.8) (for small 6, say 5 0.01) to be some neighborhood of the optimal source classifier h" ho- Then (r
supter (es(h) e - 6, and S is (ōg, 5 + 65)) ( transsfraaee to T on I for any ós > O according
Prop. Note that e'S
3 Computing Transferability
1
In the last section we proposed a new con- cept called transferability. However, although providds a theoretically sound result for transferability, it hard to verify it in practice, since we cannot exhaust all approximately good classifiers, especially for rich models such as deep neural networks. Nevertheless, Prop. 3 compute
target domain
1
y
y= 1
y ==-1
and Prop. []] provide a framework transferability through transfer measures, de- spite their simplicity. In this section we dis- cuss how to compute these quantities by mak- ing necessary approximations based on transfer measures. There are two difficulties we need to overcome: (1) In practice we only have fi- nite samples drawn from true distributions; (2) We need a surrogate loss such as cross entropy for training and the 0-1 loss for evaluation. In [3.1 we show that our transfer measures can be estimated with enough samples, and in $3.2 we discuss transferability with surrogate loss.
source domain
Figure 2: Visualization of Example
g Source
domain: Ps(Y
1,-1
I
X
0)
0.1.
Ps(Y = -1.0 < X
< 1) = 0.9.
Target
domain: Pr(Y 1.-1
X
< 0)
0.9,
Pr(Y -1,0 < x
< 1J
= 0.1. The dark
and light colors show the intensity the probabil-
ity mass. The vertical axis denotes whether it is the target or source domain (above or below ar-isis..
These results will be used in our algorithms in the next section.
3.1 Estimation of transferability
We show how to estimate the transfer measure (r(T)) from finite samples. Other versions of transfer measures in Def. follow analogously (see Appendix for more details).
Lemma 9 (reduction of estimation error). Given general loss cp in (I. suppose S and Ô are i..dd sample distributions drawn from distributions of and T, then for any I S H we have:
(r(S S (T()) ( + r
with the estimation errors cS) = SUPNET (cs(h) - (g))), e = supNer (h) (0)0)-c7(()) (h))-
This lemma tells that estimating transferability is harder than computing the estimation errors of both domains. If the function class has uniform convergence property [44]. then we can guarantee efficient estimation of transferability. We first bound the sample complexity through Rademacher complexity, which is a standard tool in bounding estimation errors [
Theorem (estimation error with Rademacher complexity). Given the 0-1 loss eD = 6D suppose S and i are sample sets with and k samples drawn i...d. from distributions S and T. respectively. For any P C H the following holds with probability
(gg(4/))
log 44/)
Tr(SIIT) Tr ( + 49hm (Fr) + 4Fr + +2
+ 2V
2m
2k
where := = ((x,)) - 1 (1((z) + y), T.. furthermore T is a set of binary classifiers with
labels {-1, 13, then 29 (Jr) Ri (T), (Fr) = (T).
5

---

We also provide estimation error results using Vapnik- Chervonenkis (VC) dimension and Natarajan dimension in Appendix [B.3] It worth mentioning that the VC dimension of neural networks has been upper bounded in Bartlett et al. [7]. Since transfer measures can be estimated, in later sections we do not distinguish the sample sets S, T and the underlying distributions S, T.
3.2 Transferability with a surrogate loss
Due to the intractability of minimizing the 0-1 loss, we need to use a surrogate loss [6] for training in practice. In this section, we discuss this nuance w.r.t. transferability. We will focus on the most commonly used surrogate loss, cross entropy (CE), although some of the results can be easily adapted to other loss functions. To distinguish a surrogate loss from the 0-1 loss, we use cp from now on for a surrogate loss and 0-1 for the 0-1 loss. One of the difficulties is the non-equivalence between 6-minimal sets w.r.t. the 0-1 loss and a surrogate loss, i.e. argmin(eD: oD) might be quite different from orgmin(e ; .D). Moreover, it is not practical to find all elements in argmin(e ( since the loss is nonconvex and nonsmooth. In light of these difficulties, we propose a more practical notion of transferability based on surrogate loss ep:
Proposition 11 (transfer measure with surrogate loss). Given surrogate loss cD CD on a gen- eral domain D. Suppose I = argmin(es, 6s) and denote ey- = infner ET (h), ess = infrer Es(),, 0-1((h) () = infNEH ET (h). If the following holds:
0-1
(S|TT = sup eT (h) G7 - (es(h) -t3) S 5,
(8)
heT
then we have arginin(es: 6s) C argmin(e 0-1 ,
5 + ōs + E
This proposition implies that the transfer measure is small, then a near-optimal classifier of the surrogate loss in the source domain would be near-optimal in the target domain for the 0-1 1 loss. It also gives us a practical framework to guarantee transferability, which we will discuss in more depth in Section 4 Assume eD H - IR to be Lipschitz continuous and strongly convex, which is satisfied for the cross entropy loss (see Appendix B.4p. We are able to translate the 6-minimal set to Tup balls in the function space:
G h 112. S (D()) < C2llh -2 111,,
(9)
where C1 and C2 are absolute constants and h* is an optimal classifier. The function norms || 11.. and || lla.D are the usual L, norms over distribution D. Since the classifier h = q(0, -) is usually parameterized with, say a neural network, we further upper bound the function norms by the distance of parameters, that is, for S < co, h x(0,-) -) and h' 2(0',,). )) we have || h Ipp. < L| - 0' 0'112, with L some Lipschitz constant of q (Appendix B.4. Combined with we obtain:
(b) - eD (h') < LC2 110 - 01122-
(10)
In other words, if the parameters are close enough, then the losses should not differ too much. We denote || ||2 as the Euclidean norm, and for later convenience we will omit the subscript in || l|2.
Algorithms for Evaluating and Improving Transferability
The notion of transferability is defined w.r.t. domains, hence by learning feature embeddings that induce certain feature distributions, one can aim to improve transferability of two given domains. In this section we design algorithms to evaluate and improve transferability by learning such transforma- tions. To start with, let g : X + Z be a feature embedding (a.k.a. featurizer), where Z is understood to be feature space. By joint distribution DD (or $ T9) we mean distribution on g(X') x ). Formally, we are dealing with p of distributions:
S = (g. iS T = (T,
(1))
where (g,id) id): (x, (g(x), 2) is a function on X' y. S and T here are joint distributions on X' x ), and here we specify X' to be the space of the original signal such as an image. Since S and T cannot be changed, what we evaluating here is the feature embedding g. The key quantity is transfer measures as in (B)
(T9) = sup ET (h) - - (ESs (h) -G60); == argmin(ess,, 6s=)
(12)
ke

---

Although I' is hard to compute, we can use (TO) to obtain a lower bound of (12 That is, given
parametrization of the classifier h q(O, and the optimal classifier h" q(6", we have:
Tr(59179) > sup (h) - cga (h) <To + c5.
|0 00-00 0- 15
sup
eTo (h) cs=(h) cT: (h*)
1--8" |56
sup
CT (h) cg. (h) cT (h*)
(13)
10-0
where 5 > depends I and the constant in In the second and the third lines, approximated the optimal errors T and Ess with 0 S Ess S e5o (h*), 0 S eto S eT (hr), and we use the learned
classifier h* g(0",,) as a surrogate for the optimal classifier. As result, if the r.h.s. of (13 is large, then S is not quite transferable to 7
Algorithm 1: Algorithm for evaluating transferabilityy among multiple domains
Input: learned feature embedding g. learned classifier h- q(0 -), target sample training set T = So, sample training sets St. .... Sn, ascent optimizer, minimal errors es. Rd es, (h*), adversarial radius 8 Initialize: a classifier h = g(0,) -) and 0-, gap = -0 for t in 1.... T do
Find max; es, (h o) and min; cs. (h o g) and corresponding indices j and it Run an ascent optimizer on h to maximize gapo es, (h g) cs, (h o g)
Project 0 onto the Euclidean ball ||0 0-1 < 5 if gapa gap then
gap gäpp. save accuracies and losses of each domain
Output: j. h, cs, o g) cs. (h o g), cs, (h*), cs. (h-)
We can thus design algorithm to evaluate the transferability in Section 4.1 By computing the lower bound in (13). we can disprove the transferability as in Prop. 5 5|nnd Prop. I Computing the lower bound in 13) can be regarded as an attack method: there is an adversary trying to show that 8 is not transferable to T For this attack, we could also design a defence method aiming to minimize the lower bound and learn more transferable features.
4.1 Algorithm for evaluating transferability
In domain generalization we have target domain and more than one source domains. To ease the presentation, we denote Sn = T (and thus Sy = T) and extend the index set to be (0. 1. ... n). We need to evaluate the transferability (13 between all pairs of S and Sy. Algorithm I gives an efficient method to compute the worst-case gap sup 11 Esş (h) Esg (h) among all pairs of (i,j). Essentially, it finds the worst pair of (i at each step such that the gap Es! (h) Esy (h) takes the largest value, and then maximize this gap over parameter 0 through gradient ascent.
Note that the computation of (13) also depends on the information from the target domain. This is valid since we are only evaluating but not training over these domains.
4.2 Algorithm for improving transferability
The evaluation sub- boob-rrcceure provides us a way to pick pair of non-transferable domains (S,, S?), which turn could be used to improve the transferability among all source domains by updating the feature embedding such that the gap supiee- <sş (h) t for (i.j) E [M] x [n].
Simultaneously, we also require that the feature embedding g preserves information for the target task of interest. With the parametrization h = (0,,,), h = =(00, )) the overall optimization problem can be formulated as:
min max
Σ Cs (h + (max,es, (h' o) - min,cs, (h'og))
(14)
0 56 tml
7

---

Intuitively, we want to learn a common feature embedding and a classifier such that all source errors are small and the pairwise transferability between source domains also small. If the optimization problem is properly solved, then we have the following guarantee:
Theorem 12 (optimization guarantee), Assume that the function q(-, x) is La Lipschitz continuous for any I. Suppose we have learned a feature embedding 9 and classifier h such that the loss functional cs H + IR. is Lg Lipschitz continuous w.r.t. distribution S for E [n] and
1
n
max 00-011 S n2
ss (h)g) + (max,es, (N' og) mi,,es, e, (h's (h') S n.
(15)
where 0,0 0' are parameters of l and ht. Then for any H ET (a(0',-) : 10 0'1 S 6}, we haver
TF (T") S fs, (V' o g) n + LeLad, 6T(H' g) s 2n + L1
(16)
for any TP, T T9 conv(S)....... S.) and any i [n].
The Lipschitzness assumption for Eg. is mild and can be satisfied for cross entropy loss (c.f. Ap- pendix B.4.I). Here conv(.) denotes the convex hull in the same sense Albuquerque et al. [2]. i... each element is a mixture of source distributions. Thm [2teels us that we can solve the optimization problem (14) properly, we can guarantee transferability on a neighborhood of the classifier, as an approximation of the d-minimal set. We thus propose Algorithm 2 which shares similarity with exist- ing frameworks, such DANN [16] and Distributional Robust Optimization
43 451 in the sense
that they all involve adversarial training and minimax optimization. However, the objective in our case is different and provide more detailed comparison with existing methods in Appendix B.5
Algorithm 2: Transfer algorithm for domain generalization
Input: samples sets of source domains S Sn,, feature embedding g. classifier = q(0, +), adversarial classifier h' = q(0',, +), surrogate loss éD. adversarial radius ö, ascent optimizer, descent optimizer, weight parameter X, number of epochs T for t in 1...T do
Compute max; Cs. (h o g) and min, cs. g) Initialization h (or 0' 0)
for k in ........... do
Run the ascent optimizer on H maximize max; es, (h' o.g) min; es, (h' o g) fixing g
Project 0' onto the Euclidean ball 110' - 0ll < 5
Fixing h', run the descent optimizer on g, minimize
error = + >. Es, (h g) + (max; es, (h' g) min; es, (h' o og))
Output: feature embedding g. classifier h
5 Experiments
Gulrajani and Lopez-Paz 20 did extensive experiments on comparing DG algorithms, using the same neural architecture and data split. Specifically, they show that with data augmentation, ERM perform relatively well among a large array of algorithms. Our experiments are based on their settings. We run Algorithm I on standard benchmarks, including RotatedMNIST [18), PACS 28) Office-Home 5I and WILDS- with nearly half million images. Detailed experimental settings can be seen Appendix C
(23 (c.f. Appendix C.I.. Specifically, WILDS-FMo W is a large dataset
Evaluating transferability From Figure it can be seen that at neighborhood of the learned classifier, there exists a classifier such that the target accuracy is degraded significantly, whereas some source domain still has high accuracy. This poses questions to whether current popular algorithms such as ERM (50), DANN [17] and Mixup [53] 54 are really learning invariant and transferable features. If so, the target accuracy should be high given a high source accuracy. However, for the PACS dataset and Mixup model (the second column of Figure the target accuracy decreases by more than 30% while the source accuracy remains roughly at the same level. We can also, e.g., read from the first column that with a small decrease of the source (test) accuracy by ~ 2% (at 8 = 2), the target accuracy of DANN drops by 10%.
8

---

0.5
0.9
0.8
0.60
target accuracy
Transfer
0.50
Transfer EONAL
Tassfer
0.40
0.40
0.35
0.0
0.1
0.2
0.3
0.00
0.05
0.10
0.15
0.00
0.05
011
0.15
RotatedMNIST
PACS
OfficeHome
WILDSFMoW
Source accuracy
0.98
0.95
0.96
0.85
0.5
6
6
Figure Top row: test accuracy of the target domain; bottom row: test accuracy of one of the source domains. Each column is for a given dataset with the name in the middle, and the legends on the bottom row are the same as those on the top row. 5 the parameter in Algorithm
From Figure 3 we can also see that Correlation Alignment [CORAL, 47 and Spectral Decomposition (SD, 40] have better transferability that other algorithms. In some sense, they are fact learning robust classifiers, i.e., all the classifiers on the neighborhood of the learned classifier can achieve good accuracies. With this robust classifier, the target accuracy does not decrease much even the classifier is perturbed.
Improving transferability Algorithm 2 has good performance among all four datasets that tried, comparable to CORAL and SD. Note that CORAL and SD do not always perform well, such in the Office-Home and WILDS-FMot datasets, but Transfer algorithm does. However, in our experiments we find there are two limitations of Algorithm 2 (1) we need a large number of training time. This is similar adversarial robustness [33] which is slower than usual training. In order to overcome this difficulty we used pretraining from other algorithms in the experiments
inner maximization steps compute the gap, which needs
Office- Home and WILDS-FMoW: (2) Moderate hyper-parameter tuning is needed. For example, we need to tune N is Algorithm 2 the learning rate (1r) SGA and the choice of We find that taking N 20 or 30 usually good choice, and 5 can be quite large such that the projection step is not taken. We take Ir 0.01 for RotatedMNIST and 0.001 for other datasets.
Label shift In order to show the difference with the well-known H-divergence [ we compute label shifts in the PACS dataset. As shown in Zhao et al. [59]. the optimal joint error (h") + es (h") (h* is the optimal classifier that minimizes es (h) + eT (h)) can be large under the shift of label distributions. We follow [59] and compute the label shift between pairs of domains in the PACS dataset, measured by total variation. From Table we can see that the label shift is large in this case, and thus the H-divergence bound [ can be quite loose. Comparably, transfer measure bound Prop. is tighter (c.f. Prop. 27) and therefore still useful in practice.
Table 1: Label shift between pairs of domains in the PACS dataset. TV: total variation; A: art painting; C: cartoon; photo; S: sketch. The total variation is always between zero and one.
TV
A
C
P
S
A
0.0
0.12
0.11
0.3
0.12
0.0
0.18
0.24
0.11
0.18
0.0
0.37
S
0.3
0.24
0.37
0.0

---

6 Related Work
Multi-task learning Multi-task learning (MTL) [56] is related to but different from DG. In MTL, there are several tasks, and one hopes to improve the performance of each task by jointly training all the tasks simultaneously, utilizing the relationships between them. This is different from DG in the sense that in DG the target domain is unknown a priori, whereas in MTL the focus is more on better generalization on existing tasks that appear in training. Hence, there is no distribution shift in MTI. per se. Furthermore, for MTL, the output spaces of different tasks are not necessarily the same.
Zero-shot learning Few-shot learning Meta-learning DG is different from zero-shot learning [27]. In zero-shot learning, one has labeled training data and the goal is to make predictions on a new unseen label set. However, in DG the label set remains the same for the source and the target domains. On the other hand, the focus of few-shot learning is on fast adaptation, in the sense that the test distribution remains the same as the training distribution, but the learner can only have access to a few labeled samples. Domain generalization also shares similarity with meta-learning. However, in meta-learning, the learner allowed to fine-tune over the target domain. In other words, the protocol of meta-learning allows access to a small amount of labeled data from future unseen domains. Meta-learning is more or less one specific method that is used to tackle few-shot learning. Because of the similarity, some meta-learning algorithms can be applied to DG [29].
Self-supervised learning Self-supervised learning (SSL) is popular unsupervised feature learning approach [3] (9]211] The goal of SSL is to learn invariant representations w.r.t. different views of the same image. Although it is a promising feature learning method, it differs from our DG settings in the sense that no labels are used in SSL.
Domain generalization There have been lot of old and new algorithms proposed for domain generalization. The simplest one is Empirical Risk Minimization (ERM), where we simply minimize the empirical risk of (the sum of) all source domains. In Blanchard et al. [10], Muandet et al. (35), kernel methods for DG were proposed. Arjovsky et al. 3 proposed Invariant Risk Minimization (IRM) which aims to learn invariant predictors across source domains, and follow-up discussions can be found in 25 41. Another approach is called distributional robustness 43 [2], where the model optimized over a worst-case distribution under the constraint that this distribution generated from a small perturbation around the source distributions. In Albuquerque et al. (2), DG scheme based on distribution matching was proposed. Moreover, many domain adaptation algorithms can be directly adapted to the task of domain generalization, such as CORAL [47 and DANN [16). Last but not least, we mention a concurrent work (55) on the theory of domain generalization, which focuses more on proposing model selection rule based on accuracy and variation.
Adversarial robustness Our evaluation and training methods in $4 are reminiscent of the adversarial training method (33) in the literature of adversarial robustness. Perturbing the classifier in our case corresponds perturbing the input data in adversarial robustness. From this perspective, our Transfer algorithm is parallel to the adversarial training method. It would be interesting to design certified robust feature embeddings, by analogy with certified robust classifiers [5].
7 Conclusions
In this paper we formally define the notion of transferability that we can quantify, estimate and compute. Our transfer measures can be understood as a special class of IPMs. They are weaker than total variation and even very dissimilar distributions could be transferable to each other. Our definition of transferability can also be naturally used to derive a generalization bound for prediction error on the target domain. Based on theory, we propose algorithms evaluate and improve the transferability by learning feature representations. Experiments show that, somewhat hat surprisingly. many existing algorithms are not quite learning transferable features. From this perspective, our transfer measures offer a novel way to evaluate the features learned from different DG algorithms. We hope that our proposal of transferability could draw the community's 's attention to further investigate and better understand the fundamental quantity that allows robust models under distribution shifts.
Broader Impact Reliable domain generalization models are important for practice use. Our work points out the reliability issue of DG algorithms. It is worth mentioning that our evaluation method can only disprove the transfcrability and survival of our attack method should not be treated warranty. Misunderstanding of it could lead to potential harm in practical applications.
10

---

Acknowledgements and Funding Transparency Statement
We thank the anonymous reviewers for their constructive comments as well as the area chair and the senior area chair for overseeing the review process. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute. We thank NSERC and the Canada CIFAR AI Chairs program for funding support. GZ is also supported by David R. Cheriton scholarship and research grant from Vector Institute. HZ supported by startup funding from the Department of Computer Science at UTUC. Finally, we thank Vector Institute for providing the GPU cluster.
References
[1] Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk minimization games. In International Conference on Machine Learning, pages 145-155. PMLR, 2020.
[2] Isabela Albuquerque, João Monteiro, Mohammad Darvishi, Tiago H Falk, and Ioannis Mitliagkas. Generalizing to unseen domains via distribution matching. arXiv preprint arXiv:1911.00804, 2019.
[3] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- mization. arXiv preprint arXiv: 19Xix:990..0220 190700299,, 2019.
[4] Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (GANs). International Conference on Machine Learning, pages 224-232. PMLR, 2017.
[5] Peter I. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-482, 2002.
[6] Peter L Bartlett, Michael I Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138-156 2006.
[7] Peter L Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight VC- dimension and pseudodimension bounds for piecewise linear neural networks. J. Mach. Learn. Res,, 20(63):1-17, 2019.
[8] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In Advances neural information processing systems, pages 137-144, 2007.
[9] Shai Ben- David, John Blitzer, Koby Crammer, Alex Kulesza. Fernando Pereira, and Jen- nifer Wortman Vaughan, A theory of learning from different domains. Machine learning. 79 (1-2):151-175, 2010.
[10] Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classifica- tion tasks to new unlabeled sample. Advances in neural information processing systems, 24: 2178-2186, 2011.
[11] Gilles Blanchard, Aniket Anand Deshmukh, Urun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. Journal of Machine Learning Research, 22(2):1-55, 2021.
[12] John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman. Learning bounds for domain adaptation. In Advances in neural information processing systems, 2007.
[13] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, pages 1597-1607, 2020. URI.Bttp::/prrceeddnng mlr press/vv19/ http9//cccnnppp// html

---

[14] Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 6172-6180, 2018.
[15] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning, pages 1310-1320. PMLR, 2019.
[16] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pages 1 1180-1189. PMLR, 2015.
[17] Yaroslav Ganin, Evgeniya Ustinova, Hama Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversaria training of neural networks. The Journal of Machine Learning Reseurch, 17(1):209--0030 2016.
[18] Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain gen- eralization for object recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision, pages 2551-25592 2015.
[19] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Moham- mad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent a new approach to self-supervised learning. In NeurlPS, 2020. URL /ttpss://rooceddiin wtwpoceedinnnsnnuris.....o newrins. wttpc/cpppeee2////eeee Abstract. html
[20] Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
[21] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020.
[22] Zeyi Huang, Haohan Wang, Eric P. Xing, and Dong Huang. Self-challenging improves cross- domain generalization. In ECCV, 2020.
[23] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie. Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. WILDS: A benehmark of in-the-wild distribution shifts. arXiv preprint 072441 2020.
[24] Vladimir Koltchinskii. Rademacher complexities and bounding the excess risk in active learning. The Journal of Machine Learning Research, 11 11:2457-2485, 2010.
[25] Masanori Koyama and Shoichiro Yamaguchi. When is invariance useful in an out-of-distribution generalization problem?, 2021.
[26) David Krueger, Ethan Caballero, Joem--enrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribuiie generalization via risk extrapo- lation (Rex). arXiv preprint arXiv:2003. 00688, 2020.
[27] Christoph Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-clate attribute transfer. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 951-958. IEEE, 2009.
[28] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pages 5542-5550, 2017.
[29] Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta- learning for domain generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[30] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 5400-5409, 2018
12

---

[31] Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference Computer Vision (ECCV), pages 624-639, 2018.
[32] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In International conference on machine learning, pages 2208-2217. PMLR, 2017.
[33] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
[34] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.
[35] Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference Machine Learning, pages 10-18. PMLR, 2013.
[36] Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, pages 429-443, 1997.
[37] Hyconseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap via style-agnostic networks. arXiv preprint arXiv:191011164, 2019.
[38] Balas K Natarajan. On learning sets and functions. Machine Learning, 4(1):67-97, 1989.
[39] Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction:
identification and confidence intervals. Journal of the Royal Statistical Society. Series B (Staristical Methodology), pages 947-1012, 2016.
[40] Mohammad Pezeshki, tékou-Oumat Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks. arXiv preprint MXX::2011.09460 2020.
[41] Elan Rosenfeld, Pradeep Kumar Ravikumar, and Andrej Risteski. The risks of invariant risk minimization. In International Conference on Learning Representations, 2021. URL
https:///peereeiee. seeett/frrmiiddBDDTTTt
[42] Walter Rudin. Real and complex analysis. McGraw-HF Education, 1987.
[43] Shiori Sagawa Pang Wei Koh*, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020. URL
[44] Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
[45] Aman Sinha, Hongseok Namkoong, Riccardo Volpi, and John Duchi. Certifying some dis- tributional robustness with principled adversarial training. arXiv preprint 17X1v:1710.10571, 2017.
[46] Bharath K Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Berharr Schölkopf, Gert RG Lanckriet, et al. On the empirical estimation of integral probability metrics. Electronic Journal of Statistics, 6:1950-1599, 2012.
[47] Baochen Sun and Kate Saenko. Deep CORAL: Correlation alignment for deep domain adapta- tion. In European conference on computer vision, pages 443-450. Springer, 2016.
[48] Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, and Geoffrey J Gordon. Domain adaptation with conditional distribution matching and generalized label shift, Advances in Neural Information Processing Systems, 33, 2020.
13

---

[49] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167-7176, 2017.
[50] Vladimir Vapnik.
Principles of risk minimization for learning theory. In Advances in neural information processing systems, pages 831-838, 1992.
[51] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In (IEEE) Conference Computer Vision and Pattern Recognition (CVPR), 2017.
[52] Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing unseen domains via adversarial data augmentation. In NeurIPS, 2018.
[53] Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 6502-6500 2020.
[54] Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXiv:2001.0000 2020.
[55] Haotian Ye, Chuanlong Xie, Tianle Cai, Ruichen Li, Zhenguo Li, and Liwei Wang. Towards theoretical framework of dittr--distribution generalization, 2021.
[56] Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, pages 1-1, 2021. doi: 10. 109/TKDE.2021.3007000
[57] Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning, pages 1404-7413, PMLR, 2019.
[58] Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey Gordon. Adversarial multiple source domain adaptation. Advances in neural information processing systems, 11:8559-85700 2018
[59] Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In International Conference on Machine Learning, pages 7523-7532. PMLR, 2019.
14


Reviews:
[{"review_id": "YjHc53yDAaa", "reviewer": "Reviewer_AD3E", "summary": "Authors introduced the notion of transferability between pairs of domains (i.e. joint distributions over input x output spaces) along with a discrepancy measure, which can be used to assess how dissimilar are two data sources relative to a given class of predictors. Such a quantity may be useful in either assessing how well is a trained model likely to perform on new data, or as a training signal so that more preference is given to predictors able to generalize out of the training data distribution. The main strength of this work in my opinion is the fact that the proposed transferability measure accounts for the joint distributions over data and labels. However, while I appreciate the depth of the discussion and results in terms of the properties of the proposal, I believe a key missing discussion is the effect that commonly used sets of assumptions have in the proposed transferability measure. Moreover, the evaluation lacks in evidencing the benefits of the proposal relative to current strategies. Please refer to the following for further details.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n+The proposed discrepancy measure accounts for the joint distribution over data and labels, which seems to remove the need for assumptions such as covariate shift, required in common alternatives that only account for the data marginal distribution.\n\n+Authors present a comprehensive set of results and in-depth discussion of the properties of the proposed measure.\n\n+An interesting evaluation is carried out where transferability is estimated for features obtained under different settings appearing in recent literature.\n\n+Algorithm 2 requires a single extra classifier to estimate the worst-case pairwise discrepancy (h’). Past similar approaches use N or even N^2 extra classifiers, where N is the number of training domains.\n\nWeaknesses/suggestions:\n\n-The write-up lacks in discussing the effects of different kinds of distribution shifts in the transferability measures. I would expect to see a discussion similar to [1] where the effect of, for instance, conditional shifts would be analyzed. It seems severe conditional shift, c.f. example in figure 1 in [1], would make the transferability measure explode, which would result in no advantage over more standard discrepancy measures over data marginals only such as the H-divergence. Please clarify whether that’s indeed the case.\n\n-Regarding specifically the empirical evaluation, it lacks in evidencing the effectiveness of the proposed measure since it focuses on analyzing the transferability of features obtained under different algorithms. It would be important to see evidence showing that the estimation errors are not significantly affecting results in practice. As a suggestion, it seems all evaluations are carried out in datasets where covariate shift can be assumed (i.e. only data marginals shift). In that case, we would expect the proposed transferability to correlate with, for example, the H-divergence. Verifying that would serve as evidence that the measure and the algorithm used to implement it actually yield informative results, and are not relevantly affected by, for example, sub-optimality of gradient ascent.\n\n-Another missing piece of evidence in my opinion is in how indicative of generalization the proposed transferability measure is. From a practical perspective, it would be relevant to see evidence that lower transferability implies better generalization.\n\n-Finally, the claim that the proposed transferrability measure is helpful in allowing the analysis to shift from the marginals to the joint distributions should be supported by empirical evidence as well. Are there cases where more standard notions such as the H-divergence wouldn't work and the proposal would?\n\nMinor suggestions:\n\nSince transferability is a property of a pair of domains and a hypothesis class, perhaps that should be indicated in the notation; i.e. (\\delta_S, \\delta_T, H)-transferable should be used instead of (\\delta_S, \\delta_T)-transferable.\n\n[1] Zhao H, Des Combes RT, Zhang K, Gordon G. On learning invariant representations for domain adaptation. International Conference on Machine Learning 2019 May 24 (pp. 7523-7532). PMLR.\n\nPost rebuttal edit:\n\nI went through the authors' responses which clarified the effect that conditional shift would have on the proposed measure. I decided to update my score to 'accept'. I do like the idea since it's the first domain-divergence measure that I'm aware of that accounts for the joint distribution over data and labels, as opposed to only one of the marginal distributions.\n\nOne concern that remains to me is the coverage of the evaluation, which I think is very limited. While the main strength of the approach, in theory, is the fact that it accounts for the joint distributions, all evaluations are carried out on cases where only the data marginals shift. Moreover, only the proposed transferability is evaluated, and it's not compared with more standard divergence measures such as the H-divergence. If the proposal yields meaningful domain divergence results, we would expect a strong correlation between their proposal and the standard measures in the considered evaluations, and that would give confidence that the proposal works in practical settings. For example, on PACS, pairwise H-divergences could be compared with the proposed transferability, and ideally, they would correlate more or less strongly depending on the label marginals shift.\n\nIn summary, I think the idea is very interesting, novel, and solves an important issue, the theoretical results are strong and insightful, but the evaluation is limited. All in all, I think the merits of the paper outweigh its flaws.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "S4Cn8Rusrx9", "reviewer": "Reviewer_NJSd", "summary": "This paper defines a notion of \"transferability\" between features from different domains, which is different from common distribution discrepancy measures such as total variation and Wasserstein distance. This measure of transferability is measured with labeled samples from both domains, and provides a bound on the target error. The empirically test transferability of the features learned by current domain generalization algorithms, and some don't do that well in this transferability measure. They propose an algorithm based on optimizing the measure of transferability across training domains and test it on some domain generalization datasets, including a satellite dataset.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Quality\n\nThe general structure of the work seems to be to find a metric (transferability) that plugs into a generalization bound, then convert this into an algorithm by optimizing the metric. This paper gives some interesting results about source vs target accuracy when subject to parameter space pertubations, but these comparisons are somewhat better suited for their method since they optimize something related to this directly. They give a comprehensive theoretical analysis including finite sample results. The main concerns are in the formulation and the empirical results. For formulation, it's not immediately clear what this method leverages better than other domain generalization methods since it also only uses the given training domain information for enforcing invariance - is the bound tighter than others, etc.? More intuition on this could help. For empirical results, from Figure 3 it seems that the SD baseline is comparable to the proposed algorithm in target accuracy across all the datasets. Also, error bars aren't given in the plots and the results could be within noise.\n\n- As the authors mention in line 258, the measure of transferability depends on labeled samples from the target domain. They circumvent this in the algorithm by optimizing transferability between a number of source domains. However, there's a bit of a conceptual gap here, since it is straightforward to say that we optimize a bound on the target accuracy of a particular target distribution that we measure transferability on, but it's less clear how optimizing for transferability on a number of source domains will generalize to a new target.\n- In line 242 and equation 13, $\\Gamma$, the $\\delta_S$-minimal set of $\\epsilon_S$, is replaced by a $\\delta$-ball in parameter space. This relies on an assumption on the neural network to be Lipschitz in the parameter space. However, for wide neural networks, it seems that small perturbations in parameter space can cause very large changes. This could make the bound in equation 13 very loose. It's also unclear that comparisons wrt to the $\\delta$-ball in parameter space like in Figure 3 are valid due to the unknown (and possibly very small) range of valid $\\delta$. The overall finding that target accuracy decreases much faster with parameter space perturbations is interesting, however.\n- In section 2.1, the paper states that the defined metric is different from IPMs since it relies on an underlying function class, but this is exactly what the \"neural network distance\" IPMs used in theory of GANs do as well [1].\n\n- Minor: in the self-driving example in the intro, night/day shouldn't be a shift where we should be applying domain generalization techniques, since it's so easy to collect data in both the night and day. Time shift might be more appropriate here (you can't collect data from the future).\n\nSignificance\n\nDomain generalization methods generally don't seem to work much better than ERM, according to some empirical works, so advancements in this field are necessary. The overall approach in this paper is nice and comes from a principled point of view, however the empirical results are inconclusive whether this paper's method improves upon the other baselines.\n\nOriginality\n\nThe final algorithm seems related, though different, to invariance based methods such as IRM or group DRO. The principle seems to be to make the errors on every training domain to be the same within a ball in the parameter space (learn parameter-stable networks). The ball in parameter space seems to be a limitation of the technique / inability to characterize the $\\delta$-minimal set of hypothesis, however, and not necessarily fundamental. The measure of transferability is nice, although it requires labeled data in both source and target so it isn't necessarily surprising (whereas using only unlabeled target data to estimate transferability would be tougher). \n\n\nClarity\n\nThe paper could be structured more hierarchically - it dives into math as part of the essential explanation. I think the clarity could benefit from giving a high level overview and then going into the details. \n- Fig 1: from the description of transferability, it's unclear why we're considering $\\delta$ perturbations in the parameter space. This is only introduced in Section 4.\n- Def 2 is named quantifiable transfer measures, but I believe only the third one is computable since it's the only one that doesn't require knowing optimal errors. Whether something should be considered quantifiable if it is not computable is unclear.\n- Since there are 3 transfer measures defined, it's a bit hard to keep track of which one is being used at the moment, and what the purpose of each of the measures are. Perhaps the paper could state which one will be the main one to work with, and then using these propositions we can make it computable. \n- In Fig 3, the brown color is used for many different algorithms, consider deduplicating the color.\n\n\n[1] Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, Yi Zhang. Generalization and Equilibrium in Generative Adversarial Nets (GANs), 2017.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\nThe authors mention that their algorithm requires a large number of inner maximization steps, increasing training time, and that moderate hyperparameter tuning is needed for additional hyperparameters. In terms of broader impact, the authors warn that just because an algorithm passes their transferability check, it does not guarantee good domain generalization performance.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "HqPXM-MI7mu", "reviewer": "Reviewer_RBZG", "summary": "This paper aims to provide a formulation of transferability. The main intuition is that any near-optimal source classifier should be also near-optimal on the target domain. So the authors quantify transferability by measuring the relation between near-optimal classifiers in source and target domains and derive genralization bounds based on this. In addition, new algorithm is proposed based on the intuition.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The problem of quantifying transferability is not well studied and the reviewer agrees that proposing a formulation of transferability or invariance is important and novel (though there are many other concurrent papers doing this, e.g., https://arxiv.org/abs/2008.01883, https://arxiv.org/abs/2106.04496, which might be better to be included later)\n\nQuality: The basic formulation of transferability in this paper is like a relaxation of the IRM objective with quantifiable surrogates. The relaxation is reasonable though the quantitative part is similar to the DANN-style discrepancy. Given the improvements on the empirical performance compared to DANN and other distributional matching baselines, the reviewer tends to vote for accept.\n\nClarity: The writing of this paper is clear.\n\nSignificance: The formulation and algorithm proposed in this paper are similar to the previous work in the form but bring new thoughts and improvements to the area.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not applicable.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "1__dUxpWfu", "reviewer": "Reviewer_hkRr", "summary": "This paper aims at identifying and quantifying transferability in learning algorithms. The paper proposes a new upper bound on the target error. The proposed algorithm is an adversarial optimisation problem that empirically proved to be effective. ", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper tackles an interesting and important problem. However, the main issue I have with the paper is that it is hard to read and barely provides intuitive explanations on the results. It is also a very notation-heavy paper, but they seem to be used without much introduction and discussion. The paper could significantly benefit from a section on problem setup and notations. For instance, before section 4, z is used but it is introduced in Section 4. Also, S_i is not introduced, I am assuming it is one of the source domains.  \n\nIn addition, the main contribution seems to be to draw connection between IPM measures and the transferability that is already known in the literature. The results section is also poorly presented and is not clear what the experimental setup is. It is also not better explained in the supplements.\n\nI have some concerns about the problem set up and the overclaim of the paper. I understand though that this paper follows similar ones in the domain transfer literature, but I believe for more modern deep learning settings it is not sufficient. It would be interesting to see how these bounds be different if the i.i.d assumption is removed and a more realistic setting is used. In fact, I believe the main obstacle in using such bounds is that the main question in transferability is how to address these confounders. For instance, if we had a dataset of cats vs dogs where all the cats have a blue background and all the dogs a red one, we expect with the simplicity bias in neural networks, the model focuses on the background alone. Therefore the transferability to another problem with colour as the main discriminative signal is intuitively easier than to a cat vs dog dataset where images are taken in the wilds. \n\nWith the rise of self-supervised and pre-training, I believe this paper could be more impactful if it included discussions on those lines of work. They aim at learning the invariances and that is how they achieve better generalisation.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "All in all, I believe the technical quality of the paper may be good but poorly presented.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on transferability in context of OOD generalization. The authors define the term, point out connection and differences with current discrepancy methods and then prove that this measure can be transferability can be estimated with enough samples and give a new upper bound for the target error based on our transferability. Empirically, they evaluate the transferability of the feature embeddings learned by existing algorithms for domain generalization. They also propose a new method for learning transferable features and test it over some benchmarks. The idea is interesting and the theoretical investigations are worthwhile. The authors addressed the questions and concerns raised by reviewers and I believe their responses are satisfactory. I request the authors to update the paper as promised in the discussion period and add the requested information and clarifications as well as comparison with the related works raised by the reviewers. 
Decision: Accept (Poster)